SpagifyPriceBot

What it is

Spagify runs Google Shopping campaigns for Shopify stores. When one of those stores advertises a product that other retailers also sell, Spagify reads the public price of that product on the other retailers' storefronts so the merchant can see how their own price compares. SpagifyPriceBot is the crawler that does that reading. It runs from Spagify's servers, on behalf of one Spagify customer at a time, and identifies itself with this user agent:

SpagifyPriceBot/1.0 (+https://spagify.com/bot; price comparison for a retailer that sells the same products)

It never signs in, never submits a form, and never reads anything that is not already public.

What it reads

• robots.txt first, every visit.
• On Shopify stores, the public catalog JSON at /products.json.
• On other stores, the sitemap and at most 25 product pages whose address mentions a brand or part number the customer sells, reading the page's JSON-LD Product and Offer markup, with Open Graph price tags as the fallback.

From a product page it keeps the product name, brand, GTIN, MPN, SKU, price, currency, availability, condition and URL. Nothing else on the page is stored, and no page is stored.

How politely

• It honours robots.txt Disallow and Allow rules: the SpagifyPriceBot group if there is one, otherwise *.
• It waits 400 milliseconds between requests and gives up on any request after 10 seconds.
• It makes at most 40 requests per site per visit, and visits a site at most once a day.

How to block it

Add this to your robots.txt. The crawler reads the file on every visit, so the rule takes effect within a day.

User-agent: SpagifyPriceBot
Disallow: /

To block only part of a site, use a Disallow path instead of /. If you would rather not changerobots.txt, email legal@spagify.com with your domain and it is added to the crawler's own exclusion list.

Contact

Questions about the crawler or about Spagify: legal@spagify.com. How Spagify handles data in general is in the Privacy Policy.