01What it fetches
TasteIndexBot screenshots and measures the homepages in The Taste Index. It names itself in its user agent:
Mozilla/5.0 (…) Chrome/<version> Safari/537.36 TasteIndexBot/0.1 (+https://headoftaste.ai/bot)- Pages
- The homepage only.
- Requests
- Each edition it fetches robots.txt once per host, then the homepage three times: once for speed, once as a desktop browser and once as a phone. The images, fonts and scripts the page loads come with it.
- Screens
- Google Chrome at 1440 × 900, and at 390 × 844 with a mobile user agent.
02How it behaves
It does not follow links, fill in forms, sign in, click anything, or get past bot challenges, logins and paywalls.
- robots.txt
- It follows RFC 9309 and obeys the rules for
TasteIndexBot, or the*rules if none name it. It checks before loading anything and again wherever redirects land. If robots.txt can’t be reached because of a server error, rate limit or timeout, the crawler treats the site as disallowed and stays away. - Pace
- Speed loads run one site at a time. Captures run three sites at once at most, with a 1.5 s pause between sites.
- Cookie banners
- We hide known ones in our own browser. They stay unanswered, so no consent is given or refused.
- Measuring code
- It runs only inside our own browser.
03How to block it
Add this to /robots.txt on your domain:
User-agent: TasteIndexBot
Disallow: /The next crawl skips the site. A page we have already published stays up until you ask for removal at hello@headoftaste.ai.
04Contact
For anything about the crawler, email hello@headoftaste.ai. Include your domain and the time of the request.