AI crawlers and robots.txt: what to allow and what to block

The main AI companies publish the names of their crawlers and what each one is for. Most now run separate bots for training and for search, so you can treat them differently.

Any request can claim any user agent, so a name in your logs is not proof. OpenAI and Perplexity publish the IP ranges their crawlers use, and Google publishes the ranges for Googlebot, which lets you or your hosting provider check that a visit claiming to be one of these bots is genuine.

No. This is the most common misunderstanding. Google-Extended is a product token, not a crawler: Googlebot does the crawling, and the token tells Google whether that content may be used to train future Gemini models and for grounding in Gemini apps. Google states that Google-Extended does not affect a site's inclusion or ranking in Google Search.

AI Overviews and AI Mode are part of Search and use pages crawled by Googlebot. If you want to limit how your content appears in them, Google points to the same controls as ordinary snippets: nosnippet, max-snippet, data-nosnippet or noindex. Blocking Googlebot removes you from Search entirely.

It depends on what your content is for. A publisher whose business is selling original reporting may reasonably block training crawlers. A service business that wants to be recommended when someone asks ChatGPT for a supplier usually wants the opposite: its pages fetched, understood and cited.

A balanced policy for most businesses is to allow the search and user-request crawlers, and make a deliberate choice about training crawlers. Blocking training crawlers will not remove content that has already been collected, and it does not stop an AI engine mentioning you based on what other sites say about you.

1. Check who is visiting. Look at your server or CDN logs for the user agents above to see which AI crawlers already reach the site.
2. Decide a policy per crawler. Training, search and user-request bots, each allowed or blocked on purpose.
3. Write the rules. Each crawler gets its own User-agent group in robots.txt at the root of the domain.
4. Check your CDN and firewall. Bot protection can block crawlers before robots.txt is ever read. In July 2025 Cloudflare announced it would block known AI crawlers by default for new domains, so check the setting if you use it.
5. Verify and review. Confirm the file loads, watch the logs over the following weeks and review the list of crawlers twice a year, because new ones appear.

A policy that allows AI search but declines model training looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

robots.txt is a voluntary standard, formalised as RFC 9309 in 2022. The major AI companies say their declared crawlers respect it, but nothing technically forces a crawler to comply, and requests made on behalf of an individual user are sometimes treated differently from automated crawling. If you need hard enforcement, it has to happen at the server or CDN level.

robots.txt and llms.txt do different jobs. robots.txt sets access rules; llms.txt is a proposed file that summarises your site for language models. Our guide to what an llms.txt file is explains where it fits.