AI crawlers list with what each bot feeds and which ones to allow

The AI crawlers list that OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon and Microsoft publish holds 23 names, and 21 do one of three jobs: training, search or fetching a page for one person’s question. That job decides what a robots.txt block costs, since a blocked search crawler or fetcher keeps a site out of that company’s answers.

What AI crawlers feed

An AI crawler is a program an AI company sends to read web pages, and what it feeds depends on its job: a future model, the index behind AI search answers or the one answer a person is waiting for.

Two of the six training names crawl nothing. Google-Extended and Applebot-Extended are robots.txt tokens telling Google and Apple how they may use pages their regular crawlers collect, for training and, at Google, for grounding in Gemini Apps. The two others are OpenAI’s OAI-AdsBot, which checks pages submitted as ChatGPT ads, and Google-CloudVertexBot, which crawls at a site owner’s request.

The 23 names the eight companies document, counted by job:

Search crawlers and user-triggered fetchers make up 15 of the 23 names, and those 15 put pages into answers today.

Googlebot feeds AI answers too. Google says rules for Googlebot cover Google Search and all its features, and AI Overviews is one of them, so a Googlebot block also removes a site from AI Overviews.

AI crawlers by company

Each company’s documentation names its AI crawlers and their jobs, as of 2 November 2026:

• OpenAI: OpenAI’s crawler documentation lists GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for pages a user’s question opens, and OAI-AdsBot for ad landing pages.
• Anthropic: Anthropic’s help article names ClaudeBot for training, Claude-SearchBot for Claude’s search results and Claude-User for pages fetched for a user’s question.
• Google: Google’s common crawlers page covers Googlebot for Google Search, Google-Extended for Gemini training and grounding, and Google-CloudVertexBot for site owners’ Vertex AI agents.
• Google fetchers: Google’s user-triggered fetchers page adds Google-Agent, for agents acting on a user’s request, and Google-GeminiNotebook, for pages users add as sources.
• Perplexity: Perplexity’s crawler documentation lists PerplexityBot for Perplexity search, which it says collects nothing for model training, and Perplexity-User for user actions.
• Apple: Apple’s Applebot page says Applebot crawls for Siri, Spotlight and Safari search and may feed model training, which Applebot-Extended lets a site refuse.
• Meta: Meta’s web crawlers page lists Meta-ExternalAgent for AI training and product indexing, Meta-WebIndexer for Meta AI search, and Meta-ExternalFetcher for links a user asks for.
• Amazon: Amazon’s Amazonbot page lists Amazonbot, whose data may train Amazon AI models, Amzn-SearchBot for search such as Alexa, and Amzn-User for live Alexa answers.
• Microsoft: Bing’s crawler help page calls Bingbot its standard crawler and lists none built for AI answers, and Microsoft is one of the search providers behind ChatGPT search.

The number of AI crawler names each company documents:

Six of the eight companies give model training a robots.txt name of its own, apart from search, while Perplexity documents no training crawler and Microsoft documents Bingbot alone. Each engine also cites different websites, so no other engine makes up for a block.

Which AI bots follow robots.txt

Every training name and search crawler on the list follows robots.txt by its company’s account, and six of the seven user-triggered fetchers may skip it. OpenAI says robots.txt may not apply to ChatGPT-User, Google and Perplexity say their fetchers generally ignore it, Meta says Meta-ExternalFetcher may bypass it and Amazon says Amzn-User may not follow all of it.

Anthropic says its three bots honor robots.txt, so Claude-User is the one fetcher a robots.txt line stops. Google says its common crawlers, Google-CloudVertexBot among them, always obey robots.txt when crawling automatically, and OpenAI’s OAI-AdsBot carries no robots.txt statement.

How the 23 names stand on robots.txt, by each company’s documentation:

Robots.txt reaches 16 of the 23 names, and the 6 that may skip it open pages while a person waits for an answer.

Google plans to read a shared training token, User-agent: __usage_ai-training, in 2027 as a Google-Extended disallow when a site sets no Google-Extended rule, ending grounding in Gemini Apps too. Microsoft reads the same token as training only from 1 February 2027, as the shared AI training opt-out sets out.

A robots.txt file for AI bots

A robots.txt file for AI bots should name every bot that puts pages into answers, keep training names in their own group and repeat the site’s Disallow lines in every group, as in this example, where /admin/ stands for the site’s rules:

# Answer bots
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
Disallow: /admin/
# Training names
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
Disallow: /admin/
# Everyone else, Googlebot included
User-agent: *
Disallow: /admin/

RFC 9309, the robots.txt standard, makes a crawler obey the group that names it and use the User-agent: * group only when none does. A Disallow line kept only in the catch-all group never reaches a named bot, and a later Disallow: / there cannot lock the named answer bots out.

Closing the training group costs more than training at three companies: Google-Extended also controls grounding in Gemini Apps, Meta-ExternalAgent also indexes content for Meta’s products, and Amazonbot’s data also improves Amazon’s products. AI Overviews and AI Mode have their own Search Console control, covered in getting pages cited in Google AI Overviews.

Checking firewalls and server logs

A firewall or a CDN, the service in front of a site that filters its traffic, can turn AI bots away before robots.txt matters. Perplexity’s crawler documentation asks sites behind a web application firewall to allow its bots, and says custom rules cannot override Cloudflare’s Bot Fight Mode.

Server logs show which bots got through: a 200 status next to OAI-SearchBot, PerplexityBot or Claude-SearchBot means the page was served, and a 403 means it was refused. OpenAI, Anthropic, Google, Perplexity, Apple, Amazon and Microsoft publish IP lists or a verification tool, since anyone can fake a bot’s name, and AI visibility tracking shows whether a visit ended in a citation.

Training crawlers can make most of a log’s AI visits. On one B2B website, 17 AI bots made 1,391 verified visits in the month to 27 September 2026, 1,022 of them by training crawlers, according to an AI crawler list published on agenzy.lt. OAI-SearchBot and ChatGPT-User, the two bots behind ChatGPT answers, made 50 between them.

What it means

The head of marketing should have the web team check AI crawler access, in this order:

1. Name OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Meta-WebIndexer and Amzn-SearchBot in robots.txt, copy every existing Disallow line into each named group, and keep Googlebot, Bingbot and Applebot open.
2. Check CDN and firewall rules, Bot Fight Mode included, and allow the IP addresses the companies publish.
3. Leave the six training names open if the company sells something, since the AI recommendation is the most valuable moment in a purchase.
4. Add an explicit Google-Extended rule before adding the shared training token.
5. Check the logs a day later for 200 responses to the answer bots.

FAQ

How long does a robots.txt change take to reach AI answers?

About a day. OpenAI gives about 24 hours for ChatGPT search, Perplexity up to 24 hours and Meta up to 24 hours of robots.txt caching, while Amazon’s bots may use a copy up to 30 days old. A first citation takes longer, as how long GEO takes in ChatGPT answers measures.

What happens when robots.txt fails to load?

It depends on the error. Under RFC 9309, a server error from 500 to 599 means complete disallow, while a missing file or another status from 400 to 499 lets crawlers read any page. Amazon says its bots treat a file they cannot fetch as missing.

Does a rule for one bot apply to another?

Sometimes. Apple says Applebot follows rules written for Googlebot when a file names Googlebot and not Applebot, and Amazon says Amzn-SearchBot follows rules given to other search bots when the file leaves it out. A Googlebot block can therefore close Applebot too.