Which AI crawlers should a local business allow in robots.txt?

Allow the AI crawlers that search the web and the ones that fetch a page when someone asks a question. OpenAI, Anthropic and Perplexity tie their search crawlers to appearing in their search results. Training crawlers have their own controls, so you can decide about them separately.

What are the three kinds of AI crawler?

1. Search crawlers collect and index pages so an assistant can answer questions from them later.
2. Training crawlers collect content that may be used to train AI models.
3. User fetchers visit a page when a person asks an assistant a question.

The kind decides what a block costs. OpenAI says sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers.” It also says a site can allow OAI-SearchBot “in order to appear in search results while disallowing GPTBot.”

Which crawlers does each company document?

Which crawlers should a local business allow?

If you want assistants to name your business, allow the search crawlers and the user fetchers. They are how an assistant finds your pages and reads them while it answers. Keep Googlebot and Bingbot open too, since Google’s AI features and Copilot rely on their crawling.

Access does not decide the answer: Google says “indexing and serving isn’t guaranteed.” How AI assistants choose local businesses covers what is known about how they pick.

Training crawlers are a business choice. Allowing GPTBot or ClaudeBot means your pages may be used to train those companies’ models. OpenAI says its settings are independent, and Apple says pages that disallow Applebot-Extended “can still be included in search results.” Google-Extended also controls grounding in Gemini Apps, so blocking it affects more than training.

What does the robots.txt look like?

To allow every crawler, one group and a Sitemap line are enough. Ultruxy’s own robots.txt works this way.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Google says Google, Bing and other major search engines support the sitemap line, which must be a full URL.

To keep search and user access but opt out of training, add a group for each training-only crawler or token:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Under RFC 9309, the robots.txt standard, a crawler obeys the group that names it and uses the User-agent: * group only when none does. Google-Extended and Amazonbot are left out because Google and Amazon describe uses beyond training. CCBot is left out because Common Crawl describes it as building an open repository of web crawl data.

Can a hosting setting block AI crawlers?

Yes. Google’s guidance lists “ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure.”

Cloudflare lets customers block AI bots by behavior: Search, Agent or Training. Its Training category includes “mixed-purpose crawlers that are used both for Training and for Search.” Its documentation names September 15, 2026 as the date of new defaults for new domains: Training and Agent bots blocked on pages that display ads, and Search still allowed. Perplexity notes that a site behind a web application firewall may need to allow its bots explicitly.

Does robots.txt stop a crawler?

No. RFC 9309 says its rules “are not a form of access authorization,” and Google says robots.txt instructions “cannot enforce crawler behavior to your site.” To keep a page private, Google suggests password protection. The RFC adds that listing a path in robots.txt makes it discoverable.

How do you check your own site?

1. Read the file. Open yourdomain.com/robots.txt and look for Disallow lines under the crawler names above or under User-agent: *. Check subdomains separately: Google says robots.txt files “are not inherited by subdomains or parent domains.”
2. Use Google Search Console. Its robots.txt report, for domain-level properties, shows the files Google found, when it last crawled them and any warnings or errors. The URL Inspection tool tests whether a specific URL is blocked for Google.
3. Use Bing Webmaster Tools. Its robots.txt tester shows whether a URL is blocked, which statement blocks it and for which user agent.
4. Check your host or CDN. In Cloudflare, the AI bot policies are under Security Settings.

The AI visibility checklist puts these checks alongside the rest of AEO. If you would rather have it checked for you, Ultruxy offers a free visibility report. How we approach the work is on the AEO service page.