AI companies use different crawlers for different jobs: some collect content to train models, others fetch pages to answer live search questions. Knowing which is which lets you stay visible in AI search while deciding separately whether your content may be used for training.
Key takeaways
- Training crawlers (such as GPTBot) and search crawlers (such as OAI-SearchBot) are separate and can be controlled separately.
- To be cited in AI search, allow the search crawlers and make sure your CDN or firewall does not block them.
- llms.txt is a proposed convention, not a standard: useful as a curated map, not a ranking factor.
The main AI crawlers
| User agent | Company | Purpose | Block it if… |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | You do not want your content used to train models |
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search results | You do not want to appear in ChatGPT search (rarely wise) |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to | Rarely needed |
| ClaudeBot | Anthropic | Collects content for model training | You do not want training use |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | You do not want to appear in Perplexity |
| Google-Extended | Control token for Gemini training and grounding (not a separate crawler) | You do not want Gemini to use your content; Search is unaffected | |
| Applebot-Extended | Apple | Control token for Apple AI training | You do not want Apple AI training use |
| Bingbot | Microsoft | Bing index, which powers Copilot | Almost never; blocking it removes you from Bing and Copilot |
A sensible default robots.txt policy
For most businesses that want visibility but are cautious about training use, a policy like this works well. Adjust it to your own view on training:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Sitemap: https://example.com/sitemap_index.xmlRobots.txt is a request, not a lock. Reputable crawlers respect it; others may not. If you need hard blocking, do it at the CDN or firewall level, and test that you have not blocked the search crawlers by accident.
Not sure what your site is blocking?
Get a free technical check of robots.txt, firewall rules and AI crawler access. Call or WhatsApp +91-8840146999.
What llms.txt is and whether you need it
llms.txt is a plain-text file at the root of a site that lists its most important pages with short descriptions, written for language models. It was proposed in 2024 as a convention. It is quick to create and does no harm, and some developer tools and AI agents read it. But no major AI search engine has said it uses llms.txt to choose sources, so treat it as a tidy map of your best content, not as a ranking lever.
Check your setup in 15 minutes
- Read your robots.txt. Look for broad Disallow rules or AI user agents you did not intend to block.
- Check the CDN and security plugin. Look for “block AI bots”, bot-fight or challenge settings that affect crawlers.
- Search your logs. Filter by the user agents above and confirm they receive 200 responses, not 403s or challenges.
- Test rendering. View the page with JavaScript disabled: your main content should still be there.
- Add llms.txt. List your key services, about page, pricing, contact and best guides.
Frequently asked questions
Should I block GPTBot?
It depends on whether you are comfortable with your content being used to train OpenAI models. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot. Many businesses block training crawlers and allow search crawlers.
Does Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content is used for Gemini model training and grounding. It does not affect Google Search rankings or inclusion in Search.
Is llms.txt a standard?
No. llms.txt is a proposed convention for listing a site’s most useful pages for language models. It is cheap to add and harmless, but no major AI search engine has confirmed it uses the file for ranking or citations.
How do I know which AI bots visit my site?
Check your server or CDN logs for their user agents, and verify important ones by IP ranges published by the provider, because user agents can be faked.
Can my firewall block AI bots without me knowing?
Yes. Some CDNs and security tools block AI crawlers by default or through bot-fight modes. Check those settings if you want to be cited.