Will AI companies respect my robots.txt?

The major operators, OpenAI, Anthropic, Google, Microsoft, Apple, and Perplexity among them, document their user agents and state that they honor robots.txt directives. Smaller data brokers vary. For bots that ignore the file, blocking happens at the server or CDN level, not in robots.txt.

Does this replace my existing robots.txt?

No. The generated file covers AI crawlers only. Merge its rules into your existing robots.txt, keeping your current sitemap line and any non-AI rules unchanged.

Should I block Common Crawl (CCBot)?

CCBot feeds an open dataset used to train many models, so blocking it is the broadest single training opt-out available. The trade-off: the same dataset also powers research and archives. Sites that want to limit model training tend to block it; sites optimizing for reach leave it open.

What happens to pages already crawled before I block a bot?

robots.txt controls future fetches. Content already collected does not disappear from existing datasets or model weights; some operators offer separate removal processes.

Robots.txt generator for AI crawlers

Decide bot by bot which AI systems may read your site. Start from a preset, flip individual crawlers, and copy a robots.txt that says exactly what you chose. Runs entirely in your browser.

OpenAI

GPTBotTraining
ChatGPT-UserAssistant fetch
OAI-SearchBotSearch index

Anthropic

ClaudeBotTraining
Claude-UserAssistant fetch
Claude-SearchBotSearch index
anthropic-aiTraining
Claude-WebTraining

Perplexity

PerplexityBotSearch index
Perplexity-UserAssistant fetch

Google

GoogleOtherOther
Google-ExtendedTraining
Google-CloudVertexBotOther

Microsoft

BingbotSearch index
msnbot / MSNBot-MediaSearch index

xAI

GrokBotTraining

Your robots.txt

# AI crawler rules generated with citlyze.com/free-tools/ai-robots-txt-generator

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Google-CloudVertexBot
Allow: /

User-agent: bingbot
Allow: /

User-agent: msnbot
User-agent: MSNBot-Media
Allow: /

User-agent: GrokBot
Disallow: /

User-agent: meta-externalagent
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: FacebookBot
Disallow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Allow: /

User-agent: MistralAI-User
User-agent: MistralAI-User/1.0
Allow: /

User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: DuckAssistBot
Allow: /

User-agent: Bytespider
User-agent: ByteSpider
Disallow: /

User-agent: Baiduspider
User-agent: Baiduspider-render
Allow: /

User-agent: PetalBot
Allow: /

User-agent: Sogou web spider
User-agent: Sogou inst spider
User-agent: Sogou
Allow: /

User-agent: 360Spider
User-agent: HaoSouSpider
Allow: /

User-agent: YandexBot
User-agent: YandexAdditional
Allow: /

User-agent: CCBot
Disallow: /

User-agent: AI2Bot
User-agent: AI2Bot-Dolma
Disallow: /

User-agent: YouBot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: Diffbot-User
Allow: /

User-agent: Timpibot
Allow: /

User-agent: Omgilibot
User-agent: Omgili
User-agent: omgilibot
Disallow: /

User-agent: ImagesiftBot
Allow: /

User-agent: VelenPublicWebCrawler
Allow: /

# Merge with your existing robots.txt; this file covers AI crawlers only.

Merge these rules into the robots.txt at your domain root. Keep your existing sitemap line and non-AI rules.

how it works

Three decisions, one file

Pick a starting point

Allow everything, block only training crawlers, or block all AI bots. The presets encode the three positions most sites take.

Adjust individual bots

Every crawler is labeled by what it does: model training, search indexing, or on-demand fetching for assistants. Flip any of them; the file updates live.

Publish your robots.txt

Copy or download the result and merge it into the robots.txt at your domain root. Existing non-AI rules stay yours; this file only covers AI crawlers.

Training, search, or assistant: why the category matters

Blocking a training crawler like GPTBot only affects what future models learn; your site keeps appearing in AI search answers. Blocking a search indexer like OAI-SearchBot or an assistant fetcher like ChatGPT-User removes you from live answers and link reading. Most sites that block anything block training and keep search, which is exactly what the middle preset does.

What robots.txt can and cannot do

robots.txt is a published request, not an enforcement mechanism. The major operators listed here document their tokens and generally honor them; a crawler that ignores robots.txt needs server-side blocking instead. The generator lists each bot's operator so you can decide who you are addressing.

Blocking AI is a trade, not a default

If AI answers are becoming a discovery channel for your market, blocking every bot makes you invisible exactly where buyers are asking. The honest way to decide is to measure: see which AI crawlers already visit you, and whether AI engines cite your pages, before turning anything off.

the bot directory

Every AI crawler in this generator

Who operates each bot, and whether it collects for model training, builds a search index, or fetches pages on demand for an assistant.

Bot	Operator	Category	What it does
GPTBot	OpenAI	Training	Foundation model training crawler
ChatGPT-User	OpenAI	Assistant fetch	User-triggered browsing and actions
OAI-SearchBot	OpenAI	Search index	ChatGPT search indexing and retrieval
ClaudeBot	Anthropic	Training	Claude training crawler
Claude-User	Anthropic	Assistant fetch	User-triggered retrieval
Claude-SearchBot	Anthropic	Search index	Claude search indexing
anthropic-ai	Anthropic	Training	Legacy or compatibility Anthropic token
Claude-Web	Anthropic	Training	Legacy Anthropic robots.txt token from earlier Claude crawling. Kept for completeness; current crawling uses ClaudeBot.
PerplexityBot	Perplexity	Search index	Perplexity search index
Perplexity-User	Perplexity	Assistant fetch	User-triggered retrieval
GoogleOther	Google	Other	Generic Google crawler for public content
Google-Extended	Google	Training	Gemini model training and grounding usage control
Google-CloudVertexBot	Google	Other	Vertex AI agent crawler for site-owner-requested generative content
Bingbot	Microsoft	Search index	Bing index that also powers Copilot and Bing generative search
msnbot / MSNBot-Media	Microsoft	Search index	Legacy Bing/MSN crawler still seen on some properties
GrokBot	xAI	Training	Grok/xAI crawler token
meta-externalagent	Meta	Training	Meta AI training and external agent
Meta-ExternalFetcher	Meta	Assistant fetch	Meta user-triggered fetcher
FacebookBot	Meta	Training	Meta AI and Facebook crawler
Applebot	Apple	Search index	Apple search, Siri, and Spotlight suggestions crawler
Applebot-Extended	Apple	Training	Apple generative AI data usage control
Amazonbot	Amazon	Search index	Amazon crawler powering Alexa and Amazon AI answers
MistralAI-User	Mistral AI	Assistant fetch	User-triggered retrieval for Le Chat
cohere-ai	Cohere	Training	AI crawler token
cohere-training-data-crawler	Cohere	Training	Training data crawler token
DuckAssistBot	DuckDuckGo	Assistant fetch	Fetches pages to ground DuckAssist, DuckDuckGo's AI answers. Blocking removes your content from those answers.
Bytespider	ByteDance	Training	ByteDance AI and search crawler (Doubao)
Baiduspider	Baidu	Search index	Baidu search and Ernie AI crawler
PetalBot	Huawei (Petal)	Search index	Huawei Petal search and AI crawler
Sogou Spider	Sogou (Tencent)	Search index	Sogou search and AI crawler
360Spider	Qihoo 360	Search index	Qihoo 360 search and AI crawler
YandexBot	Yandex	Search index	Yandex search and YandexGPT crawler
CCBot	Common Crawl	Training	Open web corpus crawler widely used for LLM training
AI2Bot	Allen Institute for AI	Training	Open language-model training data crawler
YouBot	You.com	Search index	You.com AI search crawler
Diffbot	Diffbot	Other	Knowledge graph and web search crawler
Diffbot-User	Diffbot	Assistant fetch	User-triggered Diffbot retrieval
Timpibot	Timpi	Search index	Decentralized search and AI data crawler
Omgilibot (Webz.io)	Webz.io	Training	Web data feeds used for AI training datasets
ImagesiftBot	ImageSift	Other	Image-oriented AI crawler
VelenPublicWebCrawler	Velen	Other	Public web crawler aggregating AI training data

before you block

See which AI bots already visit your site

Citlyze records AI crawler visits server-side: which bots came, what they fetched, and how often. Decide your robots.txt policy from data instead of guesswork.

Start the 14-day trial Book a demo

FAQ

AI crawlers and robots.txt, answered

Want a demo of Citlyze? Book a demo

Use the block training, allow search preset: it disallows GPTBot (training) while leaving OAI-SearchBot (the search index) and ChatGPT-User (on-demand page reading) allowed. OpenAI documents these as separate tokens precisely so sites can make that split.

Does AI recommend you when buyers ask?

Track how ChatGPT, Gemini, Perplexity, DeepSeek, ByteDance, and Grok mention, cite, and rank your brand. Review tracked answers and the gaps to close.

Start free trial Book a demo

No credit card required