AI Crawlers and Robots.txt: How to Let ChatGPT, Perplexity and Other AI Bots Read Your Site
By the Leadee team at Netbase JSC · Updated
AI crawlers fall into three groups: training bots (GPTBot, Google-Extended) that gather text for model training, search/user-fetch bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) that fetch pages to answer a live question, and general crawlers whose purpose is unclear. Each has its own user-agent token, so robots.txt can allow or block them separately. A firewall can still block a bot robots.txt allows — that is a different fix.
Key facts
- OpenAI runs three separate crawlers under three tokens: GPTBot (training), OAI-SearchBot and ChatGPT-User (both used for live ChatGPT search and link-fetch) — disallowing one does not disallow the others. (OpenAI, 2026)
- Cloudflare's "AI Scrapers and Crawlers" toggle (Security > Bots) blocks AI bots behind a single switch, free on every plan including the free tier, since July 3, 2024. (Cloudflare, 2024)
- Google's own robots.txt documentation says the file cannot enforce crawler behaviour — it is a request a crawler can choose to ignore, and "respectable" crawlers such as Googlebot follow it while others might not. (Google Search Central, 2025)
- Leadee's own crawlability check reads a tracked list of 174 AI-related user agents, split into search, training and general-purpose bots, and reports exactly which ones robots.txt blocks on your site today — see the AI crawlability check.
Which AI crawlers exist, and what do they actually do?
AI bots fall into three practical groups. Training bots (OpenAI's GPTBot, Google's Google-Extended, Anthropic's ClaudeBot, Common Crawl's CCBot) fetch pages to build or retrain a model — blocking one of these has no effect on whether that company's live product can answer a question about your site, because training and answering are different pipelines. Search and user-fetch bots (OpenAI's OAI-SearchBot and ChatGPT-User, Anthropic's Claude-SearchBot and Claude-User, PerplexityBot and Perplexity-User) fetch a page in response to an actual question someone typed — blocking these is closer to blocking a search engine. A third group of general crawlers has an unclear or mixed purpose and gets treated conservatively.
Each bot identifies itself with its own user-agent string, so robots.txt treats GPTBot and OAI-SearchBot as two completely different visitors even though both come from OpenAI. Leadee's AI crawlability check runs this exact classification against a site's live robots.txt and reports, bot by bot, whether it is allowed, blocked by robots.txt, or blocked by something else entirely (see the firewall section below).
How do I write robots.txt rules for AI bots?
A robots.txt rule is a `User-agent` line naming the bot, followed by `Allow` or `Disallow` lines for the paths that bot may or may not fetch. Rules are grouped per user agent, and a bot with no group of its own falls back to the `*` (wildcard) group.
The three common patterns: allow everything (no `Disallow` line, or an explicit `Allow: /`), block one specific bot by name while leaving everyone else alone, and block an entire category by listing every bot in it. A policy of "opt out of training, stay visible to search/user-fetch bots" is common and legitimate — it is a choice, not a mistake, as long as it is the choice you actually meant to make.
- Allow every AI bot (the default if robots.txt says nothing about them): `User-agent: *` then `Allow: /`.
- Block only GPTBot from training on your content, leave search bots alone: `User-agent: GPTBot` then `Disallow: /`.
- Block an entire category: repeat the same two-line block for every bot in that category — there is no wildcard for "all training bots," each user agent needs its own group.
- Point every crawler, AI or not, at your sitemap from the same file: add `Sitemap: https://example.com/sitemap.xml` on its own line — see sitemaps for AI search.
What do noindex, nosnippet and noai mean in meta robots?
A noindex directive — in a `<meta name="robots">` tag or an `X-Robots-Tag` HTTP header — tells any compliant crawler not to index the page at all, which Leadee's `geo.meta_robots` check treats as a high-severity failure: an AI engine that respects it cannot read or cite that page, full stop. This is the most common unintentional AI-visibility killer, usually left over from a staging-site tag that never got removed.
noai, noimageai and nosnippet are lighter-weight: they do not block indexing, only specific uses of the content (training, image training, and showing a text snippet, respectively). A page can be fully indexed and still carry one of these as a deliberate policy choice — the fix is to check whether that policy matches what you actually intend, not to remove it automatically.
- `<meta name="robots" content="noindex">` or header `X-Robots-Tag: noindex` — blocks indexing entirely. Remove it unless the page is genuinely meant to stay hidden.
- `<meta name="robots" content="noai">` — asks AI systems not to use the page for training. Valid policy choice; verify it is intentional.
- `<meta name="robots" content="nosnippet">` — blocks AI/search systems from quoting a text snippet from the page, even though the page can still be indexed and linked to.
- These directives can be set per page or site-wide via the server's default headers — check both the page HTML and the response headers, since either one alone can trigger the block.
Why does my firewall block AI bots even when robots.txt allows them?
robots.txt is a request a crawler chooses to honour; a web application firewall (WAF) or CDN edge rule is an actual gate that can reject a request before your server or CMS ever sees it, regardless of what robots.txt says. Leadee's `geo.waf_block` finding exists specifically for this gap: a live fetch using a bot's real user-agent string comes back with a 403, a 429, or a "just a moment, checking your browser" challenge page, even though the same bot's robots.txt rule says `Allow`.
Cloudflare is the most common cause, mainly its "AI Scrapers and Crawlers" toggle (Security > Bots in the dashboard) and its Bot Fight Mode, both of which can block a bot by signature independently of any robots.txt rule you wrote. The fix is an allow rule for that bot's user agent or published IP range in the firewall itself, not another edit to robots.txt.
How do I allow AI bots on WordPress, Shopify, Wix, Webflow, nginx or Apache?
Where the robots.txt rule lives, and how it is edited, depends entirely on the platform — some let you edit the raw file, some generate it from settings, and some require a workaround.
- WordPress with Yoast SEO: SEO > Tools > File editor, then delete the `Disallow` line for the blocked bot. With RankMath: General Settings > Edit robots.txt.
- Shopify: Online Store > Themes > Edit code > `templates/robots.txt.liquid`. Remove or override the default group for the blocked bot, or add an explicit `User-agent` / `Allow: /` block above the theme's default groups.
- Wix: Settings > SEO Tools > SEO Settings > Advanced SEO > robots.txt, or per page in Site SEO Overview. Wix does not allow a fully custom robots.txt on every plan — contact Wix support if the option is greyed out.
- Webflow: Project Settings > SEO > Indexing lets you edit robots.txt directly; remove the `Disallow` line and republish the site for the change to take effect.
- nginx: edit the `/robots.txt` file served from your server block's document root, then reload with `nginx -s reload`.
- Apache: edit `/robots.txt` directly in your `DocumentRoot` — it is a static file, no restart needed.
- Cloudflare (firewall, not robots.txt): Security > Bots — turn off "AI Scrapers and Crawlers" or Bot Fight Mode for the bots you want through, or add an explicit allow rule under Security > WAF > Custom rules for that bot's user agent.
What should I fix first?
- Run the AI crawlability check and look for any search or user-fetch bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) marked blocked — these affect whether your site can be cited in a live answer, not just a training run.
- Check for a site-wide noindex left over from a staging environment — it is the single most common and most severe finding.
- If a bot shows blocked by robots.txt as allowed but still fails a live fetch, look at your CDN/WAF next, starting with Cloudflare's Security > Bots settings.
- Decide, deliberately, whether you want to opt out of AI training (GPTBot, Google-Extended, ClaudeBot) — a valid choice — while leaving search/user-fetch bots untouched.
- Re-run the check after each change; a robots.txt edit and a firewall allow rule both take effect immediately, with no deploy needed on most platforms.
Questions
If I block GPTBot, will my company still show up in ChatGPT's answers?
Usually yes. GPTBot is OpenAI's training crawler; the bots behind a live ChatGPT answer are OAI-SearchBot and ChatGPT-User, which carry their own separate user-agent tokens and are not affected by a GPTBot-only Disallow rule.
What's the real difference between a training bot and a search bot?
A training bot gathers text to help build or improve a model offline, on its own schedule, unconnected to any specific question. A search or user-fetch bot fetches a page in direct response to a live question someone is asking right now — blocking it can remove you from that specific answer.
Does Cloudflare block AI bots by default on every site?
No — "AI Scrapers and Crawlers" and Bot Fight Mode are settings under Security > Bots that a site owner turns on or off; Cloudflare does not silently enable them without the account holder choosing to.
Do I need a different robots.txt rule for Googlebot than for GPTBot?
Only if you want different policies for each. They are separate user agents, so a rule aimed at one never affects the other — a common setup is to allow Googlebot and every search/user-fetch AI bot, while opting specific training bots out.
How do I check whether a bot is really blocked, beyond reading robots.txt myself?
Run a live crawlability check that fetches your homepage using that bot's real user-agent string, the way [Leadee's free AI crawlability check](/tools) does — robots.txt only tells you the stated policy, not whether a firewall is quietly overriding it.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
See which AI bots can actually read your site
Run the free check to see, bot by bot, what robots.txt, meta tags and your firewall allow today — then get it fixed.