Robots.txt tester
Paste a robots.txt file or enter a URL and Leadee shows an allow/block matrix for every major AI bot (training crawlers like GPTBot and CCBot, and search/answer bots like OAI-SearchBot and PerplexityBot) at any path you choose, flags syntax errors, an oversized file (over the 500 KiB RFC 9309 limit) and a missing or relative Sitemap: line.
Run it now
How to use
- Enter your site's URL, or paste your robots.txt file text directly.
- Pick the path you want to test (defaults to the homepage, "/").
- Review the bot matrix: which AI bots are allowed or blocked at that path, and why.
- Fix any syntax, size or Sitemap: issues the checker flags, then re-test.
What we check
Last updated 29 October 2026.
| Rule | Source | Checked |
|---|---|---|
| AI search/answer bots allowed | Leadee check access.robots_search_bots | 29 October 2026 |
| AI training bots — reported, not penalized | Leadee check access.robots_training_bots | 29 October 2026 |
| File under 500 KiB, valid syntax, reachable | RFC 9309 §2.5 (check access.robots_txt_health) | 29 October 2026 |
| Sitemap: line present and an absolute URL | RFC 9309 §2.3 (check robots.sitemap_declared) | 29 October 2026 |
This tool reads your robots.txt the same way most crawlers do — one file, evaluated per RFC 9309 — it does not simulate a live fetch with each bot's real user agent against your server. For that, see the free AI crawlability check.
See the full methodology at /methodology for how every Leadee check is scored and weighted.
Questions
Why does robots.txt matter for AI search visibility?
Many sites accidentally block AI bots with a blanket "Disallow: /" rule meant for something else, or a security plugin that blocks "unknown" user agents by default. A single misplaced rule can remove a site from ChatGPT, Perplexity or Google AI Mode answers entirely, with no warning.
What is the difference between an AI "training" bot and a "search" bot?
Training bots (GPTBot, CCBot, Google-Extended, Applebot-Extended) crawl content to train future AI models — blocking them is a legitimate policy choice with no immediate visibility impact. Search/answer bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch pages in real time to answer a specific user question — blocking them removes you from that answer right away.
Does this tool check my actual server, or just the robots.txt text?
It reads your robots.txt file (fetched live or pasted) and computes what every bot is allowed to do per RFC 9309 — the same rules a real crawler follows. It does not probe your server live with each bot's real user agent; for that, see the free AI crawlability check.
What counts as a robots.txt syntax error?
A line that has no ":" at all (not a valid directive) is flagged as a syntax error. A directive name this tool does not recognize is flagged separately as informational, since crawlers simply ignore directives they don't understand rather than treating them as errors.
Why does the tool flag my Sitemap: line?
A Sitemap: line must be an absolute URL (starting with http:// or https://) — a relative path like "/sitemap.xml" is invalid there and most crawlers will not resolve it. The tool also flags when no Sitemap: line exists at all, since it is the first place most crawlers look.
Can I test a path other than my homepage?
Yes — enter any path (for example "/products/" or "/blog/some-post") and the bot matrix re-evaluates every bot against that specific path, since a Disallow rule for one section of a site often does not apply to others.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
Monitor your robots.txt and AI bot access, not just once.
Leadee tracks AI crawler access alongside the rest of your AI visibility — not a one-off check.