Robots.txt generator
Pick a preset — allow AI search bots and block AI training bots, block all AI bots, or allow everyone — or choose a rule per bot yourself, and Leadee writes a robots.txt that keeps every rule you already have for non-AI bots, adds your Sitemap: line, and shows a diff against your current file so you can see exactly what changed before you publish it.
Run it now
How to use
- Optionally enter your site's URL (or paste your current robots.txt) so your existing rules are preserved.
- Pick a preset: allow AI search and block AI training, block all AI, allow everyone, or choose per bot.
- Add your sitemap URL so it is included as a Sitemap: line.
- Review the generated file and the diff against your current one, then copy or download it to publish.
What we check
Last updated 29 October 2026.
| Rule | Source | Checked |
|---|---|---|
| Non-AI-bot rules kept exactly as they parse | Leadee generator (own engine) | 29 October 2026 |
| AI bot groups replaced per the chosen preset | Leadee generator (own engine) | 29 October 2026 |
| Sitemap: lines preserved + yours appended | RFC 9309 §2.3 | 29 October 2026 |
| Result re-checked with the robots.txt tester | Leadee check access.robots_txt_health | 29 October 2026 |
Leadee never publishes anything to your site — you copy or download the generated file and upload it yourself.
See the full methodology at /methodology for how every Leadee check is scored and weighted.
Questions
Will this overwrite rules I already have for other bots?
No — the generator only replaces the User-agent groups for AI bots it knows about (from the same list the tester uses); any other rule (Googlebot, Bingbot's own group, a wildcard "*" group, Crawl-delay, Host) is kept exactly as it already parses, and re-appended unchanged.
Which preset should I use?
"Allow AI search, block AI training" is the most common choice for sites that want to appear in ChatGPT/Perplexity/AI Mode answers but opt out of having their content used to train future models — it is the default here. "Block all AI" opts out entirely; "Allow all" places no AI-specific rules at all (the same as having none).
What does the diff show me?
A standard unified diff (the same format Git uses) between the robots.txt you started with and the one the generator produced, so you can see exactly which lines were added, removed or left alone before you publish anything to your live site.
Does the generator publish the file to my site automatically?
No — Leadee never writes to your site. You copy or download the generated robots.txt and upload or paste it into your own site, CMS or plugin yourself.
Can I set a different rule for each bot myself?
Yes — choose the "Custom" preset and pick allow or disallow for each bot individually, instead of using one of the three presets.
Is the generated file checked for errors before I download it?
Yes — every generated file is run back through the same engine as the robots.txt tester, so the result you copy has already been confirmed free of syntax errors and oversized-file problems.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
Keep AI bot access right as your site changes, not just once.
Leadee tracks AI crawler access alongside the rest of your AI visibility — not a one-off check.