Skip to content
Leadee
Free tool

Claude bot access test

Enter a URL and Leadee checks robots.txt for Anthropic's three documented bots — ClaudeBot, Claude-SearchBot and Claude-User — then live-fetches the homepage as ClaudeBot and Claude-SearchBot to tell a robots.txt rule apart from a firewall or CDN block. Not affiliated with or endorsed by Anthropic.

Updated

  • No email needed
  • Free, no sign-up
Test bot access

1 of 3 blocked

  • ClaudeBot: blocked by robots.txt
  • Claude-SearchBot: allowed
  • Claude-User: allowed (robots.txt only)

Example result

Run it now

Leadee is not affiliated with, endorsed by, or sponsored by Anthropic. “Claude”, “ClaudeBot” and the other bot names on this page are Anthropic's own, used only to describe what this tool measures.

At a glance

3 items
What it checks
4
Questions answered
No email needed
To get a result
Can AI read my site?
Answers

What does it check?

  • robots.txt rule for ClaudeBot

    robots.txt rule for ClaudeBot, Claude-SearchBot and Claude-User

  • A live fetch with the ClaudeBot and

    A live fetch with the ClaudeBot and Claude-SearchBot user agents, to catch a WAF/CDN block robots.txt would not show

  • Which bot is blocked, and whether the

    Which bot is blocked, and whether the block is robots.txt, a firewall, or neither

What does a result look like?

1 of 3 blocked

  • ClaudeBot: blocked by robots.txt
  • Claude-SearchBot: allowed
  • Claude-User: allowed (robots.txt only)

Sample from a real run, 10 Nov 2026.

How do I read and fix the result?

Anthropic splits its bots by job: ClaudeBot collects training data, Claude-SearchBot indexes pages to improve Claude's search results, and Claude-User fetches a page when a Claude user asks about it. Blocking ClaudeBot only opts out of training; blocking Claude-SearchBot or Claude-User is what can keep your pages out of Claude's answers.

  1. Enter the URL you want to check.

  2. Read the robots.txt verdict for all three bots and the live-fetch result for ClaudeBot and Claude-SearchBot.

  3. If a bot is blocked by robots.txt, update the rule; if blocked by a firewall/CDN, allow-list that bot there instead.

  4. Re-run the test after you change anything, to confirm the block is gone.

Who is it for?

An engineer in a hard hat and safety vest reading a tablet on the shop floor

Manufacturers

Product and spec pages AI engines need to parse correctly.

Three consultants reviewing a document on a tablet

Professional services firms

Service pages that need clear Organization and entity markup.

A businessman in a suit shaking hands with a supplier in an office

B2B suppliers

Catalogue and partner pages AI crawlers must be able to read.

What are the limits?

Good to know

What are the limits?

The live fetch spoofs the User-Agent header from Leadee's own server; Anthropic publishes its bots' IP ranges (claude.com/crawling/bots.json), so a firewall that verifies by IP may treat the real bot differently. Claude-User is not live-fetched — its row carries the robots.txt verdict only.

Questions

Is this tool affiliated with or endorsed by Anthropic?

No. Leadee is not affiliated with, endorsed by, or sponsored by Anthropic. ClaudeBot, Claude-SearchBot and Claude-User are Anthropic's own bot names, documented in Anthropic's help center article on its crawlers.

Do Anthropic's bots respect robots.txt?

Anthropic states that all three bots honor robots.txt directives, including the non-standard Crawl-delay. So a robots.txt rule is the documented way to control each one — Anthropic does not recommend blocking its bots by IP address.

Will blocking ClaudeBot remove my site from Claude's answers?

Not by itself. ClaudeBot collects training data; Claude-SearchBot and Claude-User are the bots tied to search and user-requested fetches. Blocking only ClaudeBot opts out of training while leaving the other two free to read your pages.

Why is Claude-User not live-fetched?

Claude-User only fetches a page when a person asks Claude about it, so Leadee's live probe covers the two crawlers (ClaudeBot and Claude-SearchBot) and reports the robots.txt verdict for Claude-User.

Can GEO guarantee results?

No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.

How does Leadee measure AI visibility?

Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.

Who do I talk to?

Email support@leadee.ai or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.

Monitor AI bot access on a schedule.

Sign up for a free account to get notified the moment a bot gets blocked, instead of checking by hand.

Create free accountTalk to an expert