Claude SEO: How Anthropic's Claude Finds and Reads Websites
Claude can search the web and fetch pages, and Anthropic runs three separate bots for that: ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for pages a person asks about. Anthropic says all three honor robots.txt, so one rule can hide a site from Claude's answers or only opt it out of training. Test all three for free below.
By the Netbase JSC team · Updated
How does Leadee work with Claude?
- Uses the official API: Anthropic API (web search tool)
Can Anthropic's bots reach your site?
Leadee is not affiliated with, endorsed by, or sponsored by Anthropic. “Claude”, “ClaudeBot” and the other bot names on this page are Anthropic's own, used only to describe what this tool measures.
What are Anthropic's crawlers, and what does each one do?
Anthropic says all three bots honor robots.txt, including the non-standard Crawl-delay directive, and recommends robots.txt rather than IP blocking as the way to control them. It publishes its bots' IP ranges at claude.com/crawling/bots.json.
| robots.txt token | What it does | Blocking it means |
|---|---|---|
| ClaudeBot | Collects public web content that may be used to train Anthropic's models. | Opting out of training only. |
| Claude-SearchBot | Crawls pages to improve the quality of Claude's search results. | Claude's search may not index the site. |
| Claude-User | Fetches a page when a Claude user asks a question that needs it. | Claude cannot read the page a user asked about. |
Sources: Anthropic help center: does Anthropic crawl data from the web?; RFC 9309: Robots Exclusion Protocol (read 2026-11-10).
How does Leadee measure Claude?
Leadee's tracker sends buyer questions through Anthropic's Messages API with its web search tool and records the brands named and the domains cited. Search costs 10 USD per 1,000 searches plus tokens, about 0.056 USD per tracked question, which is why Claude is measured in paid plans only and has no free answer check.
Once GA4 is connected, visits referred by claude.ai count as AI traffic.
Tutorial
Insight
Tool
Service
Disclosure
Leadee is not affiliated with or endorsed by Anthropic.
Related
Platform guidePlatforms
Free toolClaude bot access test
Free toolAI Crawlers and Robots.txt
GEO guideHow AI Chooses Suppliers
GEO guideGEO vs SEO
Free toolRobots.txt generator
What do people ask about Claude SEO?
If I block ClaudeBot, will Claude stop citing my site?
Not by itself. ClaudeBot collects training data; Claude-SearchBot and Claude-User are the bots tied to search and user-requested fetches. Blocking only ClaudeBot opts out of training and leaves the other two free to read your pages.
Why is Claude not in Leadee's free checks?
Each tracked Claude question costs about 0.056 USD in API search and token fees, more than the other engines, so Claude answers are measured in paid plans. The bot access test is free because it never calls Claude.
Does Anthropic support Crawl-delay?
Yes. Anthropic says its bots honor the non-standard Crawl-delay directive in robots.txt, along with normal Allow and Disallow rules.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email support@leadee.ai or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.