Index checker
Updated
Paste a URL and Leadee fetches the page, its robots.txt and its sitemap to check what decides whether a crawler can index it — HTTP status, a noindex tag, a robots.txt block, a canonical that does not match the page, and whether it is listed in the sitemap — then checks robots.txt again for four named AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) and adds a public-search estimate of whether Google already has the page indexed.
Run it now
How to use
- Paste the URL you want to check.
- "Can it be indexed?" is the gate — a noindex tag, a robots.txt block, an HTTP error or a canonical pointing elsewhere keeps a page out no matter what else is right.
- Check the AI crawler row for GPTBot, ClaudeBot, PerplexityBot and Google-Extended — a block there is separate from whether Google can index the page.
- Read the Google line as a rough estimate, not a fact — it is a public search, not a Search Console report, unless you are signed in with a connected property.
- Fix the highest-impact issue first (usually noindex or a robots.txt block), then re-check the same URL.
Example result
1 fail, 1 warn
- Page indexable: HTTP 200, no noindex, robots.txt allows it
- PerplexityBot blocked by robots.txt
- Not listed in the sitemap yet
How to read your result
Treat "Can it be indexed?" as the gate — a fail there (noindex, a robots.txt block, an HTTP error) keeps a page out regardless of anything else. The AI crawler row is separate: a page can be fully indexable while still blocked for one AI bot, or the other way round. The Google line is always labelled as either a real Search Console verdict (shown only when you are signed in with a connected property) or a public-search estimate — never presented as the same thing.
Limitations
The "Is it in Google?" line is a public site: search estimate, not a Google Search Console fact, and the shared daily search budget can hide it entirely — the indexability and AI crawler checks above it still work either way. This reads your own page, robots.txt and sitemap live; it does not simulate a live fetch with each bot's own user agent (see the robots.txt tester for that), and each URL is cached for 24 hours.
Questions
Is this the same as Google Search Console?
No. Everything under "Can it be indexed?" and "Can AI crawlers read it?" is Leadee's own live read of the page, its robots.txt and its sitemap — the only thing anyone can know for certain about a URL without owning it. "Is it in Google?" is a public site: search estimate unless you are signed in with a Leadee Index Push connection for that property, in which case it switches to a real URL Inspection verdict read from your own Search Console data.
Why would an AI crawler be blocked separately from Google?
robots.txt rules are per bot: a site can allow Googlebot while blocking a specific AI crawler (or the reverse) with its own User-agent group. This tool reads the four AI crawlers named in Leadee's methodology — GPTBot and Google-Extended (training), ClaudeBot (training) and PerplexityBot (search) — separately from the Googlebot rule used for the indexability check.
What exactly counts as "indexable" here?
The HTTP status code, a noindex directive in either the meta robots tag or the X-Robots-Tag header, whether robots.txt allows Googlebot at that path, whether the canonical tag (if any) points back to the page itself, and whether the page is listed in the sitemap with a lastmod date — the same facts a crawler itself relies on before it will index a page.
Why did the IndexNow key check come back empty?
The IndexNow key presence check only ever returns a result for a domain Leadee already tracks through Index Push — for any other domain it honestly reports that Leadee has no way to know, rather than guessing.
How often can I run this, and is the result cached?
Each URL is cached for 24 hours, and this tool is rate-limited to 20 runs per hour per visitor — the same shared-budget limits as Leadee's other free tools.
Can GEO guarantee results?
No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.
How does Leadee measure AI visibility?
Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.
Who do I talk to?
Email [email protected] or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.
Related
Keep learning
Methodology
Try it
Free tools
Robots.txt tester
Watch your whole sitemap, not one URL at a time.
Leadee’s Index Push submits changed URLs to Google and Bing and flags pages that drop out of the index — not a one-off check.