Can AI see your site?
GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript — and your WAF may be silently blocking them. Free 60-second audit: reachability, speed, and whether your content exists in raw HTML. A site can rank #1 in Google and be invisible to every AI answer engine.
By scanning you confirm you own or are authorized to test this domain. ~15 polite requests, cached 24h.
What the scan checks
Four things have to be true before any AI system can cite you, in this order. Almost everything the generative-engine-optimization industry sells — schema markup, entity coverage, citation-friendly phrasing — sits downstream of all four. If you fail gate one, nothing downstream of it runs.
Reachability
Does the crawler get a 200?Bot-management rules ship with defaults, and defaults do not know which crawlers you want. A 403 or a challenge page to a retrieval crawler means you are absent from answers that cite your competitors. This gate fails silently — a crawler that never got a page leaves no trace in your analytics.
Speed
How slow is a cold request?Crawlers land on long-tail URLs your cache has never seen, so the cold-start number is the one that applies to them, not the warm number your monitoring reports. Slow origins get abandoned mid-fetch, which your log records as a 499 and nothing else notices.
Readability
How many words survive with JavaScript off?GPTBot, ClaudeBot and PerplexityBot do not run JavaScript. Googlebot, which feeds Gemini, is the only major AI-adjacent crawler that renders. A client-rendered site can rank first in Google and be a blank page to every other AI system.
Permission
What does robots.txt actually say?Including the tokens that never fetch. Google-Extended and Applebot-Extended are opt-out signals, not crawlers — they will never appear in your logs because they never make a request. Googlebot and Applebot do the fetching; the token governs the training use.
The 12 crawlers we probe
Each is sent as a separate request and compared against a baseline browser request to the same URL, seconds apart. A difference between the two is the finding — that is what distinguishes bot-specific filtering from a site that is simply down or slow.
| User-agent | Type | Operator and purpose |
|---|---|---|
| GPTBot | training | OpenAI — model training corpus |
| OAI-SearchBot | retrieval | OpenAI — index behind ChatGPT search |
| ChatGPT-User | user fetch | OpenAI — live fetch when a user shares a link |
| ClaudeBot | training | Anthropic — model training corpus |
| Claude-SearchBot | retrieval | Anthropic — search index |
| Claude-User | user fetch | Anthropic — live fetch on request |
| PerplexityBot | retrieval | Perplexity — answer index |
| Perplexity-User | user fetch | Perplexity — live fetch on request |
| bingbot | retrieval | Microsoft — feeds Copilot as well as Bing |
| Amazonbot | retrieval | Amazon — Alexa and shopping surfaces |
| CCBot | training | Common Crawl — upstream of most open datasets |
| meta-externalagent | training | Meta — Llama training corpus |
Google-Extended and Applebot-Extended are checked in robots.txt but never probed — they are permission tokens that never issue a request.
What this found across 18 major sites
Measured 7 August 2026. The full dataset is committed in the open-source repo, including the caveats and the sites where the probe could not reach a conclusion.
Licensing deals are visible in the status codes
The Guardian, which has a content deal with OpenAI, served GPTBot, OAI-SearchBot and ChatGPT-User a clean 200 — and returned 403 to ClaudeBot, PerplexityBot and CCBot. The New York Times, in litigation with OpenAI, returned 403 to nearly the entire field; only bingbot and Amazonbot got through.
robots.txt is a request; the WAF is the enforcement
Reddit's robots.txt disallows fourteen AI tokens with no exceptions. In practice GPTBot got a 403 and ClaudeBot and CCBot got 429s, while OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot and meta-externalagent all received a 200 from the same IP seconds apart. Figma runs the mirror image: it disallows seven tokens in robots.txt and returns 200 to every one of them.
Famous sites are blank to AI by accident
Airbnb's homepage carries 94 words of raw HTML. LinkedIn serves 23 words to anything without a session. Reddit's homepage contains one word before JavaScript runs. For contrast, Stripe serves 1,957 words and a 200 to every crawler tested, and Anthropic answered fastest at 0.102s warm.
The full write-up, including two findings that did not survive review of the data, is at Your firewall is your AI policy.
Questions
Why doesn't my analytics show this?
Because analytics runs in JavaScript, and the crawlers that matter here do not execute it. A bot that receives a 403, a challenge page, or an empty shell never fires your tracking. The failure is invisible from inside the browser — it is only visible from the request side, which is what this scan reads.
Isn't this just SEO?
It overlaps at the reachability layer and diverges at rendering. Googlebot renders JavaScript, so a client-rendered page can rank perfectly well in Google while being empty to GPTBot and ClaudeBot. Passing a Lighthouse or Search Console check tells you nothing about whether an AI crawler could read the page.
Does llms.txt help?
Almost certainly not, and this scan weights it at zero. Measured adoption data shows the overwhelming majority of llms.txt files receive no AI-crawler requests at all. It costs nothing to publish and it is not a substitute for being fetchable. We check for it and report it as information, not as a score component.
Is a spoofed user-agent the same as the real crawler?
No, and this is the honest limit of any active probe. We send requests from our own IP with a claimed identity in the header. Vendors verify their crawlers by published IP range, so a site can treat a real GPTBot differently from a request calling itself GPTBot. When our own baseline browser request is filtered, we withhold the score instead of reporting a number we cannot stand behind. Server-log analysis is the only way to settle what real crawlers received.
What is a 499 and why does it matter?
It is the status your log records when the client hung up before your origin answered. For AI crawlers it means the fetch was abandoned — no error page, no alert, no citation, and nothing in your uptime monitoring. It is the most common silent failure on slow origins and it cannot be detected by probing from outside.