What I measured

Period: 19 to 25 October 2026. Four new, small sites that I operate. For every request whose user agent contains a bot token, I checked whether the client IP lies inside the IP list that the operator publishes (Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, Ahrefs), fetched the same day. I removed my own server's requests and my own manual tests.

Applebot and AhrefsBot were 100%.

There were 332 requests that did not match. 287 of them came from a single hosting network, and that network claimed eight different bots: Bingbot, CCBot, ChatGPT-User, ClaudeBot, Googlebot, GPTBot, OAI-SearchBot and PerplexityBot. Its reverse DNS did not confirm forward. A further 45 had no reverse DNS at all.

What this does not show

• It is one week on four new, small sites. Bigger or older sites will see a different mix.
• Published IP lists can lag behind new crawler addresses. "Unverified" means "not in the list", not "proven fake". The PerplexityBot and ChatGPT-User numbers in particular could be partly list lag.
• One host claiming eight unrelated crawlers from non-operator addresses is hard to explain as anything but user-agent spoofing. That is the only claim here I would call solid.

The practical point: a user-agent string is a claim, not an identity. If you decide anything based on it (allow, block, rate limit, count it in analytics), the operator's IP list or reverse DNS is the check that matters. Google and others document reverse-DNS verification, and OpenAI, Anthropic, Perplexity, Bing, Apple and Common Crawl publish JSON lists of ranges.

The other half: what do real bots get from your site?

Separately from spoofing, there is a more common problem: robots.txt says one thing and the server does another. A WAF rule returns 403 to a user agent that robots.txt allows. A redirect sends a bot to a login page. A User-agent: * group blocks a path, and a bot-specific group further down quietly overrides it (a bot with its own group ignores the * group).

crawlerpeek checks both layers for each bot: the robots.txt decision under RFC 9309 matching rules, and what the page returns when fetched with that bot's user agent.

npx crawlerpeek check https://example.com

Example output for a site with a WAF rule and a restrictive robots.txt:

https://example.com
robots.txt: ok (HTTP 200) at https://example.com/robots.txt

Bot                 robots    HTTP
GPTBot              blocked   200  FAIL
ClaudeBot           allowed   403 (block page)  FAIL
Google-Extended     allowed   -

Here GPTBot is closed by robots.txt even though the server would answer, and ClaudeBot is allowed by robots.txt but gets a block page. Both are things you would want to know before wondering why a product is never cited.

More forms:

npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked
npx crawlerpeek check https://example.com --bots all --json
npx crawlerpeek check https://example.com --no-http          # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot      # one bot, full detail

One limit matters for the topic of this post: crawlerpeek sends the bot's user-agent string from your machine. It is not the real bot and does not come from the operator's IPs. If your site verifies bot IPs, a request that only claims to be Googlebot may be blocked, so a 403 for Googlebot can be correct. A 403 for an unusual bot while a browser gets 200 usually points at a WAF or CDN rule.

Put it in CI

Exit code 0 means ok, 1 means the --fail-on condition was met, 2 means a usage or request error. Unknown options are an error, so a typo like --failon cannot pass a build silently.

With the composite action:

- uses: crawlerpeek/crawlerpeek/action@v0
  with: { url: "https://example.com", fail-on: blocked }

Or call the CLI yourself after a deploy:

- run: npx crawlerpeek check https://example.com --fail-on blocked

--fail-on blocked fails when robots.txt closes the path for a bot, or the page answers that bot with 401, 403, 406, 451 or a challenge page. any-non-200 is stricter and also fails on 404, 5xx and redirects that end elsewhere. Run it against the pages you care about, not only the home page.

A robots.txt that cannot be fetched is reported as unknown and does not count as blocked. Under RFC 9309 a 5xx robots.txt means "disallow all" for many crawlers, so treat unknown as something to fix.

Use it from Claude Code, Cursor or another MCP client

crawlerpeek mcp starts an MCP server over stdio with three read-only tools: check_ai_access, see_as_bot and generate_robots_txt. It calls no LLM; the tools only fetch the URL you give them, and results are cached for 10 minutes.

{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": ["-y", "crawlerpeek", "mcp"] } } }

Then you can ask your assistant "can ClaudeBot read /pricing on my staging site?" and it runs the check instead of guessing. It is also listed in the official MCP Registry as io.github.crawlerpeek/crawlerpeek.

To generate a robots.txt that names the AI crawlers explicitly:

npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml

Disallowed paths are repeated inside each bot's group on purpose, for the reason above.

Safety

One request per bot plus one for robots.txt, sequential, with a 300 ms pause; a 10 second timeout; at most 5 redirects, each re-checked; bodies capped at 2 MB; no cookies. Private, loopback and metadata addresses are refused after DNS resolution, as are ports other than 80 and 443.

Feedback

Source is MIT on GitHub: https://github.com/crawlerpeek/crawlerpeek. Package: https://www.npmjs.com/package/crawlerpeek (0.1.2). I would like to hear about missing bots, wrong user-agent strings and false block-page detections. Issues are open.