Crawler Reality Check

robots.txt is what you ask crawlers to do. This is what your server actually does to them — fetches your URL with real crawler user-agents and diffs the responses against a normal browser.

The failure this catches

The most common serious technical SEO incident of the CDN era: a WAF rule, bot-fight setting, or rate limiter starts serving 403s or challenges to Googlebot, and nobody notices until traffic falls off a cliff weeks later. The page looks fine in your browser — that’s the whole problem. The same thing now silently decides whether AI assistants can read and cite you at all.

What it compares

For a normal browser plus five major crawlers (Googlebot Smartphone, Bingbot, GPTBot, ClaudeBot, PerplexityBot), the tool records and diffs:

  • Final status code and the full redirect chain
  • Title, meta robots / X-Robots-Tag (a noindex served only to bots is a classic cloaking accident)
  • Canonical URL
  • Response size, as a coarse signal that bots get meaningfully different content

Reading the results honestly

A 403 or challenge served to a spoofed Googlebot user-agent doesn’t always mean real Googlebot is blocked — good bot protection verifies crawlers by IP, and this tool’s requests come from Cloudflare’s edge, not Google’s network. The tool labels these cases separately and tells you how to confirm with your server logs or Search Console’s Crawl Stats instead of guessing.

Requests are sent from Cloudflare's edge with each crawler's published user-agent string. Sites that verify crawlers by IP address (as Google and others recommend) may challenge these requests even though they'd serve the real bot normally — the tool flags that case rather than calling it a block. Nothing is stored.