AI Crawler Access Checker

Your robots.txt was written for Google. This tells you what it says to the AI crawlers — which ones you're blocking, which you're allowing, and what each decision actually costs or protects.

Why this matters now

Blocking “AI bots” is not one decision — it’s several. Blocking GPTBot keeps you out of OpenAI’s training data but has nothing to do with whether ChatGPT can cite you: that’s OAI-SearchBot (indexing) and ChatGPT-User (live lookups). Google-Extended only controls Gemini training — blocking it doesn’t touch your Google Search rankings, and allowing it doesn’t help them. Many sites block crawlers they meant to allow, or vice versa, because the names are genuinely confusing.

What it checks

  • Your live robots.txt, parsed with proper longest-match precedence, * wildcards, and $ anchors — the same way compliant crawlers parse it
  • Sixteen crawlers across three groups: AI training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent), AI search & assistants (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User), and traditional search (Googlebot, Bingbot, DuckDuckBot)
  • Whether an llms.txt file exists
  • Whether your robots.txt declares a sitemap

What it doesn’t check

robots.txt is a request, not a firewall. A blocked bot that ignores robots.txt can still crawl you, and a WAF can block a bot your robots.txt allows. To see what actually happens when these bots hit your server, use the Crawler Reality Check.

This tool fetches your live robots.txt and evaluates it per RFC 9309 (longest-match rules, wildcards, group merging) against each crawler's documented user-agent token. It also checks for an llms.txt file. Nothing is stored.