Policy signals

Robots.txt for AI

See which AI user agents appear in robots.txt directives and how publishers are allowing or blocking them.

Range
7d 30d 90d
Current Signal 7d
GPTBot 652
ClaudeBot 548
Google-Extended 519
CCBot 514

Robots.txt user agents

Updated: 2026-06-08T00:00:00Z

Use this to: check which AI user agents need explicit robots.txt directives.

GPTBot 652
ClaudeBot 548
Google-Extended 519
CCBot 514
Bytespider 437
Googlebot 412
PerplexityBot 386
meta-externalagent 384
Amazonbot 381
Applebot-Extended 357
facebookexternalhit 351
ChatGPT-User 347
OAI-SearchBot 284
bingbot 269
anthropic-ai 244

What To Do

Robots.txt policy analyzer

Check AI user agents against the policy before shipping allow or block rules.

GPTBot Agent policy
ClaudeBot Agent policy
Google-Extended Agent policy
CCBot Agent policy
Bytespider Agent policy

Agent policy

7d

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

robots.txt is becoming the public negotiation layer between site owners and AI crawlers.

What To Do

  • Name the exact user agents you care about.
  • Separate allow, disallow, and crawl-delay intent.
  • Re-test managed robots.txt features after CDN or WAF changes.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.