Can robots.txt enforce crawler behavior?

It declares policy. Enforcement still depends on crawler compliance and any edge controls you operate.

Should I block GPTBot and allow search bots?

That depends on your content economics. Many publishers distinguish training crawlers from search/referral crawlers.

Policy signals

Robots.txt for AI

See which AI user agents appear in robots.txt directives and how publishers are allowing or blocking them.

Range

7d 30d 90d

Current Signal30d

GPTBot827

ClaudeBot726

Google-Extended690

CCBot654

Robots.txt user agents

Updated: 2026-08-24T00:00:00Z

Use this to: check which AI user agents need explicit robots.txt directives.

GPTBot827

ClaudeBot726

Google-Extended690

CCBot654

Bytespider580

meta-externalagent525

Amazonbot505

Applebot-Extended497

PerplexityBot477

ChatGPT-User430

Googlebot427

facebookexternalhit378

OAI-SearchBot370

bingbot297

anthropic-ai283

What To Do

Robots.txt policy analyzer

Check AI user agents against the policy before shipping allow or block rules.

GPTBotAgent policy

ClaudeBotAgent policy

Google-ExtendedAgent policy

CCBotAgent policy

BytespiderAgent policy

Robots.txt

Agent policy

30d

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

robots.txt is becoming the public negotiation layer between site owners and AI crawlers.

What To Do

Name the exact user agents you care about.
Separate allow, disallow, and crawl-delay intent.
Re-test managed robots.txt features after CDN or WAF changes.

Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.