Can robots.txt enforce crawler behavior?

It declares policy. Enforcement still depends on crawler compliance and any edge controls you operate.

Should I block GPTBot and allow search bots?

That depends on your content economics. Many publishers distinguish training crawlers from search/referral crawlers.

Policy signals

Robots.txt for AI

See which AI user agents appear in robots.txt directives and how publishers are allowing or blocking them.

Range

7d 30d 90d

Current Signal90d

GPTBot725

ClaudeBot628

Google-Extended576

CCBot526

Robots.txt user agents

Updated: 2026-10-12T00:00:00Z

Use this to: check which AI user agents need explicit robots.txt directives.

GPTBot725

ClaudeBot628

Google-Extended576

CCBot526

PerplexityBot522

ChatGPT-User493

Bytespider461

OAI-SearchBot445

Googlebot431

facebookexternalhit383

Amazonbot374

meta-externalagent370

Applebot-Extended362

anthropic-ai341

bingbot320

What To Do

Robots.txt policy analyzer

Check AI user agents against the policy before shipping allow or block rules.

GPTBotAgent policy

ClaudeBotAgent policy

Google-ExtendedAgent policy

CCBotAgent policy

PerplexityBotAgent policy

Robots.txt

Agent policy

90d

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

robots.txt is becoming the public negotiation layer between site owners and AI crawlers.

What To Do

Name the exact user agents you care about.
Separate allow, disallow, and crawl-delay intent.
Re-test managed robots.txt features after CDN or WAF changes.

Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.