Robots.txt user agents
Updated: 2026-08-31T00:00:00Z
Use this to: check which AI user agents need explicit robots.txt directives.
What To Do
Robots.txt policy analyzer
Check AI user agents against the policy before shipping allow or block rules.
GPTBotAgent policy
ClaudeBotAgent policy
Google-ExtendedAgent policy
CCBotAgent policy
BytespiderAgent policy
Agent policy
7dCountry activity
Use geography as a planning signal before changing crawl policy or regional infrastructure.
Why This Matters
robots.txt is becoming the public negotiation layer between site owners and AI crawlers.
What To Do
- Name the exact user agents you care about.
- Separate allow, disallow, and crawl-delay intent.
- Re-test managed robots.txt features after CDN or WAF changes.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.