Crawl/referral signal
Updated: 2026-06-12T07:30:00Z
Use this to: judge whether crawler demand is returning enough audience value.
Client and content type
Updated: 2026-06-12T07:30:00Z
Use this to: separate browser, search, and AI-agent demand before policy changes.
Country activity
Use geography as a planning signal before changing crawl policy or regional infrastructure.
Why This Matters
A high crawl-to-refer ratio can mean infrastructure cost without matching audience return.
What To Do
- Measure crawler volume beside referral sessions.
- Treat answer engines differently from training-only crawlers.
- Protect expensive pages with policy rather than blanket blocks.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.