Crawler demand

AI Bot Traffic

Track which AI crawlers are requesting pages, what they are trying to do, and where demand concentrates.

Range
7d30d90d
Current Signal7d
Googlebot25.1
ClaudeBot17.3
Meta-ExternalAgent12.7
GPTBot9.7

AI bot share

Updated: 2026-07-15T17:30:00Z

Use this to: decide which crawlers deserve monitoring, allow rules, or tighter limits.

Googlebot25.1
ClaudeBot17.3
Meta-ExternalAgent12.7
GPTBot9.7
Bingbot8.8
Applebot6.4
Amazonbot5.8
Bytespider5.4
other5.3
Claude-SearchBot3.3

Crawl purpose

Updated: 2026-07-15T17:30:00Z

Use this to: separate answer/search traffic from training and unknown crawler activity.

Training45.6
Mixed Purpose40.4
Search10.9
User Action2.5
Undeclared0.6

Top locations

Updated: 2026-07-15T18:30:00Z

Use this to: spot regional crawl pressure before changing blocks, caching, or infrastructure.

Gibraltar (GI)91.8
Iran (IR)75.7
Singapore (SG)74.9
Ireland (IE)73.6
Netherlands (NL)69.3
Finland (FI)63.2
Isle of Man (IM)57.4
Hong Kong (HK)55.8
United States (US)49.3
Russian Federation (RU)48.7
China (CN)48.7
Myanmar (MM)48.5
Germany (DE)47.6
Belgium (BE)44.8
Luxembourg (LU)44.6

Country activity

Use geography as a planning signal before changing crawl policy or regional infrastructure.

Why This Matters

AI crawler mix tells publishers whether traffic is coming from training, search, assistants, or emerging agent workflows.

What To Do

  • Separate training bots from answer/search bots.
  • Watch status-code shifts after robots.txt changes.
  • Compare top locations before blocking entire regions.
Data source: Cloudflare Radar. Processed by LLMBase for publisher and agent-readiness workflows.