Crawler-labelled requests.

A recent sample of up to 10,000 requests matching 25 AI and search user-agent labels. These labels are supplied by the requester; they do not verify who made the request. This log stores no IP addresses; this view shows aggregates only.

10,000Matching GET requestsin this recent sample
18 of 25User-agent labels matchedin this recent sample
458Distinct paths requestedin this recent sample
Sample totals, most requests first – (UTC)
User-agent labelRequestsPathsLatest in sample
meta-externalagentMeta6,536288
GooglebotGoogle719205
GPTBotOpenAI546182
AmazonbotAmazon469252
BytespiderByteDance379159
OAI-SearchBotOpenAI33150
CCBotCommon Crawl256244
ChatGPT-UserOpenAI20147
PerplexityBotPerplexity19886
BingbotMicrosoft15377
Claude-UserAnthropic6123
ApplebotApple5822
ClaudeBotAnthropic3528
DuckAssistBotDuckDuckGo3012
YouBotYou.com1312
Claude-SearchBotAnthropic77
Perplexity-UserPerplexity66
cohere-aiCohere22

Totals cover the latest 10,000 matching records available at the last successful read. Older records may fall outside this sample.

Earliest match in this sample
  1. Bytespider
  2. meta-externalagent
  3. Amazonbot
  4. Googlebot
  5. Claude-User
  6. ChatGPT-User
  7. Bingbot
  8. OAI-SearchBot
  9. GPTBot
  10. CCBot
  11. Applebot
  12. ClaudeBot
  13. PerplexityBot
  14. DuckAssistBot
  15. YouBot
  16. Claude-SearchBot
  17. Perplexity-User
  18. cohere-ai

The earliest request for each user-agent label in this recent sample, not its first-ever match.

No recorded matches in this sample (7)
  • Claude-Web · Anthropic
  • anthropic-ai · Anthropic
  • Google-Extended · Google
  • FacebookBot · Meta
  • Applebot-Extended · Apple
  • cohere-training-data-crawler · Cohere
  • MistralAI-User · Mistral

These records are written before the downstream response. They do not show whether content was retrieved successfully, indexed, cited or recommended. A user-agent label can also be supplied by a test or another client. Successful reads refresh hourly; failed reads are retried on a later request.