Crawler-labelled requests.
A recent sample of up to 10,000 requests matching 25 AI and search user-agent labels. These labels are supplied by the requester; they do not verify who made the request. This log stores no IP addresses; this view shows aggregates only.
| User-agent label | Requests | Paths | Latest in sample |
|---|---|---|---|
| meta-externalagentMeta | 6,536 | 288 | |
| GooglebotGoogle | 719 | 205 | |
| GPTBotOpenAI | 546 | 182 | |
| AmazonbotAmazon | 469 | 252 | |
| BytespiderByteDance | 379 | 159 | |
| OAI-SearchBotOpenAI | 331 | 50 | |
| CCBotCommon Crawl | 256 | 244 | |
| ChatGPT-UserOpenAI | 201 | 47 | |
| PerplexityBotPerplexity | 198 | 86 | |
| BingbotMicrosoft | 153 | 77 | |
| Claude-UserAnthropic | 61 | 23 | |
| ApplebotApple | 58 | 22 | |
| ClaudeBotAnthropic | 35 | 28 | |
| DuckAssistBotDuckDuckGo | 30 | 12 | |
| YouBotYou.com | 13 | 12 | |
| Claude-SearchBotAnthropic | 7 | 7 | |
| Perplexity-UserPerplexity | 6 | 6 | |
| cohere-aiCohere | 2 | 2 |
Totals cover the latest 10,000 matching records available at the last successful read. Older records may fall outside this sample.
- Bytespider
- meta-externalagent
- Amazonbot
- Googlebot
- Claude-User
- ChatGPT-User
- Bingbot
- OAI-SearchBot
- GPTBot
- CCBot
- Applebot
- ClaudeBot
- PerplexityBot
- DuckAssistBot
- YouBot
- Claude-SearchBot
- Perplexity-User
- cohere-ai
The earliest request for each user-agent label in this recent sample, not its first-ever match.
- Claude-Web · Anthropic
- anthropic-ai · Anthropic
- Google-Extended · Google
- FacebookBot · Meta
- Applebot-Extended · Apple
- cohere-training-data-crawler · Cohere
- MistralAI-User · Mistral
These records are written before the downstream response. They do not show whether content was retrieved successfully, indexed, cited or recommended. A user-agent label can also be supplied by a test or another client. Successful reads refresh hourly; failed reads are retried on a later request.