Crawler-labelled requests.
A recent sample of up to 10,000 requests matching 25 AI and search user-agent labels. These labels are supplied by the requester; they do not verify who made the request. This log stores no IP addresses; this view shows aggregates only.
| User-agent label | Requests | Paths | Latest in sample |
|---|---|---|---|
| meta-externalagentMeta | 8,381 | 169 | |
| OAI-SearchBotOpenAI | 1,230 | 260 | |
| PerplexityBotPerplexity | 162 | 125 | |
| GooglebotGoogle | 51 | 14 | |
| AmazonbotAmazon | 49 | 49 | |
| BingbotMicrosoft | 24 | 14 | |
| ApplebotApple | 21 | 15 | |
| Claude-UserAnthropic | 16 | 8 | |
| ChatGPT-UserOpenAI | 12 | 7 | |
| BytespiderByteDance | 10 | 8 | |
| DuckAssistBotDuckDuckGo | 9 | 6 | |
| GPTBotOpenAI | 7 | 6 | |
| CCBotCommon Crawl | 5 | 5 | |
| Google-ExtendedGoogle | 5 | 5 | |
| cohere-aiCohere | 5 | 5 | |
| ClaudeBotAnthropic | 4 | 4 | |
| Perplexity-UserPerplexity | 3 | 3 | |
| Claude-SearchBotAnthropic | 2 | 2 | |
| MistralAI-UserMistral | 2 | 2 | |
| YouBotYou.com | 2 | 2 |
Totals cover the latest 10,000 matching records available at the last successful read. Older records may fall outside this sample.
- meta-externalagent
- Bingbot
- Amazonbot
- Googlebot
- Bytespider
- OAI-SearchBot
- PerplexityBot
- Applebot
- ChatGPT-User
- GPTBot
- Claude-User
- DuckAssistBot
- Claude-SearchBot
- Perplexity-User
- ClaudeBot
- Google-Extended
- CCBot
- cohere-ai
- YouBot
- MistralAI-User
The earliest request for each user-agent label in this recent sample, not its first-ever match.
- Claude-Web · Anthropic
- anthropic-ai · Anthropic
- FacebookBot · Meta
- Applebot-Extended · Apple
- cohere-training-data-crawler · Cohere
These records are written before the downstream response. They do not show whether content was retrieved successfully, indexed, cited or recommended. A user-agent label can also be supplied by a test or another client. Successful reads refresh hourly; failed reads are retried on a later request.