By bot
Every crawler and assistant seen in the window, its vendor, why it fetched (training, AI search index, live fetch for a user, or assistant), hits, pages and last seen.
Your access log already records every visit from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers. Upload it, or let the WordPress plugin report for you, and MrKeyword shows which bot fetched which page, when, and whether AI answers cite a page no bot ever read.
Works with logs you already have · 90-day history
GPTBot and the other AI crawlers do not run JavaScript. Your analytics script never fires, so the access log is the only place their visits show up. MrKeyword reads that log and keeps the bot lines only.
Every crawler and assistant seen in the window, its vendor, why it fetched (training, AI search index, live fetch for a user, or assistant), hits, pages and last seen.
The pages AI bots fetched most, with the bots that read each one. Pages that AI answers cite are marked, so a crawler hit and a citation can be compared.
A daily bar chart of bot requests over the window, with the split by vendor on hover. Useful after a launch, a migration or a robots.txt change.
Pages AI Visibility saw cited in an answer that no AI bot fetched in the same window. The answer may be using older training data or a cache.
Hits are kept for 90 days. Upload logs for several days in a row and the counts build up instead of replacing each other.
Only known bot lines are stored. Human visits are dropped while the log is parsed, before anything is written, and only the hits you upload are kept.
The view lists each bot with its purpose and counts. This is a sample table to show the columns; your own view is built from your log.
| Bot | Vendor | Purpose | Hits | Pages |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | 412 | 68 |
| OAI-SearchBot | OpenAI | AI search index | 186 | 54 |
| ChatGPT-User | OpenAI | Live fetch for a user | 97 | 41 |
| ClaudeBot | Anthropic | Training | 73 | 39 |
| PerplexityBot | Perplexity | AI search index | 51 | 28 |
| Googlebot | Search index | 1,204 | 112 |
Any one of these routes is enough. The upload accepts up to 20 MB of plain text at a time.
In the app, open AI Crawlers and paste log lines or choose the file. Nginx and Apache combined logs, Cloudflare Logpush JSON lines and CSV exports with a user-agent column are read. Only bot lines are kept.
grep -iE 'gptbot|claudebot|perplexity|google-extended|bingbot|applebot|ccbot' access.log > ai-bots.log
Create a Logpush job for HTTP requests with EdgeStartTimestamp, ClientRequestURI, ClientRequestUserAgent and EdgeResponseStatus, then upload the NDJSON files it writes.
The MrKeyword Connector can report bot visits itself. In the AI Crawlers view, create a key, then paste it in WordPress under Settings → MrKeyword and turn reporting on. Hits are sent every ten minutes.
Purpose labels come from the crawler's own documented role, not a guess. They explain why a page was fetched.
Your own server access logs, or the WordPress plugin when it is turned on. AI crawlers request pages like any other visitor but they do not run scripts, so a JavaScript analytics tag never sees them. The access log is the record that does.
OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended, GoogleOther), Apple (Applebot, Applebot-Extended), Common Crawl (CCBot), Amazon (Amazonbot), Meta (meta-externalagent, FacebookBot), ByteDance (Bytespider), DuckDuckGo (DuckAssistBot), You.com (YouBot), Cohere, Mistral, Diffbot, Timpi and Huawei. Googlebot and Bingbot are kept too, so you can compare AI fetches with search-engine crawls.
No. Only lines whose user agent matches a known bot signature are kept. Human visits are dropped while the log is parsed, inside the request, before anything is written to the database.
90 days. Uploads older than that are removed, and a new upload replaces nothing: hits are added to the same site so several days of logs can be uploaded one after another.
No. It reports what actually fetched your pages. A bot stopped by robots.txt or a firewall never appears, so an empty row means it did not fetch in the window, not that it tried and failed. The robots.txt tester shows the rule that would decide it.
No. Upload an Nginx or Apache access log, or Cloudflare Logpush files. The plugin is an alternative for WordPress sites: turn reporting on and it sends known bot visits every ten minutes. Both routes feed the same view.
When AI Visibility has run for the site, the view lists pages that AI answers cite and checks whether an AI bot fetched them in the same window. A cited page that no bot fetched may be answered from older training data or a cache, so recent edits are not what the AI is reading.
Upload one access log and the AI Crawlers view is populated. No third-party analytics and no script on your pages.