Skip to content
MrKeyword
AI SEO · AI crawlers

See which AI bots actually read your pages

Your access log already records every visit from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers. Upload it, or let the WordPress plugin report for you, and MrKeyword shows which bot fetched which page, when, and whether AI answers cite a page no bot ever read.

Works with logs you already have · 90-day history

Reads from
  • Nginx access logs
  • Apache access logs
  • Cloudflare Logpush
  • WordPress plugin
  • AI Visibility citations

Bot visits that analytics never sees

GPTBot and the other AI crawlers do not run JavaScript. Your analytics script never fires, so the access log is the only place their visits show up. MrKeyword reads that log and keeps the bot lines only.

By bot

Every crawler and assistant seen in the window, its vendor, why it fetched (training, AI search index, live fetch for a user, or assistant), hits, pages and last seen.

By page

The pages AI bots fetched most, with the bots that read each one. Pages that AI answers cite are marked, so a crawler hit and a citation can be compared.

Hits by day

A daily bar chart of bot requests over the window, with the split by vendor on hover. Useful after a launch, a migration or a robots.txt change.

Cited but never crawled

Pages AI Visibility saw cited in an answer that no AI bot fetched in the same window. The answer may be using older training data or a cache.

90 days of history

Hits are kept for 90 days. Upload logs for several days in a row and the counts build up instead of replacing each other.

Private by default

Only known bot lines are stored. Human visits are dropped while the log is parsed, before anything is written, and only the hits you upload are kept.

Sample data

The crawler table

The view lists each bot with its purpose and counts. This is a sample table to show the columns; your own view is built from your log.

Sample data: AI crawler hits for example.com over 30 days.
BotVendorPurposeHitsPages
GPTBotOpenAITraining41268
OAI-SearchBotOpenAIAI search index18654
ChatGPT-UserOpenAILive fetch for a user9741
ClaudeBotAnthropicTraining7339
PerplexityBotPerplexityAI search index5128
GooglebotGoogleSearch index1,204112

How to get your logs

Any one of these routes is enough. The upload accepts up to 20 MB of plain text at a time.

Upload an access log

In the app, open AI Crawlers and paste log lines or choose the file. Nginx and Apache combined logs, Cloudflare Logpush JSON lines and CSV exports with a user-agent column are read. Only bot lines are kept.

grep -iE 'gptbot|claudebot|perplexity|google-extended|bingbot|applebot|ccbot' access.log > ai-bots.log

Cloudflare Logpush

Create a Logpush job for HTTP requests with EdgeStartTimestamp, ClientRequestURI, ClientRequestUserAgent and EdgeResponseStatus, then upload the NDJSON files it writes.

WordPress plugin

The MrKeyword Connector can report bot visits itself. In the AI Crawlers view, create a key, then paste it in WordPress under Settings → MrKeyword and turn reporting on. Hits are sent every ten minutes.

How the WordPress connector works →

What one bot row tells you

Purpose labels come from the crawler's own documented role, not a guess. They explain why a page was fetched.

Training
The page is being collected for a model's training set, such as GPTBot, ClaudeBot, Google-Extended or CCBot.
AI search index
The crawler builds or refreshes a search index used by an AI answer, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot.
Live fetch for a user
A person asked the assistant about something and it fetched this page for that answer, such as ChatGPT-User or Perplexity-User.
Assistant
A general assistant or agent fetch, such as Claude-Web, GoogleOther or Amazonbot.
Not covered
The view reports what reached your server. It cannot see bots stopped before the request, and it does not control what a crawler does with a page it read.

Questions and answers

Where does the data come from?

Your own server access logs, or the WordPress plugin when it is turned on. AI crawlers request pages like any other visitor but they do not run scripts, so a JavaScript analytics tag never sees them. The access log is the record that does.

Which bots are recognised?

OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, anthropic-ai), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended, GoogleOther), Apple (Applebot, Applebot-Extended), Common Crawl (CCBot), Amazon (Amazonbot), Meta (meta-externalagent, FacebookBot), ByteDance (Bytespider), DuckDuckGo (DuckAssistBot), You.com (YouBot), Cohere, Mistral, Diffbot, Timpi and Huawei. Googlebot and Bingbot are kept too, so you can compare AI fetches with search-engine crawls.

Do you store human visits?

No. Only lines whose user agent matches a known bot signature are kept. Human visits are dropped while the log is parsed, inside the request, before anything is written to the database.

How long is the data kept?

90 days. Uploads older than that are removed, and a new upload replaces nothing: hits are added to the same site so several days of logs can be uploaded one after another.

Can it tell me whether an AI bot was blocked?

No. It reports what actually fetched your pages. A bot stopped by robots.txt or a firewall never appears, so an empty row means it did not fetch in the window, not that it tried and failed. The robots.txt tester shows the rule that would decide it.

Do I need the WordPress plugin?

No. Upload an Nginx or Apache access log, or Cloudflare Logpush files. The plugin is an alternative for WordPress sites: turn reporting on and it sends known bot visits every ten minutes. Both routes feed the same view.

What does the citation cross-reference show?

When AI Visibility has run for the site, the view lists pages that AI answers cite and checks whether an AI bot fetched them in the same window. A cited page that no bot fetched may be answered from older training data or a cache, so recent edits are not what the AI is reading.

See which AI bots read your pages

Upload one access log and the AI Crawlers view is populated. No third-party analytics and no script on your pages.

Open AI Crawlers Reads your own logs.