Which AI crawlers read your site?
Paste lines from your server or CDN access log, or upload the file, and see which AI crawlers requested which pages, how often, and what status they got. Your log never leaves your browser.
Example result from a sample log. Paste your own above to see yours.
74 of 176 requests came from 7 AI crawlers. Search engine crawlers made 28 more.
Your site refused 3 AI crawler requests with 401, 403 or 429. Check your firewall, CDN bot rules and rate limits.
Log lines from Nov 16, 2026, 12:00 AM to Nov 18, 2026, 11:53 PM UTC.
Counts trust the user agent each request sends, and anyone can fake one. Check important traffic against the IP ranges each operator publishes.
20 requests
Build the index AI search answers draw on.
OAI-SearchBot OpenAI
14
- /5
- /pricing4
- /blog/choosing-a-crm3
2 requests to /old-features returned 404.
and 1 more path
- /old-features2
PerplexityBot Perplexity
6
- /blog/choosing-a-crm4
- /compare/acme-vs-globex2
24 requests
Read a page because someone asked an assistant about it.
ChatGPT-User OpenAI
18
- /pricing9
- /blog/choosing-a-crm6
- /integrations/slack3
Perplexity-User Perplexity
6
- /pricing4
- /docs/getting-started2
30 requests
Collect pages that may be used to train future models.
GPTBot OpenAI
19
- /blog/choosing-a-crm7
- /docs/getting-started5
- /docs/api4
3 requests to /blog/archive returned 503.
and 1 more path
- /blog/archive3
ClaudeBot Anthropic
9
- /docs/api4
- /pricing3
- /blog/choosing-a-crm2
3 requests to /pricing returned 403.
CCBot Common Crawl
2
- /2
28 requests
Classic search crawlers. Google's and Bing's AI answers draw on the same index.
Googlebot Google
21
- /8
- /pricing6
- /blog/choosing-a-crm5
and 1 more path
- /docs/getting-started2
Bingbot Microsoft
7
- /4
- /pricing3
How does the log analyzer work?
Export a slice of your log
Download an access log from your server, host or CDN. A day or a week is enough. Plain text and .gz both work, in combined log format or as JSON lines.
Match each request to a crawler
The tool reads every line in your browser and matches its user agent against the registry of documented AI crawlers our other tools use, from GPTBot to PerplexityBot.
Read the result by purpose
Bots are grouped by what they do: indexing for search, fetching a page for a user, or training. Search engine crawlers get their own group, and errors and refused requests are flagged for each bot.
What your log shows that robots.txt can't
robots.txt says what you allow. Your access log shows what happened: which bots came, which pages they asked for and what your server answered. A crawler you allow can still get a 403 from a firewall rule, and only the log shows it.
Why assistant fetches matter most
A request from ChatGPT-User or Perplexity-User means someone asked an assistant a question and it opened your page to answer. These requests come closest to a visit from AI search, and their paths show which of your pages AI answers draw on.
User agents can be faked
The analysis trusts the user agent each request sends, and anyone can send GPTBot's. Before you act on a number, check the request IPs against the ranges each operator publishes. Most large operators, OpenAI and Google among them, publish theirs.
From a one-off export to a daily view
A log export covers the day or week you exported. Citlyze's AI Crawler Tracking records recognized AI crawler visits through a server-side integration for Cloudflare, Vercel, WordPress or a Node server, and keeps the history, so you see when a bot stops coming.
Which AI answers cite those pages?
Your log shows which bots read your site. Citlyze shows what AI engines then say about your brand: mentions, the pages they cite, and your share of voice against competitors, tracked on a schedule.
Log analysis, answered
No. The tool reads and analyzes the file in your browser, and it never sends your log to Citlyze or anyone else. It also drops query strings from paths before showing them, since they can carry tokens or personal data.
Apache and nginx access logs in the combined format, which most servers and hosts write by default, and JSON lines with user agent, path and status fields, such as Cloudflare Logpush exports. Gzipped files work too. The result counts lines in other formats as skipped.
They are classic search engine crawlers, so counting them as AI crawlers would inflate the numbers. They still matter for AI search: Google's AI Overviews and AI Mode and Bing's Copilot draw on the indexes those crawlers build.
Up to 100 MB of log text, roughly half a million requests. The tool reads a larger file up to that point and says where it stopped. For a busy site, export one day at a time.