New: search demand, Claude tracking, WordPress drafts, every workspace on one page

Which AI crawlers read your site?

Paste lines from your server or CDN access log, or upload the file, and see which AI crawlers requested which pages, how often, and what status they got. Your log never leaves your browser.

Plain text or .gz, in combined log format or as JSON lines with user agent, path and status fields. Up to 100 MB of log text. The analysis runs in your browser, and your logs are not sent anywhere.

Example result from a sample log. Paste your own above to see yours.

74 of 176 requests came from 7 AI crawlers. Search engine crawlers made 28 more.

Your site refused 3 AI crawler requests with 401, 403 or 429. Check your firewall, CDN bot rules and rate limits.

Log lines from Nov 16, 2026, 12:00 AM to Nov 18, 2026, 11:53 PM UTC.

Counts trust the user agent each request sends, and anyone can fake one. Check important traffic against the IP ranges each operator publishes.

Search index

20 requests

Build the index AI search answers draw on.

  • OAI-SearchBot OpenAI

    14

    • /5
    • /pricing4
    • /blog/choosing-a-crm3

    2 requests to /old-features returned 404.

    and 1 more path
    • /old-features2
  • PerplexityBot Perplexity

    6

    • /blog/choosing-a-crm4
    • /compare/acme-vs-globex2
Assistant fetch

24 requests

Read a page because someone asked an assistant about it.

  • ChatGPT-User OpenAI

    18

    • /pricing9
    • /blog/choosing-a-crm6
    • /integrations/slack3
  • Perplexity-User Perplexity

    6

    • /pricing4
    • /docs/getting-started2
Training

30 requests

Collect pages that may be used to train future models.

  • GPTBot OpenAI

    19

    • /blog/choosing-a-crm7
    • /docs/getting-started5
    • /docs/api4

    3 requests to /blog/archive returned 503.

    and 1 more path
    • /blog/archive3
  • ClaudeBot Anthropic

    9

    • /docs/api4
    • /pricing3
    • /blog/choosing-a-crm2

    3 requests to /pricing returned 403.

  • CCBot Common Crawl

    2

    • /2
Search engines

28 requests

Classic search crawlers. Google's and Bing's AI answers draw on the same index.

  • Googlebot Google

    21

    • /8
    • /pricing6
    • /blog/choosing-a-crm5
    and 1 more path
    • /docs/getting-started2
  • Bingbot Microsoft

    7

    • /4
    • /pricing3

How does the log analyzer work?

  1. Export a slice of your log

    Download an access log from your server, host or CDN. A day or a week is enough. Plain text and .gz both work, in combined log format or as JSON lines.

  2. Match each request to a crawler

    The tool reads every line in your browser and matches its user agent against the registry of documented AI crawlers our other tools use, from GPTBot to PerplexityBot.

  3. Read the result by purpose

    Bots are grouped by what they do: indexing for search, fetching a page for a user, or training. Search engine crawlers get their own group, and errors and refused requests are flagged for each bot.

  • What your log shows that robots.txt can't

    robots.txt says what you allow. Your access log shows what happened: which bots came, which pages they asked for and what your server answered. A crawler you allow can still get a 403 from a firewall rule, and only the log shows it.

  • Why assistant fetches matter most

    A request from ChatGPT-User or Perplexity-User means someone asked an assistant a question and it opened your page to answer. These requests come closest to a visit from AI search, and their paths show which of your pages AI answers draw on.

  • User agents can be faked

    The analysis trusts the user agent each request sends, and anyone can send GPTBot's. Before you act on a number, check the request IPs against the ranges each operator publishes. Most large operators, OpenAI and Google among them, publish theirs.

  • From a one-off export to a daily view

    A log export covers the day or week you exported. Citlyze's AI Crawler Tracking records recognized AI crawler visits through a server-side integration for Cloudflare, Vercel, WordPress or a Node server, and keeps the history, so you see when a bot stops coming.

Which AI answers cite those pages?

Your log shows which bots read your site. Citlyze shows what AI engines then say about your brand: mentions, the pages they cite, and your share of voice against competitors, tracked on a schedule.

Log analysis, answered

No. The tool reads and analyzes the file in your browser, and it never sends your log to Citlyze or anyone else. It also drops query strings from paths before showing them, since they can carry tokens or personal data.

Want this tracked over time?

No credit card required