AI Crawlers

AI Crawler Tracking shows which AI engines are actually visiting your site: GPTBot, ClaudeBot, PerplexityBot, Bingbot, and more. It complements GEO Audits: audits tell you whether bots can reach a page; crawler tracking tells you whether they do.

Why server-side capture

AI training crawlers do not run JavaScript, so a Google Tag Manager or JavaScript pixel never sees GPTBot or ClaudeBot. Citlyze captures crawler visits server-side, where the real request User-Agent is visible, so even non-JavaScript crawlers are counted.

The human half is different: visitors referred by AI answers do run JavaScript, so for AI Traffic a browser snippet or Tag Manager install works fine (see Install AI Traffic tracking). The server-side installs on this page capture both kinds at once.

Install tracking

Go to AI Crawlers → Install tracking and generate a site key. The key has two parts, a key ID and a signing secret, shown once; copy both. Then add the matching snippet for your platform (for click-by-click walkthroughs of every option, see Install AI crawler tracking):

• Cloudflare (recommended): paste the Worker snippet in front of your site.
• Vercel / reverse proxy: add the middleware snippet.
• WordPress: download AI Crawler Tracker by Citlyze, upload it in wp-admin → Plugins, then in its settings fill all three fields: Tracker base URL (https://app.citlyze.com), Key ID, and Signing secret. Use Send test event to verify delivery. The plugin does not report until the tracker base URL is filled in.
• Custom server / Node: for self-hosted sites (AWS, GCP, Azure, bare metal, containers): add the Express-style middleware snippet (Node 20+). It adapts to Fastify, Koa, or plain http, and other languages can implement the signed-beacon protocol directly.

Each event is signed with your secret so the tracker can reject forged beacons. That signature proves the report came from your site; it says nothing about who the visitor was. Confirming that a visitor really was GPTBot is a separate step, covered in How verification works.

Standard Shopify stores cannot run server-side capture, and Shopify does not support putting a proxy (such as Cloudflare) in front of a store, so full crawler tracking is not available on standard Shopify. Headless Hydrogen or Oxygen storefronts run server-side code and can use the custom server snippet. Webflow hosting cannot run server code either; on Webflow Enterprise, a self-managed reverse proxy can run the Cloudflare Worker or the custom server snippet at the proxy layer.

The Cloudflare Worker and the custom server middleware report the HTTP status of each crawled request, which powers the error insights below. The WordPress plugin runs before the page renders, so it distinguishes only 404s from everything else, and Vercel middleware runs before the response exists, so it cannot report status at all.

WordPress full-page caches and CDNs may answer a request without running the plugin. For heavily cached sites, use the Cloudflare Worker to avoid undercounting.

Custom servers and other languages

The custom server tab ships a Node middleware, but any stack can report: a beacon is a single signed HTTPS POST to https://app.citlyze.com/api/track with a JSON body (at most 32 KB) and Content-Type: application/json.

Required headers:

Computing the signature:

1. Derive the signing key: the lowercase 64-character hex SHA-256 of your signing secret (the full ctk_... string). Use that hex string's UTF-8 bytes as the HMAC key; do not hex-decode it.
2. Build the message: "2\n" + timestamp + "\n" + nonce + "\n" + bodyDigest, where bodyDigest is the lowercase hex SHA-256 of the exact body bytes you send. Any re-serialization after signing invalidates the signature.
3. x-aeo-signature is the lowercase hex HMAC-SHA256 of that message with the signing key from step 1.

Body fields (all required unless noted):

• userAgent: the visitor's User-Agent, up to 1024 characters.
• path: the request path with a leading / and no query string or fragment, up to 2048 characters.
• visitorIp: the client IP as seen by your server. Behind a load balancer or reverse proxy, take it from the forwarding header your own proxy sets; this field is what crawler identity verification checks.
• referrer: an empty string, or an https:// referrer URL with query and fragment stripped. Only meaningful for human visits arriving from AI answers.
• status: the three-digit HTTP status of the response, or unknown if you report before the response exists.
• method: the uppercase HTTP method.

Report only requests whose user agent looks automated or whose referrer is an AI answer engine, and send the beacon after the response, fire-and-forget; a tracker outage should never slow your site.

To verify your integration, send a beacon with "test": true, a userAgent starting with citlyze-connection-test/, and "path": "/citlyze-test". A correctly signed test event returns HTTP 200 and stores nothing; real events always return 204, whether or not the visit ends up in your reports.

Rotate or revoke a key

If a signing secret may have leaked (for example it was committed to a repo or shared in a screenshot), use Rotate next to the key. Rotation issues a new secret for the same key: the key ID, name, domain, and all recorded history are kept, but the old secret stops verifying immediately, so update your snippet or plugin with the new secret right away. Revoke is different: it disables the key permanently and stops tracking for that site.

Reading the analytics

The AI Crawlers → Analytics page shows, for the selected time frame:

• headline tiles: total crawler hits, distinct crawlers, top crawler, and trend
• crawler visits over time (daily totals)
• crawler trends over time (one line per crawler)
• by crawler breakdown with organization, purpose, hits, and trend
• human visits from AI answers: visitors who clicked through from ChatGPT, Perplexity, Gemini, Copilot, and others, with their top landing pages; the full referral report, including conversions, lives on AI Traffic
• most-crawled pages (top 10, with a link to the full list)
• unrecognized bots: bot-like user agents that matched no known AI crawler, grouped under "Unknown bot" so new crawlers surface early

Use the presets (7/30/90 days), the custom date range, and the crawler filter to focus the view. Workspaces tracking more than one site also get a site filter.

How verification works

Any client can put GPTBot in its user agent. So a user-agent string is a claim, not proof, and Citlyze treats it that way: headline crawler numbers count only traffic we could confirm independently.

Each visit gets one of three confidence levels:

• Verified: we confirmed the visitor's network identity against something the operator publishes. Only these count toward your crawler totals.
• Probable: corroborating evidence, such as your CDN flagging the request as a known bot, but no independent confirmation.
• Unverified: the user agent named a crawler and nothing contradicted it, but we could not confirm it. Shown separately, never added to your totals.

Verification uses whichever of these the operator supports:

Operators that publish nothing to check against can only ever reach unverified. That is a property of the crawler, not a problem with your setup, and it is why the separate views exist rather than one blended number.

Two things worth knowing:

• Agent traffic is counted separately. Tools people drive themselves, like ChatGPT Agent, appear under agent activity rather than in crawler totals, because one person clicking is not the same signal as a crawler indexing you.
• Occasionally we cannot run a check: an operator's IP list may be temporarily unreachable. Those visits stay unverified rather than being counted, so your totals never include something we did not confirm.

Crawled pages

AI Crawlers → Crawled pages is the full drill-down: every page AI crawlers fetched in the selected window, with search, sorting, and pagination. Each row shows hits, share of all crawler traffic, trend versus the prior window, errors, the top crawler, and last-seen date; expand a row for the per-crawler split.

Two insights appear above the table when relevant:

• Crawled but never cited: pages AI engines fetch but never cite in your tracked answers; candidates for clearer, more citable content.
• Pages returning errors: pages that answered crawlers with 4xx/5xx responses. Broken pages cannot be read or cited. The full range needs the Cloudflare or custom server install; a WordPress install surfaces 404s but not 5xx.

The table can be downloaded as CSV on plans that include data export.

Access the data via API and MCP

Crawler visits are available programmatically once tracking is installed:

• REST: GET /api/v1/crawler-events; see the AI Crawler Events resource.
• MCP: the list_crawler_events tool; see the MCP tools reference.

Both are read-only and scoped to your workspace. The REST resource supports filtering by crawler_id, tracked site, and exact path; human visits from AI answers are exposed at GET /api/v1/ai-referrals.