AI Crawlers

AI Crawler Tracking shows which AI engines are actually visiting your site — GPTBot, ClaudeBot, PerplexityBot, Bingbot, and more. It complements GEO Audits: audits tell you whether bots can reach a page; crawler tracking tells you whether they do.

Why server-side capture

AI training crawlers do not run JavaScript, so a Google Tag Manager or JavaScript pixel never sees GPTBot or ClaudeBot. Citlyze captures crawler visits server-side, where the real request User-Agent is visible, so even non-JavaScript crawlers are counted.

Install tracking

Go to AI Crawlers → Install tracking and generate a site key. The key has two parts — a key ID and a signing secret — shown once; copy both. Then add the matching snippet for your platform:

• Cloudflare (recommended) — paste the Worker snippet in front of your site.
• Vercel / reverse proxy — add the middleware snippet.
• WordPress — download the plugin, upload it in wp-admin → Plugins, then paste your key ID and signing secret in its settings.

Each event is signed with your secret so the tracker can reject forged beacons. Shopify and Webflow (standard plans) cannot run server-side capture; put Cloudflare in front of those sites and use the Cloudflare snippet.

Reading the analytics

The AI Crawlers → Analytics page shows, for the selected time frame:

• headline tiles: total crawler hits, distinct crawlers, top crawler, and trend
• crawler visits over time (daily totals)
• crawler trends over time (one line per crawler)
• by crawler breakdown with organization, purpose, hits, and trend
• most-crawled pages

Use the time-frame (7/30/90 days) and crawler filters to focus the view.

Access the data via API and MCP

Crawler visits are available programmatically once tracking is installed:

• REST: GET /api/v1/crawler-events — see the AI Crawler Events resource.
• MCP: the list_crawler_events tool — see the MCP tools reference.

Both are read-only, scoped to your workspace, and support filtering by crawler_id and tracked site.