Bot Access
Continuous checks that AI bots can reach your pages, with incident alerts and causes.
A page that AI bots cannot fetch is a page AI answers cannot cite. AI Crawlers → Bot access continuously tests whether AI bots can actually reach your key pages, compares your robots.txt policy against live responses, and opens an incident when a bot loses access.
This is different from a one-time audit: firewall rules, CDN bot protections, and robots.txt files change, and a page that passed last month can be silently blocked today.
Monitored pages
Your homepage and most-cited pages are seeded automatically; use Add page to monitor more, within your plan's limit (5 pages on Starter, 25 on Growth, 100 on Pro and above). Each page row expands to per-bot detail: the HTTP result, what robots.txt says for that bot, and the check's outcome.
Checks run from Citlyze servers presenting each bot's user agent. Sites that allow-list verified bot IP ranges can therefore show as blocked here even when the real bot gets through; pair this page with AI Crawlers analytics to confirm what actually visits.
Incidents and causes
When a bot that could reach a page no longer can, an incident opens with the cause:
- robots.txt block: your robots.txt disallows the bot. If that is intentional, no action is needed; deliberate policy is respected and training-only bots are not alerted on.
- Firewall/CDN block: the request was rejected before your site answered. The vendor is named when identifiable (for example Cloudflare), so you know which dashboard to open.
- Timeout or error: the page did not answer usably.
Incidents resolve automatically when access recovers. Use Re-check now after changing robots.txt or firewall rules instead of waiting for the next scheduled check.
Alerts
The Bot access incident alert rule notifies you when an incident opens, once per incident. See Alerts.
Exporting
Check results and incidents are available through the API: bot-access-results and bot-access-incidents.