Crawlers and bot access
Which AI bots visit your site, what each one does, how to allow or block them, and how to find them in your logs.
11 articles, newest first
RSS feed
Tools
Best WordPress Plugins for AI Search in 2026, by Job
WordPress plugins for AI search compared: SEO plugins with llms.txt and schema, AI crawler control, and llms.txt tools, by what each actually does.
, 5 min read

Crawlers
How to Get Indexed by AI Search Engines
Prepare pages for ChatGPT, Copilot, Perplexity and Google AI search. Check crawler access, indexing and snippet controls, then verify engines can read them.
, 8 min read

Crawlers
Agentic SEO: Audit Your Site With Lighthouse's Agent Checks
Agentic SEO gets your site ready for AI agents that complete tasks. Run Lighthouse's Agentic Browsing audit, read the result, and fix each check.
, 8 min read

Crawlers
Applebot and Siri AI: What iOS 27 Changes for Brands
Siri AI answers from the web in iOS 27. What Apple says about Applebot and source links, what is still unknown, and how to test Siri in your category.
, 7 min read

Data
llms.txt Statistics 2026: What 584,107 Files Show
Common Crawl found 584,107 llms.txt files, 68% from templates. We validated a random 1,000: 75.2% fail the format. The data, and what a good file needs.
, 8 min read

Crawlers
Cloudflare's AI Crawler Block: When It Stops Googlebot
Since September 15, Cloudflare's Block setting for AI training also stops Googlebot. Who is affected, a five-minute check, and settings by site type.
, 9 min read

Engines
Gemini Rank Tracking: How to Track Your Brand in Gemini
How to track brand mentions in Gemini answers, why there is no GeminiBot to unblock, and how Gemini differs from AI Overviews and AI Mode.
, 5 min read

GEO
llms.txt: What It Is and Whether It Actually Works
What llms.txt is, how it differs from robots.txt and sitemaps, whether AI engines read it, and why three in four files we checked fail the format.
, 5 min read

GEO
The GEO Audit: A Step-by-Step Guide to Making Your Pages Citable
A GEO audit checks whether AI engines can reach, read, and reuse your pages. The full checklist: access, robots, rendering, structure, and content readiness.
, 5 min read

Crawlers
A Crawler Visit Is Not a Citation: From Crawl to AI Answer
GPTBot fetching your page does not mean ChatGPT cites you. The four stages from crawl to citation, which bot visits matter, and how to check each end.
, 5 min read

Crawlers
GPTBot, ClaudeBot, PerplexityBot: A Marketer's Guide to AI Crawlers
Which AI crawlers visit your site, what each one does, when to allow or block them, and how to see their activity without reading server logs.
, 5 min read