WordPress plugin

AI Crawler Control by Citlyze is a free WordPress plugin with two parts. The free tools work without an account: they show which AI crawlers your robots.txt lets in, name what blocks the rest, and let you block AI training while staying in AI search. Connecting the plugin to Citlyze adds tracking of AI crawler visits and visits from AI answers, covered in Install AI crawler tracking.

Install

1. In WordPress admin, go to Plugins → Add New, search for AI Crawler Control by Citlyze, then click Install Now and Activate.
2. Open Citlyze → AI crawler access in the admin menu. The check runs straight away.

The free tools need no account, key or setup.

Check which AI crawlers can read your site

Citlyze → AI crawler access reads the robots.txt your visitors actually receive, including anything a cache or CDN changes, and evaluates it for every known AI crawler. Crawlers are grouped by what blocking them changes:

• AI search crawlers build the indexes that ChatGPT search, Perplexity, Claude and other AI answers cite. Blocking them removes you from those answers.
• AI assistants fetch a page when someone asks an assistant about it.
• AI training crawlers collect content to train models. Blocking them keeps your content out of future training; it does not remove you from AI answers.
• Search engines such as Googlebot and Bingbot are listed so you can see them, but the plugin never blocks them.

For each blocked crawler, the table says where the rule came from: WordPress, a named plugin or theme, the robots.txt file on your server, Cloudflare's managed robots.txt, or your server or CDN.

Above the table, the check lists anything in the way:

• Search engine visibility turned off under Settings → Reading.
• A robots.txt file on your server, which replaces the one WordPress generates.
• A robots.txt that loads slowly or not at all.
• Search engines or AI search crawlers blocked by another plugin.
• Hosts and security plugins that can turn crawlers away before robots.txt is read, such as Cloudflare, WP Engine or Wordfence.

Choose AI crawler rules

Under Your AI crawler rules, pick one:

• Don't add rules (the default): your robots.txt stays as it is.
• Block AI training, stay in AI search: blocks training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot, and leaves OAI-SearchBot, PerplexityBot and the assistant fetchers alone.
• Block all AI crawlers: also removes you from AI answers.
• Custom: choose crawlers one by one.

Before you save, the preview shows the exact lines and how many crawlers in each group stay allowed. Rules go in a marked block at the end of your robots.txt, after anything other plugins write, and crawlers you allow keep following your existing rules. Undo last change puts back your previous choice.

Google-Extended controls whether Gemini uses your content for training and for grounding answers in the Gemini app and Vertex AI. It does not affect Google Search, AI Overviews or AI Mode.

Turn on Also declare my choice with a Content-Signal line to add a Content-Signal line stating your preference for crawlers that read it. It declares a preference; it does not enforce one.

If your server has a robots.txt file, or WordPress runs in a subfolder, WordPress cannot serve these rules. The screen shows them with a Copy rules button so you can paste them into the file.

Rules for parts of your site

Under Parts of your site, below the choice for the whole site, add a path such as /members/ or /downloads/ and choose what to keep out:

• AI training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and the other training crawlers.
• All AI crawlers: AI search crawlers and assistants too, so pages in that section drop out of AI answers.

Search engines are never included. Paths work as they do in robots.txt: they start with a slash, * matches any characters, and a $ at the end marks the end of the address, so /*.pdf$ covers every PDF. You can add up to 20 sections.

A crawler that gets a group of its own in robots.txt stops following the rules written for every crawler (User-agent: *), so the plugin repeats those rules in its group. Outside your sections, nothing changes for that crawler. Under Crawler by crawler, a crawler kept out of a section shows as Limited, with the sections it cannot read.

AI readiness in the editor

Each post and page gets an AI readiness panel in the block editor's sidebar, or an AI readiness box in the classic editor. It scores the page for AI answers from 0 to 100 with the same checks as the Citlyze Chrome extension and the free AEO grader, so a published page gets the same score in all three.

• As you write, the panel checks your content: question-shaped headings, a short answer under each heading, and tables and lists.
• After you save, it scores the whole page as crawlers receive it: AI crawler access for that exact URL, whether the page can be indexed, one clear H1, schema markup, and the title and meta description. Drafts are scored from their preview, ignoring the noindex WordPress adds to previews.

Each check that falls short says what to change. Before you publish, the pre-publish panel lists the content checks that still fail.

AI suggestions

On WordPress 7.0 and later, with an AI provider set up in WordPress, the panel can help with two checks. Under Question-shaped headings, Suggest a question rewrites a heading as the question a reader would ask. Under Short answers under headings, Draft a short answer writes a 40 to 60 word answer from the text under the heading. You read each suggestion first: Insert puts it in the post as plain text, and the editor's Undo takes it back out.

Only when you click, the heading and up to about 4,000 characters of the text under it go to the AI provider set up in WordPress. Nothing goes to Citlyze. Without a provider, the panel shows no suggestions.

What the free tools send

Nothing to Citlyze. The checks run in your WordPress admin and your browser, and read only your own site. AI suggestions in the editor send a heading and the text under it to the AI provider set up in WordPress, and only when you click. Data goes to Citlyze only if you connect Citlyze for tracking.

Connect Citlyze

Robots.txt says what crawlers may read, not what they actually read. To see verified AI crawler visits and the AI answers that send you visitors, create a site key in Citlyze and enter it under Citlyze → Connect Citlyze. The steps are in Install AI crawler tracking. For daily checks that AI crawlers can actually fetch your pages, see Bot Access.