citlyze docs
Using Citlyze

WordPress plugin

Free AI crawler controls for WordPress, plus optional tracking in Citlyze.

AI Crawler Control by Citlyze is a free WordPress plugin with two parts. The free tools work without an account: they show which AI crawlers your robots.txt lets in, name what blocks the rest, and let you block AI training while staying in AI search. Connecting the plugin to Citlyze adds tracking of AI crawler visits and visits from AI answers, covered in Install AI crawler tracking.

Install

  1. In WordPress admin, go to Plugins → Add New, search for AI Crawler Control by Citlyze, then click Install Now and Activate.
  2. Open Citlyze → AI crawler access in the admin menu. The check runs straight away.

The free tools need no account, key or setup.

Check which AI crawlers can read your site

Citlyze → AI crawler access reads the robots.txt your visitors actually receive, including anything a cache or CDN changes, and evaluates it for every known AI crawler. Crawlers are grouped by what blocking them changes:

  • AI search crawlers build the indexes that ChatGPT search, Perplexity, Claude and other AI answers cite. Blocking them removes you from those answers.
  • AI assistants fetch a page when someone asks an assistant about it.
  • AI training crawlers collect content to train models. Blocking them keeps your content out of future training; it does not remove you from AI answers.
  • Search engines such as Googlebot and Bingbot are listed so you can see them, but the plugin never blocks them.

For each blocked crawler, the table says where the rule came from: WordPress, a named plugin or theme, the robots.txt file on your server, Cloudflare's managed robots.txt, or your server or CDN.

Above the table, the check lists anything in the way:

  • Search engine visibility turned off under Settings → Reading.
  • A robots.txt file on your server, which replaces the one WordPress generates.
  • A robots.txt that loads slowly or not at all.
  • Search engines or AI search crawlers blocked by another plugin.
  • Hosts and security plugins that can turn crawlers away before robots.txt is read, such as Cloudflare, WP Engine or Wordfence.

Choose AI crawler rules

Under Your AI crawler rules, pick one:

  • Don't add rules (the default): your robots.txt stays as it is.
  • Block AI training, stay in AI search: blocks training crawlers such as GPTBot, ClaudeBot and CCBot, and leaves OAI-SearchBot, PerplexityBot and the assistant fetchers alone. Google-Extended stays allowed unless you tick Also block Google-Extended.
  • Block all AI crawlers: also removes you from AI answers.
  • Custom: choose crawlers one by one.

Before you save, the preview shows the exact lines and how many crawlers in each group stay allowed. Rules go in a marked block at the end of your robots.txt, after anything other plugins write, and crawlers you allow keep following your existing rules. Undo last change puts back your previous choice.

Google-Extended controls whether Gemini uses your content for training and for grounding answers in the Gemini app and Vertex AI. Block AI training, stay in AI search leaves it allowed, because blocking it also keeps your pages out of Gemini app answers. It does not affect Google Search, AI Overviews or AI Mode.

Turn on Also declare my choice with a Content-Signal line to add a Content-Signal line stating your preference for crawlers that read it. It declares a preference; it does not enforce one.

If your server has a robots.txt file, or WordPress runs in a subfolder, WordPress cannot serve these rules. The screen shows them with a Copy rules button so you can paste them into the file.

Rules for parts of your site

Under Parts of your site, below the choice for the whole site, add a path such as /members/ or /downloads/ and choose what to keep out:

  • AI training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and the other training crawlers.
  • All AI crawlers: AI search crawlers and assistants too, so pages in that section drop out of AI answers.

Search engines are never included. Paths work as they do in robots.txt: they start with a slash, * matches any characters, and a $ at the end marks the end of the address, so /*.pdf$ covers every PDF. You can add up to 20 sections.

A crawler that gets a group of its own in robots.txt stops following the rules written for every crawler (User-agent: *), so the plugin repeats those rules in its group. Outside your sections, nothing changes for that crawler. Under Crawler by crawler, a crawler kept out of a section shows as Limited, with the sections it cannot read.

AI readiness in the editor

Each post and page gets an AI readiness panel in the block editor's sidebar, or an AI readiness box in the classic editor. It scores the page for AI answers from 0 to 100 with the same checks as the Citlyze Chrome extension and the free AEO grader, so a published page gets the same score in all three.

  • As you write, the panel checks your content: question-shaped headings, a short answer under each heading, and tables and lists.
  • After you save, it scores the whole page as crawlers receive it: AI crawler access for that exact URL, whether the page can be indexed, one clear H1, schema markup, and the title and meta description. Drafts are scored from their preview, ignoring the noindex WordPress adds to previews.

Each check that falls short says what to change. Before you publish, the pre-publish panel lists the content checks that still fail.

AI suggestions

On WordPress 7.0 and later, with an AI provider set up in WordPress, the panel can help with two checks. Under Question-shaped headings, Suggest a question rewrites a heading as the question a reader would ask. Under Short answers under headings, Draft a short answer writes a 40 to 60 word answer from the text under the heading. You read each suggestion first: Insert puts it in the post as plain text, and the editor's Undo takes it back out.

Only when you click, the heading and up to about 4,000 characters of the text under it go to the AI provider set up in WordPress. Nothing goes to Citlyze. Without a provider, the panel shows no suggestions.

Keep watching

Once a day, the plugin checks the robots.txt crawlers receive, from your server. If something you did not choose starts blocking AI search crawlers, AI assistants or search engines, robots.txt stops loading, or WordPress starts asking search engines not to index the site, a notice appears for administrators on the Dashboard, the Plugins screen and the Citlyze screens, and Tools → Site Health reports it. Rules you chose under Your AI crawler rules never raise an alert, and the notice goes away once the change is undone.

To get an email too, turn on Also email me when that happens in the Keep watching card. It goes to the site's administration email address, once per change. Site Health → Info has an AI Crawler Control section with the latest results, which helps when you ask for support.

llms.txt

Citlyze → llms.txt finds the llms.txt your site serves, whatever generates it, and checks it against the llmstxt.org format: title, summary, sections, link format and size.

If your site has none, turn on Serve an llms.txt from this site. The plugin then serves a short file listing your pages and recent posts, updated as you publish, and leaves out posts your SEO plugin marks noindex and password-protected posts. If a file on your server or another plugin, such as Yoast SEO or Rank Math, already provides llms.txt, the plugin does not serve a second one.

No major AI engine has documented reading llms.txt, so treat it as optional. Your robots.txt matters more, because it decides whether AI crawlers can read your site at all.

AI agents

On WordPress 6.9 and later, the plugin registers read-only abilities that AI agents connected through the WordPress Abilities API and MCP adapter can use: list the AI crawlers the plugin knows, read your AI crawler access and what shapes it, and read your llms.txt status. Only administrators' agents can use them, and none of them changes anything on your site.

What the free tools send

Nothing to Citlyze. The checks run in your WordPress admin and your browser, and read only your own site. AI suggestions in the editor send a heading and the text under it to the AI provider set up in WordPress, and only when you click. Data goes to Citlyze only if you connect Citlyze for tracking.

Connect Citlyze

Robots.txt says what crawlers may read, not what they actually read. To see verified AI crawler visits and the AI answers that send you visitors, create a site key in Citlyze and enter it under Citlyze → Connect Citlyze. The steps are in Install AI crawler tracking. For daily checks that AI crawlers can actually fetch your pages, see Bot Access.

Drafts from Citlyze

With the site connected, a content draft can go from Citlyze to WordPress without copy and paste. Open the draft in Citlyze, choose Export → Send to WordPress and pick the site. It arrives under Posts as a WordPress draft, with its title, text, excerpt and suggested slug. Your site looks for new drafts once an hour; Check for drafts now under Citlyze → Connect Citlyze fetches them right away.

Nothing is published: you review and publish in WordPress. Sending the same draft again updates that WordPress draft while it is still unpublished, and WordPress keeps the earlier text as a revision. A published post is never changed. Citlyze has no password for your site. To stop receiving drafts, untick Receive drafts from Citlyze on the same screen.

On this page