HitMind – AI Crawler Check

txt: the one that names it. 1).

It is an easy mistake to make. txt files that block paths and name an AI crawler left those paths open to that crawler.

txt the way crawlers do and shows:

• The trap, if you have it: every path blocked under User-agent: * that a named AI crawler can still fetch, and the exact lines to add to its group.

• All 20 AI crawlers, one row each, from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl, ByteDance and DuckDuckGo: what each one is for (AI answers that can link to you, model training, or other), whether its operator says it follows robots.txt, which group it reads, and whether it can fetch your home page.

• Where to change the file: a real file in your site root, the one WordPress generates (and which SEO plugin adds to it), Cloudflare’s managed robots.txt, or a WordPress install in a subfolder.

• Lines crawlers skip, such as a misspelled directive.

• A Site Health check, so the trap shows up under Tools → Site Health without opening the plugin.

• A box to try changes before you publish them.

What it does not do:

• It never changes your robots.txt. Which crawlers to block is your call; the plugin shows what your file says and how to make it say what you mean.

• It does not block or allow any crawler, and adds nothing to your pages.

• It stores nothing, tracks nothing and needs no account.

txt generator and checker and its AI crawler study. txt files from that study.

External services

txt. It does not contact any other server.

ai, which open only when you click them:

• “Run a free SEO audit” opens HitMind’s free audit with your domain filled in (the domain is part of the link). Terms of service, privacy policy.

• The 29.8% link opens HitMind’s AI crawler study.

Both links carry utm_source=wordpress-plugin so HitMind can count visits from the plugin. Nothing else is sent.