About this plugin
Bravo Crawl Hunter is a powerful AI crawl blocker plugin that blocks AI crawlers, AI scrapers, bad bots, spam bots, unwanted bots, and other automated crawlers from your WordPress site.
txt or denied outright with an HTTP 403 response.
You can also monitor and review crawler activity from the Bravo Crawl Hunter >> Dashboard menu in your WordPress dashboard.
The plugin provides detailed crawler activity, including total requests, blocked requests, top crawlers, most targeted pages, and crawler categories.
Every crawler the plugin knows about comes from a curated registry organized by purpose, so you can understand what a bot is used for before deciding how to handle it.
🤖 Block AI crawlers and scrapers
Bravo Crawl Hunter includes a curated registry of AI crawlers, scrapers, AI assistants, and AI search bots from major AI companies and services.
AI Crawler bots are blocked by default, while AI assistants and AI search bots can be selected individually. txt rules, or HTTP 403 hard blocking.
You can also add custom User-Agent strings when a bot or scraper is not included in the built-in registry.
🐺 A curated crawler registry
More than 75 known crawlers ship with the plugin, grouped into seven categories:
- AI Crawler – Crawlers that collect content for AI-related purposes, including GPTBot, CCBot, ClaudeBot, and Bytespider.
- AI Assistant – On-demand fetchers behind assistants and chat features, such as ChatGPT-User and Claude-User.
- AI Search – Crawlers used by AI-powered search and answer services.
- Search Engine Crawler – Traditional search engine crawlers such as Googlebot, Bingbot, and DuckDuckBot.
- SEO Tool – Crawlers used by third-party SEO analysis and ranking tools, including AhrefsBot and SemrushBot.
- Archiver – Crawlers used by web archiving services.
- Other Crawler – Other recognized automated traffic, such as monitoring services and link-preview bots.
Only the AI Crawler category is blocked out of the box. Search engines, SEO tools, archivers, and other crawlers stay untouched until you opt in, so search visibility never changes without your explicit choice.
🛡️ Two ways to control access
- robots.txt rules – Publishes Disallow rules for the crawlers you select. Well-behaved bots honor them, though not every bot does.
- Hard block (403) – Returns an HTTP 403 response when a selected crawler reaches WordPress, whether or not it follows robots.txt.
The two methods are independent. Use either one, or both together, from Settings → General Settings.
🔗 Covers WordPress content endpoints
Hard blocking applies to front-end requests as well as WordPress REST API and XML-RPC requests, helping prevent selected crawlers from accessing your content through alternate WordPress endpoints.
⚙️ Custom User-Agents
Bots and scrapers that are not in the registry can be blocked by adding their User-Agent string, making it easy to use Bravo Crawl Hunter as a custom bot blocker for unwanted crawlers.
Up to 50 entries are supported, each matched anywhere in the incoming User-Agent header and case-insensitively, and enforced with the same protection methods as the registry.
🚀 Works with caching and CDNs
Server-side detection runs when a request reaches WordPress. A page served entirely from a server cache, caching plugin, or CDN may never reach WordPress, so it cannot be classified that way.
For those requests, the optional JavaScript Detection feature records known crawlers that load a cached page containing the plugin’s lightweight detection script.
It is disabled by default and can be enabled from Settings → General Settings.
📊 Dashboard
The dashboard summarizes crawler activity over the date range you pick:
- Total, blocked, allowed, and unknown requests, plus the number of crawlers seen.
- Activity trends across the selected range.
- A breakdown by individual crawler and by crawler category.
- The pages receiving the most blocked crawler requests.
- Recent crawler activity.
📋 Crawler Logs
The Crawler Logs screen is a searchable, filterable, paginated history of classified requests. Each entry records the time, crawler, path, action, and user agent.
Logging is on by default and can be switched off under Settings → General Settings; dashboard totals and charts keep updating either way.
Log retention is capped by a maximum entry count under Settings → Advanced, and the oldest entries are removed first.
💾 Export, import, and data on uninstall
Settings can be exported to a JSON file and imported back, to keep a backup or move a configuration to another site, from Settings → Advanced.
The same screen lets you reset everything to the recommended defaults, and choose whether settings, logs, and stats are kept or removed if the plugin is deleted.