SEOmarketeer Bot Flagger

Most analytics trust what a visitor says it is. Attack tools know that, so they call themselves Chrome, or Googlebot, or an AI crawler, and walk straight past. SM Bot Flagger checks what they actually do: a request for your cloud credentials is an attack, whatever name it gives.

What it does

Each front-end request is written to its own table with:

• date and time, in site time and UTC

• IP address, stored whole or masked

• user agent

• referrer and referring domain

• requested path and query string

• HTTP status code

• whether the request started a new visit, which makes it a landing page

• the category and client name the rules assigned

• how long WordPress took to build the page

Nothing is written to postmeta, and there is one INSERT per request.

Categories

Traffic is sorted into six buckets:

• Regular traffic — looks like a person in a browser

• Search bots — Googlebot, Bingbot, DuckDuckBot and friends

• AI bots — GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot and around thirty others

• Malicious traffic — sqlmap, Nikto, WPScan, plus around 75 path signatures:

config and env files, backup leftovers, SSH and private keys, AWS, Azure,

Google Cloud, Oracle, OVH, Linode and Hetzner credentials, Docker and

Kubernetes, Terraform state, Vite and Spring Actuator file disclosure, path

traversal and SQL injection, and credential files for AI coding agents

• Other bots — SEO crawlers, uptime monitors, link previews, HTTP libraries

• Caching — your own cache plugins warming pages: WP Rocket preload and bot,

WP Fastest Cache, LiteSpeed crawler, W3 Total Cache, WP Super Cache,

FlyingPress, NitroPack, Cache Enabler, Swift Performance, Cloudflare Always

Online

• Unclassified — empty or unrecognised user agents

Cache warming is the site requesting its own pages, so it is not a visit and it

is not really a bot either. Keeping it separate stops preload runs from inflating

the traffic numbers. Filter the log to Caching to check a preload actually ran.

Settings has a “Do not log” row where you can tick categories to discard. Those

requests are still classified, then dropped before anything is written, so they

cost one pass over the rules and no database queries at all. This is worth doing

for Caching on a busy site, where a preload run can easily be the largest source

of requests. Nothing is stored, so skipped traffic cannot be recovered later and

will not show up in the log, the charts or the API.

A rule matches on one of three things: the user agent, the request path, or the

IP address. User agent and path patterns match anywhere in the value, with *

as a wildcard and an optional regular expression. Address patterns understand an

*, and

0/24 or 2001:db8::/32. Prefixes only match on a

4.

This is how you handle a crawler that lies about its user agent, or a scraper

running from one hosting range: match the address instead. It is also how you

mark an internal monitoring path or a staging endpoint as something other than

real traffic.

Rules live under SM Bot Flagger → Classification. Each rule matches a substring

or a regular expression against the user agent, the path or the IP. They are

evaluated top to bottom and the first match wins, so put narrow rules above broad

ones. You can edit, reorder, disable, delete and restore them.

Performance

Bot Flagger → Performance shows how long WordPress takes to build your pages, from the build time already recorded with every request:

• the typical page (median) and the slowest 5%, compared with the period before

• a chart per day, or per hour for the last 24 hours

• the slowest pages, each linking to its requests in the traffic log

• build times per visitor type, so you can see whether bots get slower pages

It starts with Regular traffic only, and you can add any other visitor type. The Dashboard has a small Performance card that follows its own filters. Pages served from a page cache never reach WordPress, so they are not measured.

Privacy

SM Bot Flagger processes personal data in the sense of the GDPR, because an IP

address is personal data. The defaults are chosen to keep as little as

possible:

• IP addresses are masked before storage, to /24 for IPv4 and /48 for IPv6.

Full addresses are stored only if you switch that on.

• Visitors are counted through a hash of the masked address, the user agent

and a secret that changes every day, so nobody can be followed from one day

to the next.

• Values that look like secrets in URLs, such as password reset keys and

tokens, are removed before a row is written.

• Administrators are not logged by default.

• Entries are deleted after 90 days by default.

• Nothing leaves your server. There is no tracking script, no external

service, no call home. The plugin author never sees your data.

The plugin adds suggested text to your site’s privacy policy guide under

Settings > Privacy > Policy Guide. Adapt it to your configuration. If you

store full addresses or keep entries longer, say so there, and check your

legal basis: for most sites that is a legitimate interest in security, which

should be recorded in your processing register.

The optional MCP endpoint lets a tool you connect read the log. Whoever holds

a token can read the same data you can, so treat tokens as passwords and

revoke the ones you no longer use.

The MCP endpoint

Enable it under Settings, then create a token under MCP API.

tld/wp-json/sm-bot-flagger/v1/mcp

0 over POST

Auth: Authorization: Bearer <token>, or X-MCP-Token: <token> if a proxy

strips the Authorization header

Tools:

• get_traffic_summary — totals and the category split for a period

• get_timeseries — hourly or daily counts per category

• get_top — ranks pages, landing pages, referrers, referring domains, user

agents, clients, categories, status codes, IPs, rules or hour of day

• search_hits — individual requests, with filters

• classify_user_agent — test a user agent against the current rules

• list_classification_rules — the rules in evaluation order

• upsert_classification_rule — add or change a rule, write token only

• delete_classification_rule — remove a rule, write token only

Read tokens can only read. Give a token write access only if you want the client

to change how traffic is categorised.

htaccess on Apache:

*)" HTTP_AUTHORIZATION=$1

Hooks

Extension points for add-ons:

sm_bot_flagger_categories (array) — add your own traffic categories. They appear in the

rules editor, the charts, the filters and the MCP schemas automatically.

sm_bot_flagger_classification (array) — override the category, label or rule for a

request, after the rules have run.

sm_bot_flagger_mcp_tools (array) — register MCP tools. Include a callback key and the

free plugin will dispatch to it.

sm_bot_flagger_stats_where (array) — append conditions to every stats query.

sm_bot_flagger_admin_menu (action) — add a submenu page under SM Bot Flagger.

sm_bot_flagger_dashboard_cards / sm_bot_flagger_dashboard_panels (actions) — add to the dashboard.

sm_bot_flagger_log_columns (array) and sm_bot_flagger_log_column (action) — add columns to the log.

sm_bot_flagger_settings_fields (action) and sm_bot_flagger_sanitize_settings (array) — add settings.

SM_Bot_Flagger_Logger::current_classification() classifies the current request once and

caches it, so an add-on can act on a crawler early rather than at shutdown.

sm_bot_flagger_should_log (bool) — return false to skip logging a request. sm_bot_flagger_hit_data (array) — change the row before it is written, or return an empty

array to drop it. Useful for adding a country lookup.

sm_bot_flagger_mcp_tools (array) — add your own MCP tools. Give each one a handler named

tool_ on a class of your choosing and hook tools/call accordingly.

Notes on accuracy

• Logging happens on shutdown, so requests served entirely from a page cache

or a CDN never reach PHP and are not counted. On a cached site, treat these

numbers as a lower bound.

• Bot detection relies on self-reported user agents. Anything determined to look

like a browser will be counted as regular traffic.

• Only pick a proxy header for the IP if a proxy actually sets it. A client can

send X-Forwarded-For itself, which would let anyone forge their address.

Security notes

• Query string values for parameters like key, token, code, password and nonce

are replaced with [redacted] before a row is written, so password reset links

and OAuth callbacks do not end up sitting in the log table.

• CSV exports prefix any cell starting with =, +, – or @ with an apostrophe, so

a crafted user agent cannot run as a formula when the file is opened.

• The MCP endpoint blocks an address for 15 minutes after 20 failed token

attempts.

• Tokens are not accepted as a?token= query parameter by default, because

query strings leak into access logs and referrers. Enable it with the

sm_bot_flagger_mcp_allow_query_token filter if a client cannot send headers.

• A regular expression that does not compile is switched off rather than being

treated as literal text, so a typo cannot silently widen what a rule matches.

• Rule patterns are matched against at most the first 1000 characters of a

header, which bounds the cost of a badly written expression.

A write-scoped MCP token can change classification rules, and those rules run on

every front-end request. Treat a write token as trusted, and hand out read

tokens for anything that only needs to look at the numbers.

Behaviour rules

Pattern rules judge a request on what it looks like. Behaviour rules judge a

client on what it has been doing. They run at log time, because they need the

response status and a short memory, and they need a stored IP address, so they

do nothing when IP storage is off. Confirmed search and AI crawlers are never

demoted by them.

Random paths. A 404 on a single path segment of 4 to 12 alphanumeric characters

with no vowels and mixed case, such as /6bXX29bt, is treated as scanning. Real

slugs contain vowels; generated tokens do not. The 404 is required, so a short

consonant heavy page of your own is never caught.

Repeated 404s. Flagged on both a count and a share: by default 8 misses within

10 minutes, and at least half of that client’s requests in the window missing.

The share is what makes this safe. A busy legitimate client can collect a

handful of 404s among hundreds of good requests, and counting alone would flag

it. Scanners miss on almost everything they ask for.

Login attempts. php,

10 per hour by default. Deliberately not a rate: credential stuffing is usually

slow and spread over many addresses, so a per-second limit never sees it.

Request rate, off by default. On real traffic the fastest clients are usually

legitimate, while scanners and credential stuffing run slowly enough to pass any

threshold worth setting. Counting 404s separates the two far better than

counting requests. Measure your own traffic before switching this on.

Once a client trips a rule it stays flagged for an hour by default, and what it

already did in that window is recategorised too.

Path precedence

A user agent is a claim the client makes about itself. Anything can send

“OAI-SearchBot” or “Googlebot”, and plenty of attack tooling does, because it

gets past naive filtering.

Settings has “Trust the path”, on by default, which tests path and address rules

before user agent rules. aws/credentials is then recorded

as an attack whatever it claims to be, and your genuine crawler rules stay

correct for genuine crawlers.

Turn it off if you would rather rules were evaluated strictly in the order shown

on the Classification screen.

Filtering

Dashboard tiles toggle categories, and several can be combined. Rows in the top

lists are links: click a page, landing page, status code, client or referring

domain to open the traffic log filtered to it.

The log itself filters on category, status family (2xx to 5xx), an exact status

code, a page path, landing pages only, and a free text search across path, user

agent, referrer, IP and client. Combining a path with “landing pages only” gives

you the entry traffic for one page. Filters survive paging and are applied to

CSV exports.

Correcting a classification

Open the Traffic log and click the client name on any row. You get the user

agent as an editable pattern, a name, and a category. Shorten the pattern to the

distinctive part before saving, for example MyScraper/2. rather than the whole

string, so the rule catches other versions too.

Saving puts the rule first in the evaluation order, so it beats the shipped

defaults. Tick the box to also rewrite requests that are already in the log,

which is matched on the user agent containing your pattern.

The chart above the table follows the filters, and can be grouped by category,

by client or by the raw user agent. The eight busiest values get their own band

and the rest are folded into “Everything else”. “Show full user agent” prints

the whole string under each client name.