Context
A static site on Cloudflare’s Free plan published llms.txt and Markdown copies of its posts for agents. Its robots.txt said Allow: /. Requests with GPTBot, ClaudeBot, CCBot, Amazonbot or PerplexityBot user agents still got 403 Your request was blocked on every path except robots.txt.
Insight
The zone’s bot_management object showed ai_bots_protection: disabled and bot_preference_sync_enabled: false, which reads as “not blocking”. The setting that blocked was ai_training: "disallow", enforced by a managed rule named “Block AI training crawlers for Bot Preference Sync disallow”.
In the dashboard this is the domain’s Security → Settings → Bot traffic → AI Crawler and Agent access policies, row Training. Setting it to “Allow (do not block)” removed the 403s.
Bot Fight Mode is a second, separate source of challenges. On the Free plan it cannot be exempted per bot; it is only on or off.
Diagnosis
-
Request the same URL with several bot user agents and compare status codes:
Terminal window curl -s -o /dev/null -w '%{http_code}\n' -A '<bot UA>' https://example.com/llms.txt -
Name the rule that fired with a GraphQL query on
firewallEventsAdaptive, grouped bydescriptionanduserAgent.
Did any agent fetch the Markdown files?
The same GraphQL API answers this without extra logging: httpRequestsAdaptiveGroups, filtered on clientRequestPath and verifiedBotCategory. It works on the Free plan with 31 days of retention.
Count only rows with a verifiedBotCategory. Most unverified requests carrying ChatGPT-User or Claude-User user agents were scanners probing for .env files.