Context

A static site on Cloudflare’s Free plan published llms.txt and Markdown copies of its posts for agents. Its robots.txt said Allow: /. Requests with GPTBot, ClaudeBot, CCBot, Amazonbot or PerplexityBot user agents still got 403 Your request was blocked on every path except robots.txt.

Insight

The zone’s bot_management object showed ai_bots_protection: disabled and bot_preference_sync_enabled: false, which reads as “not blocking”. The setting that blocked was ai_training: "disallow", enforced by a managed rule named “Block AI training crawlers for Bot Preference Sync disallow”.

In the dashboard this is the domain’s Security → Settings → Bot traffic → AI Crawler and Agent access policies, row Training. Setting it to “Allow (do not block)” removed the 403s.

Bot Fight Mode is a second, separate source of challenges. On the Free plan it cannot be exempted per bot; it is only on or off.

Diagnosis

  1. Request the same URL with several bot user agents and compare status codes:

    Terminal window
    curl -s -o /dev/null -w '%{http_code}\n' -A '<bot UA>' https://example.com/llms.txt
  2. Name the rule that fired with a GraphQL query on firewallEventsAdaptive, grouped by description and userAgent.

Did any agent fetch the Markdown files?

The same GraphQL API answers this without extra logging: httpRequestsAdaptiveGroups, filtered on clientRequestPath and verifiedBotCategory. It works on the Free plan with 31 days of retention.

Count only rows with a verifiedBotCategory. Most unverified requests carrying ChatGPT-User or Claude-User user agents were scanners probing for .env files.