SourceCited / Crawler licensing
Guide · Crawler control
The crawler question moved from allow-or-block to license-and-verify. Most site owners still want in — but Cloudflare now verifies it at the edge.
Cloudflare now blocks AI crawlers by default for new domains and runs pay-per-crawl over HTTP 402; RSL adds machine-readable license terms to robots.txt with major-publisher backing. If you want AI citations — most site owners do — keep retrieval bots allowed and verify 200s at the edge; treat monetization as a publisher play with real enforcement gaps.
Three shifts define the 2026 crawler landscape. Cloudflare blocks AI crawlers by default for new domains (since July 2025), so your robots.txt can say Allow while the edge serves 403s. Cloudflare's pay-per-crawl revives HTTP 402 Payment Required — reportedly over a billion 402 responses a day. And its Content Signals policy lets you declare permitted uses (search, AI input, AI training), while the IETF's Web Bot Auth proposal adds cryptographic bot verification.
RSL (Really Simple Licensing) extends robots.txt with machine-readable license terms — free, attribution, subscription, pay-per-crawl, pay-per-inference. It is led by RSS co-creator Eckart Walther with Doug Leeds, backed by Reddit, Yahoo, Medium, Quora, Ziff Davis, O'Reilly and MIT Press, enforced in principle at CDNs (Fastly, Akamai, Cloudflare), and drew tentative Creative Commons support in December 2025. The honest read: declaring terms is easy; enforcement and pay-per-inference tracing remain unsolved, reported license-omission rates run above 70%, and revenue for a typical site is unproven.
| If your goal is | Then |
|---|---|
| AI visibility (most site owners) | Allow retrieval bots, keep training access unless the content is strategic, and verify 200s in edge logs monthly |
| Protecting premium, licensable content | Declare terms (Content Signals / RSL), consider pay-per-crawl — and accept reduced AI visibility |
| Either | Log-verify: what robots.txt says and what your CDN does can differ |
Blocking retrieval crawlers means opting out of AI answers for that content. Make that choice deliberately — never by CDN default.
Whatever stance you choose, verify it where it is enforced. Open your CDN's bot-management settings and find the AI-crawler toggle; pull thirty days of logs and grep for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Agent status codes; then compare what the edge actually did against what your robots.txt intends. Silent drift here is the norm, not the exception.
Here is my robots.txt, my CDN bot-management settings (pasted or exported), and a 30-day log sample. Build one table per AI crawler: what robots.txt says | what the edge actually returned (status codes seen) | verdict — aligned, silently blocked, or silently allowed. For every silent block, name the exact CDN setting to change.
Only if protection or monetization outweighs visibility for that content. For most sites seeking citations, no: allow retrieval bots explicitly and verify at the edge that they actually get 200s.
The rails exist — Cloudflare pay-per-crawl and RSL license terms — but enforcement is immature, omission rates are high, and typical-site revenue is unproven. Watch it as a publisher play; do not budget on it.
Your CDN may override you. Cloudflare has default-blocked AI crawlers on new domains since July 2025, so check the edge settings and confirm in logs, not just in the file.