An unfamiliar crawler appears in your server logs. Do you allow it because it might bring traffic, or block it because it might be collecting your work for an AI model?
That question is too broad to produce a useful answer. A crawler building a search index creates a different exchange from an AI agent retrieving a page for one user. A company assembling a training dataset has another purpose entirely.
On July 1, 2026, Cloudflare announced an evolution from Pay per crawl toward Pay per use. Cloudflare also said that beginning September 15, 2026, for all new domains onboarding to Cloudflare, Training and Agent will be blocked by default on pages that display ads, while Search will remain allowed by default. The lasting lesson is not which product switch to use. It is that “AI bot” is no longer a precise enough policy category.
A better question is: What is this crawler trying to do, which parts of the site can it reach, what does that access cost, and what value—if any—comes back to the site?
Start with purpose, not the bot’s name
Treat a crawler as software that reads pages automatically. A traditional search crawler usually indexes those pages and may send identifiable visitors back. Other crawlers may retrieve information for an AI assistant in real time, collect material for training or datasets, or perform work for your own team or a known partner.
Those purposes call for different defaults:
- Search indexing is usually an exposure trade: indexable content in exchange for discoverability and measurable referrals.
- Real-time agent access is an operational issue: which pages can be read, how frequently, and whether the agent can trigger actions.
- Training or dataset collection is closer to a licensing decision because it may reuse the content without sending readers back.
- Internal or partner automation depends on clear ownership, an agreed scope, and a way to handle changes or incidents.
Our earlier lesson, Does Your Website Need an AI Entry Point? Start With This Checklist, explains how a service can expose an entrance that software understands. This lesson addresses the next question: once a crawler reaches the door, what should it be allowed to do?
Make the first decision with this table
Set a default by purpose before investigating individual bot names. User-Agent strings can be misleading and source addresses can change. You can still revise a policy using declared purpose, observed behavior, requested paths, and measurable cost.
| Crawler purpose | Default decision | Keep or adjust that decision when… | Block or escalate when… | First metric to watch |
|---|---|---|---|---|
| Search indexing | Allow | The purpose is clear, referrals are identifiable, and the crawl rate stays at no more than 2× that of a normal search bot | Identity is unclear, sources change repeatedly, or generated summaries appear to replace visits; if AI referrals remain below 1% of total content-page visits for 30 days, move the crawler to observation or commercial review | Weekly AI referrals, pages crawled, and server cost |
| Real-time agent retrieval | Observe or rate-limit | It reads public pages only and does not sign in, place orders, submit forms, or escape per-user request limits | It bypasses a login barrier, repeatedly triggers search or filters, submits forms, or simulates sensitive actions | Daily request peak, error rate, and requests to high-value paths |
| Training or dataset collection | License or charge | A public license already permits reuse, the site has explicitly agreed, or the material was published for reuse | The operator does not state its purpose, combines search and training under one identity, or offers no opt-out and deletion process | Words collected, repeat-crawl ratio, and licensing response rate |
| Internal or known-partner bot | Allow for the agreed use | It has a stable identity, a named owner, and separate test and production traffic | No one owns it, its purpose expands without notice, or traffic surges without an accountable responder | Partner request volume and response time to anomaly notices |
These are defaults, not permanent labels. “Allow” means the exchange currently works for you. “Observe or rate-limit” preserves a useful function while controlling cost and behavior. “License or charge” applies when access directly consumes a content asset. “Block” is appropriate when the purpose is unknown, access exceeds its authority, or the risk persists after intervention.
Review every crawler category after 30 days. Otherwise, a bot whose behavior has changed may continue operating under a decision made for an earlier purpose.
Protect valuable paths before refining the whole policy
Publicly readable does not have to mean available for unlimited automated extraction.
For a first policy, identify the parts of the site where automated access could create disproportionate cost or risk. That may include membership areas, courses, pricing, original research, internal search, filters, and API-like routes. An API-like route looks like an ordinary web page but can be queried repeatedly to return structured results.
Then apply the policy at the appropriate layer:
- Use
robots.txtto express general crawling preferences, but not as mandatory access control. - Use a WAF or bot-management system to observe traffic patterns, impose rate limits, or block abusive access.
- Move repeated queries for pricing, inventory, research data, or search results to an API with identification, logging, and rate limits.
- Handle training datasets, paid content, and uses outside an existing partnership through contracts or licensing rather than treating technical access as automatic consent.
Your first version does not need to settle every edge case. It only needs to record three things: the default for each purpose, extra restrictions for high-value paths, and the person responsible for the 30-day review.
During that review, check crawler request volume, identifiable referrals, access to high-value paths, error rates, and cost changes. Reconsider an allow rule when the crawler runs at more than 2× the rate of a normal search bot or when AI referrals stay below 1% of total content-page visits for 30 days. Depending on the purpose, the next decision may be rate-limiting, licensing, or blocking.
If you are also deciding what AI-generated material may enter your publishing workflow, use AI Content Labels Will Not Save Your Time; Your Entry Points Matter More. One policy governs what your site receives; the other governs how machines may use what your site publishes.
Create an “AI crawler access policy v0.1” today. Add the four purposes, their default decisions, your high-value paths, the enforcement layer, an owner, and a review date 30 days from now. robots.txt, WAF rules, bot management, APIs, rate limits, and licensing are not six competing answers. They are different ways to enforce the same purpose-based policy.
AI Synthesis Card
Act as a policy facilitator for my content site. Build an AI crawler access policy v0.1 using only information I provide; mark missing evidence as “To confirm” instead of guessing. First establish which site areas are public, paid, membership-only, research-heavy, searchable, filterable, or API-like, then identify the available crawler logs and the people responsible for content and technical decisions. Classify observed access as search indexing, real-time agent retrieval, training or dataset collection, or internal/known-partner automation. For each category, produce one row containing purpose, paths reached, default decision, permission conditions, warning signals, measurement, enforcement layer, and owner. The decision must be Allow, Observe or rate-limit, License or charge, or Block. Flag any crawler running above 2× the rate of a normal search bot and any case where 30-day AI referrals are below 1% of total content-page visits. Map enforcement to robots.txt, WAF, bot management, API/rate limits, or contract/licensing as appropriate. End with one action I can complete today and a specific checklist for a review 30 days from now.
Everyday four-panel comic
Think of your site as a library with a reading room, a service desk, and an archive.
Someone who checks the catalog and directs visitors to the library can enter under ordinary rules. An assistant retrieving many items for one reader may need limits on speed and which rooms it can access. Someone who wants to copy the entire archive into a separate database should negotiate permission first.
They may all arrive wearing an “AI” jacket, but the jacket should not determine the access policy. Their purpose should.
- Three robots approach a small content archive: one carries a magnifying glass, one carries a single item on a tray, and one pulls a large empty cart.
- The search robot checks a public catalog, the agent delivers one requested item to a seated reader, and the third robot begins loading stacks into its cart.
- The archive owner leaves the search lane open, places a metered turnstile before the agent, and blocks the bulk-collection cart with a red rope.
- Search continues in the public area, the agent waits at the metered gate, and bulk collection remains behind the rope while the owner reviews the decision.
References
Cloudflare Blog: Your site, your rules: new AI traffic options for all customers — https://blog.cloudflare.com/content-independence-day-ai-options/ (2026-07-01)
Cloudflare Press Release: Cloudflare Allows the Agentic Internet to Flourish with a Simple Philosophy: Your Content, Your Rules — https://www.cloudflare.com/press/press-releases/2026/cloudflare-allows-the-agentic-internet-to-flourish-with-a-simple-philosophy-your-content-your-rules/ (2026-07-01)
TechCrunch: Cloudflare’s new policy pushes AI companies to pay for publishers’ content — https://techcrunch.com/2026/07/01/cloudflares-new-policy-pushes-ai-companies-to-pay-for-publishers-content/ (2026-07-01)
Help Net Security: Cloudflare changes AI crawler access rules — https://www.helpnetsecurity.com/2026/07/02/cloudflare-ai-crawler-controls/ (2026-07-02)
Cloudflare Blog: Introducing pay per crawl: Enabling content owners to charge AI crawlers for access — https://blog.cloudflare.com/introducing-pay-per-crawl/ (2025-07-01; background on the per-crawl mechanism)
