The new AI crawler rules changed on 15 September: check your data collection now

Site owners are not blocking bots as a matter of taste. They are protecting the traffic their business runs on. Cloudflare reports that most sites still want search, and that AI summaries cost them visits.

17%

Sites that enable some mechanism to block AI training

under 1%

Sites that block search bots entirely

over 40% more likely

Readers of AI summaries who end a search without visiting a website

Those three figures are all from Cloudflare. Read together they tell you what you are negotiating with. A publisher wants to be found and wants the click. It is far less relaxed about content that feeds an answer somewhere else. If your collector looks like the second thing, expect friction, even when your use is modest and internal.

Cloudflare names Apple, Google and Microsoft as accountable operators of crawlers that serve more than one purpose, and sets out four requirements they meet.

• An opt-out for AI training, expressed through robots.txt.
• An opt-out for AI summaries.
• Visibility at the level of the individual URL into how content is used.
• Assurance that opting out of training will not affect search rankings.

You are not Google, and nobody is asking you to publish a policy. But the shape of that list is a useful standard to hold your own collection to. Can you say, per source, what you take and what you do with it? Can you honour a robots.txt rule you disagree with? Could you show a publisher which URLs you fetched last month and why? If the answer is no, you are relying on not being noticed, which is not a plan.

This is the failure we see most often, and it is not an outage. A blocked or challenged request usually returns a page. It has a status code that looks fine and a body that contains a notice instead of listings. Your parser finds no rows, writes nothing, and logs no error. Nothing breaks.

A week later the dashboard shows fewer new listings, fewer job ads, fewer price changes. That reads as a softening market. Someone briefs it as a market signal. Nobody asks whether the feed is intact, because the feed did not complain. The cost of that is not a technical one. It is a decision made on a chart that was describing your collector, not your market.

Three checks catch it early, and none of them are expensive to build.

• A coverage check per source: how many pages did we expect to see, and how many did we parse?
• An expected-row-count alert that fires on a drop, not only on an error.
• A freshness check per source, so a feed that has quietly stopped updating is visible on the same screen as the numbers it feeds.

The same change cuts the other way. Cloudflare says the preset configurations apply to domains onboarding from 15 September 2026, and that ad-supported sites default to Disallow AI Training. If your organisation publishes listings, a jobs board, research reports or open data, your own settings are now part of your publishing strategy rather than a network detail.

So confirm it rather than assume it. Find out who administers the domain, look at what each category is currently set to, and decide deliberately. A default that suits an advertising business may not suit a council that wants its service information quoted as widely as possible.

1. Inventory your sources. List every site a collector touches, with the business decision it feeds.
2. Test one URL per source with your real collector identity, not from a browser. A browser will often succeed where your pipeline does not.
3. Compare row counts per source with the same week a month ago. Investigate any quiet decline before you explain it as a trend.
4. Declare a user agent and a contact address, and keep the story behind it true. A publisher who can reach you has a reason to talk instead of block.
5. Check your own domain's settings for Search, Training and Agent, and write down the choice you made and why.

Start with the inventory, because you cannot judge exposure you have not written down. Then pick the one source that feeds your most-used report and test it properly this week. If you would rather have someone look at it with you, read the RS Core case study at /subs-services/rscore, then send us the list of sources you depend on and the decisions they support.

1. Have it both ways: stay discoverable in search while disallowing AI training (accessed 2026-09-23)