Cloudflare implements changes in policy regarding web crawlers
The hosting platform Cloudflare is changing its approach to interacting with web crawlers, blocking their operations on sites starting from this September. The company is trying to return control over content to its owners.
According to new rules that will come into effect on September 15, 2026, Cloudflare will provide sites with settings that allow indexing for search engines but block the use of pages for training AI models. This applies to new clients and new sites of existing users. Mixed web crawlers that do not allow for control over content use for AI will also be blocked.
Matthew Prince, co-founder and CEO of Cloudflare, emphasized the importance of quickly creating a sustainable ecosystem in conditions where most internet traffic does not belong to humans. The new tools aim to encourage bot operators to separate search and model training functions.
Cloudflare is also relaunching its Pay Per Crawl feature under the name Pay Per Use. Now, payments to site owners will be made for displaying content in AI chatbot responses, not just for actual page scanning.
Experts point out that such a policy could affect companies like Google, whose Googlebot simultaneously indexes pages and collects information for AI training. Cloudflare’s new policy aims to change this practice, providing greater control for publishers.
It is also worth noting that Cloudflare recently announced the reduction of over 1,100 employees as part of a transition to an AI-focused model.
| Implementation Date | Process | Consequences |
|---|---|---|
| September 15, 2026 | Blocking web crawlers | Content control for owners |
| 2025 (relaunch) | Pay Per Use | Payments for appearance in AI responses |
