If you’ve published content online lately, chances are an AI bot’s already scraped it.
From chatbots to search assistants, the demand for quality training data has pushed AI companies to scrape websites with little concern for consent – or compensation.
But that could be changing fast. Cloudflare is an internet infrastructure giant that helps keep millions of websites online and running smoothly. And it’s just made a major move – announcing that AI scrapers will now be blocked by default across their platform. For context, these are the bots that quietly hoover up your content without your say-so.
If your website relies on content to attract visitors, rank in search results and drive business (and let’s face it, whose doesn’t?), this shift could be huge. At Wrise, we help clients create content that people and search engines love – and moves like this help protect that investment. So, what’s actually changed?
What Cloudflare has done
Until recently, Cloudflare gave users the option to block AI bots. Handy if you knew about it. But now, they’ve gone one step further – blocking these crawlers by default for all new domains on their platform. That means if you’re using Cloudflare, you’re automatically getting protection unless you choose to turn it off.
They’re also testing something called “Pay Per Crawl”. It’s exactly what it sounds like – a way for website owners to charge AI companies for access to their content. If the bots want your data, they’ll have to pay for it.
It’s a bold move. And it could be a game-changer for small publishers and content creators who’ve been quietly fuming as their hard work gets scraped and reused without so much as a credit – let alone a cheque.
Why it matters for website owners
Let’s face it – content isn’t cheap. Whether you’ve written it yourself, paid someone (like us!) to do it, or spent years building up your brand’s authority, it’s your intellectual property. And when AI bots swoop in and grab it without permission, it’s more than annoying – it’s potentially undermining your business.
Related: Google Is Stealing Your Traffic: Here’s How!
Some of these AI crawlers act aggressively – hammering your server with requests that slow down your site or even knock it offline. Others simply ignore polite requests in your robots.txt file to stay away. Cloudflare’s new default settings add a proper layer of defence – and a clear message: ask first, or don’t come in.
The big picture – tensions with AI companies
Naturally, not everyone’s thrilled. OpenAI, the maker of ChatGPT, reportedly declined to take part in Cloudflare’s preview of these changes. Their argument? Cloudflare is becoming a “middleman” between content creators and AI firms. They claim to respect robots.txt – but studies show plenty of crawlers still ignore it.
The bigger issue here is control. Up to now, AI companies have operated with a sense of entitlement – scraping first, negotiating later (if at all). But Cloudflare’s move flips the script. It gives creators more leverage. If AI tools want access to premium content, they’ll need to start treating it like the valuable resource it is.
The economics of AI data
Here’s where it gets really interesting – and possibly a bit brutal. Experts warn that many generative AI companies might not survive this new environment. If they suddenly have to pay for high-quality data, their business models could fall apart. And with fierce competition from countries like China, Western firms in particular could struggle to stay profitable.
As Ilia Kolochenko, a cybersecurity expert, bluntly puts it: “This may be fatal to the current business models of GenAI.” It’s not just a tech story – it’s a shift in how value is assigned to digital content.
What this means for your business
If you’re a content-driven business – whether you run a blog, publish industry insights or just care about your SEO – this is a moment to pay attention to.
First off, Cloudflare’s move gives you more control. You get to decide if AI companies can access your content, and potentially profit from it. That’s a huge win, especially for smaller businesses and publishers who’ve been effectively locked out of these negotiations until now.
Secondly, there’s a possible new revenue stream. The “Pay Per Crawl” model isn’t widely adopted yet, but it’s a signal that the tide is turning. AI companies may have to pay to play, and if your site’s content is valuable, you might find yourself with a new bargaining chip.
Lastly, it’s a reminder to stay on top of your bot settings. Whether you’re using Cloudflare or another CDN, it’s worth reviewing who has access to your content and why. Your words are valuable – and it’s time they were treated that way.
Stay on the right side of AI developments
Cloudflare’s default block on AI crawlers is more than a technical tweak – it’s a big shift in how content is valued, protected and potentially monetised. For content creators, publishers and businesses like yours, it opens the door to greater control and new opportunities.
Of course, AI isn’t going anywhere. But how it interacts with the rest of the internet is starting to change. And that’s good news if you’ve spent time and money crafting quality content.
At Wrise, we’re all about making content that works for you – attracting the right people, boosting your rankings and getting results. If you want help protecting your content, creating new blog posts that search engines love, or just figuring out what the hell “Pay Per Crawl” means for your business – we’ve got you.
Let’s make sure your content works for you – not just the bots. Contact us on hello@wrise.co.uk or call 0161 533 0593 today.
Affordable SEO Services for Small Businesses 🚀
Outsmart your competition with a low-cost SEO package that delivers real results. Stop dreaming. Start ranking!
✅ No Contracts
✅ No Set Up Fees
✅ Free Review
Affordable SEO Services for Small Businesses 🚀
Outsmart your competition with a low-cost SEO package that delivers real results. Stop dreaming. Start ranking!
Get a free SEO review of your website! 🧐
Want to boost your website’s search engine rankings and organic traffic? Tell us about your project to get started. We’ll provide a free preliminary review and SEO action plan.
