Cloudflare’s Plan to Save the Internet from AI Scraping
Matthew Prince, the CEO of Cloudflare, showed up to a recent conference wearing all black and motorcycle boots. On a scorching 32-degree day. His fashion choice was actually mourning attire for what he sees as the death of the internet as we know it.
And when you hear the numbers he shared, you might want to dig out your own black outfit.
Here’s what’s happening: The fundamental economics of content creation on the web is collapsing. Fast. And if we don’t act soon, the entire ecosystem that’s powered the internet for the past 35 years could disappear.
The Numbers That Should Terrify Every Content Creator
Let’s talk data, because the numbers tell a story that’s both shocking and urgent.
Ten years ago, the relationship between Google and content creators was relatively fair. For every two pages Google scraped from your website, they’d send you one visitor. It was a trade: they’d take snippets of your content to show in search results, and in exchange, they’d send traffic your way.
Fast forward to today. That ratio has shifted dramatically:
- Six months ago: 6 pages scraped for 1 visitor
- Today: 18 pages scraped for 1 visitor
That’s a 9x deterioration in just a decade. But here’s where it gets truly alarming.
The AI companies make Google look generous:
- OpenAI six months ago: 250 pages scraped for 1 visitor
- OpenAI today: 1,500 pages scraped for 1 visitor
- Anthropic six months ago: 6,000 pages scraped for 1 visitor
- Anthropic today: 60,000 pages scraped for 1 visitor
Why such dramatic changes in just six months? People are trusting AI more. They’re getting their answers directly from ChatGPT or Claude instead of clicking through to read the original content.
Why This Matters More Than You Think
You might be thinking, “So what? Technology changes. We adapt.” But this isn’t just another tech disruption. It’s an existential threat to how information is created and shared online.
Prince breaks it down simply: People create content for one of three reasons:
- To make money (through ads or subscriptions)
- To gain fame or recognition
- To build their ego or reputation
When AI companies scrape your content and serve it up without sending traffic back, all three incentives disappear. No visitors means no ad revenue. No readers means no recognition. No audience means no ego boost.
And here’s what’s even interesting: six months ago, 75% of Google queries got answered directly on Google’s page. Prince estimates it’s probably 90% today. When you add AI overviews that mash together content from multiple sources, the original creators become invisible.
The Domino Effect Nobody’s Talking About
This isn’t just about publishers losing money. It’s about the entire information ecosystem breaking down.
Think about it: If content creators can’t monetize their work, they stop creating. If they stop creating, where does fresh, original content come from? AI models can only regurgitate and recombine what already exists. They can’t conduct interviews, break news stories, or share lived experiences.
We’re heading toward an information desert where AI feeds on AI, creating an ever-degrading cycle of recycled content.
Cloudflare’s Bold Plan (And Why It Might Actually Work)
This is where Prince’s black outfit starts to make sense. He’s not just mourning, he’s mobilizing.
Cloudflare sits in front of about 20-30% of the web, giving them a unique vantage point and, more importantly, leverage. Their plan is surprisingly straightforward: Create scarcity.
Right now, AI companies can scrape content freely from most websites. Publishers might strike deals with OpenAI for $20 million, but then let every other AI company take their content for free. As Prince points out, you can’t have a functioning market without scarcity.
Cloudflare is planning to offer publishers tools to block AI scrapers completely. Not for money… Prince was clear about that. They’re giving it away because they believe the survival of the open web depends on it.
A Vision for a Better Future
But Prince isn’t just playing defense. He has a vision for how content creation could evolve that’s genuinely exciting.
Instead of chasing pageviews with increasingly sensational headlines (which he partly blames for our current political climate), imagine if content creators were compensated based on how much their work advances human knowledge.
Picture specialized AI models (one for teenage girls interested in pop culture, another serving as the world’s perfect medical advisor). Each would value different content. A Taylor Swift song announcement would be valuable to the first but not the second. A new study on chemotherapy treatments would be the opposite.
If AI companies had to pay for content based on its actual value to their specific models, we’d create a real market. Content creators would be incentivized to fill knowledge gaps rather than chase clicks.
What This Means for You
If you’re a content creator, publisher, or anyone who depends on the web for your livelihood, this isn’t a distant threat. It’s happening now. The 60,000-to-1 scraping ratio from Anthropic isn’t a future projection, it’s today’s reality.
The good news? This is a solvable problem. But it requires collective action. Publishers need to work together to create scarcity. Cloud service providers need to offer tools to control access. And yes, AI companies need to be part of the solution too.
Prince made an interesting point: He goes to war every day with state-sponsored hackers from China, Russia, Iran, and North Korea. Stopping “some nerd with a Delaware C-corp in Palo Alto” should be comparatively easy.
The question isn’t whether we can save the internet. It’s whether we’ll act fast enough to do it.
If you’re in the content creation space and haven’t started thinking about this, it’s time to start. The web that’s sustained businesses, careers, and the free flow of information for 35 years is under threat. But with the right approach (creating scarcity, demanding fair compensation, and reimagining how we value content) we might just build something better.
At least that is what Prince believes…
Oh wow, this is fascinating.
And terrifying!
Thanks Tamara.