This Week in NET

The Future of Content and AI: Pay per Crawl and What’s Next

This Week in NET

Show Notes

In this episode, host João Tomé is joined by Will Allen, Cloudflare’s VP of Product Management, to discuss Pay-Per-Crawl and our new permission-based model for AI bots. These updates, launched on July 1, 2025 — what we call Content Independence Day — aim to reshape how AI models access and reward content, shifting from an opt-out to an opt-in approach. Cloudflare’s CEO, Matthew Prince, also appears in a video at the start, reflecting on Content Independence Day. We explore how AI Overviews are changing the old “traffic for content” model and how Cloudflare is helping creators take control through tools like AI Audit. Plus: the future of trustworthy content, bot authentication, and the rise of a fairer content economy. Send us your questions for a future episode on these topics at [email protected]. Check the mentioned blog posts here: https://blog.cloudflare.com/tag/pay-per-crawl/

More from This Week in NET

May 1, 2026

In this episode of This Week in NET, JQ Lau and Victor Hwang from our Network & Infrastructure Strategy team walk us through Cloudflare's 13th generation of servers — the machines that power a significant part of the internet across 330+ cities worldwide.

Apr 29, 2026

In this episode of This Week in NET, Vinti Batiste, VP of Sales for US Enterprise at Cloudflare, shares her 30-year journey across IBM, Cisco, and now Cloudflare — and what it really means to be in enterprise sales.

Apr 22, 2026

In this special Agents Week edition of This Week in NET, host João Tomé is joined by Ming Lu (Principal Product Manager) and Anni Wang (Product Manager) to recap Cloudflare’s first-ever Agents Week.

Apr 14, 2026

In this World Quantum Day special edition of This Week in NET, host João Tomé is joined by Bas Westerbaan (Principal Research Engineer) and Sharon Goldberg (Senior Director, Product) to explain why the timeline for post-quantum cryptography may be arriving sooner than expected.

Apr 10, 2026

In this short edition of This Week in NET, host João Tomé joins from the island of Madeira for a quick preview of Cloudflare’s first Agents Week.

See all episodes →