Show Notes
In this episode of Search Off the Record, Martin and Gary from the Google Search Relations team take a deep dive into how Googlebot and web crawling work—past, present, and future. Through their humorous and thoughtful conversation, they explore how crawling evolved from the early days of the internet, when scripts could index a chunk of the web from a single homepage, to the more complex and considerate systems used today. They discuss the basics of what a crawler is, how tools like cURL or Wget relate, and how policies like robots.txt ensure crawlers play nice with web infrastructure.
The conversation also covers Google's internal shift to unified infrastructure for all crawling needs, highlighting how different teams moved from separate crawlers to a shared system that enforces consistent policies. They explain why some fetches bypass robots.txt (like user-initiated actions) and the rising impact of automated traffic from new products and AI agents. With a nod to initiatives like Common Crawl, the episode ends with a look at the road ahead, acknowledging growing internet congestion but remaining optimistic about the web's capacity to adapt. Resources: Episode transcript → https://goo.gle/sotr092-transcript
Chapters: Chapters: 0:00 - Intro 0:53 - What is a Web Crawler? 3:11 - Building a Minimal Crawler 6:12 - Ethical Crawling: Robots.txt & Host Health 7:42 - BackRub and Early Crawling Challenges 11:02 - The Anatomy of a Search Engine Paper 13:09 - Crawling Across Google Products 16:51 - New Crawlers & User Agent Strings 22:38 - Crawlers Beyond Google 23:17 - The Evolution of Crawlers 26:32 - Bad Actors and Overpowering Servers 27:31 - Reducing the Footprint on the Internet 28:44 - The Future of Crawlers 31:29- Conclusion
Listen to more Search Off the Record → https://goo.gle/sotr-yt Subscribe to Google Search Channel → https://goo.gle/SearchCentral
Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
#SOTRpodcast #SEO #SearchOfTheRecord
Speakers: Martin Splitt, Gary Illyes Products Mentioned: Googlebot, Search
More from Search Off the Record
Should you publish your content as a PDF or stick strictly to standard HTML? Does Google treat PDF files differently when it comes to indexing, ranking, and AI search features? In this episode of Search Off the Record, Martin Splitt and John Mueller from the Google Search Relations team demystify how Google Search handles PDF documents. They break down some of Google's internal pipelines, explain why putting an LLMs.txt file inside a PDF is a bad idea, explore common technical traps like rasterized pages and canonicalization issues, and share practical advice on managing PDFs for SEO, AEO (SEO), and user experience. In this episode, you'll learn: Google's PDF converter pipeline: How Google converts binary PDF containers into HTML pages during indexing so the rest of the pipeline can process them normally.
Sitemaps have been a core element of web development and search engine optimization for two decades, yet Search Console warnings like "Couldn't fetch" continue to generate concern among site owners. Are XML sitemaps still necessary with modern search crawlers, or can search engines and AI bots figure out site structure on their own? In this episode of Search off the Record, Google Search Relations team members Martin Splitt and John Mueller break down the history, technical specifications, and some misconceptions surrounding sitemaps, RSS feeds, LLMs.txt, canonicalization signals, and Search Console indexing behaviors. In this episode, you'll learn: The Origins of Sitemaps: How early web crawling challenges led to the sitemap standard and how building a sitemap generator started John Mueller's career at Google.