S01E06 - Hermes Agents and Penmanship. How an analogue hobby helped us shape our agents.

In Ep 13, we discuss where probabilistic AI tools like JEV fit alongside large language models, how they might help make faster and cheaper decisions in web scraping pipelines, and what we learned from experimenting with JEV. We also talk about building scraping workflows with Scrapy and notebooks, and the practical trade-offs of scraping product variants at scale. Topics include:• JEV and probabilistic classification• Using AI to decide when a website needs browser rendering• Quality of input signals, cost, speed, and debugging• Scrapy workflows, notebooks, and data review• Product variants, extraction, and the cost of extra requests Subscribe for more conversations about web scraping, data extraction, and the tools and techniques behind them. #WebScraping #AI #DataExtraction

When should you use a browser for web scraping? John and Ayan join Zyte’s Paweł to explore CDP, Playwright, and the trade-offs between running browsers locally and using managed infrastructure. From fingerprinting and session management to pricing and AI agents, discover what it takes to run browser-based scraping at scale.

In this episode, Ayan and John explore the rapidly evolving world of AI agents, coding harnesses and custom developer tools. They discuss emerging AI models, modular agent ecosystems, web scraping, prompt-to-data workflows, AI evaluation and the security risks of putting agents into production. They also share new Scrapy projects, reflect on the power of open-source software and Linux, and look at the rise of proactive desktop agents. Plus, there’s time for model pricing, developer workflows and a well-earned appreciation of croissants.

Episode 10 of the Zyte podcast is out, and it is one of the widest-ranging conversations we have recorded this year. John Rooney sits down with Neha Setia Nagpal and Ayan Pahwa for an unfiltered, all-in-style chat covering everything the team has been building, reading, and arguing about lately.

This episode Ayan and John talk to Ian Johnson, staff engineering and author of the Harness Engineering book. We talk through the subject, what it is and what it means to us as builders.Ian Johnson: https://tacoda.devHarness Engineering book: https://leanpub.com/harness-engineering Upcoming Webinar with Ian Johnson: https://www.eventbrite.com/e/how-to-grow-from-junior-to-senior-in-the-age-of-ai-tickets-1994040543945 Access the web's data. Clean, ready, and at scale | https://zyte.com