Show Notes
In this talk, Hugo Bowne-Anderson, an independent data and AI consultant, educator, and host of the podcasts Vanishing Gradients and High Signal, shares his journey from academic research and curriculum design at DataCamp to advising teams at Netflix, Meta, and the US Air Force. Together, we explore how to build reliable, production-ready AI systems—from prompt evaluation and dataset design to embedding agents into everyday workflows.
You’ll learn about: How to structure teams and incentives for successful AI adoption Practical prompting techniques for accurate timestamp and data generation Building and maintaining evaluation sets to avoid “prompt overfitting”- Cost-effective methods for LLM evaluation and monitoring Tools and frameworks for debugging and observing AI behavior (Logfire, Braintrust, Phoenix Arise) The evolution of AI agents—from simple RAG systems to proactive, embedded assistants How to escape “proof of concept purgatory” and prioritize AI projects that drive business value Step-by-step guidance for building reliable, evaluable AI agents
This session is ideal for AI engineers, data scientists, ML product managers, and startup founders looking to move beyond experimentation into robust, scalable AI systems. Whether you’re optimizing RAG pipelines, evaluating prompts, or embedding AI into products, this talk offers actionable frameworks to guide you from concept to production.
LINKS Escaping POC Purgatory: Evaluation-Driven Development for AI Systems - https://www.oreilly.com/radar/escaping-poc-purgatory-evaluation-driven-development-for-ai-systems/ Stop Building AI Agents - https://www.decodingai.com/p/stop-building-ai-agents How to Evaluate LLM Apps Before You Launch - https://www.youtube.com/watch?si=90fXJJQThSwGCaYv&v=TTr7zPLoTJI&feature=youtu.be My Vanishing Gradients Substack - https://hugobowne.substack.com/ Building LLM Applications for Data Scientists and Software Engineers https://maven.com/hugo-stefan/building-ai-apps-ds-and-swe-from-first-principles?promoCode=datatalksclub
TIMECODES: 00:00 Introduction and Expertise 04:04 Transition to Freelance Consulting and Advising 08:49 Restructuring Teams and Incentivizing AI Adoption 12:22 Improving Prompting for Timestamp Generation 17:38 Evaluation Sets and Failure Analysis for Reliable Software 23:00 Evaluating Prompts: The Cost and Size of Gold Test Sets 27:38 Software Tools for Evaluation and Monitoring 33:14 Evolution of AI Tools: Proactivity and Embedded Agents 40:12 The Future of AI is Not Just Chat 44:38 Avoiding Proof of Concept Purgatory: Prioritizing RAG for Business Value 50:19 RAG vs. Agents: Complexity and Power Trade-Offs 56:21 Recommended Steps for Building Agents 59:57 Defining Memory in Multi-Turn Conversations
Connect with Hugo Twitter - https://x.com/hugobowne Linkedin - https://www.linkedin.com/in/hugo-bowne-anderson-045939a5/ Github - https://github.com/hugobowne Website - https://hugobowne.github.io/
Connect with DataTalks.Club: Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ Check other upcoming events - https://lu.ma/dtc-events GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/
More from DataTalks.Club
Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.
In this talk, Dewang Sultania, ML Engineer at Netflix and co-author of Designing AI Systems, shares his technical journey from building early inference platforms at Adobe to deploying multimodal AI models at scale. We explore the complete architecture of enterprise AI platforms, focusing on evaluation datasets, tool calling orchestration, and the practical realities of shipping production agents.
In this talk, Suhas Suresha, Senior Machine Learning Engineer at Adobe and co-author of Designing AI Systems, shares his expertise in building large-scale generative AI infrastructure from deploying real-time ECG analysis models at his previous startup, QALY, to architecting enterprise LLM platforms. We explore the massive engineering effort required to take an AI application from a weekend prototype to a scalable production system, and how to build a unified AI platform that eliminates "AI sprawl."
In this talk, Sahil Walia, Senior Technical Architect at Snowflake, shares his extensive data engineering expertise from writing simple SQL queries early in his career to architecting enterprise-scale Apache Iceberg lakehouses today. We explore the evolution of the modern data platform, the mechanics of open lakehouses, and how decoupling storage from compute is transforming data architecture.
In this talk, Philip Christos, Associate at M2VC, shares his expertise in frontier AI investing from his early days as a data analyst to evaluating cutting-edge deep tech startups. We explore how AI agents are reshaping modern work, the realities of venture capital in the LLM era, and why the future of software engineering relies more on high-level systems thinking than writing syntax.You’ll learn about:- How venture capitalists formulate AI investment strategies and evaluate startup pitch decks.- The rapid adoption of AI in software engineering compared to traditional sectors like accounting.- The crucial difference between data progress and algorithmic progress in frontier models.- Why LLMs still struggle with high-level software architecture, business logic, and design "taste."- How AI is accelerating iterations and breakthroughs in classical engineering and scientific research.- What specific deep tech and AI infrastructure ideas are actually attracting VC funding today.TIMECODES:00:00 Venture Capital Career Journey05:14 VC Analyst vs Data Analyst Roles10:13 AI Investment Thesis Strategy16:00 Emotion Analytics and Voice AI Startups22:09 LLM Impact on Venture Capital28:19 Software Engineering AI Adoption34:18 Data vs Algorithmic AI Progress40:15 Classical Engineering AI Acceleration46:12 Software Architecture AI Limitations51:11 Future Software Engineering Careers57:55 Systems Thinking in DevOps1:03:00 Pitching Deep Tech StartupsThis talk is perfect for software engineers, AI founders, and technical product managers who want to understand the evolving tech job market and venture landscape. It provides essential insights for anyone looking to build VC-backed AI startups or future-proof their career as AI coding tools become ubiquitous.Connect with Philip- Linkedin - https://www.linkedin.com/in/philipchristos/- Twitter - https://x.com/phil_christos- Website - https://t.me/stacked_channelConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/