Show Notes
How do you know if your AI product is actually any good? Hamel Husain has been answering that question for over 25 years. As a former machine learning engineer and data scientist at Airbnb and GitHub (where he worked on research that paved the way for GitHub Copilot), Hamel has spent his career helping teams debug, measure, and systematically improve complex systems.
In this episode, Hamel joins Teresa Torres to break down the craft of error analysis and evaluation for AI products. Together, they trace his journey from forecasting guest lifetime value at Airbnb to consulting with startups like Nurture Boss, an AI-native assistant for apartment complexes. Along the way, they dive into: - Why debugging AI starts with thinking like a scientist - How data leakage undermines models (and how to spot it) - Using synthetic data to stress-test failure modes - When to rely on code-based assertions vs. LLM-as-judge evals - Why your CI/CD set should always include broken cases - How to prioritize failure modes without drowning in them
Whether you’re a product manager, engineer, or designer, this conversation offers practical, grounded strategies for making your AI features more reliable—and for staying sane while you do it.
More from Just Now Possible
What does it take to build an AI voice agent that people will talk to for 44 minutes about their own blind spots? Your360 AI found out, one call at a time.
What happens when you hand your opportunity solution tree to an AI? Vistaly rebuilt its entire product to find out—and the agents were the easy part.
What does it take to build an AI app builder specifically for product managers—not engineers—inside an already crowded market?
How do you build trustworthy AI diagnostic tools in one of medicine's most historically under-researched areas?
What does it take to reinvent a 14-year-old company—not once, but twice?