Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon
Lenny's Podcast: Product | Career | GrowthShow Notes
Aishwarya Naresh Reganti and Kiriti Badam have helped build and launch more than 50 enterprise AI products across companies like OpenAI, Google, Amazon, and Databricks. Based on these experiences, they’ve developed a small set of best practices for building and scaling successful AI products. The goal of this conversation is to save you and your team a lot of pain and suffering.
We discuss: 1. Two key ways AI products differ from traditional software, and why that fundamentally changes how they should be built 2. Common patterns and anti-patterns in companies that build strong AI products versus those that struggle 3. A framework they developed from real-world experience to iteratively build AI products that create a flywheel of improvement 4. Why obsessing about customer trust and reliability is an underrated driver of successful AI products 5. Why evals aren’t a cure-all, and the most common misconceptions people have about them 6. The skills that matter most for builders in the AI era — Brought to you by: Merge—The fastest way to ship 220+ integrations: https://merge.dev/lenny Strella—The AI-powered customer research platform: https://strella.io/lenny Brex—The banking solution for startups: https://www.brex.com/product/business-account?ref_code=bmk_dp_brand1H25_ln_new_fs — Transcript: https://www.lennysnewsletter.com/p/what-openai-and-google-engineers-learned — My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/183007822/referenced — Get 15% off Aishwarya and Kiriti’s Maven course, Building Agentic AI Applications with a Problem-First Approach, using this link: https://bit.ly/3V5XJFp — Where to find Aishwarya Naresh Reganti: • LinkedIn: https://www.linkedin.com/in/areganti • GitHub: https://github.com/aishwaryanr/awesome-generative-ai-guide • X: https://x.com/aish_reganti — Where to find Kiriti Badam: • LinkedIn: https://www.linkedin.com/in/sai-kiriti-badam • X: https://x.com/kiritibadam — Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ — In this episode, we cover: (00:00) Introduction to Aishwarya and Kiriti (05:03) Challenges in AI product development (07:36) Key differences between AI and traditional software (13:19) Building AI products: start small and scale (15:23) The importance of human control in AI systems (22:38) Avoiding prompt injection and jailbreaking (25:18) Patterns for successful AI product development (33:20) The debate on evals and production monitoring (41:27) Codex team’s approach to evals and customer feedback (45:41) Continuous calibration, continuous development (CC/CD) framework (58:07) Emerging patterns and calibration (01:01:24) Overhyped and under-hyped AI concepts (01:05:17) The future of AI (01:08:41) Skills and best practices for building AI products (01:14:04) Lightning round and final thoughts — Referenced: • LevelUp Labs: https://levelup-labs.ai/ • Why your AI product needs a different development lifecycle: https://www.lennysnewsletter.com/p/why-your-ai-product-needs-a-different • Booking.com: https://www.booking.com • Research paper on agents in production (by Matei Zaharia’s lab): https://arxiv.org/pdf/2512.04123 • Matei Zaharia’s research on Google Scholar: https://scholar.google.com/citations?user=I1EvjZsAAAAJ&hl=en • The coming AI security crisis (and what to do about it) | Sander Schulhoff: https://www.lennysnewsletter.com/p/the-coming-ai-security-crisis • Gajen Kandiah on LinkedIn: https://www.linkedin.com/in/gajenkandiah • Rackspace: https://www.rackspace.com • The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code | Dan Shipper (co-founder/CEO of Every): https://www.lennysnewsletter.com/p/inside-every-dan-shipper • Semantic Diffusion: https://martinfowler.com/bliki/SemanticDiffusion.html • LMArena: https://lmarena.ai • Artificial Analysis: https://artificialanalysis.ai/leaderboards/providers • Why humans are AI’s
More from Lenny's Podcast: Product | Career | Growth
Emmanuel Ameisen is an interpretability researcher at Anthropic, where he looks inside LLMs like Claude to understand how they think. Before joining Anthropic, he spent years in applied ML at Stripe and Insight, and is the author of Building Machine Learning Powered Applications. His research has produced some of the most striking findings in the field: that models form general representations of concepts, plan many tokens ahead without stating those plans explicitly, and may exhibit something structurally similar to the distinction between conscious and unconscious processing in humans.
Josh Woodward is the head of Google Labs, the Gemini app, and AI Studio. He has spent over 17 years at Google and oversees its longest-running and largest labs team, whose products include NotebookLM (now Gemini Notebook), Project Genie, and Flow. His job is to find, experiment with, and scale new AI products, and he has built one of the most distinctive product cultures in the industry for generating ideas, running experiments, and, even more importantly, knowing when to quit.
Tibo Sottiaux leads ChatGPT and Codex at OpenAI. Under his stewardship, OpenAI has shipped some of its most consequential consumer launches: Codex, ChatGPT Work, and most recently, the new Dots personal agent platform. He joined me at DevDay, hours after his team launched more than 20 new products, to talk about where things are heading.
Molly Graham is back for round two, and this one is even more powerful. Molly has spent more than 20 years helping organizations and the humans inside them navigate growth and change. She’s held leadership roles at Google, Facebook, Quip, and the Chan Zuckerberg Initiative and is the host of TED’s WorkLife podcast (which she took over from Adam Grant). She also runs Glue Club, a leadership community for senior operators, and writes a popular newsletter called Lessons.
Peter Sellis was the first product manager at Snapchat, where he spent seven years building one of the most beloved consumer products in history. He went on to serve as Head of Product at Discord, where he helped drive some of the fastest user growth the platform had seen. Having left his role at Discord, he gets to be especially real and unfiltered — nothing to promote, no PR, no comms team.