
Show Notes
Engineering teams have access to more delivery data than ever. Jira can measure cycle time and velocity. GitHub can track pull requests and code changes. Deployment tools can show how often software reaches production and how reliably those releases perform. These metrics are useful, but they only tell us how efficiently a team produces software. They do not tell us whether that software made the product better. In this episode of The Product Engineering Podcast, James explores the difference between output and outcome metrics, and why engineering leaders need both. The episode begins with the McNamara fallacy: the tendency to focus on what is easy to measure while overlooking information that may be more important. Engineering organisations can fall into the same trap when they treat delivery activity as the primary measure of success. James explains why output metrics work well for predictable work such as migrations and platform upgrades, where the intended destination is already understood. Product development is different. Teams are often working with uncertain assumptions about customers, behaviour and business impact. Efficient delivery cannot compensate for building the wrong thing. The episode also covers: The difference between output and outcome metrics Why delivery metrics are easier to measure and attribute When metrics such as cycle time, deployment frequency and story points are useful Why product engineering cannot be managed like a manufacturing line How feature factories emerge when teams are rewarded mainly for shipping The growing importance of outcome accountability as AI makes implementation faster Why teams should decide how they will measure success before delivery begins The difference between leading and lagging indicators How outcome measurements can guide iteration and product decisions James shares an example involving two engineering teams. One completed a predictable React migration, while the other delivered a navigation redesign that users disliked. Both teams delivered their planned work successfully, but only one produced the intended result. The example demonstrates why delivery success and product success are not the same thing. The question engineering teams should ultimately be able to answer is simple: Did the product get better? Learn more about The Product Engineering Podcast at doproductengineering.com.
More from The Product Engineering Podcast
Where should the line between product management and engineering sit? And in a world where AI is making it easier for anyone to build software, do we even need that boundary anymore? In this episode of The Product Engineering Podcast, James is joined by Phil Hornby, product leadership coach, founder of For Product People and host of the Talking Roadmaps podcast, to explore the increasingly blurred boundaries between product management and engineering. Drawing on Phil's experience across both engineering and product leadership, we discuss whether organisations should define responsibilities around job titles or build teams around the strengths of the people within them. We explore what it really means to be a Product Engineer, why engineers should get closer to customers and business decisions, and whether AI could eventually enable individuals to take on responsibilities that previously required entire cross-functional teams. We cover: Where the boundary between product management and engineering should sit, and whether we need one at all Why organisations should be designed around people's individual strengths rather than rigid job descriptions How engineers can participate directly in product discovery without becoming product managers The difference between understanding users and understanding the customers who actually pay for a product Why product management is as much about business strategy, pricing and commercial viability as it is about user experience Whether AI makes the idea of a one-person product engineering team realistic Using AI and personal knowledge management systems to capture context, accelerate thinking and improve writing Why storytelling is an essential skill for communicating customer problems to engineering teams The danger of using AI to turn feature requests directly into working software without understanding the underlying problem The trade-offs between generalist Product Engineers and deeply specialised engineering teams How different organisational structures, from functional organisations to cross-functional teams, can both be successful Why engineers should become more curious about customers, business models and the commercial outcomes their work supports A recurring theme is that high-performing teams aren't created by finding people who fit neatly into predefined roles. They're created by combining people with complementary strengths and giving them the freedom to contribute where they can add the most value. We finish with Phil's advice for engineers who want to become more product-focused: stop worrying so much about the terminology and start developing genuine curiosity about your customers, your users and your business. Find Phil: For Product People: Phil's product leadership coaching and training Talking Roadmaps: Phil's podcast exploring product roadmaps and product leadership LinkedIn: Search for Phil Hornby
What does engineering management look like in a world where AI can write much of the code, and engineers are increasingly expected to think like product managers? In this episode of The Product Engineering Podcast, James is joined by Ryan Murphy, founder of EM Accelerator and Crying on the Sofa, to explore the changing role of engineering managers and what it means to lead an effective Product Engineering team. We discuss the relationship between engineering managers and product managers, why managers sometimes become unnecessary barriers between engineers and the wider business, and how to build teams that focus on outcomes rather than simply delivering features. We also explore how AI is changing engineering management, why communication and stakeholder management are becoming more important, and whether engineering managers will need to become more hands-on again. We cover: How the relationship between engineering managers and product managers affects team effectiveness Why engineering managers shouldn't act as gatekeepers between engineers and product How misunderstandings between product and engineering create unnecessary conflict Why engineers should spend time with sales, marketing and customer success to develop product understanding Celebrating product outcomes and learning rather than just technical excellence and delivery How AI coding tools are changing the skills engineers and engineering managers need Helping introverted engineers develop their communication and stakeholder management skills Using AI as a management assistant, sounding board and organisational tool The debate around AI-generated writing and preserving authenticity Why the human side of engineering management may become more important as AI takes on more technical work Supporting employees struggling with their mental health, and the importance of active listening One of the central questions is whether AI will turn engineering managers back into hands-on developers, or whether the increasing importance of coaching, communication and team wellbeing will push the role further towards people leadership. We finish by discussing mental health in engineering, including Ryan's experience and his work with Crying on the Sofa, an initiative aiming to start one million conversations about mental health. Find Ryan: EM Accelerator: https://emaccelerator.com Crying on the Sofa: https://www.cryingonthesofa.org
AI can get you to something that looks finished remarkably quickly. But looking finished and being production-ready are not the same thing. In this episode of The Product Engineering Podcast, James explores the gap between an AI-built prototype and software that is actually ready to run reliably in production. A prototype might get you 80% of the visible result very quickly, while the remaining 20% — security, scalability, observability, maintainability and operational support — can require far more work than people realise. We cover: Why an impressive AI prototype isn't necessarily production-ready The hidden engineering work users never see Scalability, security, observability and maintainability Why software needs to be designed for the right level of scale The difference between code that demonstrates an idea and code you can support long term Why prototypes are still incredibly valuable learning tools When it's perfectly reasonable to build something you intend to throw away The role engineers play when product managers prototype with AI Why “production-ready” depends on the needs of your organisation The danger of both under-engineering and over-engineering Why prototypes shouldn't be slowed down by unnecessary production requirements Making the gap between prototype and production explicit A prototype doesn't need to answer every engineering question if its purpose is to answer a product question. If it helps you discover whether an idea is worth pursuing, it has already delivered value — even if none of its code eventually reaches production. The important distinction comes once you've decided to keep it. At that point, the team needs to understand exactly what still has to happen before the software can safely and sustainably become part of the product.
AI can write code incredibly quickly. But simply using AI to make your existing software delivery process faster misses much of the opportunity. In this episode of The Product Engineering Podcast, James looks at how AI changes Product Engineering when you treat it as a learning tool, rather than simply a faster way to turn requirements into production code. AI is excellent at implementing a well-defined plan, but human judgement is still needed to decide whether that plan represents the right thing to build. We cover: Why using AI to accelerate a legacy delivery model isn't enough Where human product judgement still matters Why AI changes the economics of experimentation Moving engineers and product managers further upstream into product decisions Using AI to explore several possible solutions rather than committing immediately to one Creating prototypes and thin product slices that may never reach production Why engineers can use AI to test ideas directly with users How AI can compress the feedback loop between an idea and evidence Why killing ideas quickly can be more valuable than shipping them quickly Why discarded AI-generated code isn't necessarily waste A useful diagnostic: how much of the code you're generating actually reaches production? Before AI, code was expensive enough that teams had strong incentives to decide what to build before implementation began. AI lowers that cost dramatically. That means teams can afford to build something partly to learn whether it should exist at all. The goal isn't simply to produce more code. It's to explore more possibilities, gather evidence faster and make better product decisions.