Show Notes
As AI becomes embedded in software development, many of the metrics that engineering organizations have relied on for years are starting to break down. In this session from DX Annual, Uber's Ty Smith and Abhishek Tibrewal share how their approach to measuring AI's impact on developer productivity has evolved over time. They walk through the different phases of their measurement journey, from adoption and engagement to measuring impact, ROI, and agentic value, explaining what they chose to measure at each stage, what worked, what failed, and how their thinking changed along the way. They also discuss the role of qualitative feedback before telemetry existed, the challenge of identifying meaningful engagement signals, why "developer years saved" failed as an ROI metric, and how AI agents forced them to rethink traditional productivity measurements. Finally, they introduce Uber's emerging framework built around feature velocity and explore the unanswered questions that remain as software development becomes increasingly agent-driven.
Where to find Abhishek Tibrewal • LinkedIn: https://www.linkedin.com/in/aabhishektibrewal Where to find Ty Smith: • LinkedIn: https://www.linkedin.com/in/tyvsmith
In this episode, we cover: (00:00) Intro (01:30) Steve Yegge’s 8 stages of AI-assisted development (03:22) Uber’s shift to a generative AI-powered company (04:20) Uber’s pre-AI productivity metrics (06:55) Important questions from stakeholders that previous metrics didn’t answer (08:25) How Uber measures AI before telemetry exists (11:11) Metrics used to measure adoption (12:49) Measuring engagement (14:30) Measuring impact (16:32) The challenge of measuring AI ROI (19:32) Rethinking adoption, engagement, and impact for agentic AI (26:01) The new north star: Feature velocity (28:41) PR classification + feature velocity: the questions it can answer (33:01) What comes next and what’s still unanswered (34:30) Lessons learned and what they'd do differently (37:11) Q&A #1: How Uber defines a feature (38:50) Q&A #2: Measuring success and AI ROI
Referenced: • Welcome to Gas Town • Dara Khosrowshahi (Uber CEO)
More from Engineering Enablement by DX
As AI agents become more capable, context quality is increasingly the limiting factor in their performance. In this episode of Engineering Enablement, DX researcher Brian Houck by his CAFE(S) is joined by co-authors Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX), and Margaret-Anne Storey (University of Victoria) to discuss CAFE(S), a new framework for improving context quality. The panelists introduce the five dimensions of CAFE(S): Clarity, Actionability, Fidelity, Efficiency, and Security. They walk through common context failures, how poor context accrues technical and cognitive debt, what it costs in developer productivity and operating spend, how AI is changing the skills engineers need, and why context review could become as important as code review.
Meta researcher Moritz Beller joins Brian Houck to discuss how Meta measures developer productivity. Moritz explains diff authoring time and how Meta uses it to evaluate tools and guide engineering decisions. They explore how AI is changing engineers’ work and revealing gaps in traditional metrics. They also revisit Moritz’s “Mind the Gap” study and discuss the promise and risks of AI agents for testing.
In this episode of Engineering Enablement, Robert Lucero, Chief Architect at Okta, joins host Brian Houck to discuss how agentic AI is changing identity and access management. They explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access can enable agents to operate autonomously without creating unacceptable risk. Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. They also discuss AI’s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.
Max Kanat-Alexander is an Executive Distinguished Engineer at Capital One, where he helps lead developer experience and AI-assisted software development. In this episode of Engineering Enablement, Max joins Brian Houck to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever. They explore how engineering skills are evolving, why AI can amplify weaknesses in the development lifecycle, and how teams should rethink code review, quality, and testing. Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why developing the next generation of senior engineers remains one of the biggest open questions in software engineering.
AI adoption among software developers is approaching 100%, AI-authored code now makes up more than half of merged code, and developers report saving more time with AI every quarter. But those gains aren’t translating evenly into better outcomes. In this episode of Engineering Enablement, host Brian Houck, Distinguished Scientist at DX, sits down with Justin Reock, Deputy CTO at DX, to unpack findings from DX’s latest AI Impact Report. They explore where AI is improving engineering velocity and developer experience, where concerns are emerging around PR size, change confidence, and failure rates, and why rising AI spend has yet to produce a comparable increase in innovation. They also discuss how AI is changing the meaning of code maintainability and where developers’ AI-driven time savings may actually be going.