Show Notes
AI engineering tools are evolving fast. New coding assistants, debugging agents, and automation platforms emerge every month. Engineering leaders want to take advantage of these innovations while avoiding costly experiments that create more distraction than impact.
In this episode of the Engineering Enablement podcast, host Laura Tacho and Abi Noda outline a practical model for evaluating AI tools with data. They explain how to shortlist tools by use case, run trials that mirror real development work, select representative cohorts, and ensure consistent support and enablement. They also highlight why baselines and frameworks like DX’s Core 4 and the AI Measurement Framework are essential for measuring impact.
Where to find Laura Tacho: • LinkedIn: https://www.linkedin.com/in/lauratacho/ • X: https://x.com/rhein_wein • Website: https://lauratacho.com/ • Laura’s course (Measuring Engineering Performance and AI Impact): https://lauratacho.com/developer-productivity-metrics-course Where to find Abi Noda: • LinkedIn: https://www.linkedin.com/in/abinoda • Substack: https://substack.com/@abinoda
In this episode, we cover: (00:00) Intro: Running a data-driven evaluation of AI tools (02:36) Challenges in evaluating AI tools (06:11) How often to reevaluate AI tools (07:02) Incumbent tools vs challenger tools (07:40) Why organizations need disciplined evaluations before rolling out tools (09:28) How to size your tool shortlist based on developer population (12:44) Why tools must be grouped by use case and interaction mode (13:30) How to structure trials around a clear research question (16:45) Best practices for selecting trial participants (19:22) Why support and enablement are essential for success (21:10) How to choose the right duration for evaluations (22:52) How to measure impact using baselines and the AI Measurement Framework (25:28) Key considerations for an AI tool evaluation (28:52) Q&A: How reliable is self-reported time savings from AI tools? (32:22) Q&A: Why not adopt multiple tools instead of choosing just one? (33:27) Q&A: Tool performance differences and avoiding vendor lock-in
Referenced: Measuring AI code assistants and agents QCon conferences DX Core 4 engineering metrics DORA’s 2025 research on the impact of AI Unpacking METR’s findings: Does AI slow developers down? METR’s study on how AI affects developer productivity Claude Code Cursor Windsurf Do newer AI-native IDEs outperform other AI coding assistants?
More from Engineering Enablement by DX
As AI agents become more capable, context quality is increasingly the limiting factor in their performance. In this episode of Engineering Enablement, DX researcher Brian Houck by his CAFE(S) is joined by co-authors Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX), and Margaret-Anne Storey (University of Victoria) to discuss CAFE(S), a new framework for improving context quality. The panelists introduce the five dimensions of CAFE(S): Clarity, Actionability, Fidelity, Efficiency, and Security. They walk through common context failures, how poor context accrues technical and cognitive debt, what it costs in developer productivity and operating spend, how AI is changing the skills engineers need, and why context review could become as important as code review.
Meta researcher Moritz Beller joins Brian Houck to discuss how Meta measures developer productivity. Moritz explains diff authoring time and how Meta uses it to evaluate tools and guide engineering decisions. They explore how AI is changing engineers’ work and revealing gaps in traditional metrics. They also revisit Moritz’s “Mind the Gap” study and discuss the promise and risks of AI agents for testing.
In this episode of Engineering Enablement, Robert Lucero, Chief Architect at Okta, joins host Brian Houck to discuss how agentic AI is changing identity and access management. They explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access can enable agents to operate autonomously without creating unacceptable risk. Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. They also discuss AI’s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.
Max Kanat-Alexander is an Executive Distinguished Engineer at Capital One, where he helps lead developer experience and AI-assisted software development. In this episode of Engineering Enablement, Max joins Brian Houck to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever. They explore how engineering skills are evolving, why AI can amplify weaknesses in the development lifecycle, and how teams should rethink code review, quality, and testing. Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why developing the next generation of senior engineers remains one of the biggest open questions in software engineering.
AI adoption among software developers is approaching 100%, AI-authored code now makes up more than half of merged code, and developers report saving more time with AI every quarter. But those gains aren’t translating evenly into better outcomes. In this episode of Engineering Enablement, host Brian Houck, Distinguished Scientist at DX, sits down with Justin Reock, Deputy CTO at DX, to unpack findings from DX’s latest AI Impact Report. They explore where AI is improving engineering velocity and developer experience, where concerns are emerging around PR size, change confidence, and failure rates, and why rising AI spend has yet to produce a comparable increase in innovation. They also discuss how AI is changing the meaning of code maintainability and where developers’ AI-driven time savings may actually be going.