Measuring AI code assistants and agents with the AI Measurement Framework

As AI agents become more capable, context quality is increasingly the limiting factor in their performance. In this episode of Engineering Enablement, DX researcher Brian Houck by his CAFE(S) is joined by co-authors Max Kanat-Alexander (Capital One), Eirini Kalliamvakou (DX), and Margaret-Anne Storey (University of Victoria) to discuss CAFE(S), a new framework for improving context quality. The panelists introduce the five dimensions of CAFE(S): Clarity, Actionability, Fidelity, Efficiency, and Security. They walk through common context failures, how poor context accrues technical and cognitive debt, what it costs in developer productivity and operating spend, how AI is changing the skills engineers need, and why context review could become as important as code review.

Meta researcher Moritz Beller joins Brian Houck to discuss how Meta measures developer productivity. Moritz explains diff authoring time and how Meta uses it to evaluate tools and guide engineering decisions. They explore how AI is changing engineers’ work and revealing gaps in traditional metrics. They also revisit Moritz’s “Mind the Gap” study and discuss the promise and risks of AI agents for testing.

In this episode of Engineering Enablement, Robert Lucero, Chief Architect at Okta, joins host Brian Houck to discuss how agentic AI is changing identity and access management. They explore how organizations should define agent identities, why authentication and authorization remain foundational, and how sandboxing, fine-grained permissions, and just-in-time access can enable agents to operate autonomously without creating unacceptable risk. Robert shares what Okta has learned from driving AI adoption among security-minded engineers, what makes a repository ready for AI agents, and why strong testing, CI, and review processes matter even more as AI generates more code. They also discuss AI’s role in software validation, the need for human judgment, and whether AI will ultimately give the advantage to security teams or attackers.

Max Kanat-Alexander is an Executive Distinguished Engineer at Capital One, where he helps lead developer experience and AI-assisted software development. In this episode of Engineering Enablement, Max joins Brian Houck to discuss how AI is changing software development and why strong engineering fundamentals matter more than ever. They explore how engineering skills are evolving, why AI can amplify weaknesses in the development lifecycle, and how teams should rethink code review, quality, and testing. Max also shares how leaders can assess whether their organizations are ready for more advanced AI workflows and why developing the next generation of senior engineers remains one of the biggest open questions in software engineering.

AI adoption among software developers is approaching 100%, AI-authored code now makes up more than half of merged code, and developers report saving more time with AI every quarter. But those gains aren’t translating evenly into better outcomes. In this episode of Engineering Enablement, host Brian Houck, Distinguished Scientist at DX, sits down with Justin Reock, Deputy CTO at DX, to unpack findings from DX’s latest AI Impact Report. They explore where AI is improving engineering velocity and developer experience, where concerns are emerging around PR size, change confidence, and failure rates, and why rising AI spend has yet to produce a comparable increase in innovation. They also discuss how AI is changing the meaning of code maintainability and where developers’ AI-driven time savings may actually be going.