From Supervising AI to Building Systems for It
Agents and Engineers | Agentic AI, Software & Agentic EngineeringShow Notes
Dan and Eleanor open by discussing how fast software engineering has changed. In the last six months, Eleanor's practice flipped from treating AI as a messy assistant that needs close supervision to building systems that put the agents on the path to success. She now writes essentially no code herself, arguing that the models have become good enough that her involvement mostly makes the results worse.
This journey starts from babysitting agents locally to delegating to async, cloud-based agents like GitHub Copilot, Cursor, Devin, OpenHands, or Factory. Eleanor warns that the home-grown terminal "loops" everyone is building right now are great for learning but too brittle to scale.
Next up, what does an agent engineering system actually need? Eleanor recommends starting with a sandboxed, execution environment (usually containers), careful configuration over how the agent reaches the outside world (MCP servers and selective network access), a way to see across multiple repositories, and layered rules via AGENTS.md and skills.
Eleanor makes the case that async delegation is a forcing function for better specifications. Deterministic feedback like static analysis and test suites are the single biggest factor in work quality because "you can't control AI with AI." She has moved to fully test-driven development and notes that current-generation models no longer find unintended workarounds to tests (e.g., deleting them) the way Claude 4 and early GPT-5 once did.
Dan and Eleanor turn to adoption and skills, including how to get better at using AI with deliberate practice. Eleanor explains why she moved using Python, which she was most familiar with from use over her career, to statically typed languages like TypeScript and Go for agent work, why supply chain risk at her healthcare company has her questioning every dependency, and why she dislikes the term "junior developer." Curiosity and systems thinking, not tenure, are what matter now.
The episode closes on verification and scale. Eleanor distrusts any output she can't verify, doesn't miss hand-writing code, and argues that inventing new ways to verify, including more formal methods, is the real bottleneck now that models are cheap and strong. On team size, she pushes back on the "small teams" consensus, pointing to the success of large open-source communities. Eleanor remarks that software development has become a sub-branch of systems engineering, and anyone not practicing this now will be shocked in a matter of months.
Full episode notes
Click here to view the episode transcript.
Chapters
(00:00) - Introduction
(00:58) - The flip: from supervising AI to getting out of its way
(03:14) - Cloud-based agents vs. rolling your own
(06:08) - The primitives every agent system needs
(07:43) - Why async delegation beats local babysitting
(11:02) - Writing specs: Codex, Repo Prompt, and markdown
(12:21) - Guardrails: AGENTS.md, skills, and deterministic checks
(14:19) - Going fully test-driven
(17:19) - How engineers really adopt (and hide) AI
(19:38) - Getting better through deliberate practice
(21:12) - From experiment to reusable skill to library
(24:26) - Choosing a language: Python, TypeScript, Go
(26:56) - Supply chain risk and distributing specs
(29:06) - Beyond 'junior': curiosity over tenure
(31:03) - Systems thinking as the durable skill
(38:02) - Where Eleanor still doesn't trust AI
(39:40) - Not missing the keyboard
(42:43) - Keeping up with a fast-moving field
(44:50) - What teaching reveals
(48:27) - Verification as the real bottleneck
(50:41) - Team size and open source at scale
(55:48) - Closing: take agents seriously
Links from the show
--------------------
GitHub Copilot coding agent
Devin
OpenHands
Factory
Codex
Repo Prompt
AGENTS.md
Model Context Protocol (MCP)
Anthropic 'when AI builds itself'
Lovable
Vercel
Formal verification
UML
Jimini Health
Guests
-------
Eleanor Berger, Member of the Technical Staff, Jimini Health
More from Agents and Engineers | Agentic AI, Software & Agentic Engineering
Allen Hutchison is the co-founder and CTO of Vycari, building AI agents for everyday tasks with Chris Perry. He spent 21 years at Google, most recently as a Distinguished Engineer at Google DeepMind, helped found Gemini CLI, and built the Antigravity SDK. His earlier leadership work included Google's mapping products, Core ML, and the AI Developer organization. He writes Letters from Silicon Valley. Dan and Allen discuss why consumer agents need familiar interfaces. Vycari embeds agents in mini apps, including a calendar booking tool that works with meeting preferences and buffer time. Allen wants users to think about the assistant's usefulness rather than the technology behind it. Vycari's two founders have landed well over a thousand PRs in roughly a month. Planning documents guide implementation, nightly jobs keep those documents aligned with code, and review follow-ups become issues that agents can implement. Independent reviews and reflection on user trajectories help identify problems, while Allen retains the final merge decision. Their engineering choices favor quick feedback and manageable costs. Vycari is separating microservices into their own repositories and runs Python and Postgres on VPSs. Allen explains how inexpensive models, stronger models writing their procedures, and deterministic integrations support the goal of a $5 monthly subscription. Full episode notes Transcript Chapters
Ben Morss is a Developer Advocate at Mistral. He previously spent 9.5 years at Google as a Product Manager for advanced APIs in Chrome and a Developer Advocate for the web. He earned a bachelor's degree in Computer Science and Music from Harvard University and a doctorate in Music Composition from the University of California at Davis. He continues to make music and write about music theory.
Cory O'Daniel is CEO and co-founder of Massdriver, an internal developer platform and platform orchestrator for governed self-service infrastructure using Terraform, OpenTofu, and Helm. He has spent more than 20 years building teams and startups and working with cloud infrastructure. He created Bonny, an Elixir-based Kubernetes operator framework, and is a co-founder of OpenTofu. Dan and Cory discuss what infrastructure agents need to act safely. Massdriver models environments and resource relationships so agents can query context directly, while separate agent identities and attribute-based permissions constrain their access. Cory describes a staging workflow that tests infrastructure changes through real deployments before he promotes a release to production. Cory explains how a tested, consistent codebase supports his team's Claude Code workflow, why enforceable tooling matters more than long instruction files, and why human review still matters. He argues that engineers should build quality controls and deployment paths for domain experts who can now create software themselves, with collaboration and operational ownership still needing attention. Full episode notes Transcript Chapters
Stephanie Jarmak is an AI Engineer at Omni working on multi-agent orchestration and code intelligence. She is a research affiliate for NASA's Science Explorer (SciX) and a maintainer of Gas City, an open-source orchestration builder for multi-agent coding workflows.
Eric Ma is a Senior Principal Data Scientist at Moderna, where he leads the Data Science and Artificial Intelligence (Research) team. Previously, he conducted biomedical data science research at the Novartis Institutes for Biomedical Research and earned his doctorate in Biological Engineering from MIT. Eric is also an open-source software developer known for leading pyjanitor and nxviz and contributing to NetworkX and PyMC.