Show Notes
In this episode, I sit down with Ankur Goyal, founder and CEO of Braintrust, the AI evals and observability platform used by teams like Notion, Stripe, Vercel, and Zapier. This one is for the senior engineers, staff engineers, VPs of engineering, and CTOs in my audience. We get into how coding agents can take on deeply technical architecture and infrastructure work that no single human engineer could tackle before, and then we demystify evals so you can use them to make your AI products better without touching the implementation.
What you’ll learn: How Ankur uses Codex to run week-long benchmark experiments across database indexes, column store formats, and execution engines to speed up slow queries Why he argues there’s no excuse to skip rigorous benchmarking now that agents can run them tirelessly The “agent line” framework: how to decide which decisions, directions, and interactions you can hand off to an agent How I think about the practical vs. theoretical quality of AI on hard technical problems, and why human attention decays on tedious work Why evals are the modern version of a PRD, and how to encode “what good looks like” so a model can figure out the “how” How to build a scoring function live and let an agent improve your prompt inside a safe playground How Ankur turned his designer David’s taste into a repeatable eval so quality scales beyond one person Why fixing your CI is the highest-leverage way to speed up engineering velocity — Brought to you by: Guru—The AI layer of truth Persona—Trusted identity verification for any use case — In this episode, we cover: (00:00) Introduction to Ankur Goyal (03:00) Using AI agents for database optimization (06:10) Running exhaustive benchmarks with coding agents (09:03) Why staff engineers are wrong about AI limitations (11:30) The “agent line” framework for delegation (14:00) Ankur’s workflow: running 4 to 6 concurrent agents (17:16) Technical setup: foreground agents, background agents, and cloud environments (20:32) Spending time with AI tools (23:06) Demystifying evals (26:02) Live demo: Building an eval for documentation answers (30:20) The alternative to evals: vibe checks and whack-a-mole (32:09) Capturing designer taste in scoring functions (33:13) Quick recap (33:44) Managing velocity and throughput (35:40) Why CI/CD investment is critical for AI-accelerated teams (37:30) Ankur’s prompting strategy when agents fail (39:10) Closing thoughts and how to connect — Tools referenced: • Braintrust: https://www.braintrust.dev/ • Codex: https://openai.com/codex/ • GPT 5.4: https://developers.openai.com/api/docs/models/gpt-5.4 • Claude: https://claude.ai/ — Other references: • GPT 5.5 just did what no other model could: https://www.lennysnewsletter.com/p/gpt-55-just-did-what-no-other-model • Paul Graham’s Maker vs. Manager Schedule: http://www.paulgraham.com/makersschedule.html • tmux: https://github.com/tmux/tmux • Chris Tate at Vercel: https://www.linkedin.com/in/ctatedev/ — Where to find Ankur Goyal: LinkedIn: https://www.linkedin.com/in/ankrgyl/ — Where to find Claire Vo: ChatPRD: https://www.chatprd.ai/ Website: https://clairevo.com/ LinkedIn: https://www.linkedin.com/in/clairevo/ X: https://x.com/clairevo — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].
More from How I AI
Anthropic's new personal agent, Claude Duet, lives on your phone, inherits all your Claude plugins and connectors, and is pitched at small business owners and busy professionals. It's the most polished agent launch I've seen: thoughtful onboarding, task lists, first-class artifacts, and Slack-style channels. But polish isn't enough when you're competing with every other personal agent for attention. I put it to work on my inbox and Slack, my sales pipeline, a Google Docs sales deck, a launch video, and finding Warriors tickets for my kids, and I'll walk you through what it nails, where it stalls, and what Anthropic needs to fix before it can become anyone's number one agent.
Brooke Lacey has been making videos since before Facebook had video, and she built a TikTok audience of more than 290,000 by learning to cut short-form videos herself. With over 25 years in tech, she now builds her own apps, including Zosha, a free mobile app for learning AI. In this episode, Brooke walks me through the Claude Code, FFmpeg, and Whisper workflow that edits her TikToks in about 15 seconds, how she mines her comments for new scripts, and how she runs a weekly short-form operation with no editor.
I put Meta’s new Muse Wonder (and Wonder Full) frontier models through the How I AI bench, which covers nearly 20 tasks across prototyping, agentic workflows, coding, writing, and creative work. I scored everything with an LLM-as-a-judge pass and my own voice notes, then pulled together the first-look results you see here.
Alex Cornell is a design director at Meta, where he leads design on Muse, Meta’s consumer AI agent. He came to the role with a long background in software product design, including time as a founder, and has spent the past year building one of the most thoughtful agent interfaces in the consumer AI space. Muse ships with a custom avatar, a three-layer status display, a personalized ideas surface, and a generative feed. Alex has been sharing the reasoning behind each of those design decisions publicly on Threads.
Kath Korevec is a member of the Product staff at OpenAI working on Codex, and she spent over a year building and using ChatGPT Sites internally before its public launch. She’s been on the front lines of shipping Plugin Insights, MCP plugin hosting, and the connector ecosystem, which now includes around 60 integrations.