Show Notes
Coding Chats episode 76 - John talks to Laura Dietz - a computer science professor whose work focuses on whether AI evaluation metrics actually tell the truth. She's known for her critical take on "LLM as a judge" — not because she thinks it's useless, but because she wants numbers that mean something rather than numbers that just make a system look good.
The conversation tackles some uncomfortable realities for software engineers: using an LLM to write code and another to review it is a circular trap, prompt engineering shouldn't be a computer scientist's day job, and every time you reject your code AI's output, you're quietly generating the training data that shapes its successor.
Chapters 00:00 Introduction to Laura Dietz and Her Journey 03:12 Exploring LLMs as Judges 06:16 Challenges in Evaluating Search Systems 08:49 The Evolution of User Queries and Expectations 11:46 The Role of LLMs in Information Retrieval 14:44 Defining Quality in Search Results 17:27 The Complexity of User Intent 19:54 Human-AI Collaboration in Code Review 22:53 The Future of LLMs in Software Development 25:23 Balancing Human and AI Roles 28:20 Innovative Approaches to AI Evaluation 34:10 The Art of Assembling Ideas 36:39 Balancing Cost and Quality in LLMs 39:09 Evaluating LLM Performance 43:50 The Future of LLMs and Training Data 49:19 Exploring New Architectures in AI 55:16 Understanding In-Context Learning 01:00:45 The Role of AI in Creative Expression 01:06:59 Exploring Related Content
Laura's Links: https://www.cs.unh.edu/~dietz/https:// www.linkedin.com/in/laura-dietz-47036516/ John's Links: John's LinkedIn: https://www.linkedin.com/in/johncrickett/ John’s YouTube: https://www.youtube.com/@johncrickett John's Twitter: https://x.com/johncrickett John's Bluesky: https://bsky.app/profile/johncrickett.bsky.social
Check out John's software engineering related newsletters: Coding Challenges: https://codingchallenges.substack.com/ which shares real-world project ideas that you can use to level up your coding skills.
Developing Skills: https://read.developingskills.fyi/ covering everything from system design to soft skills, helping them progress their career from junior to staff+ or for those that want onto a management track.
Takeaways Using an LLM to both generate and evaluate outputs is circular — like a student grading their own homework. If your evaluation metric can go up without your system actually improving, it's not a real metric. A better human-in-the-loop isn't one that rubber-stamps AI suggestions — it's one that's guided to look in the right place. LLMs don't get bored, which makes them genuinely useful for code review — but that's not the same as making them accurate. "Faith-based engineering" — trusting AI output without validation — is a real and growing problem in software teams. Prompt engineering is a workaround, not a discipline; real engineers should be building systems, not crafting incantations. Every rejection you give your code AI is training signal — your frustration today is someone else's better tool tomorrow. The transformer attention mechanism is a weighted sum, and a sum isn't always the right operation — some problems need an AND, not an OR. AI tools are lowering the barrier to coding for people who were previously too intimidated to try, and that's worth celebrating. The same network effect that makes a platform valuable also makes monopoly in AI training data genuinely dangerous.
More from Coding Chats
Coding Chats Episode 83 - In this episode, VB from OpenAI’s developer experience team explains why he thinks plan mode is obsolete and how coding agents have shifted from step-by-step instructions to outcome-driven, long-running work. We cover how to write a good agents.md, why tests still matter when agents write the code, spec-driven development, and why cloud agents may be the future of dev tooling.
Coding Chats Episode 82 - John talks with Dave Westgarth, head of delivery at a Databricks consultancy. Dave shares his path from a psychology degree and McDonald's into tech, and they discuss why the Scrum Master role is so misunderstood, how it differs from a delivery manager, and whether either needs to be technical.They also look at how AI is changing delivery work. Faster code just moves the bottleneck upstream, John argues there's no AI-specific skill you can't learn in a day, and engineering fundamentals are what make AI useful.
Coding Chats Episode 81 - Ben Howdle talks to John Crickett about being a first time CTO in a start-up. Ben, author of The First CTO, shares what changes when you move from building technology to leading a technical team: the pressure, the trade-offs, and the realization that writing code is only part of the job.Drawing on startup experience and lessons learned the hard way, Ben discusses how to balance quality against runway, manage people, and communicate technical decisions in terms that founders and other non-technical stakeholders can understand. He also explores why resourcefulness, resilience, and comfort with ambiguity matter so much—and why CTOs often spend more time selling ideas than they expect.Whether you’re stepping into your first CTO role or leading a technical team at an early-stage company, this conversation offers candid insights into the realities of technical leadership.
Coding Chats Episode 80 - start-up advisor Alexander Berkovich shares his expertise on building successful start-ups, hiring strategies, CTO roles, and the importance of communication between technical and business teams. Discover practical tips for navigating the challenges of early-stage companies and how to align technical excellence with business goals.
Coding Chats Episode 79 - Richmond Alake, Director of AI Developer Experience at Oracle, joins John to discuss agent memory — how AI agents store, retrieve, and adapt to information. He argues that developers building memory on flat files are naively reinventing the database, and that once you factor in concurrency, security, and scalability, a proper database is inevitable. The conversation covers the full memory stack and how Oracle's AI database keeps embeddings and data together without shipping sensitive information to external providers.