Show Notes
What happens when the technology we built to evaluate AI becomes too limited to keep up with AI itself? In this episode, we explore a fundamental shift in how we assess artificial intelligence. For years, we relied on large language models to judge other models—a paradigm known as LLM-as-a-Judge. But as AI systems tackle increasingly complex, multi-step tasks, this approach is breaking down. The solution? Turning judges into agents—autonomous systems that can plan, use tools, collaborate, and verify their assessments against real-world evidence. We unpack what this means for AI development pipelines, from code generation to medical diagnosis, and why the future of AI evaluation may determine the future of AI itself. Inspired by the work of Runyang You, Hongru Cai, Caiqi Zhang, Yongqi Li, Wenjie Li, and colleagues at Hong Kong Polytechnic University, Cambridge, and Huawei, this episode was created using Google's NotebookLM.Read the original paper here: https://arxiv.org/pdf/2601.05111
More from AI Odyssey
🎧 Did AI Agents Just Break Math? Ten thousand AI agents. Eighty-eight hours. OpenAI reports a proof for a forced version of the Navier–Stokes equations, followed by verification in Lean. This episode follows the shift from individual problem-solving to teams of AI agents that explore ideas, challenge arguments and submit proofs to a checker. From Anthropic’s formalization of Fermat’s Last Theorem to the concerns raised by 25 Fields medalists, it examines what happens when producing results outruns our ability to understand them. Lean checks a formal argument against precise definitions and assumptions. It does not explain why the result matters or how people can build on it. Drawing on research announcements and the mathematical community’s responses, this episode was created using Google's NotebookLM. Research and debate: https://openai.com/index/navier-stokes-solution/ and https://mathandai.org/ Watch the video explainer on YouTube: https://youtu.be/GEDuTadiqnk?si=r7Eod_SsodeqUel7
🎧 REA: Has AI Made All Software Open Source?
🎧 Personal AI Agents That Act for You: Hermes, OpenClaw, Grok Bot and Muse AI agents are moving beyond answering questions. You can give them a task, let them work across apps, and step in when needed. This episode looks at four options for individuals. Hermes Agent and OpenClaw offer control on your own devices, with setup required. Grok Bot works in a cloud computer for eligible subscribers. Meta’s Muse is built for everyday tasks: it can use a browser and connected apps, keep working after you close the app, and asks for approval before sending an email or making a purchase, according to Meta. Muse is rolling out in the US. We compare access, practical uses, and permissions. We also cover the unconfirmed OpenAI DevDay rumor without treating it as a launch. This AI Odyssey episode was created with Google’s NotebookLM from many sources, including official product information and reporting.