Optimising for Trouble – Game Theory and AI Safety | with Jobst Heitzig
Game Changer - the game theory podcastShow Notes
What happens when an AI system faithfully follows a flawed goal? In this episode, we explore how even well-designed algorithms can produce dangerous outcomes — from amplifying hate speech to mismanaging infrastructure — simply by optimising a reward function which, like all reward functions, fails to encode all that matters. We discuss the hidden risks of reinforcement learning, why over-optimisation can backfire, and how game theory helps us rethink what it means for AI to act "rationally" in complex, real-world environments.
Jobst Heitzig is a mathematician at the Potsdam Institute for Climate Impact Research and an expert in AI safety and decision design.
More from Game Changer - the game theory podcast
In this episode, we speak with Dorsa Amir about randomness. Dorsa explains why acting randomly can be evolutionarily and strategically relevant for humans. She addresses the common perception that humans are "bad at being random" and, drawing on her research, sheds light on this question: our ability to be and act randomly may depend on the strategic nature and context of the situation in which randomness is required. We then dive into anthropology, where Dorsa shares insights into how humankind has developed randomization devices across the world, offering many examples and anecdotes about their development. Dorsa Amir is an Assistant Professor of Psychology and Neuroscience at Duke University. Her research explores how culture and cognition shape one another, drawing on developmental psychology, anthropology, and behavioural economics.
In this episode, we speak with Elif Kubilay about a study she conducted with Sule Alan on how adolescents can be empowered to change the social climate in schools. We discuss student-teachers, peer influence, bullying, self-persuasion, and what happens when students are trusted with real responsibility. The paper is available here. Elif Kubilay is an Associate Professor in Economics at the University of Essex. Her research focuses on development economics, labour economics, education, and behavioural economics, with a particular interest in how school environments shape children's cognitive and non-cognitive skills.
In this episode, we explore the challenges of multi-party negotiations and why reaching agreement becomes exponentially harder as more stakeholders enter the room. Our guest Lawrence Susskind explains the principles of the Mutual Gains Approach and we discuss applications ranging from local disputes over renewable energy projects to the intricacies of building data centres and setting up other large-scale infrastructure investments. Along the way, we examine why organisations often underestimate stakeholder concerns and what they can do to build durable agreements. Lawrence Susskind is the Ford Professor of Urban and Environmental Planning at MIT and co-founder of the Program on Negotiation at Harvard Law School.
In this episode, we explore what it really means to make a "good" prediction. Starting with something as ordinary as a weather forecast, we discuss why forecasting poses deep economic and game theoretic questions. Together with our guest Rakesh Vohra, we unpack the concept of calibration, examine how forecasts can be evaluated, and revisit the surprising insight that even someone without meteorological expertise can produce calibrated weather predictions.
In this episode, we explore how digital platforms transformed markets by dramatically reducing search costs and information asymmetries. Together with Sarah Auster, we dive into the "digital paradox" of endless choice, the unintended consequences of frictionless search, and what this means for the future design of platforms like Netflix and Amazon. Along the way, we uncover the broader economic story behind the evolution of the internet economy and the game-theoretic forces shaping digital markets today. Sarah Auster is a Professor of Economics at University of Bonn. You can find the papers relevant to this episode here: Simultaneous Search and Adverse Selection and Pandora's Box Reopened: Robust Search and Choice Overload.