Show Notes
James Bowler, Head of Research Partnerships at AE Studio, is joined by two guests to explore emergent communication in multi-agent AI systems: what happens when agents under pressure develop communication protocols no human designed and few humans can read. Hale Sirin leads the AI agents program at Schmidt Sciences' AI and Advanced Computing Institute and is an assistant research professor at Johns Hopkins. Elias Stengel-Eskin is an assistant professor at UT Austin and a lead PI on the program. The core concern is under-appreciated: as more agents are deployed by more actors across shared infrastructure, how those agents talk to each other matters as much as what any individual agent can do. Hale frames this through Schmidt Sciences' agents program, a pilot studying how inter-agent communication evolves, stabilizes, or diverges. The worry isn't just that agents become more capable, but that their protocols may become opaque, making human oversight impossible exactly when it's most needed. Elias adds a second motivation: these systems give linguistics a new subject of study, a chance to watch meaning and even syntax change through interaction. To study this rigorously, the team, led by Elias and language evolution expert Simon Kirby (University of Edinburgh), built a test bed around a deliberately out-of-distribution scenario: a sci-fi medical emergency involving a fictional alien patient called Veyru. Because LLMs have no training data on Veyru's anatomy, the specialist genuinely knows things the field agent does not, forcing real communication rather than single-agent collapse. Under token-budget pressure, agents consistently developed inscrutable protocols, not just abbreviation dictionaries but compositional languages where message order encodes meaning, the kind a cryptanalyst would need grounding data to decode. The key ingredient was a postmortem stage, an unconstrained channel after each round where agents could deliberate on what went wrong. No one told them to build a language there. They did it anyway. Two results stand out. First, the same pair of models produces meaningfully different languages across runs, providing the variability needed to study transmission: which protocols survive when a new agent is swapped in mid-task, like a shift change in an emergency room. Second, because these are LLM-based agents, newcomers can take an active role in language acquisition, asking clarifying questions when a symbol is too ambiguous to infer and updating their model before acting. This active learning dynamic is largely absent from earlier literature. The conversation closes on how AE Studio's research engineers, paired with academic leads, can accelerate research and build open source infrastructure for the wider community. In this episode: What emergent communication is and why it matters for AI oversight How information asymmetry between agents drives genuine communication What emergent communication reveals about how LLMs understand language How a postmortem channel pushes agents from abbreviations to compositional protocols Why compositional languages are harder to decode but easier to transmit to new agents How LLM agents actively probe for missing vocabulary The Schmidt Sciences open call, Scaling AI Safety for a Multi-Agent World Why multi-agent setups are necessary for partial goal alignment or asynchronous execution
Learn more: https://ae.studio/alignment Read the blogpost: https://www.schmidtsciences.org/glossogen/ AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ Schmidt Sciences: https://www.schmidtsciences.org/ James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Hale Sirin LinkedIn: https://www.linkedin.com/in/hale-sirin/ Elias Stengel-Eskin LinkedIn: https://www.linkedin.com/in/elias-stengel-eskin/ Explore GlossoGen: emergentcomms.ai Contact us: [email protected]
More from The AI Alignment Podcast
In this episode, James is joined by Brian Abeyta, Senior Policy Advisor for International AI Governance at the Machine Intelligence Research Institute (MIRI), to ask what decades of nuclear nonproliferation can teach us about governing frontier AI. Brian spent two decades on nonproliferation and CBRN risk at Sandia National Laboratories, the National Nuclear Security Administration, the State Department and the White House National Security Council.
In this episode, James is joined by Mithuna Yoganathan, founder of the Looking Glass Universe YouTube channel and PhD in applied math and theoretical physics from Cambridge University, to discuss how to effectively communicate AI risk to a broader public. Mithuna recently pivoted her channel from physics education to covering AI risks, and the conversation explores what that pivot looks like in practice: who the audience should be, what narratives actually shift people's views, and what it takes to attract more talent into alignment work. The two dig into the jagged nature of AI capabilities and why that makes public communication so difficult. Most people dismiss existential risk arguments because AI still can't reliably draft a good email, yet the same systems are contributing to proofs of Millennium Prize problems over a weekend. Mithuna describes two default reactions she encounters: the sci-fi believer who assumes AI danger is obvious, and the self-described serious person who dismisses the whole conversation as hype. For the second group, she finds that concrete recent incidents, including the Hugging Face hack in which thousands of AI agents coordinated to breach another company's systems in pursuit of a higher benchmark score, are far more persuasive than abstract arguments about future risk. The episode also works through the strategic tension Mithuna faces with her channel: optimizing for broad public awareness versus deep technical content aimed at recruiting people into the field. A video seen by 2,000 of the right people, she argues, can be more impactful than one seen by a million. James draws a parallel from his own path, moving from Tesla to a quantitative hedge fund to AE Studio, and they discuss how people with non-research backgrounds can contribute meaningfully to alignment: as operators, communicators, and even co-authors on research papers. Both push back against the assumption that only top-decile mathematicians have a role to play, noting that much of alignment research looks more like replication, extension, and model-probing than it does like inventing a new architecture. The conversation closes on practical advice for people considering the transition: explore part-time arrangements before quitting outright, run small side projects using modern tooling to test your fit, apply broadly and expect rejection, and reach out to people in the field with thoughtful questions. The bottleneck, James argues, is less talent than it is funders willing to deploy capital quickly enough to absorb the people who are already motivated to work on the problem. In this episode: 00:00 Intro 01:17 Mithuna's journey from physics PhD to AI risk communicator 04:33 Which AI risks matter most and why X-risk gets dismissed 08:33 AI solving hard math: the Navier-Stokes moment 10:30 Academic reactions to AI progress and attracting new talent 12:36 Narratives that work and the Hugging Face incident 19:54 Channel goals: public awareness vs. recruiting researchers 25:33 What action should audiences actually take 31:28 Comparing approaches: Dwarkesh and reaching a broader public 35:06 Lessons from growing a niche YouTube channel 40:55 Two channel strategies and who to make content for 47:30 The pipeline mismatch: too few jobs, too many applicants 58:06 Advice for individuals considering the career transition 1:06:33 Personal upsides and downsides of working on AI alignment 1:14:13 Closing Learn more: https://ae.studio/alignment AE Studio is hiring: https://www.ae.studio/join-us Subscribe to our newsletter: https://aestudio.beehiiv.com/ Mithuna's Channel - Looking Glass Universe: https://www.youtube.com/@LookingGlassUniverse James Bowler LinkedIn: https://www.linkedin.com/in/james-bowler-84b02a100/ Mithuna Yoganathan LinkedIn: https://www.linkedin.com/in/mithuna-yoganathan-061a96275/ Contact us: [email protected]
In this episode, James Bowler is joined by Melanie Plaza, Chief Technology Officer at AE Studio, to explore the commercial case for AI alignment, arguing that the techniques researchers care about most are the same ones that make production AI systems actually trustworthy and valuable.
In this episode, James Bowler, Head of Research Partnerships at AE Studio, is joined by Erick Martinez, Alignment Researcher at AE Studio, to explore why pre-training is a largely neglected frontier in alignment research, and what it actually takes to do alignment work there.
This episode is a deep dive into a post on Anthropic's research blog: https://www.anthropic.com/research/off-switch-dual-use