Show Notes
Behind every catastrophic system failure is a room full of stressed-out people trying to fix it. Launched in 2021, the Downtime Project was a podcast run by two seasoned distributed system engineers, who dissected post-mortems published after notable outages. Today, those two engineers, Jamie Turner and Tom Kleinpeter join David Wynn on the Bug Bash podcast to talk about their collaboration and what they learned. The two Dropbox veterans look back at the war rooms of the internal incidents behind their service and others. We dive into the emotional toll of managing outages, why zero-blame cultures are mandatory for survival, and the stories that we only tell over beers years after the fact. And of course, AI worms its way into the conversation as well. It’s a look at the pride, the panic, and the people keeping real world services online, and the lessons we learn from going offline. And if you like conversations like this, you’ll want to join us at the first ever European edition of the Bug Bash conference, happening on September 30 in Copenhagen. Check it out at bugbash.antithesis.com The Downtime Project is still online, and you can find all 8 episodes on Spotify or wherever you get your podcasts. If you’re enjoying the Bug Bash podcast, you’ll probably enjoy the Downtime Project too.
More from The BugBash Podcast
In this episode we sit down with Jay Lorch and Chris Hawblitzel from Microsoft Research, along with Sr Engineer Shuxian Wang from Antithesis, all to talk about Verus. We break down how to retrofit proofs onto code you have already shipped, without starting over from scratch. Learn how modern SMT solvers check your code at compile time, why Rust ownership maps naturally to separation logic, and what happens when AI coding agents try to cheat their way through mathematical proofs.
Many elegant theoretical proofs exist for distributed systems, but local hardware and production realities rarely play by the rules. Today, we're joined by UIUC professors Ram Alagappan and Aishwarya Ganesan, along with Antithesis Sr Engineer Michael Vaughn, to explore what it actually takes to build resilient architectures. We look at how recovery protocols can actively backfire under disk faults, how novel storage interfaces let us achieve high performance without sacrificing guarantees, and why the emergence of autonomous agents is forcing us to rethink data infrastructure from the ground up.
For decades, formal methods were considered theoretical exercises locked in academic labs—too brittle, complicated, and expensive for most production software. But as vital systems and AI-driven infrastructure face ever greater challenges, just testing edge cases by hand is no longer enough. Today, we're joined by Anjana Rajan and Jonathan Ring, co-founders of Atalanta, to explore how they are bridging the software understanding gap. They break down how to take high-level mission objectives and translate them into code-level proofs without needing a PhD in mathematics. From securing autonomous nuclear reactors to creating resilient space architectures, discover how formal verification is moving from an academic ideal to an accessible engineering reality.
Can thinking like an idealistic scientist make you a better developer? In this episode, Deepak Cherian, oceanographer turned software engineer joins the show to discuss the massive data deluge facing scientists—and what’s being done to keep up. Discover how Deepak’s team is building Ice Chunk to bring database-level transactions to n-dimensional datasets, how property-based testing acts as the ultimate automated peer reviewer, and why software engineering might benefit from embracing the scientific method a little more vigorously. And if you like conversations like this, you’ll want to join us at the first ever European edition of the Bug Bash conference, happening on September 30 in Copenhagen. Check it out at bugbash.antithesis.com
Building a deterministic hypervisor is an incredibly difficult feat of engineering. Doing it as a bachelor’s thesis in just four months is even more so. In this episode, David Wynn sits down with Peter Graugaard and Nicholas Kristensen, the creators of Dhyv—an open-source deterministic hypervisor they wrote for their undergraduate thesis at the Danish Technical University. Inspired by Antithesis, these two software engineering students dove straight into CPU manuals, custom kernel patches, and the brutal realities of virtualizing time. Listen to how they survived the crunch, wrangled the chaotic spectrum of determinism, and built "AI-friendly" CLI tools to automate their massive testing pipelines. Joining him for the conversation is Alex Pshenichkin, the architect of Antithesis’ own deterministic hypervisor. And if you like conversations like this, you’ll want to join us at the first ever European edition of the Bug Bash conference, happening on September 30 in Copenhagen. Check it out at bugbash.antithesis.com