
Inside the 29.5 Million DARPA AI Cyber Challenge: How Autonomous Agents Find & Patch Vulns
AI Security PodcastShow Notes
What does it take to build a fully autonomous AI system that can find, verify, and patch vulnerabilities in open-source software? Michael Brown, Principal Security Engineer at Trail of Bits, joins us to go behind the scenes of the 3-year DARPA AI Cyber Challenge (AICC), where his team's agent, "Buttercup," won second place. Michael, a self-proclaimed "AI skeptic," shares his surprise at how capable LLMs were at generating high-quality patches . However, he also shared the most critical lesson from the competition: "AI was actually the commodity" The real differentiator wasn't the AI model itself, but the "best of both worlds" approach, robust engineering, intelligent scaffolding, and using "AI where it's useful and conventional stuff where it's useful" . This is a great listen for any engineering or security team building AI solutions. We cover the multi-agent architecture of Buttercup, the real-world costs and the open-source future of this technology .
Questions asked: (00:00) Introduction: The DARPA AI Hacking Challenge(03:00) Who is Michael Brown? (Trail of Bits AI/ML Research)(04:00) What is the DARPA AI Cyber Challenge (AICC)?(04:45) Why did the AICC take 3 years to run?(07:00) The AICC Finals: Trail of Bits takes 2nd place(07:45) The AICC Goal: Autonomously find AND patch open source(10:45) Competition Rules: No "virtual patching"(11:40) AICC Scoring: Finding vs. Patching(14:00) The competition was fully autonomous(14:40) The 3-month sprint to build Buttercup v1(15:45) The origin of the name "Buttercup" (The Princess Bride)(17:40) The original (and scrapped) concept for Buttercup(20:15) The critical difference: Finding vs. Verifying a vulnerability(26:30) LLMs were allowed, but were they the key?(28:10) Choosing LLMs: Using OpenAI for patching, Anthropic for fuzzing(30:30) What was the biggest surprise? (An AI skeptic is blown away)(32:45) Why the latest models weren't always better(35:30) The #1 lesson: The importance of high-quality engineering(39:10) Scaffolding vs. AI: What really won the competition?(40:30) Key Insight: AI was the commodity, engineering was the differentiator(41:40) The "Best of Both Worlds" approach (AI + conventional tools)(43:20) Pro Tip: Don't ask AI to "boil the ocean"(45:00) Buttercup's multi-agent architecture (Engineer, Security, QA)(47:30) Can you use Buttercup for your enterprise? (The $100k+ cost)(48:50) Buttercup is open source and runs on a laptop(51:30) The future of Buttercup: Connecting to OSS-Fuzz(52:45) How Buttercup compares to commercial tools (RunSybil, XBOW)(53:50) How the 1st place team (Team Atlanta) won(56:20) Where to find Michael Brown & Buttercup
Resources discussed during the interview: Trail of Bits Buttercup (Open Source Project) DARPA AI Cyber Challenge (AICC) Movie: The Princess Bride
More from AI Security Podcast
While attackers using AI can now operate at "machine speed," traditional enterprise security teams are often stuck moving at "meeting speed." To bridge this gap, organizations must fundamentally rethink how they structure their security operations, leverage AI for defensive scale, and eliminate the bureaucratic bottlenecks that delay critical patching and remediation. In this episode of the AI Security Podcast, Ashish and Caleb sit down to analyze the September 2026 Anthropic Threat Report and the underlying causes of recent high-profile AI sandbox escapes. Caleb explains how security leaders can fight AI with AI, sharing how he successfully used autonomous agents to conduct a 98% accurate least-privilege architectural audit across his entire organization. Caleb also speaks about the "barbell hiring" strategy eliminating middle management to build a highly agile, cross-functional Security SWAT team composed entirely of principal-level experts and junior engineers.
As the speed of automated threats outpaces traditional patching cycles, organizations must prioritize structural containment over purely reactive detection. Cloud network architectures frequently default to permissive egress, meaning workloads can reach the public internet unless explicitly restricted. In this episode, Doug Merritt, CEO of Aviatrix and former CEO of Splunk, details how to build practical containment strategies for modern cloud and AI environments. He outlines how security teams can implement egress controls in monitor mode to validate traffic without risking system downtime. The discussion covers the transition from legacy IP-based firewalls to identity-based policies for ephemeral serverless and Kubernetes workloads. By prioritizing reachability and blast radius as core metrics, organizations can systematically reduce their attack surface and improve resilience.
When non-technical employees can spin up fully functioning apps in days and AI agents are running parallel workflows that babysit their own pull requests, the traditional SDLC is dead. In this episode, Ashish sits down with Prahathess Rengasamy, Senior Security Engineer at SpaceX AI, to discuss how the staggering compression of development time requires an entirely new approach to security engineering. Prahathess explains why legacy 10-minute PR wait times are incompatible with multi-agent workflows, and how security teams can build internal Model Context Protocol (MCP) servers to inject compliance and security guardrails directly into an agent's context window. We unpack the differences between Q&A chatbots and autonomous coding agents, the critical need to preserve foundational preventative cloud controls (like AWS SCPs and GCP Org Policies), and why holding an autonomous machine accountable for management decisions is a critical architectural flaw. Finally, Prahathess breaks down the "build vs. buy" debate, revealing how small security teams can leverage AI to rapidly deploy internal platforms and paved roads that previously took months to engineer.
How does a global financial engine moving trillions annually manage cyber risk in the age of AI? In this episode, Ashish sits down with Subra Kumaraswamy, CISO of Visa. Subra discusses Visa's journey as part of Anthropic's Glasswing Project testing the Mythos model, revealing why a majority of Mythos-discovered vulnerabilities were non-exploitable due to robust zero-trust architecture and micro-segmentation. Subra explains why Visa open-sourced VVAH (Visa Vulnerability Agentic Harness) to help security teams scale vulnerability discovery, prioritization, and remediation across any model. He also breaks down how Visa triages 98% of Level 1 SOC incidents with AI agents, why "Mean Time to Adapt" (MTTA) is replacing legacy patch timelines, and how cross-functional AI governance enables innovation while maintaining strict production controls
Is the AI "vulnpocalypse" already here? According to Casey Ellis, Founder of Bugcrowd and pioneer of Disclose.io, we aren't quite in an apocalypse yet, we're actually in a "slopdemic." The cost of discovering vulnerabilities has plummeted, flooding bug bounty and SOC triage queues with low-quality, noisy submissions. Because these queues are so overwhelmed, many highly skilled researchers are simply hoarding zero-days because reporting them has become too difficult. In this episode, Ashish sits down with Casey to unpack the major themes and mindset shifts from RSA and Black Hat 2026. Casey breaks down how AI is shrinking the OODA loop for defenders, forcing the industry to adopt a true "assume breach" mentality and reconsider deception technology to frustrate active adversaries. They also explore the risks of non-technical employees "vibe coding" corporate applications and why CISOs must get hands-on with AI tools at home if they want to understand the risks their workforce is taking.