The Chief AI Officer Show

How incident.io built AI agents that draft code fixes within 3 minutes of an alert

The Chief AI Officer Show

Show Notes

Lawrence Jones, product engineer at incident.io, describes how their AI incident response system evolved from basic log summaries to agents that analyze thousands of GitHub PRs and Slack messages to draft remediation pull requests within three minutes of an alert firing. The system doesn't pursue full automation because the real value lies elsewhere: eliminating the diagnostic work that consumes the first 30-60 minutes of incident response, and filtering out the false positives that wake engineers unnecessarily at 3am. The core architectural decision treats each organization's incident history as a unique immune system rather than fitting generic playbooks. By pre-processing and indexing how a specific company has resolved incidents across dimensions like affected teams, error patterns, and system dependencies, incident.io generates ephemeral runbooks that surface the 3-4 commands that actually worked last time this type of failure occurred. This approach emerged from recognizing that cross-customer meta-models fail because incident response is fundamentally organization-specific: one company's SEV-0 is an airline bankruptcy, another's is a stolen laptop. The engineering challenge centers on building trust with deeply skeptical SRE teams who view AI as non-deterministic chaos in their deterministic infrastructure. Lawrence's team addresses this through custom Go tooling that enables backtest-driven development: they rerun thousands of historical investigations with different model configurations and prompt changes, then use precision-focused scorecards to prove improvements objectively before deploying. This workflow revealed that traditional product engineers struggle with AI's slow evaluation cycles, while the team succeeded by hiring for methodical ownership over velocity. Topics discussed: Balancing precision versus recall in agent outputs to earn trust from SRE teams who are "hardcore AI holdouts"

Pre-processing incident artifacts (PRs, Slack threads, transcripts) into queryable indexes that cross-reference team ownership, system dependencies, and historical resolution patterns

Model selection strategy: GPT-4.1 for cost-effective daily operations, Claude Sonnet for superior code analysis and agentic planning loops

Backtest infrastructure that reruns thousands of past investigations with modified prompts to objectively validate changes through scorecard comparisons

Building ephemeral runbooks by extracting which historical commands and fixes worked for similar incidents, filtered by what the organization learned NOT to do in subsequent incidents

Prioritizing alert noise reduction over autonomous remediation because the false positive problem has clearer ROI and lower risk

Why AI engineering teams fail when staffed with traditional engineers optimized for fast feedback loops rather than tolerance for non-deterministic iteration

Building entirely custom tooling in Go without vendor frameworks due to early ecosystem constraints and desire for native product integration

The evaluation problem where only engineers who invested hundreds of hours building a system can predict how prompt changes cascade through multi-step agentic workflows

More from The Chief AI Officer Show

Aug 20, 2026

98% of IT leaders say they're using AI, yet 43% aren't measuring ROI and 80% admit they're not using it to its potential. Olga Lagunova, Chief Innovation and Technology Officer at GoTo, says the industry won the adoption battle but hasn't reached the outcomes it needs. Olga breaks down GoTo's federated AI steward model, where functional leaders own roadmaps and a central enablement team provides infrastructure. She explains why SMB pragmatism shaped their practical AI principles and how 39% of employees already report AI is eroding their skills.

May 28, 2026

Eric Ries built the Lean Startup methodology, helped advise on Anthropic's long-term benefit trust structure, and is now making an argument most enterprise leaders aren't ready to hear: deploying AI agents inside a company with bad governance isn't just a compliance risk, it's an existential one. His new book, Incorruptible, makes the case that the same slow-moving forces that corrupt companies over decades will be dramatically accelerated by AI, and that the standard governance playbook most executives have been handed is the actual source of the problem. In this conversation, Eric and Ben cover the legal and structural traps that silently strip companies of mission control long before anyone notices, why Anthropic walking away from a $200M Pentagon contract turned into an unexpected competitive advantage, and what AI leaders specifically need to do before they deploy agents at scale. Eric is direct about what he thinks leaders are getting dangerously wrong right now, and he pulls no punches. Topics Discussed: Why standard "best practice" corporate documents are structurally designed to separate mission from control

May 12, 2026

Intercom launched outcome-based pricing before the market had a framework for it, before inference costs made it profitable, and before customers knew how to budget for it. Fergal Reid, Chief AI Officer, was inside that decision and he shares exactly how they modeled their way through it, what two bets he personally owned, and why they went to $1 per resolution knowing it was a loss. That pricing story is inseparable from their model strategy. Fergal walks through the production data that led them to conclude that Opus 4.5 didn't outperform Sonnet 4.0 on their RAG customer service task, what that told them about the limits of general intelligence at the application layer, and why it pushed them to build Apex, their own model trained via reinforcement learning on an open-weight base specifically for customer service. With 85% of Intercom's own support volume now fully automated, the bets held. Topics discussed: Outcome-based pricing mechanics: the $2 beta, the loss-leader move to $1, and the two assumptions Fergal had to own

Apr 9, 2026

What happens when a journalist turned Amazon product manager becomes the Chief AI Officer of one of the world's largest international broadcasters? You get someone who sees the AI threat to media not just as a distribution problem, but as a full production chain crisis that requires a fundamentally different organizational architecture. Marie Kilg, Chief AI Officer at Deutsche Welle, makes the case that legacy media's survival depends on something most AI transformation conversations ignore: data interoperability across systems that were never designed to talk to each other. With 32 languages, siloed editorial teams, and decades of layered organizational structure, Deutsche Welle's path to an AI-powered content flywheel starts at the infrastructure layer, not the model layer. Topics Discussed: Why AI threatens the full media production chain, not just distribution

Mar 26, 2026

Most security leaders treat AI as a new threat category requiring new defenses. Rohit Parchuri, SVP and Chief Information Security Officer at Yext, pushes back hard on that. His argument: if your foundational controls are solid, AI does not require you to rebuild anything. What it does is amplify whatever you already have, gaps included, which makes the real question not "what new controls do we need?" but "how well are we actually executing on what we already built?" Rohit walks host Ben Gibert through how Yext is operationalizing this at scale: threat-modeling AI as just another system with inputs, processing, and outputs; building AI security testing directly into the existing CI/CD pipeline rather than standing it up as a separate track; investing heavily in data classification and taxonomy to solve DLP before deploying any AI tool internally; and establishing an AI Excellence Committee with cross-functional representation to run a single governance funnel across every AI request in the company. He also makes the case that the CISO who earns a seat at the AI strategy table is the one who deeply understands the business value chain, not just the threat landscape. Topics discussed: Threat-modeling AI as a system instead of a threat category

See all episodes →