Episode 8 - Agent Orchestration (Track A - System)

Ninety-two percent accuracy. A green arrow on the slide. And users quietly abandoning the product underneath it. An insurer's claims-intake agent reported beautiful model metrics for two months while adjusters drifted back to reading files themselves — the errors hid in the complex claims, nobody could tell which fields were wrong, and "good" was never defined in the adjuster's language. The root cause wasn't measurement. It was definition, inherited from the vendor dashboard that shipped in the box. This episode lays out the fix: four numbers to write before the first prompt. Task success defined in the user's units and verified in the system of record. The escalation band, which fails in both directions. Cost per successful outcome, not per call. And the harm line, the wrong action that must essentially never happen. Plus who signs the bar (the business owner, before the build), why the four numbers work as tripwires that police each other, the escalation band as the leading indicator, and the floor you write down while you're still clear-headed. With a worked example (the four numbers for the bank payments agent from The Not Yet List) and the win: a retailer's returns agent whose operations director wrote the definition of good into the kickoff document, above the architecture diagram. This is Track B, Episode 3 — the Product track: what to build, and why. New here? Start with Episode 1: Beyond the Demo. Monday move: finish the sentence "this agent is good when…" in the user's units, write the four numbers and the floor, and get the business owner to sign it. If nobody will sign, you don't have a metrics problem. You have an ownership problem.

Fourteen service lines. One prior authorization agent. A go-live that lasted six weeks. The episode opens in a hospital — a health system that gave one agent all of prior auth on day one, every service line, every exception — but the story isn't really about healthcare. It's about the shape of every failed agent deployment: an agent asked to be trusted with everything, everywhere, on day one, that guessed when it should have asked for help. Then the mirror image, in financial services: a regional bank's payments agent that shipped doing exactly one job — investigate failed payments, show the evidence, move no money — survived two weeks of analysts trying to catch it being wrong, caught a small business's payroll file before it bounced, and earned each expansion after that. The argument: version one of an agent is not an MVP. An MVP tests demand; version-one scope is a trust decision with a blast radius. Why smart teams overscope anyway (demo pressure, stakeholder arithmetic, the vendor platform story), what actually counts as "one job," and the discipline that fixes it: the not-yet list — everything the agent deliberately won't do on day one, written down, read out loud, with an evidence bar for how items earn their way off. Scope expands by grant, never by drift. This is Track B, Episode 2 — the Product track: what to build, and why. New here? Start with Episode 1: Beyond the Demo. Monday move: write your agent's not-yet list — five things it will not do on day one — and read it out loud to your sponsor before the next build review. If the list is empty, you don't have a scope. You have a hope.

A refund agent with a human approval on every action sounds safe. Until reviewers are clicking approve half-asleep, and a $4,800 decimal slip sails through in under two seconds. In this episode: the handoff layer. Where the agent meets the user, and who's actually in charge at the moment something irreversible happens. When should the agent ask vs. act? (One agent guessed wrong and ordered noodles instead of milk tea.) How to gate without alert fatigue: stop where money moves or something can't be undone. Never on read-only work. Why transparency is what protects user autonomy. Plus a beat for PMs: the approval dialog, the escalation message, the activity log. That's product surface you can own and prototype yourself, starting now. Track A, Episode 2 — the System track: how you build it. New here? Start with Episode 1: Beyond the Demo, or catch the Two Tracks interlude for the full map. Monday move: take your highest-stakes agent action and list every approval it triggers. Cross out each gate that doesn't map to something irreversible or expensive. For every gate that survives, rewrite the user-facing sentence until a reviewer can make a real decision from it.

🎁 Free download: The Agent Brief template from this episode — get it here: The Brief Six weeks of building. Demo day applause. Then the pilot died quietly — because nobody had written down what the agent was supposed to do. Alex and Jordan open Track B — the Product track — with a support-triage agent that auto-resolved a billing ticket it got wrong. The post-mortem asked the obvious question; nobody had an answer on paper. The discipline: two pages before anything gets built. Page one, the announcement, written as if the agent already shipped. Page two, the FAQ of dreaded questions — what won't it do, what happens when it's wrong, who approves what, when do we kill it. A brief that gets rejected is a success: you killed the wrong thing on paper in an afternoon instead of in production six weeks later.

Every agentic system is really nine layers — whether you designed them or not. In this episode, Alex and Jordan walk all nine through a real war story: an invoice agent that worked in the pilot, then paid twelve thousand dollars to the wrong vendor in week three of production. The agent did everything it was told. The problem was everything nobody told it. The nine layers: the handoff (experience), orchestration, decision intelligence, integration, context and memory, the data foundation, the model layer, security and trust, and observability — with governance cutting across all of them. The thesis: the layers aren't the hard part. The boundaries between them are. Two skipped decisions — nobody defined what the agent may conclude, nobody owned the data it read — cost twelve thousand dollars. Plus, a beat for the product managers: you don't need to draw all nine layers yourself, but you need to know which layer your requirement lives in. This is Track A, Episode 1 — the System track: how you build it. New here? Start with Episode 1: Beyond the Demo, or catch the Two Tracks interlude for the full map of where the show goes from here. Monday move: take your most important pilot and draw the nine layers on a single page. For each layer, write the one decision you actually made. Where it's blank, that's your architecture debt.