Skills, Context, and Trust: The New Agentic Coding Stack

Dan and Jonathan Bown open with the talk Jonathan gave at ODSC, "Practical Agent Ops: From POC to Prod with MLflow 3.0." MLflow 3.0 arrived last summer as the first stable release built for generative AI rather than traditional machine learning, and Jonathan's team used it to build an agent for pre-enrollment students. The centerpiece of that work was evaluation-driven development. Instead of jumping straight into a working prototype they aligning the business up front on what quality actually looks like before signing off on a model with inherently non-deterministic output.

The initial key to success was an Excel file. In it, the data science team had already assembled 150 ground truth examples, but left them untested and set aside while engineers focused on code. Jonathan's team paused the coding work and ran a simple foundation model against those examples first, landing at what amounted to a coin flip of useful versus hallucinated answers. From there they refined the examples with the business, loaded them into MLflow's evaluation datasets built from live traces, and iterated by versioning prompts and agent configurations.

Tooling came up repeatedly. MLflow's open source repo now ships a skill file that plugs into coding tools like Claude Code, which Jonathan called a game changer for keeping up with an API that changes at roughly a release a month. The Databricks AI Dev Kit, released around March, bundles skills for the Databricks SDK, CLI, data engineering, and analytics work, usable either inside Databricks' Genie Code pane or in outside tools such as Claude Code, AWS Kiro, or Google Antigravity. Jonathan said installing it produced a dramatic jump in output accuracy compared to coding assistants working from stale or incomplete context about Databricks and MLflow APIs.

Dan raised the idea that LLMs and agentic tools are becoming users of software in their own right, alongside humans, and Jonathan tied that to broader changes at WGU: more of the business, not just engineers, now writes system prompts and builds their own copilot-style agents. His own day to day has moved from core development toward AI enablement, meaning security review, best practices, and helping non-technical staff adopt evaluation-driven habits for the prompts and agents they build themselves.

Jonathan's path to WGU ran through Pentara, a biostatistics consultancy, and Zions Bancorporation, where he did quant finance work before a stint simulating financial products for WGU students. He became a founding member of WGU's MLOps team in 2023, when the university's machine learning was still traditional work like random forests and ensembles for predicting student outcomes, well before Databricks had built out MLOps tooling. Dan connected this to Hamel Husain's essay "The Revenge of the Data Scientist", and Jonathan agreed that evaluation-driven development brings the work full circle: checking evals and correctness is the generative AI analogue of checking a confusion matrix.

The pre-enrollment agent's rollout became the clearest illustration of the method. The first release, a bare foundation model with no WGU context, drew heavy negative feedback from the employees testing it, some of whom wanted to cancel the initiative. Jonathan's team treated that feedback as fuel, folding the failed questions into an evaluation dataset and iterating until they reached roughly 82 percent correctness and near-total relevance, at which point the same employees became enthusiastic supporters. He credited MLflow's architecture for building subject matter experts directly into the agent ops workflow rather than treating evaluation as a purely technical exercise.

Jonathan was candid about where his trust runs out. He does not trust a tool's first output even after a full planning session, citing a Kiro planning cycle from the day before that failed on the first try despite extensive back and forth. He is cautious about MLflow's fast release cadence outpacing its