
From React to Dagster: Pete Hunt on Data, Infra, and AI-Ready Platforms
Platform Engineering PodcastShow Notes
Is Postgres actually a better message queue than Kafka? This provocative question is just one of many insights Pete Hunt shares in this conversation about data orchestration, platform engineering, and the evolution of infrastructure. Pete Hunt, CEO of Dagster Labs and former React co-founder at Facebook, brings his unique perspective from working at tech giants like Instagram and Twitter to discuss how different platform team approaches impact product development. Having witnessed both Facebook's clear delineation between product and infrastructure teams and Twitter's DevOps-style ownership model, Pete offers valuable comparisons of these contrasting philosophies. The conversation explores: How Dagster provides a higher-level abstraction for data teams, making it easier to track and debug data assets rather than just managing workflows The challenges of modern data platforms and why many organizations struggle with complex, distributed systems that could be simplified A practical approach to migrating from Airflow to Dagster with their "Airlift" toolkit that allows for incremental, low-risk transitions How AI development is fueling demand for better data orchestration as companies build applications that rely on properly managed data pipelines
Pete also shares his thoughtful approach to balancing technical debt and product development with a "quarter on, quarter off" cadence that allows teams to both ship features and clean up the inevitable corners that get cut under deadline pressure. For platform engineers, data teams, and technical leaders navigating the intersection of infrastructure and AI, this episode provides practical insights on creating abstractions that deliver real operational value without unnecessary complexity. Guest: Pete Hunt, CEO of Dagster Pete is the CEO of Dagster Labs, where he first joined as Head of Engineering in early 2022 and transitioned into the CEO role later that same year. Before Dagster, Pete co-founded Smyte, an anti-abuse startup acquired by Twitter, where he continued as a senior staff engineer. Earlier in his career, Pete was one of the first engineers to work on Instagram after its acquisition by Facebook in 2012. There, he led development on Instagram’s web and analytics teams and became a co-founder of the React.js project, helping transform an internal experiment into one of the most widely used front-end frameworks in the world. He was also part of the early community around GraphQL and has remained deeply engaged in open source and developer tooling. Pete brings a pragmatic, hands-on perspective to modern data infrastructure. Having been both a founder and an engineer, he focuses on reducing complexity and fatigue in data teams by building tools that actually work together. At Dagster, he remains close to the code and actively involved in technical decisions, combining leadership with deep technical fluency. Pete Hunt, X Dagster Dagster Pipes Dagster Airlift Links to interesting things from this episode: React “Postgres: a Better Message Queue than Kafka?” Airflow Kubeflow CAPES Fargate
More from Platform Engineering Podcast
We're hitting pause for the summer on new episodes of the Platform Engineering Podcast, but don’t worry, we’ve got some great episodes in the feed to keep you entertained in the meantime. If you’ve got suggestions for guests or topics, please reach out — I’m always eager to annoy people into joining the conversation! Can’t wait to reconnect with you all soon!
Network calls fail in ways function calls never do - and once a monolith becomes microservices, reliability problems show up fast: retries amplify load, latency spikes cascade, and “what talks to what?” becomes hard to answer. William Morgan, co-creator of Linkerd and the person who coined “service mesh,” breaks down what service meshes actually solve for platform teams running Kubernetes at scale. The conversation focuses on practical outcomes: improving reliability between services, getting uniform observability without rewriting every app, and handling gaps Kubernetes doesn’t cover well - like gRPC/HTTP2 load balancing and cross-environment communication. Key topics Why reliability is the first “microservices tax” (timeouts, retries, backoff, cascading failure) What Kubernetes does not solve at the networking layer—and where a service mesh fits gRPC/HTTP2 load balancing problems and why L4 balancing can fall short Service-to-service visibility: understanding traffic flows and performance without per-app instrumentation Cost and resilience tradeoffs with multi-AZ Kubernetes on AWS (and how zonal-aware balancing can help) Whether developers should ever need to interact with service mesh configuration Where zero trust and policy controls belong: platform guardrails vs application ownership
When code gets cheaper to produce, feedback becomes the limiting factor - CI, reviews, and the handoffs between tools can quietly slow everything down. Rob Zuber breaks down what platform engineers are seeing as teams adopt AI-assisted development: more branch builds, new failure modes, and growing pressure to shorten the loop between “change made” and “change validated.” He focuses on how CI can evolve from a human-first dashboard into a system that agents can interact with directly through APIs, CLIs, and MCP-style interfaces - so fixes can happen faster and with less waiting on manual triage. Along the way, Rob and Cory dig into practical questions engineering leaders are wrestling with: how PR review becomes the next major bottleneck, what “agent experience” means in a delivery pipeline, why speed isn’t only about faster compute (it’s also about doing less unnecessary work), and how teams can share learnings so “agentic velocity” doesn’t only benefit a few power users. If you’re building or running the systems that ship software, this is a clear look at where CI fits in an AI-accelerated workflow, and what needs to change to keep delivery safe, fast, and sustainable.
A lot of infrastructure and automation fails for ordinary reasons: rate limits, flaky networks, partial permissions, long-running jobs, and retries that vanish when the process restarts. Durable execution is a way to design systems that keep going anyway - without rebuilding a maze of queues, cron jobs, and manual cleanup. Cornelia Davis breaks down how durable execution works in practice: writing “normal” code while the runtime provides durable retries, state management, and the ability to pause work, wait for a human or external change (like a quota increase), and resume right where things left off. The conversation connects these ideas to platform engineering realities - Terraform workflows, long provisioning times, and “orphan” resources - and explains how Temporal workflows and activities help teams model failure handling as a first-class part of the system. You’ll also hear why this approach is showing up in AI engineering: long-running agent workflows, frequent rate limiting, and the need to avoid re-running expensive LLM calls when something breaks near the end.
What happens when a non-deterministic AI system is asked to touch production telemetry or generate changes for an SRE pipeline? The cost of being “close enough” can be lost data, downtime, or a security incident. Cribl’s Nikhil Mungel joins Cory to break down what it takes to build AI that sysadmins can actually trust. The conversation digs into harness engineering and the practical guardrails that turn probabilistic models into repeatable, verifiable outcomes. They cover why breaking work into small chunks matters, how validation and testing become the real leverage point for AI-native development, and what “code factories” mean for review, CI, and platform reliability when teams can generate a thousand PRs an hour. Platform engineers will also hear a pragmatic take on the future of the job. The focus shifts away from typing code and toward building systems for verification, simulation, and safe deployment at scale, plus clearer ways to decide what needs human scrutiny and what can ship automatically.