Show Notes
Summary In this episode of the Data Engineering Podcast Lucas Thelosen and Drew Gilson from Gravity talk about their development of Orion, an autonomous data analyst that bridges the gap between data availability and business decision-making. Lucas and Drew share their backgrounds in data analytics and how their experiences have shaped their approach to leveraging AI for data analysis, emphasizing the potential of AI to democratize data insights and make sophisticated analysis accessible to companies of all sizes. They discuss the technical aspects of Orion, a multi-agent system designed to automate data analysis and provide actionable insights, highlighting the importance of integrating AI into existing workflows with accuracy and trustworthiness in mind. The conversation also explores how AI can free data analysts from routine tasks, enabling them to focus on strategic decision-making and stakeholder management, as they discuss the future of AI in data analytics and its transformative impact on businesses.
Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. Your host is Tobias Macey and today I'm interviewing Lucas Thelosen and Drew Gilson about the engineering and impact of building an autonomous data analyst Interview Introduction How did you get involved in the area of data management? Can you describe what Orion is and the story behind it?How do you envision the role of an agentic analyst in an organizational context?
There have been several attempts at building LLM-powered data analysis, many of which are essentially a text-to-SQL interface. How have the capabilities and architectural patterns grown in the past ~2 years to enable a more capable system? One of the key success factors for a data analyst is their ability to translate business questions into technical representations. How can an autonomous AI-powered system understand the complex nuance of the business to build effective analyses? Many agentic approaches to analytics require a substantial investment in data architecture, documentation, and semantic models to be effective. What are the gradations of effectiveness for autonomous analytics for companies who are at different points on their journey to technical maturity? Beyond raw capability, there is also a significant need to invest in user experience design for an agentic analyst to be useful. What are the key interaction patterns that you have found to be helpful as you have developed your system? How does the introduction of a system like Orion shift the workload for data teams? Can you describe the overall system design and technical architecture of Orion?How has that changed as you gained further experience and understanding of the problem space?
What are the most interesting, innovative, or unexpected ways that you have seen Orion used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Orion? When is Orion/agentic analytics the wrong choice? What do you have planned for the future of Orion? Contact Info LucasLinkedIn
DrewLinkedIn
Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Enginee
More from Data Engineering Podcast
Summary In this episode, I spoke with Keerti Kulkarni about the data and AI systems behind physical logistics automation at Amazon. She explained how her team supports dock-door operations across more than 100 North American facilities by turning telemetry from physical workflows into actionable operational intelligence. We discussed the end-to-end data path from CloudWatch, Firehose, and S3 into Redshift and AI-powered analysis, along with the realities of handling petabyte-scale event streams, late and duplicate data, reconciliation, cost controls, and the need to treat physical telemetry as a first-class signal rather than noisy exhaust. Kirti also shared how these systems are designed to protect worker privacy while still surfacing the right information for safety and operational decisions. We also dug into the role of LLMs and agentic workflows in helping operations teams diagnose failures quickly through conversational interfaces. Kirti described how AI-generated summaries help identify root causes such as authorization problems, hardware issues, or connectivity failures, while human feedback loops, structured outputs, and strict vocabulary controls help manage hallucination risk. Beyond troubleshooting, we explored how the platform is evolving toward proactive maintenance, training insights, per-site cost attribution, and broader modular agent architectures that could be shared across logistics teams. It’s a detailed look at what it takes to combine data engineering, real-time operations, and AI in a large-scale physical environment.
Summary In this episode Christopher Doidge talks about his Agile Ledger Architecture (ALA) approach to data warehousing and how it aims to reduce data debt while shortening the path from raw data to trustworthy business insight. Christopher explained that ALA is not a replacement for existing warehouse patterns like medallion architecture, star schemas, or other modeling approaches, but a complementary discipline focused on pushing business definitions upstream, enforcing cleaner ledger-style transformations, and producing gold-layer tables that stakeholders can actually use without relying on analysts to repeatedly rebuild the same logic. He also discussed his “15-minute litmus test” for time-to-insight, the importance of durable documentation through a data dictionary, and why undocumented business logic living in analyst scripts is often one of the biggest hidden forms of data debt. Overall, this was a thoughtful conversation about designing warehouse systems that serve not just analysts, but the broader business as well. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Today’s episode is sponsored by Parallel - where agents find answers. Most engineers today closely follow new model releases, but don’t pay attention to their agent’s most important tool: web search. Parallel develops enterprise-grade infrastructure for agents to retrieve high-quality context from the web. Their core products are a suite of APIs for retrieving high quality information from the web with Pareto-optimal quality, cost, and speed. Whether you work on voice agents that need 200 millisecond latency, chat bots that balance speed, depth, and quality, or long-horizon agents to do thorough, overnight research for you, Parallel is a single platform for all your agentic research. Get started for free at dataengineeringpodcast.com/parallel Your host is Tobias Macey and today I'm interviewing Christopher Doidge about a data warehousing approach called Agile Ledger Architecture that is designed to drive down data debt Interview Introduction How did you get involved in the area of data management? Can you describe what the agile ledger architecture is and the story behind it? There are numerous patterns and practices that have been developed for data warehousing over the past 40 years. How does the agile ledger architecture fit in that ecosystem? (e.g. is it compatible with Kimball, Inmon, Data Vault, Anchor Modeling, etc.?) What are some examples of the types of debt that accumulate in current approaches to warehouse implementation, and the impact that it has on the utility of that asset? Digging into the architecture itself, what are the core principles that it is built on? What are the technologies or practices that it is best suited to? (e.g. event streams, lakehouse, ELT workflows, etc.) For someone who has already invested a substantial amount of effort into building a warehouse, what does adoption of the agile ledger architecture look like? Once you have started that adoption, what are the technical controls that you can put in place to ensure that this architecture is maintained and doesn't regress or get sidestepped in another portion of the warehouse? As LLMs and agents grow to become the predominant consumers (and often producers) of a warehouse, what are the benefits that the agile ledger architecture provides to ensure appropriate context and grounding to produce useful insights and accurate answers? What are the most interesting, innovative, or unexpected ways that you have seen the agile ledger architecture used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on data warehouse design and implementation? When is the agile ledger architecture the wrong choice? What do you have planned for the future of this architecture? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for
Summary In this episode Soham Mazumdar, co-founder and CEO of Wisdom.ai, talks about what “context” really means in data engineering and AI systems. He explores why context has become such an overloaded term, spanning everything from semantic layers and data catalogs to tribal knowledge, query logs, dashboards, and even agent memory. Soham explained that the big shift is that context is no longer being prepared primarily for human analysts, but for LLMs and agents that can’t reliably fill in missing gaps on their own. That change raises the bar for how context is represented, validated, benchmarked, and maintained so that AI systems can produce trustworthy outcomes.
Summary In this episode Yetunde Dada discusses Otto, Astronomer’s AI agent for Airflow, and the broader challenge of making agentic tooling actually useful for data engineers. She explored why generic coding assistants often fall short in data workflows, how Otto adds the missing context around Airflow, Astro, upgrades, and troubleshooting, and why Astronomer focused first on high-leverage use cases such as DAG authoring, investigation of pipeline failures, version migrations, and legacy scheduler modernization. She also discussed the practical realities of introducing agents into engineering teams: model choice, security boundaries, vendor lock-in concerns, validation of generated code, and the need for agents to fit into existing workflows rather than forcing users into new ones. Overall, this conversation offers a detailed look at how specialized AI agents can support data engineers today, and where Astronomer is headed next with a vision for self-healing pipelines that keep humans in control while automating more of the operational burden.