Spark Isn't Going Anywhere. So They Rebuilt It in Rust. — Shehab Amin, CEO of LakeSail

What does it actually take to build AI systems that work in production? In this episode of Data Engineering Central, I sit down with Maria Vechtomova, AI Engineering Lead, co-founder of CAUCHY, Databricks MVP, and author of MLOps with Databricks. Maria has spent more than 12 years working across data, machine learning, MLOps, and AI engineering. We talk about how production ML has evolved from the early days of homegrown model platforms to today’s world of LLMs, agents, Databricks, MLflow, and AI-assisted coding. We dig into why LLM applications can be significantly harder to evaluate and monitor than traditional ML, the growing importance of tracing and observability, and why governance becomes more complicated when agents can interact with data and other systems. Maria also explains why many of the problems we’re facing with AI aren’t actually new — AI has simply made them much harder to ignore. * We also get into AI coding tools, testing AI-generated code, Databricks and Unity Catalog, the rapidly changing LLMOps ecosystem, and what engineers should focus on as AI changes the way software is built. One of my favorite ideas from the conversation: AI is a mirror. It can help you write a lot more code, a lot faster — but whether that code is actually good still depends on the engineer behind it. Maria also shares what she thinks matters most for engineers entering the field today: common sense, soft skills, and business sense. Topics include: MLOps and LLMOps, production AI, Databricks and MLflow, AI agents, observability and tracing, model and LLM evaluation, AI governance, Unity Catalog, AI-generated code, testing, and the future of AI engineering. Thanks for reading Data Engineering Central! This post is public so feel free to share it.

I sat down with Erfan Hesami, founder of Pipeline to Insights, to talk about where data engineering is headed as AI and agents become part of everyday engineering work. We dig into the changing role of the data engineer, the growing overlap between data and AI engineering, and why strong fundamentals may actually become more important as AI makes writing code cheaper and easier. We also talk about AI agents inside modern data platforms, governance and security, AI-generated code and technical debt, controlling AI costs, the importance of human judgment, and why communication and business skills are becoming increasingly valuable for engineers. * Erfan also shares the story behind Pipeline to Insights, how writing helped accelerate his own learning, and how consistency helped him grow the publication from its first post in October 2024 into a successful data and AI community. If you’re a data engineer, analytics engineer, AI engineer, or someone trying to break into data, this conversation offers a grounded look at what skills are likely to matter as AI changes how we build software and data systems. Thanks for reading Data Engineering Central! This post is public so feel free to share it.

What happens when you spend decades building a successful career in tech, finally reach Big Tech, and then suddenly get laid off? In this episode of the Data Engineering Central Podcast, I sit down with AsianDadEnergy to talk about his journey from writing BASIC on an IBM 386 and working with COBOL mainframes to consulting, becoming a chief architect in Big Tech, and eventually losing his job after seven years. But the layoff wasn’t the end of the story. Years earlier, the 2008 financial crisis convinced him and his wife to live below their means, eliminate debt, save aggressively, and invest. By the time his Big Tech layoff arrived, those investments had grown enough to cover their living expenses—and he eventually realized he didn’t actually need to find another job.

AI can now write SQL, generate Python, review pull requests, and build entire applications with a few well-crafted prompts. Yet despite all of that progress, data engineering still feels oddly resistant to the full AI revolution. Pipelines break. Data models drift. Production data needs validation. Someone still has to close the loop. In this episode of the Data Engineering Central Podcast, I sat down with Hugo Lu, founder and CEO of Orchestra, to talk about what “agentic data engineering” actually means beyond the buzzwords. Rather than another conversation about AI replacing engineers, we dug into the infrastructure that’s still missing before autonomous data platforms become reality. * Hugo shares his unlikely path into data engineering, from investment banking to helping build data systems at Juul, before eventually founding Orchestra. Thanks for reading Data Engineering Central! This post is public so feel free to share it.

When people think about starting a software company, they usually imagine raising venture capital, hiring engineers, and growing a team as quickly as possible. Michael Drogalis chose a different path. After helping build technology in the Kafka ecosystem, founding a startup that was ultimately acquired by Confluent, and leading product for stream processing, he walked away from big tech to see if one person could build a serious B2B software company. * The result became ShadowTraffic, a product that helps engineering teams generate realistic production traffic for testing, demos, and development. Data Engineering Central is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.