Orchestrating cloud data migrations at HCL Tech
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AISummary
Banu Prakash, a technical lead at HCL Tech, shares insights on leveraging Apache Airflow for orchestrating large-scale data and platform migrations to the cloud. HCL Tech, a service-based company, specializes in providing software services to major clients in sectors like healthcare and finance. Prakash's team focuses on data engineering and big data, utilizing Airflow for various tasks including data extraction, processing, and incident management.
Prakash emphasizes the flexibility and cloud-agnostic nature of Airflow, which allows for easy integration with multiple platforms and simplifies migration between cloud services. He details the orchestration of data extraction jobs from on-premises databases to the cloud, highlighting the tool's ability to manage complex workflows with minimal code changes during version upgrades. Additionally, he discusses the automated generation of Directed Acyclic Graphs (DAGs) to streamline operations for clients, ensuring they can manage their workflows independently after project completion.
The conversation also touches on Airflow's integration with ServiceNow for incident management, where an automatic resolution engine enhances efficiency by streamlining the debugging process. Looking ahead, Prakash expresses a desire for Airflow to evolve into an AI-first platform, enabling more integrated debugging capabilities directly within its user interface.
Show Notes
Large-scale cloud migrations can result in a lot to manage when it comes to orhcestration: multi-cloud targets, on-prem extraction, dynamic workloads, and strict operational controls. In this episode, Kenten Danas is joined by Bhanu Prakash, Technical Lead at HCL Tech, to talk through how his team uses Airflow to orchestrate end-to-end data and platform migrations to the cloud, generate Dags dynamically for micro-batch workloads, and automate incident response through a ServiceNow integration...
More from The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
At Lyft, a small data engineering team supports dozens of scientists and analysts who tune driver pay configurations on demand, all under SOX controls. In this episode, Ken Obata, Senior Software Engineer at Lyft, joins Kenten Danas to walk through how his team built self-service Dags with an event-driven integration between GitHub Actions and Airflow, and how they solved the data modeling and consistency problems that came with it. Key Takeaways: 00:00 Introduction. 02:49 How Airflow r...
Running Airflow with a one-person data engineering team means every decision about cost, tooling, and best practices has to earn its keep. In this episode, Kenten sits down with Aniruddha Sengupta, Staff Data Engineer at Insignis, to talk through how the UK cash-savings fintech runs a low-cost, scalable Airflow platform. The conversation covers observability, DAG-level best practices, the internal tools Insignis built to control costs, and where AI fits into the roadmap. Key Takeaways: 00...
Scaling Airflow to over 1,000 active DAGs on a team of 3-4 engineers takes structure. In this episode, Kenten Danas talks with Matt Stavinga, Data Engineer at Idelic (now part of Descartes), about how his team standardized DAG creation and logging, built a Jenkins-driven testing workflow with per-PR Airflow environments, and is opening up DAG configuration and triggering to their own customers through the Safety Suite UI. Key Takeaways: 00:00 Introduction. 01:40 Idelic's mission: collecting...
Running LLMs over sensitive data means external APIs are off the table, but open-weight models still need reproducibility, validation, and a paper trail before downstream systems can depend on their output. In this episode, Kenten is joined by Chhayank Jain, Senior ML Engineer at Visa, to talk through how Airflow orchestrates local LLM pipelines end to end: nightly inference over millions of records, silent failure handling with Pydantic and distribution checks, and using a separate DAG to ma...
Security is one of the least visible parts of Airflow for most users, but it shapes what features teams can turn on in production. In this episode, Kenten Danas talks with Anish Giri, Senior Software Engineer and active Apache Airflow contributor focused on security, about three recent contributions that touch nearly every Airflow 3 deployment: hardening connection testing, adding JWT revocation, and splitting execution tokens by scope. Key Takeaways: 00:00 Introduction. 01:16 Anish's ins...