
Foundations of Analytics Engineer Role: Skills, Scope, and Modern Practices - Juan Manuel Perafan
DataTalks.ClubShow Notes
In this talk, Juan, Analytics Engineer and author of Fundamentals of Analytics Engineering share his professional journey from studying psychological research in Colombia to becoming one of the first analytics engineers in the Netherlands. We explore the evolution of the role, the shift toward engineering rigor in data modeling, and how the landscape of tools like dbt and Databricks is changing the way teams work.
You’ll learn about: The fundamental differences between traditional BI engineering and modern analytics engineering. How to bridge the gap between business stakeholders and technical data infrastructure. The technical "glue" that connects Python and SQL for robust data pipelines. The importance of automated testing (generic vs. singular tests) to prevent "silent" data failures. Strategies for modeling messy, fragmented source data into a unified "business reality." The current state of the "Lakehouse" paradigm and how it impacts storage and compute costs. Expert advice on navigating the dbt ecosystem and its emerging competitors.
Links: DE Course: https://github.com/DataTalksClub/data-engineering-zoomcamp Luma: https://luma.com/0uf7mmup
TIMECODES: 0:00 Juan’s psychological research and transition to data 4:36 Riding the wave: The early days of analytics engineering 7:56 Breaking down the gap between analysts and engineers 11:03 The art of turning business reality into clean data 16:25 Why data engineering is about safety, not just speed 20:53 Reimagining data modeling in the modern era 26:53 To split or not to split: Finding the right team roles 30:35 Python, SQL, and the technical toolkit for success 38:41 How to stop manually testing your data dashboards 46:34 Bringing software engineering rigor to data workflows 49:50 Must-read books and resources for mastering the craft 55:42 The future of dbt and the shifting tool landscape 1:00:29 Deciphering the lakehouse: Warehousing in the cloud 1:11:16 Pro-tips for starting your data engineering journey 1:14:40 The big debate: Databricks vs. Snowflake 1:18:28 Why every data professional needs a local community
This talk is designed for data analysts looking to level up their engineering skills, data engineers interested in the business-logic layer, and data leaders trying to structure their teams more effectively. It is particularly valuable for those preparing for the Data Engineering Zoomcamp or anyone looking to transition into an Analytics Engineering role.
Connect with Juan Linkedin - https://www.linkedin.com/in/jmperafan/ Website - https://juanalytics.com/
Connect with DataTalks.Club: Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub LinkedIn - https://www.linkedin.com/company/datatalks-club/ Twitter - https://twitter.com/DataTalksClub Website - https://datatalks.club/
More from DataTalks.Club
Tejas Kumar joins Alexey Grigorev to discuss building evaluation datasets, calibrating LLM judges against human preferences, and testing agents beyond their final answers.
In this talk, Dewang Sultania, ML Engineer at Netflix and co-author of Designing AI Systems, shares his technical journey from building early inference platforms at Adobe to deploying multimodal AI models at scale. We explore the complete architecture of enterprise AI platforms, focusing on evaluation datasets, tool calling orchestration, and the practical realities of shipping production agents.
In this talk, Suhas Suresha, Senior Machine Learning Engineer at Adobe and co-author of Designing AI Systems, shares his expertise in building large-scale generative AI infrastructure from deploying real-time ECG analysis models at his previous startup, QALY, to architecting enterprise LLM platforms. We explore the massive engineering effort required to take an AI application from a weekend prototype to a scalable production system, and how to build a unified AI platform that eliminates "AI sprawl."
In this talk, Sahil Walia, Senior Technical Architect at Snowflake, shares his extensive data engineering expertise from writing simple SQL queries early in his career to architecting enterprise-scale Apache Iceberg lakehouses today. We explore the evolution of the modern data platform, the mechanics of open lakehouses, and how decoupling storage from compute is transforming data architecture.
In this talk, Philip Christos, Associate at M2VC, shares his expertise in frontier AI investing from his early days as a data analyst to evaluating cutting-edge deep tech startups. We explore how AI agents are reshaping modern work, the realities of venture capital in the LLM era, and why the future of software engineering relies more on high-level systems thinking than writing syntax.You’ll learn about:- How venture capitalists formulate AI investment strategies and evaluate startup pitch decks.- The rapid adoption of AI in software engineering compared to traditional sectors like accounting.- The crucial difference between data progress and algorithmic progress in frontier models.- Why LLMs still struggle with high-level software architecture, business logic, and design "taste."- How AI is accelerating iterations and breakthroughs in classical engineering and scientific research.- What specific deep tech and AI infrastructure ideas are actually attracting VC funding today.TIMECODES:00:00 Venture Capital Career Journey05:14 VC Analyst vs Data Analyst Roles10:13 AI Investment Thesis Strategy16:00 Emotion Analytics and Voice AI Startups22:09 LLM Impact on Venture Capital28:19 Software Engineering AI Adoption34:18 Data vs Algorithmic AI Progress40:15 Classical Engineering AI Acceleration46:12 Software Architecture AI Limitations51:11 Future Software Engineering Careers57:55 Systems Thinking in DevOps1:03:00 Pitching Deep Tech StartupsThis talk is perfect for software engineers, AI founders, and technical product managers who want to understand the evolving tech job market and venture landscape. It provides essential insights for anyone looking to build VC-backed AI startups or future-proof their career as AI coding tools become ubiquitous.Connect with Philip- Linkedin - https://www.linkedin.com/in/philipchristos/- Twitter - https://x.com/phil_christos- Website - https://t.me/stacked_channelConnect with DataTalks.Club:- Join the community - https://datatalks.club/slack.html- Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/r?cid=ZjhxaWRqbnEwamhzY3A4ODA5azFlZ2hzNjBAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ- Check other upcoming events - https://lu.ma/dtc-events- GitHub: https://github.com/DataTalksClub- LinkedIn - https://www.linkedin.com/company/datatalks-club/ - Twitter - https://twitter.com/DataTalksClub - Website - https://datatalks.club/