Trust Rules to validate your data, AgileData Engineering Pattern #3 - Episode #69
Agile Data N’ InfoShow Notes
Join Shane Gibson and Nigel Vining as they describe and discuss the AgileData Engineering Pattern of Trust Rules to validate data.
The Trust Rules pattern provides automated data validation to ensure all incoming data is fit for purpose and trustworthy. It bakes in essential checks such as unique business keys and business effective dates, which run automatically upon data load or table refresh. Users can also define custom validation rules for specific columns. Results are collected, persisted, and surfaced via applications or alerts, with the system optimising validation for cost and speed through smart partitioning and clustered columns.
An AgileData Engineering Pattern is a repeatable, proven approach for solving a common data engineering challenge in a simple, consistent, and scalable way, designed to reduce rework, speed up delivery, and embed quality by default.
If you want a copy of the pattern template head over to: https://agiledata.substack.com/i/167964917/pattern-name Discover more AgileData Engineering Patterns over at https://agiledata.substack.com/s/agiledata-engineering-patterns
If you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.
Subscribe: Apple Podcast | Spotify | Google Podcast | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser | Deezer | Podcast Addict | Buy the Green Book now!
Simply Magical Data Ways of Working
More from Agile Data N’ Info
In this episode Shane Gibson sits down with data modeling veteran’s John Giles and Remco Broekmans The conversation explores the premise of democratising data modeling through AI, centered around a paper published by John regarding where AI can and cannot add real value. The group highlights the critical importance of conceptual and context data modeling, stressing that business terms and definitions must be established upfront before looking at physical source tables. They strongly critique the anti-pattern of placing humans “at the end of the line” merely to rubber-stamp automated outputs, arguing instead for an “AI-in-the-loop” approach where technology serves strictly as an assistant under human direction. The discussion also dives deep into complex real-world scenarios where automated tools fail—specifically when physical data or source systems do not yet exist. John shares extreme edge cases, such as establishing a brand-new government department from scratch based on a vague Act of Parliament, unifying over 70 distinct healthcare organisations into a single model, and modeling evolving visions like the Australian wool industry. Remco breaks down his multi-agent ELM framework, explaining how introducing deliberate manual friction points between LLM tasks keeps human modelers in control and why model-storming workshops, human trust, and organisational culture are far more vital than push-button automation. Whether you are curious about conceptual modeling techniques, how to effectively integrate “AI in the loop” while avoiding push-button modeling pitfalls, or strategies for aligning business definitions across complex organisations, this episode is packed with valuable insights for data professionals.
Raw and unedited Top 3 things:
Raw and unedited Top 3 things:
Raw and unedited Top 3 things:
Raw and unedited