Privacy-aware Data Pipelines with Skyflow’s Piper Keyes
Partially Redacted: Data, AI, Security, and PrivacyShow Notes
A data analytics pipeline is important to modern businesses because it allows them to extract valuable insights from the large amounts of data they generate and collect on a daily basis. This leads to better decision making, improved efficiency, and increased ROI. However, despite your best efforts, sensitive customer data tends to find its way into our analytics pipelines, ending up in our data warehouses and metrics dashboards. Replicating customer PII to your downstream services greatly increases your compliance scope and makes maintaining data privacy and security significantly more challenging. In this episode, Engineering Lead at Skyflow Piper Keyes joins the show to discuss what goes into building a privacy-aware data pipeline, what tools and technologies should you be using, and how Skyflow addresses this problem. Topics: What is a data analytics pipeline? What does it mean to build a privacy-aware data pipeline? Can you give some examples of use cases where privacy-aware data pipelines are particularly important? What’s it mean to de-identify data and how does that work? What are some common techniques used to preserve privacy in data pipelines? How does analytics work for de-identified data? How do you balance the need for data privacy with the need for actually being able to use the data? What’s it take to build a privacy-aware pipeline from scratch? What are some of the biggest challenges in building privacy-aware data pipelines? How does something like this work with Skyflow? Let’s say I have customer’s transactional data from Visa, how could I ingest that data into my data warehouse but avoid having to build PCI compliance infrastructure? Walk me through how that works. Could you build a machine learning model based on the de-identified data? Once I have the data in my warehouse, let’s say I needed to inform a clinical trial participant about an issue but I also want to maintain their privacy, how could I perform an operation like that? What other use cases does this product enable? Resources: Running Secure Workflows with Sensitive Customer Data Maximize Privacy while Preserving Utility for Data Analytics
More from Partially Redacted: Data, AI, Security, and Privacy
In this episode, Sean sat down with Jack Godau to dive deep into the world of pseudoanonymization. They started by discussing Jack's career trajectory working with highly sensitive data and how that experience shapes his engineering mindset. Jack shared how pseudoanonymization differs from anonymization, explaining its value for maintaining data utility while complying with stringent regulations like GDPR. Jack also walked us through the challenges and key components of building a pseudoanonymization engine, including the complexities of handling re-identification risks, ensuring scalability, and optimizing performance for large datasets. He shared insights on the trade-offs between data protection and usability, and whether building these systems in-house is worth the investment for startups. Finally, they explored where the field is heading, especially as data privacy concerns continue to grow.
In this episode, Sean sits down with Ben Burkert, Co-founder and CTO of Anchor, to dive into the world of certificate management and internal TLS. We explore how certificates and TLS function, the inherent difficulties in managing internal TLS certificates, and why nearly every engineer has a horror story related to it. Ben also shares insights into how Anchor is addressing these challenges and making internal TLS certificate management simpler and more reliable. Key Topics:Understanding Certificates and TLS:Basics of how certificates and TLS work.The role of TLS in securing internal communications. The Challenges of Internal TLS Certificate Management:Why managing internal TLS certificates is so difficult.Common pitfalls and challenges engineers face. Engineer Horror Stories:Real-world examples of certificate management gone wrong.The impact of these failures on teams and organizations. How Anchor is Fixing the Problem:Anchor’s approach to simplifying internal TLS certificate management.Key features and benefits of Anchor’s solution. If you've ever struggled with internal TLS certificates or are looking for a way to avoid the pain altogether, Ben’s expertise provides a clear path to overcoming the challenges of certificate management with a modern, reliable approach. Resources:https://anchor.dev/https://lcl.host/
In this episode, we sit down with Ori Rafael, CEO and Co-founder of Upsolver, to explore the rise of the lakehouse architecture and its significance in modern data management. Ori breaks down the origins of the lakehouse and how it leverages S3 to provide scalable and cost-effective storage. We discuss the critical role of open table formats like Apache Iceberg in unifying data lakes and warehouses, and how ETL processes differ between these environments. Ori also shares his vision for the future, highlighting how Upsolver is positioned to empower organizations as they navigate the rapidly evolving data landscape.
In this episode, Sean Falconer is joined by Aubrey King, solutions architect and community evangelist at F5, to discuss the top 10 security issues for LLM applications. They explore critical threats such as prompt injections, insecure output handling, and training data poisoning, among others. Aubrey provides insights into why these issues arise, the attacks being observed, and the methods used to mitigate these risks. This episode is essential listening for anyone interested in the security of large language models and their applications.
In this episode, host Sean Falconer sits down with Eric Flaningam, a researcher at Felicis Ventures, to explore the fascinating world of data warehouses. They dive into the history, evolution, and future trends of data warehousing, shedding light on its importance. Key topics discussed include an overview of the article "A Primer on Data Warehouses," and the definition and key characteristics of data warehouses. They also cover the historical evolution and major milestones in data warehousing, the shift from batch processing to real-time data, and the convergence of data warehouses and SQL. Eric and Sean discuss the impact of unstructured and complex data, advancements in technology and their effect on data warehouses, and the technical architecture and components of a typical data warehouse. They share real-world benefits and use cases of data warehouses, common challenges in implementing and maintaining data warehouses, and future trends and the influence of AI and machine learning on data warehouses. For further reading, check out Eric Flaningam’s article, A Primer on Data Warehouses: https://www.generativevalue.com/p/a-primer-on-data-warehouses