Show Notes
In this episode, we take a closer look at the Hadoop Distributed File System (HDFS), a key part of the Hadoop framework that helps store and manage huge amounts of data. We’ll explore how HDFS spreads data across many affordable servers, making it both scalable and cost-effective. You’ll learn about its main components, like the NameNode and DataNodes, and how they work together. We’ll also discuss features that keep your data safe and ensure it moves efficiently. Join us, we’ll touch on the challenges of managing large data clusters and what the future might hold for HDFS.
More from 10-Minute System Design
In this 10-minute episode, we’ll explore neural networks — the core component of many modern AI models, including LLMs like ChatGPT, Gemini, and Claude, that mimic how the human brain works. We’ll demystify the “black box” nature of the neural networks and reveal how they work. It’s a must-listen for new learners curious about AI.
In this episode, we'll take a look at Meta’s ambitious approach to scaling large language models. We'll explore the shift from handling many smaller models for recommendation engines to building colossal generative AI models, and the immense challenges that come with it. From hardware and software optimizations to managing power and dealing with inevitable hardware failures, we'll break down the critical pieces that make Meta's infrastructure tick. What does it take to run systems this large without breaking? Tune in to learn how Meta did it.
In this episode, let's explore how Netflix revamped their video processing pipeline, moving from a monolithic system to a microservices architecture. What drove such a major shift? You'll hear how their original platform, Reloaded, couldn’t keep up with Netflix’s rapid pace of innovation, and why Cosmos, their new system, is now the backbone of everything from streaming to studio operations. But what challenges did they face along the way? And is Cosmos truly the future-proof solution it promises to be? Tune in and find out.
In this episode, we'll explore the intricate system and architecture design behind Apple's iCloud. We'll break down how Apple seamlessly handles billions of users by combining Cassandra and FoundationDB to power iCloud's backbone. What prompted Apple to shift from Cassandra to FoundationDB, and how does this choice impact scalability and performance? Get a closer look at the architecture that makes iCloud tick, and discover how it enables such a smooth user experience. The surprising reason behind Apple’s tech pivot might just change the way you think about designing cloud storage systems.
In this episode, we explore the system behind Uber's driver-matching functionality, capable of handling an incredible one million requests per second. We break down the key technologies that make it work, from H3, the hexagonal grid system for location indexing, to Ringpop, which scales services across servers. You'll hear about how GPS data is transformed into road segments, and how databases like Cassandra and Redis power this high-demand platform. Whether you're curious about large-scale systems or just fascinated by Uber's tech, this episode simplifies complex engineering into something anyone can understand.