How Apache Kafka got faster by switching ext4 to XFS
The Backend Engineering Show with Hussein NasserShow Notes
Allegro improved their Kafka produce tail latency by over 80% when they switched from ext4 to xfs. What I enjoyed most about this article is the detailed analysis and tweaking the team made to ext4 before considering switching to xfs. This is a classic case of how a good tech blog looks like in my opinion.
0:00 Intro
0:30 Summary
2:35 How Kafka Works?
5:00 Producers Writes are Slow
7:10 Tracing Kafka Protocol
12:00 Tracing Kernel System Calls
16:00 Journaled File Systems
21:00 Improving ext4
26:00 Switching to XFS
Blog
https://blog.allegro.tech/2024/03/kafka-performance-analysis.html
More from The Backend Engineering Show with Hussein Nasser
I discuss the state of what will be happening in the coming few years and why I encourage the 3 of you who want to be elite engineers to keep their critical thinking sharp. To pause and think before prompting.
This is a conversation with Rashed Hasan, a PhD Candidate from the University of Nebraska. We chatted about databases, ORMs and how to use AI without losing yourself. Rashed pagehttps://www.mdrashedulhasan.me/homepage
I discuss loops engineering and how it will numb creative thinking and critical system design for engineers.
I like SQL but the standard should have included ways to hint instead of the abstraction of "let me do it for you"
In this episode I explore why you should take some time to examine the operation you are caching before doing so. And why caching is a cop-out