Systems Recipes - Issue #3
Systems Recipes - Issue #3
Welcome to the third issue of Systems Recipes! This is a newsletter about Software and Systems Architecture. We aim to bring you stories that inspire us to be better at our craft.
Every two weeks, we’ll send our top 3 picks that we’re reading at the moment 🤓.
We took a small break between the previous issue and this issue due to everything going on locally and globally. We are now back for the regularly scheduled programme!
If you have read or watched anything interesting lately that you think will be a good fit for future issues or have any feedback, simply hit reply. You can also reach out on Twitter or send an email! 📧
Building a Distributed Log from Scratch
When your application emits a log line containing a timestamp and message, that log line typically gets appended somewhere, like a specific log file or a within a journal. Logs are a totally-ordered, append only data structure.
Core systems like Kafka and NATS build on the simple log abstraction to record events with a timestamp and provide messaging systems on top. These systems run across multiple machines so there is a need to partition the event data correctly and effectively.
Writing to a log might seem simple, but can get very complex with working with distributed logs. This is a fascinating 5 part series which goes into the depths of building a distributed log from scratch.
Building a Distributed Log from Scratch, Part 1: Storage Mechanics – Brave New Geek
The Log: What every software engineer should know about real-time data's unifying abstraction
After you’ve read the previous 5-part series on designing a distributed log abstraction, have a look at how logs are used at LinkedIn.
This post goes into the use-cases and gives practical examples for how logs can be applied to:
Data Integration - Logs can be used to track events and provide a basis for integration across multiple disparate systems
Real-time data processing - Logs can be used to underpin streaming systems
Distributed system design - Systems can use logs as a primary basis for reliable replication
The Log: What every software engineer should know about real-time data's unifying abstraction | LinkedIn Engineering
Let's talk about racism
Unless you’ve been living under a rock, you will have undoubtedly seen events unfold over the past weeks and months. Black Lives Matter!
This post by our colleague Yehudi really touched us and highlights the need to have these tough discussions. These events should become real to all of us so we understand and collectively become motivated to do something about them.
It’s about time we talk about racism. The entire post is extremely important for all of us to not just read but deeply understand.
Let's talk about racism.
Check out this Black Lives Matter website which has petitions you should sign and places to donate. It’s about time we effect change. This is not a one-time issue, it’s highlighting injustices that have been occurring for generations.