Distributed systems
CAP, quorums, split brain
The problem
More than one machine = a distributed system, and distributed systems are hard for one reason: machines must agree on what's true over a network that fails. Cables get cut, packets vanish, clocks drift. Everything in this block is machinery for agreeing anyway.
CAP — the forced choice
Two datacenters, the fiber between them cut (a network partition — not hypothetical, Tuesday). A write landed on one side; a read arrives on the other. That side has exactly two options: answer anyway (possibly stale — it chose Availability) or refuse (correct silence — it chose Consistency). There is no option three; checking with the other side is what the partition forbids. Since partitions WILL happen, "pick 2 of 3" collapses to: when the split comes, is this operation CP or AP?
Choose per operation, not per system
Profile photos, feeds, like counts → AP: stale beats down, and copies reconcile later (eventual consistency). Bank balances, the last concert ticket, username uniqueness → CP: a spinner beats double-spent money. The heuristic that sounds senior because it is: default to eventual consistency; pay for strong consistency only where correctness demands it. (Between them lives causal consistency: cause always visible before effect — nobody sees a reply before its comment.)
Split brain and the majority quorum
CP systems need a leader for writes — and when the leader dies, the followers must elect one over an unreliable network. The nightmare: a partition lets both halves elect a primary — two truths diverging by the second. That's split brain. The defense is arithmetic: leadership requires a majority quorum, and a majority can only exist on one side of any split. Five nodes split 3–2: the 3-side elects, the 2-side mathematically can't — it goes quiet, enacting the CP choice by math alone. This is why clusters run odd node counts, and why the names to drop are Raft and Paxos, shipped inside ZooKeeper and etcd — used, never re-derived.
The other gremlins
Clock skew: machines' clocks disagree, so "latest write by timestamp" is meaningless across nodes — logical clocks (Lamport, vector) order events by happened-before instead of wall time. Cascading failure: one slow node times out its callers, who time out theirs — timeouts, backoff with jitter, circuit breakers. Byzantine failure: a node that lies; almost every system you'll design assumes crash-stop instead (machines work or die, never deceive) — know the assumption you're making.