Load balancing
L4 vs L7, strategies, health checks, statelessness
The problem
One server can only take so much. The moment you have several, something must decide who handles each request — otherwise some servers drown while others idle, and one death takes the system down. That something is the load balancer: it sits between clients and servers, and from the outside your whole fleet looks like one machine. (It’s a reverse proxy — it also hides your internals.)
Picking a server
Round robin: take turns — fair by count, not by cost, so one heavy endpoint can still drown a server while its neighbor naps. Least connections: send new work to whoever’s least busy — the fix for variable request cost. Weighted round robin: beefier servers get proportionally more. Random: at high volume, surprisingly close to perfectly even.
L4 vs L7 — opening the envelope
An L4 balancer never opens the envelope: it routes on IP and port, blazing fast, knows nothing. An L7 balancer reads the HTTP request — path, headers, cookies — and can route by content: send /api/videos to a beefy pool and /api/users to a cheap one. One front door, specialized rooms behind it.
The key insight: statelessness
If a server stores your session in its own memory, the balancer must pin you to it (sticky sessions) — and that server’s death logs everyone out. Move state to a shared store (Redis, a database) with the cookie holding only a claim ticket, and any server can serve anyone: dumb round robin works, deaths are invisible, and scaling is "add more servers." Stateless servers are easy to load balance; state is what makes scaling hard.
Health checks and their failure modes
The balancer pings servers and quietly removes the unresponsive ones, re-adding them on recovery — that’s how server death becomes invisible. Two classic failure modes: flapping (a slow-but-alive server gets marked down, rests, looks fine, gets marked up, chokes again — fix: require several consecutive failures before acting), and the thundering herd (a revived server has zero connections, so least-connections dogpiles it — fix: slow start, ramping traffic up gradually).
Who balances the balancer?
Every request flows through the LB, making it the new single point of failure. Production runs a pair: active-passive with heartbeat failover (or active-active at scale, pushing the problem up into DNS). Asking "what happens when this box dies?" about every box — including this one — is the habit that separates senior designs.