Messaging
Queues, pub/sub, Kafka, idempotency
The problem
Synchronous calls chain fates: if service A calls B and B is slow or down, A is stuck. And a user shouldn't wait through thumbnail generation, moderation, notifications, and analytics to see "Uploaded ✓". Messaging decouples: producers drop messages on a broker; consumers process at their own pace. The user's request returns in milliseconds; the heavy work happens behind the curtain.
Three species
Work queue (SQS, RabbitMQ): a to-do list — each message grabbed by exactly one worker, processed, deleted. Ten workers = ten-way parallelism. Pub/sub (SNS): a broadcast — one event to a topic, every subscriber gets a copy. Event stream (Kafka): pub/sub where messages don't vanish — an ordered, durable log that consumers read at their own cursor and can rewind and replay. That one property, replayability, is most of why Kafka eats the streaming world.
The two-layer composition
The canonical architecture uses both layers: pub/sub between services — the upload service publishes one "photo uploaded" event; thumbnailer, moderation, notifications, and analytics each subscribe. Next quarter's search-indexing team subscribes too, and nobody touches the upload code — that's loose coupling. Then a work queue inside each service distributes its copy of the event across its own workers. Fan-out at the top, to-do lists at the bottom. (End users aren't broker subscribers — followers get pinged by the notification service over push/WebSockets.)
Delivery guarantees and idempotency
Three strengths: at-most-once (fire and forget — may lose messages), at-least-once (retry until acknowledged — may duplicate; what almost everyone runs), exactly-once (largely a myth — in practice, at-least-once plus consumers that tolerate repeats). Tolerating repeats is idempotency: same message twice, same result as once. Two builds: an idempotency key (unique ID per logical operation; consumer skips seen IDs) or natural idempotency (deterministic output — a thumbnail written to photo-8841-small.jpg overwrites itself harmlessly).
When things break
Backpressure: producers outpace consumers and the queue grows unbounded — autoscale consumers on queue depth, and alert on depth (a growing queue is the earliest smoke detector). Poison message: one malformed job crashes every worker that touches it — retry limit, then exile to a dead-letter queue for humans. Ordering: parallel workers process out of order — when order matters per entity, partition by key (Kafka): one user's events flow through one partition in order while different users parallelize.