Messaging and event-driven systems

A synchronous call couples two services in time: if the callee is down or slow, the caller is too. A message breaks that coupling. The producer hands work to a broker and moves on; the consumer picks it up when it can. Everything hard about this style follows from that one gap in time, and this article is about the patterns that close it safely.

The map

Read this first when short on time. Every branch is a section below.

Queue, log and topic: three shapes of a broker

Every broker is one of three shapes, and the difference is who owns the read position and whether a message survives being read. Picking the wrong shape is the most expensive mistake in this whole area, because it is the hardest to undo later.

A queue holds work. One consumer wins each message, acknowledges it, and the broker deletes it. Adding workers adds throughput, and the queue itself tracks what is outstanding. A log is an append-only file that nobody consumes away: each reader keeps its own offset, so ten independent teams can read the same events, and any of them can rewind to yesterday. A topic in the pub/sub sense fans one message out to a copy per subscriber, and the copies are then usually queues.

flowchart TB
  subgraph Q["Queue: consumed once"]
    direction LR
    QP[producer] --> QB[[queue]]
    QB --> QC1[worker 1]
    QB --> QC2[worker 2]
  end
  subgraph L["Log: retained and replayable"]
    direction LR
    LP[producer] --> LB[[log]]
    LB --> LC1["group A at offset 402"]
    LB --> LC2["group B at offset 91"]
  end
  subgraph T["Topic: a copy each"]
    direction LR
    TP[producer] --> TB[[topic]]
    TB --> TC1[subscriber 1]
    TB --> TC2[subscriber 2]
  end
  Q ~~~ L
  L ~~~ T

Broker

The middleman that stores messages between producer and consumer. It is a database with a strong opinion about reading order.

Acknowledgement

The consumer telling the broker it is done. Until then the broker considers the message in flight and will hand it out again.

Offset

A reader's position in a log. Committing an offset is how a log-based consumer records progress.

Partitions: where ordering comes from

A topic is split into partitions so that it can be written and read in parallel, and this is the single fact that shapes everything else: ordering is guaranteed inside one partition and nowhere else. A global order across a topic would mean one writer and one reader, which is the throughput of one machine.

The producer picks a partition by hashing a key. Use the entity's id as the key and every event about that entity lands in the same partition in the order it happened, which is almost always the ordering you actually need. Events about different users may interleave freely, and that is fine.

flowchart LR
  P[producer] -->|"hash(user_id) mod 3"| R{{partitioner}}
  R --> P0["partition 0"]
  R --> P1["partition 1"]
  R --> P2["partition 2"]
  P0 --> C1["consumer 1"]
  P1 --> C1
  P2 --> C2["consumer 2"]

A consumer group divides the partitions among its members: one partition to one consumer, never two. That is what makes scaling predictable and what caps it. Six partitions support at most six working consumers; a seventh sits idle. When a member joins, leaves or stalls, the group rebalances and partitions are reassigned, which briefly pauses consumption.

Delivery guarantees: what "exactly once" really means

There are only two honest delivery guarantees, and everyone chooses the second. If the consumer acknowledges before processing, a crash loses the message: at-most-once. If it acknowledges after processing, a crash before the ack means redelivery: at-least-once. There is no third option, because the ack and the work cannot be made atomic across two machines.

sequenceDiagram
  autonumber
  participant P as Producer
  participant B as Broker
  participant C as Consumer
  P->>B: publish order 42
  B-->>P: stored
  B->>C: deliver order 42
  C->>C: process and write to the database
  C--xB: ack lost, consumer dies
  B->>C: redeliver order 42 after the timeout
  Note over C: at-least-once means the same work arrives twice
  C->>C: key 42 seen before, skip

"Exactly once" is therefore not a property of delivery but of effect. You get it by making the processing idempotent: an idempotency key with an atomic check-and-insert, a conditional write on a version, or a dedupe table keyed by message id. The reliability article covers those techniques in detail.

Kafka's exactly-once is a real thing with a narrow scope. A transactional producer writes the output records and the consumer's offset commit in one atomic transaction, and readers set isolation.level=read_committed so they never see records from an aborted one. That makes consume-transform-produce chains safe inside Kafka. It says nothing about a side effect in Postgres or a charge at a payment provider; those still need an idempotency key.

Retries, dead letters and backpressure

Some messages fail because the world was briefly broken, and some fail because they will never succeed. The design question is how to keep retrying the first kind without letting the second kind stop everything.

A message that cannot be processed is redelivered: the consumer nacks it, or its visibility timeout expires and the broker hands it back. If nothing changes, that repeats forever. In a queue this burns capacity; in a log it is worse, because the consumer cannot commit past the bad offset and the whole partition stops.

flowchart TD
  M[message delivered] --> P{Processed?}
  P -- yes --> A[commit and move on]
  P -- no --> R{Retries left?}
  R -- yes --> D["retry topic with a delay"]
  D --> M
  R -- no --> Z[[dead-letter queue]]
  Z --> H[alert, inspect, replay after a fix]

Two mechanisms do the work. Delay tiers: instead of retrying in place, republish to a retry topic with a delay — 5 seconds, then 1 minute, then 10 — so the main topic keeps moving while the failing message waits. A dead-letter queue: after a retry cap, move the message aside with its error and let a human look. Retries should use exponential backoff with jitter, for the same reason as any other retry.

Backpressure in a broker is not a signal you receive; it is a number you watch. Consumer lag — how far behind the newest offset a group is — is the one metric that tells you whether consumption is keeping up. Rising lag means add consumers (up to the partition count), make processing cheaper, or accept the delay. The broker's retention is the real deadline: if lag grows past it, messages are deleted unread.

Visibility timeout

How long a queue hides a delivered message from other consumers. If no ack arrives in that window, the message reappears.

Consumer lag

The gap between the last offset produced and the last offset committed by a group. The health metric for any log consumer.

The dual write problem: outbox and change data capture

A service that saves an order and then publishes OrderPlaced is writing to two systems that cannot commit together. Crash after the database commit and the event never happens; publish first and fail to commit and you have announced an order that does not exist. This is the dual write problem, and it is the most common correctness bug in event-driven systems.

The transactional outbox solves it by making the event part of the database transaction. The service inserts the order row and an outbox row atomically; a separate relay reads unsent outbox rows, publishes them, and marks them sent. The event is now as durable as the order.

CREATE TABLE outbox (
  id          BIGSERIAL PRIMARY KEY,
  aggregate   TEXT        NOT NULL,   -- 'order'
  event_type  TEXT        NOT NULL,   -- 'OrderPlaced'
  payload     JSONB       NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  sent_at     TIMESTAMPTZ
);

sequenceDiagram
  autonumber
  participant S as Order service
  participant D as Database
  participant R as Relay
  participant K as Broker
  S->>D: begin transaction
  S->>D: insert the order row
  S->>D: insert the outbox row
  S->>D: commit
  Note over D: both rows commit, or neither does
  R->>D: poll for unsent outbox rows
  D-->>R: event 8814
  R->>K: publish event 8814
  R->>D: mark 8814 as sent

Change data capture is the same idea without the outbox table: a connector tails the database's own replication log — Postgres's write-ahead log, MySQL's binlog — and turns every committed row change into an event. Nothing in the application changes, ordering matches commit order exactly, and no polling is needed. The cost is that the events are row shapes rather than domain facts, so a table refactor leaks into consumers. Many teams use CDC on the outbox table and get both.

Sagas: a transaction across services

An order touches payment, inventory and shipping, each with its own database. A single ACID transaction across them would need two-phase commit, which holds locks across services and blocks everything if the coordinator dies. A saga replaces it with a sequence of local transactions, each paired with a compensating action that undoes it.

Compensation is semantic, not a rollback. You cannot un-charge a card; you issue a refund, which is a new, visible fact. That is the trade: the saga never blocks, but intermediate states are visible to everyone, so there is a moment where the customer has been charged for an order that is about to fail.

sequenceDiagram
  autonumber
  participant O as Orchestrator
  participant P as Payment
  participant I as Inventory
  participant H as Shipping
  O->>P: charge the card
  P-->>O: charged
  O->>I: reserve stock
  I-->>O: reserved
  O->>H: book a courier
  H-->>O: none available
  Note over O,H: step 3 failed, so undo steps 2 and 1 in reverse
  O->>I: release the stock
  O->>P: refund the charge

Choose choreography for two or three steps, orchestration beyond that. The moment somebody asks "where is order 7741 stuck", choreography has no answer and orchestration does.

Event sourcing and CQRS: storing facts, not state

Event sourcing stores what happened instead of what is. A row that says balance = 40 has forgotten how it got there; a stream of Deposited 100, Withdrew 60 has not. Current state is a fold over the events, and a snapshot every few hundred events keeps that fold cheap.

What you gain is the audit trail as the source of truth, the ability to answer questions nobody had thought of when the write was made, and a way to rebuild any derived data by replaying. What you pay is real: events are immutable, so a schema change means upcasting old ones forever, and a full replay of a large stream is measured in hours.

CQRS is the natural partner: separate the write model from the read models. Writes validate and append; each read model is a projection built by consuming the log and shaped for exactly one query. The order list, the search index and the monthly totals stop competing for one schema.

flowchart TD
  C[commands] --> W["write model, validates and appends"]
  W --> E[["event log, the source of truth"]]
  E --> P1["projection: order list"]
  E --> P2["projection: search index"]
  E --> P3["projection: monthly totals"]
  P1 --> Q["read queries, eventually consistent"]
  P2 --> Q
  P3 --> Q

Projection

A read model built by consuming events. Disposable by design: delete it and replay to rebuild.

Upcasting

Translating an old event version into the current shape when it is read, because the stored event can never be edited.

Log compaction

Keeping only the newest record per key in a log, so it can be retained forever as a snapshot of current state.

Stream processing: continuous queries over a log

Stream processing is running a query that never ends. Stateless steps — map, filter, route — are easy and need nothing from the framework. The value is in the stateful ones: counting per key, joining two streams, detecting a pattern over a few minutes. Those need somewhere to keep state and a way to recover it after a crash.

The hard part is time. Event time is when the thing happened; processing time is when the record arrived. A phone offline in a tunnel delivers its events ten minutes late, so a count by event time cannot simply close at the wall clock. A watermark is the framework's declaration that it believes no event older than time T will arrive; when the watermark passes a window's end, the window closes and emits. Anything later is dropped, or handled by an allowed lateness that re-emits an updated result.

State lives in an embedded store — usually RocksDB on local disk — and is made durable by periodic checkpoints to object storage. On failure the job restarts from the last checkpoint and rewinds its source offsets to match, which is how a stream job recovers without double-counting.

Recap