Designing WhatsApp

Messaging inverts the assumption every other system in this series makes. There is no request waiting for a response: the server must push to a device that may be asleep, on a train, or offline for three days, and it must never lose the message or show it twice. On top of that, the server is not allowed to read anything it is carrying. The interesting parts of this design are the connection tier, the ordering guarantees, and what encryption takes away.

The map

Read this first when short on time. Every branch is a section below.

What it must do, and at what size

It must deliver a message from one device to one or more others exactly once, in a consistent order within a conversation, whether the recipient is online now or in three days; report delivery and read status; support groups; and keep content unreadable by the servers carrying it. It must not, given that last requirement, offer server-side search, server-side spam filtering on content, or web access without a paired device holding keys.

Users         2 billion, ~500 million devices connected at any moment
Messages      100 billion/day -> ~1.16 million/s average, ~3 million/s peak
Message size  ~100 bytes of ciphertext for text, media stored separately
Receipts      2-3 status events per message -> the control plane is
              roughly as busy as the data plane

Connection    ~10 KB of server memory per idle socket
              500M sockets x 10 KB = 5 TB of memory across the gateway tier
              at 1M sockets per gateway -> ~500 gateway machines

Fan-out       a 256-member group turns 1 send into 256 deliveries
              and 256 delivery receipts

The shape here is unusual and worth naming. The messages are tiny, so bandwidth is not the constraint. Connections are the constraint: five hundred million mostly-idle sockets is the dominant cost, and it is a memory and file-descriptor problem rather than a throughput one. The second surprise is that receipts double or triple the message rate — every message produces a delivered event and often a read event, each of which is itself a message that must be routed and stored.

flowchart TD
  D1["sender device"] --> G1["gateway, holds the socket"]
  G1 --> CS["chat service"]
  CS --> ST[["message store, per conversation"]]
  CS --> RG[["session registry: device to gateway"]]
  CS --> BR[["delivery bus, partitioned by conversation"]]
  BR --> G2["recipient's gateway"]
  G2 --> D2["recipient device"]
  CS --> PN["push service, for offline devices"]
  PN --> D2

The connection tier

Every online device holds one long-lived connection so the server can push. That tier is deliberately the dumbest part of the system: it terminates TLS, holds the socket, and forwards frames. It keeps no business logic and no durable state, which is what lets it be scaled and restarted independently of everything else.

The session registry maps a device to the gateway currently holding its socket. It is a small record with a short TTL, refreshed by the gateway's heartbeat, so a gateway that dies has its entries expire without anyone running cleanup. Delivering to a user means looking up their devices, finding each one's gateway, and publishing to it.

Keepalives are a battery decision as much as a networking one. Every ping wakes the phone's radio, and a radio wake costs far more energy than the bytes suggest, so intervals are long — minutes, not seconds — and tuned per network, because carrier NAT tables drop idle mappings on their own schedule. The client's reconnect uses exponential backoff with jitter, which matters more here than almost anywhere: when a gateway restarts, a million devices try to reconnect at once, and without jitter they arrive as one wave.

Sending one message

The send path is short and every step in it is load-bearing.

sequenceDiagram
  autonumber
  participant S as Sender
  participant G as Gateway
  participant C as Chat service
  participant R as Recipient
  S->>G: send, client id 9f2, ciphertext
  G->>C: persist and assign a server id
  C-->>G: stored, id 41207
  G-->>S: one tick, the server has it
  C->>R: deliver via the recipient's gateway
  R-->>C: delivered receipt
  C-->>S: two ticks
  R-->>C: read receipt when opened
  C-->>S: two blue ticks

Persist before delivering. The server acknowledges only after the message is durable, so a gateway crash between receipt and storage is a retry rather than a loss. This is the ordering that makes the whole system trustworthy, and reversing it for latency would be the worst trade available here.

The client id makes retries safe. The sender generates an id before sending; if the acknowledgement is lost and the client retries, the server recognises the id and returns the original result instead of storing a second copy. Without it, every flaky connection produces duplicate messages — the most visible possible bug.

Ordering is per conversation. The server assigns a monotonic id and the delivery bus is partitioned by conversation id, so every device in a conversation sees the same order. Across conversations there is no order and none is needed. Clients sort by server id rather than arrival, so a message that arrives late still lands in the right place.

Note what the three states actually mean: the first tick is "the server stored it", the second is "the recipient's device acknowledged receipt", and the third is "the application was opened on that message". They are three different events in three different places, which is why a product that tries to collapse them into one always ends up lying about something.

The store is a queue, not an archive

Here is a decision that looks like a storage detail and is really a product and privacy decision: how long does the server keep a message?

In an end-to-end encrypted messenger, the server's copy is ciphertext it cannot read, useful only for delivery. So the store behaves as a per-device outbound queue: hold the message until every one of the recipient's devices has acknowledged it, then delete it, with a hard expiry of some days for devices that never come back. Undelivered messages accumulate per device and are drained in order on reconnect, where the client asks for everything after the last id it holds.

The two models are not interchangeable options: the encryption choice decides the storage model, which decides the product. Once the server cannot read messages, server-side search and instant history on a new device stop being features that were skipped and become features that cannot exist.

Groups

A group message is one send and N deliveries. The server expands the member list and routes a copy to every device of every member, which is the same fan-out problem as the news feed with two differences that make it much easier.

First, groups are bounded. A few hundred to about a thousand members is a deliberate product limit, and it keeps the worst-case fan-out three orders of magnitude below a social feed's. Anything larger is a different product — a broadcast channel — with a different design, because it is a one-to-many publication rather than a conversation.

Second, a group is one conversation, so it is one partition, so ordering is free: every member sees the same sequence because every message passes through the same ordered stream. That is a strong guarantee cheaply obtained, and it is why group chat feels consistent in a way feeds never do.

flowchart TD
  M["one group message"] --> C["chat service, conversation partition"]
  C --> E["expand to 256 members"]
  E --> D["resolve each member's devices, about 400 sockets"]
  D --> S[["deliver per device, store for the offline ones"]]
  S --> R["256 delivery receipts come back"]
  R --> C

End-to-end encryption, and what it takes away

The server carries ciphertext it cannot decrypt. That single property reshapes the rest of the design, so it is worth understanding the mechanism well enough to know what it costs.

Starting a session without both parties online. Each device publishes an identity key, a signed prekey and a batch of one-time prekeys to the server. A sender fetches one bundle, combines it with their own keys, and derives a shared secret unilaterally — so the first message can be sent to a device that is offline. The server stores public prekeys and never a secret; the worst it can do is hand out a bundle it has tampered with, which is what identity-key verification and key-change warnings exist to expose.

A new key per message. The session then advances a ratchet: every message derives a fresh key, and the chain moves forward with each exchange. Two properties follow, and they are the point of the whole construction. Forward secrecy: compromising a device today does not decrypt yesterday's messages, because those keys are gone. Recovery: a compromise is healed once a new exchange mixes in fresh key material. Out-of-order arrival is handled by keeping a small window of skipped message keys, which is why a message that arrives an hour late still decrypts.

Groups need a different trick. Encrypting a message separately for every member's every device would mean hundreds of encryptions per send. Instead each sender generates a sender key for the group, distributes it once to each member through the pairwise sessions, and thereafter encrypts each message once with that key. Sends become cheap; the cost is that the key must be rotated and redistributed whenever a member leaves, or a former member could still read new traffic.

Prekey bundle

Public keys a device publishes in advance so that others can start an encrypted session with it while it is offline.

Forward secrecy

Old messages stay unreadable even if a device's current keys are compromised, because the keys that encrypted them no longer exist.

Sender key

A per-sender group key distributed once through pairwise sessions, so group messages need one encryption rather than one per recipient.

Presence, typing and receipts

These three features look trivial next to encryption and are collectively a larger share of the traffic than messages.

Presence is a key with a short TTL refreshed by the device's heartbeat, so a dropped connection becomes offline without anyone sending a message. The expensive part is distribution: pushing every status change to every contact is a multiplicative fan-out of events nobody is looking at. The fix is to fan out only to watchers — people with that conversation currently open — and answer everyone else on demand when they open a chat.

Typing indicators are ephemeral, never stored, throttled to at most one event every few seconds, and expire on their own after a short timeout so a client that disappears mid-sentence does not leave a permanent "typing…" on someone's screen.

Receipts are the ones that cost real money, because each is a routed, stored, acknowledged event of its own. Batch them: a device that receives twenty messages while offline sends one receipt covering all of them, and read receipts for a conversation collapse into a single high-water mark — "read up to message 41207" — rather than one event per message. That change alone removes most of the control-plane traffic.

Push notifications

A device without a live connection is reached through the platform's push service, which is the only way to wake an application the operating system has suspended.

sequenceDiagram
  autonumber
  participant C as Chat service
  participant P as Push service
  participant D as Device
  C->>C: recipient has no live socket
  C->>P: wake device, no message content
  P->>D: notification
  D->>C: reconnect and fetch after last id
  C-->>D: the queued messages
  Note over D: the client decrypts and renders the alert

Two constraints shape this. The payload cannot carry the message, both because the server cannot read it and because platform services are third parties; so the notification is a wake-up, and the client fetches and decrypts before displaying anything. And platform delivery is best effort — throttled, batched by the operating system, and occasionally dropped — so it can never be the only delivery mechanism. The message is already durably stored; push only shortens the time until the device asks for it.

Media

Photos and video do not travel through the message path. The client encrypts the file with a fresh key, uploads the blob to object storage, and sends a message containing the pointer, the key and a small inline thumbnail.

Three consequences are worth noting. The blob store sees only ciphertext, so it can be a plain CDN-fronted object store with no special handling. Download is a normal edge-cached fetch, which is what makes media fast globally without any of it being readable in transit or at rest. And forwarding does not re-upload: the same blob is referenced by a new message with the same key, so a video forwarded a million times is stored once — which is the single largest saving in the media path, and also why forwarded-message limits exist as a policy control rather than a technical one.

What breaks first

1. Connections per gateway. The dominant cost, and it is memory and file descriptors rather than throughput. Adding gateways is easy; the thing to watch is the reconnect burst when one is drained.
2. Group fan-out. Members times devices, times receipts. Capping group size is the lever, and it is a product decision rather than an engineering one — which is why the cap exists in every messenger.
3. The control plane. Receipts, presence and typing together exceed message traffic. Batching and high-water-mark receipts are what keep it bounded.
4. Push service quotas. An external dependency with its own limits and its own throttling behaviour, and the one part of the path that cannot be scaled by spending money on your own hardware.

The judgement that runs through all of it: delivery beats latency. Persisting before acknowledging, retrying until receipted, and keeping undelivered messages for days are all trades of speed for certainty, and they are the right way round for this product in a way they would not be for a feed.

Recap