Scalability and compute patterns

A service that works for a thousand users rarely works unchanged for a million. This article is the sequence of things you reach for as load grows: a bigger machine, then more machines, then a structure that lets more machines help, then the concurrency, queues and batching that keep each machine productive.

The map

Read this first when short on time. Every branch is a section below.

Where the limits come from

A single server runs out of one of four things: CPU, memory, network bandwidth, or disk I/O. Which one goes first decides the fix, and one small formula tells you how much concurrency you are really dealing with.

Two words come up constantly. Throughput is work per second. Latency is time per request. Most scaling choices trade one for the other, and batching in particular buys throughput with latency.

Vertical scaling: a bigger machine

Vertical scaling replaces a server with one that has more CPU, memory or faster disks. Nothing in the software changes, which is why it is always the first thing to try and often the right thing to keep doing for a database.

• What it fixes. CPU-bound and memory-bound workloads, and any component that is hard to split: a relational database, a stateful service, a build server.
• Where it stops. There is a largest instance, and the price climbs faster than the capacity. Resizing usually means a restart, and the one big box is still one box: when it fails, everything on it fails.
• The quiet strength. One large machine has no network between its parts. A well-tuned Postgres on 64 cores and a terabyte of RAM handles a load that many teams would reach for sharding to solve, with far less complexity.

Horizontal scaling: more machines

Horizontal scaling adds servers to a pool behind a load balancer so that capacity grows with the number of machines. It has no ceiling and survives failures, but only if the servers do not need to share anything.

flowchart TD
  C[Clients] --> LB[Load balancer]
  LB --> N1[node 1]
  LB --> N2[node 2]
  LB --> N3[node 3]
  LB -. added at 70% CPU .-> N4[node 4]
  N1 & N2 & N3 & N4 --> S[(shared state: database, cache, queue)]
  M[Metrics: CPU, requests, queue depth] --> AS[Autoscaler]
  AS -- add or remove nodes --> LB

Shared-nothing

The design rule that makes horizontal scaling work: each node owns its CPU, memory and disk and coordinates with the others only through the network. No shared memory, no shared file system, no "the leader keeps the list". Anything nodes must agree on lives in a separate system built for it: a database, a cache, a queue. That system then becomes the next thing to scale, which is what the storage article is about.

Autoscaling

An autoscaler measures a signal and changes the node count to keep it near a target. The details decide whether it helps or hurts:

• Pick a leading signal. CPU is the default, but it lags: by the time CPU is high, latency already is. Requests per node or queue depth react earlier.
• Respect the lag. A new node takes a minute or more to boot, join and warm up. Scale out early and in slowly, with cooldowns so the fleet does not oscillate.
• Keep a floor. A minimum count that survives one zone failing, and headroom for the burst that arrives before the scaler reacts.
• Scale the whole path. Doubling web nodes doubles database connections. If the database cannot take them, scaling out makes things worse.

Microservices: splitting the system into services

Microservices split an application into separately deployed services, each owning one area of the business and its own data. The point is organisational as much as technical: teams ship independently, and each piece scales on its own curve.

flowchart TB
  subgraph Mono["Monolith"]
    direction LR
    M1[users] --- M2[orders] --- M3[payments] --- M4[search]
    M1 & M2 & M3 & M4 --> MD[(one database)]
  end
  subgraph Micro["Microservices"]
    direction LR
    U[User service] --> UD[(users db)]
    O[Order service] --> OD[(orders db)]
    P[Payment service] --> PD[(payments db)]
    O -- REST or gRPC --> U
    O -- event: OrderPlaced --> Q{{event bus}}
    Q --> P
  end
  Mono ~~~ Micro

Drawing the boundaries

A good service boundary is a bounded context: a part of the domain with its own vocabulary and its own data, where most changes stay inside. "Orders" is a context; "the function that formats dates" is not. Each service owns its database; other services get the data through its API or through events it publishes, never by reading its tables. That rule is what keeps services independently deployable, and breaking it is how you end up with a distributed monolith: many deployables that must still all change together.

How services talk

• Synchronous (REST, gRPC): simple to reason about, but the caller waits, and a slow or failed callee becomes the caller's problem. Timeouts, retries and circuit breakers become mandatory (the reliability article covers them).
• Asynchronous (events on a broker): the producer publishes "OrderPlaced" and moves on; consumers react in their own time. Loosely coupled and resilient, at the cost of eventual consistency and harder debugging.
• Cross-service transactions do not exist. A saga replaces them: a sequence of local transactions with compensating actions if a later step fails (reserve stock, charge card, and if the charge fails, release the stock).

What it costs

Every function call that becomes a network call gains latency, partial failure and serialisation. You now need service discovery, per-service deploy pipelines, distributed tracing to find out where time went, and contracts that let services evolve without breaking each other. Below a certain team size that cost outweighs the benefit, which is why a modular monolith, one deployable with strict internal module boundaries, is the better starting point for most products.

Bounded context

A boundary inside which one model and vocabulary apply. The natural unit for a service.

Saga

A multi-step business operation across services implemented as local transactions plus compensating actions instead of one distributed transaction.

Conway's law

Systems end up shaped like the organisation that builds them. Service boundaries that ignore team boundaries fight this and usually lose.

Statelessness: any node can serve any request

A stateless server keeps nothing between requests that another server would need. Everything a request depends on arrives with it or lives in shared storage, so any node can handle any request and nodes can be added, removed or replaced at will.

flowchart TB
  subgraph Stateful["Stateful: session in memory, sticky routing"]
    direction LR
    C1[User] -- sticky cookie --> LB1[Balancer] --> A1["node 1<br/>session for this user"]
    LB1 -.-> A2["node 2<br/>knows nothing"]
  end
  subgraph Stateless["Stateless: session outside, any node"]
    direction LR
    C2[User] -- JWT or session id --> LB2[Balancer]
    LB2 --> B1[node 1]
    LB2 --> B2[node 2]
    B1 & B2 --> R[(Redis: sessions)]
    B1 & B2 --> S3[(object storage: uploads)]
  end
  Stateful ~~~ Stateless

Where the state goes

The payoff shows up everywhere else in this article. Autoscaling works because a new node is instantly useful. Rolling deploys work because killing a node loses nothing. A crash is an inconvenience, not data loss. And the load balancer can use any algorithm it likes, because sticky sessions are no longer needed.

Concurrency: keeping one machine busy

Concurrency is how a server makes progress on many requests at once. The model you pick decides how many connections a machine can hold, what happens when a downstream call is slow, and how the code looks.

flowchart TB
  subgraph Pool["Thread pool"]
    direction LR
    Q1[request queue] --> T1[thread 1: waiting on DB]
    Q1 --> T2[thread 2: waiting on HTTP]
    Q1 --> T3[thread 3: computing]
    Q1 --> TN[... 200 threads, mostly waiting]
  end
  subgraph Loop["Event loop"]
    direction LR
    E[loop] --> H1[start DB query, register callback]
    E --> H2[start HTTP call, register callback]
    E --> H3[run ready callbacks]
    OS[OS: epoll, kqueue] -- socket ready --> E
  end
  Pool ~~~ Loop

I/O-bound or CPU-bound

The first question about any workload. If requests spend their time waiting on databases, other services or disks, the machine can hold far more of them than it has cores; the limit is memory per connection, and event loops and lightweight threads shine. If requests spend their time computing, the limit is cores, no concurrency model creates more of them, and a pool of about one thread per core is right. Most web services are I/O-bound, which is why a thread pool of 200 usually holds 200 requests that are all waiting.

// Fetch many URLs concurrently without holding a thread per request.
// Each call suspends while waiting; a small pool of threads serves them all.
suspend fun fetchAll(urls: List<String>): List<Response> = coroutineScope {
    val inFlight = Semaphore(20)                       // bound concurrency: 20 at a time
    urls.map { url ->
        async(Dispatchers.IO) {
            inFlight.withPermit { client.get(url) }    // suspends, does not block
        }
    }.awaitAll()
}

Queues: decoupling producers from workers

A queue sits between the code that creates work and the code that does it. The producer appends a message and returns immediately; workers pull messages at their own pace. That one indirection absorbs bursts, isolates failures and lets each side scale on its own.

flowchart TD
  P1[API: place order] --> Q[["queue<br/>depth = 12,000 during the sale"]]
  P2[API: place order] --> Q
  Q --> W1[worker]
  Q --> W2[worker]
  Q --> W3[worker]
  W1 & W2 & W3 --> D[(database)]
  W2 -- fails 3 times --> DLQ[["dead-letter queue"]]
  Q -. depth drives .-> AS[autoscaler adds workers]

What a queue gives you

• Buffering. A sale sends 12,000 orders in a minute; the workers process 2,000 a minute; nobody gets an error, the backlog drains in six minutes.
• Decoupling. The producer does not know or care whether workers are running, slow, or being redeployed.
• Retry and isolation. A message that fails is retried; one that keeps failing goes to a dead-letter queue to be inspected without blocking the rest.
• Fan-out. With pub/sub, one event reaches many independent consumers (email, analytics, fraud check) without the producer listing them.

Delivery guarantees

Brokers hand a message to a worker and wait for an acknowledgement. If the worker crashes before acking, the message is delivered again. That is at-least-once delivery, the default everywhere, and it means every consumer must be safe to run twice (the reliability article covers idempotency). At-most-once acks before processing and accepts loss. Exactly-once in the strict sense does not exist across arbitrary systems; what Kafka offers is exactly-once within Kafka through transactions, and what everyone else means is at-least-once plus idempotent consumers.

Backpressure

A queue hides overload: the API stays fast while the backlog grows, until someone notices that orders take an hour to confirm. Backpressure is how the system says "slow down" instead. Bound the queue and reject or shed when it is full, alert on consumer lag (how far behind the workers are), scale workers on depth, and give each message a deadline after which processing it is pointless. A queue with no limit is not a buffer, it is a delayed outage.

Consumer group

A set of workers that share one subscription; each message goes to one of them. Two groups each get every message.

Visibility timeout

SQS's name for the window in which a delivered message is hidden from other workers while one processes it.

Consumer lag

Messages produced but not yet consumed. The single best health metric for a queue.

Batching: fewer, bigger operations

Batching groups many small operations into one larger one so that the fixed cost per operation, mostly the network round trip, is paid once. It is the most reliable way to raise throughput, and its price is always the same: the first item waits for the batch to fill.

sequenceDiagram
  participant A as App
  participant D as Database
  Note over A,D: one at a time: 3 round trips, ~3 ms each
  A->>D: INSERT row 1
  D-->>A: ok
  A->>D: INSERT row 2
  D-->>A: ok
  A->>D: INSERT row 3
  D-->>A: ok
  Note over A,D: batched: 1 round trip, one transaction
  A->>D: INSERT rows 1, 2, 3
  D-->>A: ok

Where it shows up

• Databases. Multi-row inserts and bulk loads; DynamoDB's BatchWriteItem (25 items) and BatchGetItem (100 items).
• Brokers. The Kafka producer holds records for linger.ms and sends them as one request; batches also compress far better than single records.
• APIs. A bulk endpoint instead of one call per item; GraphQL's DataLoader collects every "load user by id" made during one request and issues a single query, which is the standard fix for the N+1 problem.
• Telemetry. Metrics and logs are buffered and flushed every few seconds; sending each one would cost more than the work being measured.
• Machine learning serving. GPUs are efficient on batches of inputs, so inference servers wait a few milliseconds to group requests.

// Write 50,000 events in batches of 500: 100 round trips instead of 50,000.
events.chunked(500).forEach { batch ->
    jdbc.batchUpdate(
        "INSERT INTO events (id, user_id, kind, at) VALUES (?, ?, ?, ?)",
        batch.map { arrayOf(it.id, it.userId, it.kind, it.at) }
    )
}

# Kafka producer: trade up to 10 ms of latency for much higher throughput
linger.ms=10
batch.size=65536
compression.type=lz4

The trade-off

Every batch is bounded by two limits, a size and a time, and sends when either is hit. Small batches keep latency low; large batches keep throughput high. A partial failure inside a batch also needs an answer: retry the whole batch (fine if writes are idempotent), or retry only the failed items (if the API reports them). And a batch that is too large hits its own limits: request size caps, lock time in the database, memory in the worker.

Serverless: scaling to zero

Serverless functions are the end point of statelessness plus autoscaling: you upload code, the platform runs one instance per concurrent request and none when idle, and you pay per invocation. The scaling is instant and automatic; the costs are cold starts and limits.

• Fits. Spiky or low traffic, event-driven glue (a file lands in a bucket, a message arrives, a webhook fires), APIs whose load is unpredictable, anything that would otherwise be an idle server.
• Cold starts. The first request to a new instance pays for container start and runtime initialisation: tens of milliseconds for a small Node or Go function, seconds for a large JVM one. Provisioned concurrency, snapshots (Lambda SnapStart) and smaller packages reduce it.
• Limits. Maximum run time (15 minutes on Lambda), memory caps, no persistent local state, and per-request pricing that beats a server at low volume and loses at sustained high volume.

Recap