Designing an LLM serving system

Serving a large language model breaks the assumptions every other system in this series rests on. A request does not have a response time, it has a stream. The model is too large for one chip. The cost of a request depends on how long its answer turns out to be, which nobody knows when it arrives. And the resource that limits throughput is not compute or network but the memory holding a per-request cache that grows with every token generated. This article is about the handful of mechanisms that make that tractable.

The map

Read this first when short on time. Every branch is a section below.

What it must do, and at what size

It must accept a prompt, stream back generated tokens, serve many users concurrently on expensive hardware, and hold a latency contract. It must not, in this article, cover training, fine-tuning, or the product layer around the model — the next article builds that on top of this one.

Model      70B parameters at 2 bytes each   = 140 GB of weights
           one accelerator has ~80 GB       -> at least 2 chips, usually 4

KV cache   2 (keys and values) x layers x kv_heads x head_dim x 2 bytes
           = 2 x 80 x 8 x 128 x 2           ~= 320 KB per token
           an 8,000-token conversation      ~= 2.5 GB for ONE request

Memory     4 chips x 80 GB = 320 GB
           minus 140 GB of weights and overhead -> ~150 GB for cache
           -> roughly 60 concurrent 8k-token requests, not thousands

Latency    time to first token: ~200-500 ms is a good target
           time per output token: ~20-50 ms, must beat reading speed
Unit cost  tokens per second per accelerator is the number that matters

Read that memory line again, because it is the whole article in one number. Concurrency is capped in the tens, not the thousands, and the cap comes from a cache whose size grows with every token generated. Every technique below is either a way to make that cache smaller, to use it more efficiently, or to get more useful work out of each pass over the weights.

Prefill and decode: one request, two workloads

Generation happens in two phases with entirely different performance characteristics, and failing to separate them is the most common mistake in this area.

Prefill processes the whole prompt at once. Every token can be computed in parallel, so it is a large matrix multiplication: the accelerator is genuinely busy, arithmetic units saturated, and the time scales with the prompt length. It is compute bound.

Decode produces one token at a time, each depending on the last, so there is no parallelism across tokens within a request. Generating a single token requires reading every parameter in the model — 140 gigabytes of weights — to do a small amount of arithmetic. It is memory bandwidth bound, and it is where nearly all the wall-clock time goes.

The consequence is the central insight of LLM serving. If one request is decoded alone, the accelerator spends its time waiting on memory and its arithmetic units idle — an enormously expensive machine running at a few percent of capability. But the weights that were just read can serve any number of sequences in the same step. Batching thirty-two sequences reads the weights once and produces thirty-two tokens instead of one, at almost the same wall-clock cost.

Batching is therefore not an optimisation here; it is the design. Throughput scales nearly linearly with batch size until memory runs out, and per-token latency barely moves. Everything else in this article exists to make larger batches possible.

The KV cache

Attention needs every previous token's keys and values to compute the next token. Recomputing them each step would make generation quadratic, so they are cached — and that cache is the resource that limits everything.

Its size is fixed by the model's shape and grows linearly with sequence length: for each token, keys and values, for every layer, for every attention head. For a large model that is around 320 kilobytes per token, so a single conversation with eight thousand tokens of context holds about two and a half gigabytes — for one user. A hundred concurrent users with long contexts would need more memory than the weights themselves.

Three properties follow. Memory caps concurrency, not compute: the question "how many users can this machine serve" is answered by dividing free memory by cache-per-request. The cache grows during the request, so a sequence admitted safely can exhaust memory later, which makes admission a prediction rather than a check. And architectural choices target it directly — grouped-query attention, which lets several query heads share one set of key and value heads, exists largely to cut this cache by a factor of several, at a small quality cost.

Prefill

Processing the prompt, computing its keys and values in one parallel pass. Compute bound, scales with prompt length.

Decode

Generating one token at a time, reading all weights per step. Memory bandwidth bound, scales with output length.

KV cache

Stored keys and values for every token so far, per sequence. The resource that determines how many requests fit.

Continuous batching

Given that batching is everything, the question is how to form batches when requests arrive continuously and finish at unpredictable times.

Static batching — collect N requests, run them together until all are done — fails badly for two reasons. The batch cannot start until it is full, so early arrivals wait. And it cannot finish until the longest generation finishes, so a request producing a thousand tokens holds thirty-one slots idle while it completes alone. With output lengths varying by an order of magnitude, most of the batch's capacity is wasted.

flowchart TD
  S["scheduler runs before every token step"] --> F{Any sequence finished?}
  F -- yes --> R["release its cache blocks"]
  R --> A
  F -- no --> A{Room for a waiting request?}
  A -- yes --> AD["admit it, run its prefill"]
  A -- no --> D["run one decode step for the batch"]
  AD --> D
  D --> S

Continuous batching — also called in-flight batching — makes the batch a living set rather than a fixed group. Before each token step the scheduler removes sequences that have finished, admits waiting requests into the freed space, and runs one decode step for whatever is currently resident. A request that finishes early releases its memory immediately; a request that arrives mid-flight joins within one step rather than waiting for a batch boundary.

The effect on utilisation is large: reported throughput improvements of several times over static batching are typical, without changing the model or the hardware at all. It is the single highest-leverage technique in this article.

Paged attention

Continuous batching exposes a memory management problem. How much cache should be reserved for a request whose output length is unknown?

The naive answer — reserve enough for the maximum possible output — wastes most of the memory, because the maximum is rarely reached: a request that could generate two thousand tokens and stops after eighty has held twenty-five times what it needed. Contiguous allocation adds external fragmentation on top, leaving gaps too small to use. Together these can waste the majority of available cache memory, which translates directly into a smaller batch and lower throughput.

The fix is borrowed from operating systems: allocate the cache in fixed-size blocks with a page table per sequence. Logical positions map to physical blocks through an indirection, blocks are allocated as the sequence grows, and they need not be contiguous. Waste falls to a fraction of one block per sequence — a few percent instead of a majority.

The indirection buys a second, equally valuable property: blocks can be shared. Many requests begin with the same long system prompt, and with a page table they can point at the same physical blocks for that prefix instead of each storing a copy. When sequences diverge, the shared block is copied on write. For a product where every request carries the same few thousand tokens of instructions, this is an enormous saving in both memory and prefill compute.

Splitting the model across chips

A hundred and forty gigabytes of weights does not fit on an eighty-gigabyte accelerator, so the model must be divided. There are three ways, with different communication costs.

flowchart TB
  subgraph T["Tensor parallel"]
    direction LR
    T1["chip 1: a slice of every layer"] --> T2["collective every layer"]
    T2 --> T3["chip 2: the other slice"]
  end
  subgraph P["Pipeline parallel"]
    direction LR
    P1["node A: layers 1-40"] --> P2["activations only"]
    P2 --> P3["node B: layers 41-80"]
  end
  T ~~~ P

The practical arrangement is a hierarchy that matches the hardware: tensor parallelism inside a node, where chips are connected by a high-bandwidth interconnect that can absorb a collective per layer, and pipeline parallelism across nodes, where the link is ordinary networking and communication must be rare. Getting this backwards — tensor parallelism across a network — produces a system that spends most of its time in communication.

Pipeline parallelism introduces bubbles: while stage one works on a batch, stages two and three have nothing to do. Splitting the batch into microbatches so that several are in flight at different stages fills most of the gap, which is the same technique as instruction pipelining in a processor and has the same residual inefficiency at the start and end.

Scheduling: when prefill meets decode

Now the two phases collide. A newly admitted request needs a prefill pass, which is a large compute burst, while resident sequences want a steady rhythm of decode steps. Running the prefill blocks the decodes, and every user in the batch sees their stream pause.

This is visible to users as a stutter: tokens arriving smoothly, then a gap, then smoothly again. The longer the incoming prompt, the longer the gap, so one user pasting a large document degrades everyone else's experience simultaneously.

Chunked prefill is the standard answer: split a long prompt into pieces and interleave those pieces with decode steps, so the batch keeps generating while the prompt is absorbed over several steps. Time to first token for the new request rises slightly; the stutter for everyone else disappears. It is a direct trade of one user's latency against the smoothness of all the others, and stating which side you are protecting is part of the design.

Preemption handles the harder case. When memory runs out mid-generation — because admitted sequences grew longer than predicted — some sequence must yield. Its cache blocks are either swapped out or simply discarded and recomputed when it resumes, which is cheaper than it sounds because prefill is fast and compute-bound. Choosing the victim is a scheduling policy: most recently admitted, lowest priority, or the one whose recomputation is cheapest.

Caching and shortcuts

Three techniques buy back significant cost, in descending order of how easily they are obtained.

Prefix caching is the largest easy win. In most products every request begins with the same system prompt and often the same conversation history, so computing that prefix once and reusing its cached keys and values across requests removes both repeated prefill compute and duplicated memory. With paged attention this falls out of block sharing almost for free.

Speculative decoding attacks the memory-bandwidth bound directly. A small, fast draft model proposes several tokens ahead; the large model then verifies all of them in a single forward pass — which costs about the same as generating one token, because that pass is bandwidth bound and the extra tokens ride along. Accepted drafts are effectively free, rejected ones cost only the draft. When the draft model is well matched, two to three times the tokens per second at identical output quality is achievable, because verification guarantees the result matches what the large model would have produced.

Semantic caching — answering a new question from a stored answer to a similar one — is the most tempting and the most dangerous. It changes the output rather than accelerating it, similarity is not equivalence, and a cached answer may be stale or subtly wrong for the new phrasing. It belongs to narrow, well-understood domains rather than to a general assistant.

Autoscaling

Every scaling assumption from the rest of this series fails here, and it is worth being explicit about why.

Cold start is minutes, not seconds. A new replica must load a hundred and forty gigabytes of weights from storage into accelerator memory, which takes minutes even with fast storage and warm caches. Reactive autoscaling against a traffic spike arrives long after the spike is over.

There is no scale to zero, and capacity is bought in whole accelerators — a large, indivisible unit. Utilisation cannot be smoothed the way it can with stateless containers.

Utilisation is the wrong signal. Accelerator utilisation looks high during decode even when the machine is doing very little useful work, because it is waiting on memory. The signals that mean something are queue depth, cache occupancy — how full the KV memory is, which is the true measure of how close to capacity the replica is — and the latency targets themselves.

So the shape is: keep warm replicas sized for expected load plus headroom, scale on leading indicators with generous lead time, queue with a deadline rather than accepting unbounded work, and shed politely at the front door when full — a clear rejection with a retry hint is far better than admitting a request that will time out after occupying memory for a minute.

Cost, quantisation and routing

The unit of cost is tokens per second per accelerator, and there are three levers on it.

Quantisation stores weights in fewer bits — eight or even four instead of sixteen. Because decode is bandwidth bound, halving the bytes read per step nearly halves the time per step, so the speedup is roughly proportional. It also shrinks the weights, leaving more memory for KV cache and therefore a larger batch. The cost is some quality loss, which varies by model and task and must be measured on the actual workload rather than on public benchmarks.

Shorter contexts attack the cache directly. Trimming conversation history, summarising older turns and retrieving only relevant passages reduce cache per request linearly, which raises concurrency by the same factor.

Routing is the largest lever of all: send the easy majority of requests to a small model and reserve the large one for requests that need it. Most traffic in most products does not need the biggest available model, and the cost difference between model sizes is an order of magnitude — far larger than any serving optimisation can deliver.

What breaks first

1. KV cache memory. Always first. It caps batch size, which caps throughput, and it grows with context length and concurrency simultaneously. Relieved by paging, prefix sharing, shorter contexts and quantised weights that leave more room.
2. The interconnect. Under tensor parallelism, a collective operation runs on every layer of every step, so a slow link between chips turns into per-token latency directly.
3. Prefill interference. Long prompts stalling everyone else's decode, visible as stutter. Chunked prefill is the fix and it costs the new request a little latency.
4. Cold start on a spike. Minutes to load weights means capacity cannot be summoned reactively, so headroom must be bought in advance and queues must have deadlines.

The judgement underneath all of it: throughput and latency trade directly here in a way they do not elsewhere. A larger batch means more tokens per second per accelerator and a slightly slower stream for each user. There is no configuration that optimises both, so the design must say which one it protects — and for an interactive product the answer is usually latency, bought with more hardware.

Recap