Where the money goes when you serve a language model

Two phases, two bottlenecks

Almost every serving mistake comes from treating inference as one thing. It is two, and they have opposite characteristics.

Prefill processes the entire prompt in a single forward pass. Every token is available at once, so the work is large dense matrix multiplications — exactly what accelerators are built for. Prefill is compute-bound: the arithmetic units are the limit.

Decode generates one token at a time. Each step multiplies a single vector against the model's weights, which means reading every parameter from memory to do a trivially small amount of arithmetic. Decode is memory-bandwidth-bound: moving the weights is the limit, and the arithmetic units sit largely idle.

That asymmetry explains nearly everything about serving.

A second consequence: prefill and decode compete. A long prompt arriving mid-generation occupies the compute units and stalls decode for everyone in the batch — visible to users as a stutter in streaming output. Serving stacks handle this by chunking prefill into pieces that interleave with decode steps, trading a little prefill latency for much smoother generation across the batch.

The memory budget

Memory, not compute, is usually what decides how many users you can serve. Three things compete for it.

total = weights + kv_cache + activations

weights   = n_params * bytes_per_param
          # fp16: 2 bytes. int8: 1. int4: ~0.5.

kv_cache  = 2 * n_layers * n_kv_heads * d_head
          * seq_len * batch_size * bytes_per_element
          # linear in BOTH context length and batch size

activations # modest with memory-efficient attention; was not always

Weights are fixed once you pick the model and precision. Activations are small with modern attention implementations. So the KV cache is the term that varies, and it is the one that decides your concurrency.

Rearranged, that gives the number you actually want:

max_batch = (memory - weights) / kv_bytes_per_sequence

# And therefore, directly:
#   longer context  -> fewer concurrent users
#   fewer concurrent users -> lower throughput
#   lower throughput -> higher cost per request
# Context length is a cost decision, not only a capability one.

Work this out for your model and hardware before deployment. Discovering it under load is how you find out that your advertised 128k context window supports four concurrent users.

Reducing the cache

Grouped-query attention shares one key/value head across a group of query heads. Eight KV heads instead of sixty-four cuts the cache eightfold at a modest quality cost. This is an architectural choice made at training time, so it is a reason to prefer certain models rather than something you can retrofit.

Paged attention allocates the cache in fixed-size blocks instead of one contiguous buffer per sequence. Without it you must reserve worst-case length for every request, and a request that generates 50 tokens holds memory for 4,000. Paging typically recovers a large fraction of that waste and is the main reason modern serving engines achieve the batch sizes they do.

Prefix caching keeps the KV entries for a shared prompt prefix across requests. If every request begins with the same 2,000-token system prompt, you can compute it once. On chat products with a long fixed preamble this eliminates a substantial share of prefill work outright.

Batching, and why continuous

Static batching — collect n requests, run them together, return them together — is the obvious approach and it wastes most of its potential.

The problem is that generation lengths vary enormously. Batch eight requests where seven finish in 40 tokens and one runs to 900, and for 860 steps you are running a batch of eight to produce one useful token. The finished slots are occupied by padding.

Continuous batching — also called in-flight batching — operates at the granularity of a step rather than a request. When a sequence finishes it leaves the batch immediately and a queued request takes its slot on the very next step. The batch stays full.

# static: the batch is hostage to its slowest member
step   1: [A B C D E F G H]
step  40: [A _ _ _ _ _ _ H]   # six slots wasted
step 900: [_ _ _ _ _ _ _ H]   # seven slots wasted for 860 steps

# continuous: finished sequences leave, queued ones join
step   1: [A B C D E F G H]
step  40: [A I J K L M N H]   # refilled on the next step
step 900: [P Q R S T U V H]   # still full

On workloads with variable output lengths — which is nearly all real workloads — this is typically the single largest throughput improvement available, and it requires no change to the model. If you are serving without it, it is the first thing to fix.

The trade is that a request's latency now depends on what else is in the batch. Larger batches raise throughput and raise per-request latency, which is why interactive and bulk traffic generally belong in separate pools with different batch-size targets rather than competing in one.

Quantisation

Store weights in fewer bits. Because decode is bandwidth-bound, halving the bytes per parameter roughly halves the time spent reading weights — so quantisation buys speed as well as memory, which is the part people underestimate.

Two distinctions worth having clear.

Weight-only versus weight-and-activation. Weight-only quantisation stores compressed weights and dequantises them to compute in higher precision. It gets the memory and bandwidth savings with minimal quality impact, and it is the common choice. Quantising activations too allows genuinely lower-precision arithmetic and is more aggressive in both senses.

Post-training versus quantisation-aware. Post-training quantisation converts a finished model using a small calibration set — cheap, fast, usually sufficient. Quantisation-aware training simulates the quantisation during training so the model adapts to it, which recovers more quality at low bit-widths and costs a training run.

Speculative decoding

The insight follows directly from decode being bandwidth-bound. Reading the whole model to produce one token is wasteful; reading it to verify five candidate tokens costs almost the same.

So: a small fast draft model proposes several tokens ahead. The large model evaluates all of them in one forward pass and accepts the longest prefix consistent with what it would have generated itself, rejecting the rest.

draft  proposes:  " the quick brown fox jumps"
target verifies:  " the quick brown"  ✓✓✓   "fox" ✗

# Three tokens accepted for one target forward pass.
# The rejected suffix is discarded and the next round begins.

Done with the correct acceptance rule, the output distribution is identical to what the target model would have produced alone. It is a pure latency optimisation with no quality trade — which is unusual enough to be worth stating plainly, because most optimisations on this page do trade something.

The speedup depends on the acceptance rate, which depends on how well the draft mimics the target. A draft that is too weak gets rejected constantly and adds overhead; one that is too strong costs as much as the model it is helping. Variants avoid the separate draft model entirely — predicting several tokens from the target model's own hidden states, or drafting from the prompt text for tasks with heavy copying, which works remarkably well for summarisation and editing.

The important caveat for capacity planning: speculative decoding improves latency for a single request and can reduce total throughput under heavy load, because the rejected tokens are wasted compute. It is a good fit for interactive traffic with spare capacity, and a poor fit for a saturated batch endpoint.

The metrics that matter

"Latency" is not one number and conflating the parts leads to optimising the wrong thing.

Time to first token (TTFT). From request to the first token appearing. Dominated by queueing plus prefill, so it scales with prompt length. This is what determines whether the product feels responsive, because it is the gap where the user sees nothing.

Inter-token latency (ITL). The gap between subsequent tokens during streaming. Determines whether generation feels smooth. Beyond roughly the speed of comfortable reading, improving it further is not perceptible — which makes it a place where you can deliberately trade latency for throughput.

Total latency. TTFT plus ITL times output length. Output length is frequently the dominant term, which makes "instruct the model to be concise" a genuine latency optimisation and not just a cost one.

Throughput. Tokens per second across all requests, or requests per second. This is the number that determines cost per request, and it trades directly against per-request latency through batch size.

Track percentiles rather than means. A p50 of 200ms with a p99 of 12 seconds is a product where one request in a hundred looks broken, and the mean will not tell you.

Which lever moves which number

The table most serving discussions need and rarely have. Several of these trade against each other, and knowing which is which prevents a lot of wasted work.

The row worth dwelling on is the last. Routing by difficulty is usually the highest-return cost optimisation available, because most traffic does not need the largest model. Classify incoming requests, send the easy majority somewhere cheap, escalate the rest, and let your evaluation suite confirm quality held. It is typically a week of work for a large and permanent saving.

Self-host or buy

The arithmetic is more one-sided than people expect, in both directions depending on scale.

A hosted API charges per token with no fixed cost. You pay exactly for what you use, you get zero operational burden, and you get new models without a migration project. At low and moderate volume this is almost always cheaper than self-hosting, because a GPU costs the same whether it is busy or idle and most workloads are spiky.

Self-hosting replaces a variable cost with a fixed one. That becomes favourable only at sustained high utilisation — the crossover is where your token volume, priced at API rates, exceeds the fully loaded cost of the hardware plus the engineers who keep it running. That second term is the one omitted from most comparisons, and it is not small.

Reasons to self-host that are not about cost, and are frequently the real reasons:

Data cannot leave your environment. Regulatory or contractual. This is a genuine requirement and settles the argument on its own.

You need a model nobody hosts — your own fine-tune, or an open model with modifications.

Latency floor. Co-locating the model with your application removes a network hop, which matters at tight budgets.

Predictability. No rate limits you do not control, no deprecation timeline set by someone else, no pricing change mid-quarter.