Skip to content
~/ai-ml-handbook by @ka1manov

./calc.sh

Calculators

The arithmetic from the articles, made executable. Every formula here is the one derived in the linked article — if they ever disagree, the article is the specification and this is the bug.

tools 3|runs in your browser, nothing sent anywhere|by @ka1manov

GPU memory and KV cache

Memory, not compute, usually decides how many users you can serve. This works out whether a model fits, and what batch size you can actually run at a given context length — the number that becomes your throughput, and therefore your cost per request.

derived in inference economics|the KV cache

parameters edit anything
weight precision
KV cache precision
memory budget live
weights—
KV cache, per sequence—
KV cache, all sequences—
overhead—
total required—
fits in budget?—
max concurrent sequences—
cache as share of total—

Try it: drop KV heads from 32 to 8 and watch the cache collapse — that is the entire argument for grouped-query attention. Then double the context and watch your maximum batch halve, which is why long context is a throughput decision and not only a capability one.

Cost per successful outcome

Cost per request is the number that governs nothing. Cost per successful outcome is the one that governs the business, because a cheaper model that fails more often and triggers retries costs more overall — and only this denominator shows it.

derived in unit economics|RAG cost model

your numbers prices change — use today's
what it costs live
cost per call—
input share of that—
calls per successful outcome—
cost per successful outcome—
per month—
per year—

Try it: drop success from 85% to 60% without changing anything else. That is why quality is cost. Then cut input tokens in half — if your retrieval recall holds at five chunks instead of ten, that is free money.

Vector index memory

At scale, memory is the dominant cost of a retrieval system, and the HNSW graph is frequently as large as the vectors it indexes. This sizes the index and shows what quantisation actually buys.

derived in vector search internals|the recall triangle

corpus and index edit anything
stored precision
index type
keep full-precision copy for rescoring
index size live
vectors—
HNSW graph—
rescoring copy—
total resident—
bytes per vector—
vs fp32 HNSW—

Try it: switch fp32 to int8 at 10M vectors. Then notice the graph does not shrink at all — it stores neighbour ids, not vectors, which is why quantisation stops helping once the graph dominates. Raising M makes that worse.

what these do not tell you

All three are sizing and cost models, not performance predictions. They cannot tell you your recall, which depends on the intrinsic dimensionality of your own embeddings; your tokens per request, which depends on your prompts and retrieval; or your success rate, which depends on your evaluation set. Those come from measurement.

Use these to decide what to build and what to budget. Use an evaluation suite to find out whether it works.