Skip to content
~/ai-ml-handbook by @ka1manov
[_][□][X] ka1manov@handbook: ~/ai-ml-handbook

python -m handbook --serve --free

[00:00:00]initloading curriculum … 7 stages, 42 nodes
[00:00:00]okstage 00 · foundations — the maths and engineering you actually use
[00:00:00]okstage 01 · classical ml — the baselines you must beat
[00:00:00]okstage 02 · deep learning — training loops that converge
[00:00:01]okstage 03 · transformers & llms — attention to alignment
[00:00:01]okstage 04 · applied llm systems — rag, agents, fine-tuning
[00:00:01]okstage 05 · production & llmops — serving, eval, cost, observability
[00:00:01]okstage 06 · specialisation — pick a depth and go
[00:00:01]alsorag deep dive, 122 interview questions, startup playbook, 50 field notes, 6 long reads
[00:00:02]readyserving on this page
[_][□][X] start here — 57k words is a lot, so here is the order

Who are you right now?

If you are levelling up

The shortest route from "I can train a model" to "I am trusted with the thing that pages at 3am." Start with the gap audit — it tells you which of the rest you actually need.

  1. Audit yourself against the roadmap

    Do not read it. Scan only the prove it line of all 42 nodes and tick what you could produce this afternoon. Whatever you cannot is your actual curriculum. ~15 min

  2. RAG systems, properly

    The most common production LLM architecture and the most commonly built badly. Read the failure-mode taxonomy even if you skim the rest. ~35 min

  3. Evaluation engineering

    The single practice that separates engineers who improve systems from engineers who change them. Highest leverage thing on this whole site. ~15 min

  4. Inference economics

    Where the money actually goes, and which optimisation moves which metric. Then put your own numbers into the memory calculator. ~20 min

  5. 50 field notes

    Concentrated. Read them in one sitting; a handful will describe a mistake you are currently making. ~12 min

  6. Then go work your gaps

    Back to the roadmap, with the ticks you made in step one. Build the artifacts, not the reading list. ongoing

~1h 40m of reading, then the work. Your ticks save in this browser, so you can stop and come back.

Everything I wish someone had handed me before my first ML system went to production.

A staged roadmap you can actually check off, a RAG deep dive that covers the failure modes nobody writes about, 122 interview questions with real answers, and a playbook for founders deciding what to build. Written for engineers who ship, not for people collecting certificates.

words ~57k|read ~2h|price free, forever|signup none|tracking none

Read this first

This is a gift to my subscribers on X. There is nothing to buy, no email to hand over, and no analytics script watching you. Take it, fork it, print it, teach from it.

Three things make it different from the roadmap graphics you have already seen:

Every node has a "prove it." Not "learn backpropagation" — implement backpropagation for a two-layer network in NumPy and match PyTorch's gradients to 1e-6. An artifact you either have or do not have. Reading about a concept and being able to use it under pressure are different skills, and only one of them survives an interview or an outage.

Every node says what it prevents. Techniques are not interesting in themselves. They are interesting because of the specific way things break without them. Where I know the failure, I name it.

The time estimates are honest. They assume you have a job and study around it. If a number looks large, that is the number, not a motivational one.

how to use the roadmap

Do not start at stage 00 unless you are genuinely starting. Scan the "prove it" line of every node instead. Anything you could not produce this afternoon is a gap, wherever it sits in the sequence. Work the gaps, not the order. Your ticks are saved in this browser only.

Who this is for

The engineer levelling up. You can train a model in a notebook and you want the map to a system that stays up. Stages 02 through 05 are yours; skim 00 and 01 for gaps.

The candidate. You have a loop in two weeks. Go straight to the question bank, then read RAG and evaluation properly, because that is where senior interviews actually separate people.

The founder or CTO. Start with the startup playbook. It has the cost arithmetic and the build-versus-buy tree. Come back to the roadmap when you are hiring and need to know what to test for.

who this is not for

If you want to understand the mathematics of machine learning deeply, read Bishop, Murphy or Goodfellow instead. This handbook is about building systems that work, which is a related but genuinely different discipline. It will not make you a researcher, and it does not pretend to.

The roadmap

Seven stages, 42 nodes. Tick what you can already prove.

progress 0/42
[00]

Foundations

The maths and engineering you actually reach for. Skip what you can already prove — nobody is marking your attendance.

  1. Matrix multiplication and its shapes, dot products as similarity, matrix–vector products as linear maps, eigenvectors, SVD, and rank. Not a semester course — the working subset.

    Why it matters: almost every real bug in a training script is a shape bug. Engineers who cannot hold [batch, seq, d_model] in their head debug by permuting dimensions at random until the error goes away, which produces code that runs and is silently wrong.

    Prove it: derive the output shape of batched multi-head attention by hand, on paper, from [B, S, D] through heads and back, then verify against a real forward pass. Separately, reconstruct a matrix from its top-k SVD components and plot reconstruction error against k.

    ~15–25h · assumes secondary-school algebra

  2. Distributions, expectation and variance, conditional probability and Bayes, maximum likelihood, sampling, confidence intervals, and hypothesis tests.

    Why it matters: this is the difference between "the new model scored 91.2% versus 90.8%, ship it" and asking whether that gap survives the test set's sampling noise. Most shipped regressions I have seen were approved on differences that were inside the error bars.

    Prove it: take any two model variants, compute a bootstrap confidence interval on their metric difference over your test set, and state the minimum test-set size that would make a 0.4-point difference detectable. If you cannot do this, you cannot responsibly approve a model change.

    ~25–40h · the highest-leverage stage-00 node

  3. Derivatives, partial derivatives, gradients, the chain rule, and the Jacobian. Stop there. You do not need contour integration.

    Why it matters: exploding and vanishing gradients, dead ReLUs, and why residual connections exist are all chain-rule facts. Without this you cannot read a loss curve; you can only react to it.

    Prove it: write the gradient of cross-entropy with respect to the logits by hand and explain, in one sentence, why the softmax and the loss are fused in every serious implementation.

    ~12–20h

  4. Type hints, dataclasses, context managers, generators, virtual environments and dependency pinning, pytest, and a debugger you can actually drive.

    Why it matters: the gap between a data scientist and an ML engineer is mostly here. Notebook code that trains a good model and cannot be run by anyone else has produced nothing of value to the company.

    Prove it: take your worst notebook and turn it into an installable package with a CLI, pinned dependencies, and tests that run in CI. A colleague clones it and reproduces your number without asking you a single question.

    ~20–30h · pays back faster than anything else here

  5. SQL to window functions and CTEs, pandas or Polars fluency, vectorisation over loops, columnar formats such as Parquet, and out-of-core processing.

    Why it matters: you will spend more time on data than on models, and the difference between a query that runs in 4 seconds and 40 minutes decides how many experiments you get to run this week. Iteration speed is the real constraint on model quality.

    Prove it: write a single SQL query with a window function that produces a point-in-time-correct training label — one that uses only information available at prediction time. Getting this wrong is the most common source of leakage in industry.

    ~25–40h

  6. Git beyond commit and push, code review, Docker, CI, logging and structured observability, and enough Linux and networking to debug a container that will not start.

    Why it matters: ML systems fail as software far more often than they fail as mathematics. The 3am page is a CUDA version mismatch, a full disk, or an OOM — not a subtle optimisation failure.

    Prove it: containerise a training job so it runs identically on your laptop and on a GPU host, with dependencies pinned to exact versions, and produces a reproducible metric given a fixed seed.

    ~30–50h · the most under-invested node on this page

[01]

Classical machine learning

Skipping this to get to transformers is the single most common mistake I see. These are the baselines you have to beat, and on tabular data you often will not.

  1. What a model is actually promising: that training and deployment data are drawn from the same distribution. Bias and variance, over- and underfitting, and the assumptions that break in production.

    Why it matters: every production ML failure that is not a plumbing failure is this contract being violated. Distribution shift is not an exotic edge case; it is the default state of the world, and your model is the only party that does not know it.

    Prove it: name the three assumptions your current production model makes about its input distribution, and the monitor that would tell you when each one breaks. If you cannot name the monitors, you do not have them.

    ~8–12h

  2. Train/validation/test discipline, k-fold and stratified cross-validation, grouped splits, time-based splits, and the many faces of leakage.

    Why it matters: leakage is the defect that punishes you hardest and latest. An offline AUC of 0.95 that collapses to 0.62 in production has cost a quarter, and the cause is nearly always a feature computed with information that would not exist at prediction time.

    Prove it: deliberately introduce target leakage into a dataset, observe the inflated validation score, then build the split that catches it. Write down the three leakage checks you will run before believing any offline number again.

    ~12–18h · the node that saves careers

  3. Ordinary least squares, regularisation and what L1 and L2 actually do to coefficients, the logistic link, and reading a coefficient honestly.

    Why it matters: a regularised linear model is interpretable, trains in seconds, costs nothing to serve, and is frequently within a couple of points of the deep model that took a month. It is also the baseline that makes your gradient boosting result meaningful — without it, "0.87 AUC" is a number with no denominator.

    Prove it: on a real problem of yours, get a regularised linear baseline to within five points of your best model, and be able to say precisely what the remaining gap is buying and whether it is worth the serving cost.

    ~12–20h

  4. Decision trees, bagging and random forests, gradient boosting, and the practical differences between XGBoost, LightGBM and CatBoost — especially their categorical handling and missing-value semantics.

    Why it matters: on tabular data, gradient-boosted trees remain the thing to beat, and a large body of comparative work continues to find they are competitive with or better than deep tabular models at a fraction of the cost. If your tabular problem is being solved with a neural network, you should be able to defend that choice with a measured comparison.

    Prove it: tune a GBDT properly — learning rate against number of trees, depth, and the sampling parameters — and explain what each parameter trades. Then explain why raising the learning rate usually requires fewer trees and what that does to variance.

    ~20–30h

  5. Categorical encoding and its traps, target encoding done without leaking, scaling, binning, interactions, and handling missingness as signal rather than noise.

    Why it matters: on most tabular problems, features move the metric more than model choice does. Target encoding in particular leaks quietly unless it is fit inside the cross-validation fold, and that bug produces a beautiful offline number and a flat A/B test.

    Prove it: implement out-of-fold target encoding yourself and demonstrate the score difference against the naive version fit on the full training set. The gap is your leakage.

    ~15–25h

  6. Accuracy, precision and recall, F1, ROC-AUC against PR-AUC, log loss, calibration, and regression metrics. Threshold selection as a business decision rather than a default of 0.5.

    Why it matters: ROC-AUC is misleading under heavy class imbalance because the false-positive rate barely moves when negatives dominate; PR-AUC shows the problem plainly. And a model can rank perfectly while being badly calibrated, which breaks you the moment a downstream system treats the score as a probability — expected-value calculations, budget pacing and risk thresholds all do.

    Prove it: take an imbalanced problem, show a case where ROC-AUC looks strong and PR-AUC does not, then produce a reliability diagram and fix the calibration with Platt scaling or isotonic regression.

    ~15–20h

[02]

Deep learning

Where most self-taught engineers stall: they can call .fit() but cannot diagnose a loss curve that flattens at the wrong value.

  1. The computational graph, forward and backward passes, automatic differentiation, and why gradients vanish or explode through depth.

    Why it matters: when a model will not train, the useful question is where the gradient dies. Engineers who have only ever called loss.backward() cannot form that question, so they change the learning rate and hope.

    Prove it: implement a two-layer MLP with backprop in NumPy — forward, backward, update, no autograd — and match PyTorch's gradients to 1e-6 on the same weights and inputs.

    ~20–30h · do this once and you will never forget it

  2. SGD with momentum, Adam and AdamW, weight decay done correctly, warmup, cosine decay, and gradient clipping.

    Why it matters: learning rate is the hyperparameter that matters most and the one people tune least carefully. The AdamW distinction is not pedantry: in plain Adam, L2 regularisation added to the loss gets scaled by the adaptive per-parameter denominator, so parameters with large gradient history are effectively regularised less. Decoupling it changes results measurably.

    Prove it: run the same model with Adam and AdamW at identical weight decay and show the difference in final validation loss. Then produce a learning-rate range test and explain how you would pick the maximum rate from it.

    ~15–25h

  3. Dropout, weight decay, early stopping, data augmentation, and the normalisation layers — batch, layer and RMS — with the reasons transformers settled on layer-style normalisation.

    Why it matters: batch normalisation couples every example in a batch to every other, which is fine for vision with large batches and actively harmful for variable-length sequences and small batches. It also behaves differently in training and inference, which is a classic source of train/serve skew that only shows up after deployment.

    Prove it: take a model using batch norm, run it at batch size 2, and explain the degradation. Then state precisely what batch norm does differently at inference time and why that creates skew risk.

    ~12–20h

  4. Mixed precision, gradient accumulation, checkpointing and resumption, deterministic seeding, distributed data parallel, and profiling to find whether you are compute-, memory- or input-bound.

    Why it matters: a training run that cannot resume from a checkpoint will eventually lose you a week to a pre-emption. And a surprising share of GPU fleets are input-bound — the accelerator idles waiting on the data loader — which is invisible unless you profile and enormously expensive if you do not.

    Prove it: profile a training job and state, with evidence, whether it is compute-, memory-bandwidth- or input-bound. Then make it 1.5× faster and explain which bottleneck you moved.

    ~25–40h

  5. Convolution, pooling, receptive fields, residual connections, and the inductive biases — locality and translation equivariance — that make CNNs sample-efficient on images.

    Why it matters: even in a world of vision transformers, the architecture-as-prior lesson is the transferable one: ViTs need far more data or heavy augmentation to match CNNs on small datasets precisely because they lack that built-in bias. Knowing when a prior helps and when it constrains you is the actual skill.

    Prove it: compute the receptive field of a stack of convolutions by hand, then explain why a residual connection makes a 50-layer network trainable when a plain stack of the same depth is not.

    ~20–30h

  6. A systematic procedure: overfit a single batch, check the initial loss against its theoretical value, inspect gradient norms per layer, visualise inputs after every transform, and remove pieces until it works.

    Why it matters: this is the single most valuable practical skill in deep learning and almost nobody teaches it explicitly. If your ten-class classifier does not start at a loss near ln(10) ≈ 2.303, something is wrong before training has even begun, and knowing that saves you days.

    Prove it: write your own debugging checklist, in order, with the expected value at each step. Then use it to find a bug you have deliberately planted in a training script — a wrong axis in a loss, a label shuffle, a transform applied twice.

    ~15–25h · pure compounding return

[03]

Transformers and large language models

The architecture is genuinely simple. The engineering around it is not, and that is where the questions get asked.

  1. Queries, keys and values, scaled dot-product attention, multi-head attention, causal masking, and the quadratic cost in sequence length.

    Why it matters: the quadratic term is not trivia — it is the reason long context is expensive, the reason KV caching exists, and the reason a dozen approximate-attention schemes were invented. The scaling by √dk exists because dot products of high-dimensional vectors grow with dimension, pushing softmax into saturation where gradients vanish.

    Prove it: implement multi-head attention in NumPy including the causal mask, and match a reference implementation. Then explain, with the arithmetic, why doubling context length roughly quadruples attention cost but only doubles the feed-forward cost.

    ~20–30h · full walkthrough here

  2. Embeddings, positional information from absolute encodings through to RoPE, the attention sublayer, the feed-forward sublayer, residual connections, and pre- versus post-normalisation.

    Why it matters: pre-normalisation is what made very deep transformers trainable without delicate warmup schedules, because it keeps a clean residual path for gradients. Rotary embeddings matter because they encode relative position, which is what actually generalises when you extend context beyond what you trained on.

    Prove it: draw the block from memory with every tensor shape annotated, then state where in the block the parameters actually live and what fraction of them sit in the feed-forward sublayer.

    ~15–25h

  3. Byte-pair encoding, WordPiece, SentencePiece, byte-level fallbacks, vocabulary size trade-offs, and special tokens.

    Why it matters: a startling share of "the model is stupid" reports are tokenisation artefacts. Character-level tasks, arithmetic on long numbers, and non-Latin scripts all degrade because of how text was split, not because of reasoning. Non-English text also tends to consume more tokens per unit of meaning, which is a direct and often unnoticed cost multiplier.

    Prove it: tokenise the same paragraph in English and in a non-Latin-script language and compare the token counts. Then construct a prompt that fails purely because of tokenisation and explain the mechanism.

    ~10–15h · badly underrated

  4. Next-token prediction, masked language modelling, the compute–parameters–data relationship, and why the field moved toward training smaller models on substantially more data.

    Why it matters: scaling results tell you where to spend a fixed budget, and the compute-optimal answer is usually not "the largest model you can afford." Inference cost scales with parameters for the entire life of the deployment, so a smaller model trained longer is often the better economic choice even when both reach the same quality.

    Prove it: given a fixed training budget, argue for a specific model size and token count and defend it on total cost of ownership rather than benchmark score alone.

    ~12–20h

  5. Greedy decoding, beam search, temperature, top-k, nucleus sampling, repetition penalties, and constrained or grammar-guided decoding.

    Why it matters: sampling parameters change output quality as much as prompt wording, and teams routinely ship a default temperature they never examined. Beam search helps translation and hurts open-ended generation, where it produces bland, repetitive text — the objective it optimises is not the one you want.

    Prove it: hold a prompt fixed and sweep temperature and top-p, then state the setting you would ship for an extraction task and for a creative task, with the reason for each. Then explain why temperature 0 is not strictly deterministic in a batched serving environment.

    ~10–15h

  6. Supervised fine-tuning, reward modelling, PPO-style RLHF, direct preference optimisation, and the alignment tax.

    Why it matters: this is why a base model and a chat model behave so differently, and why a fine-tune can improve your task while degrading instruction-following. DPO matters practically because it removes the separate reward model and the reinforcement-learning loop, which is most of the operational difficulty.

    Prove it: explain the difference between a base and an instruction-tuned model to a non-specialist in three sentences, then state what DPO removes from the RLHF pipeline and what you give up for that simplicity.

    ~15–25h · decision guide here

[04]

Applied LLM systems

Where most AI engineering jobs actually live in 2026. The model is a component; the system is the work.

  1. Instruction structure, few-shot examples and their selection, chain-of-thought and when it does not help, role and system prompts, delimiters, and prompt versioning.

    Why it matters: the discipline, not the tricks. A prompt that is not versioned, not tested against a fixed set of cases, and not owned by anyone is a production dependency nobody controls. Teams discover this when someone "improves" a prompt and a downstream parser starts failing silently.

    Prove it: put your prompts in version control with a test suite of at least twenty input/expected-behaviour pairs that runs in CI. Then demonstrate a prompt change that improves one case and regresses another — and show that your suite caught it.

    ~15–25h

  2. JSON mode, schema-constrained decoding, function and tool calling, validation and repair loops, and designing schemas a model can actually fill.

    Why it matters: this is the boundary between a demo and a system. Parsing prose with regular expressions works until it does not, at 3am. Constrained decoding that enforces a grammar during generation is categorically more reliable than asking politely and retrying, because invalid tokens are never sampled in the first place.

    Prove it: build an extraction endpoint that returns schema-valid JSON on 1,000 consecutive adversarial inputs, with a defined behaviour for the cases it cannot handle. "It mostly works" is not the bar; the defined failure path is the deliverable.

    ~20–30h

  3. Chunking, embeddings, vector indexes, hybrid retrieval, reranking, context assembly, citation, and evaluation of both halves separately.

    Why it matters: RAG is the most common LLM architecture in production and the most commonly done badly. The characteristic failure is a fluent, confident, wrong answer, which is worse than no answer because it is trusted. Most teams evaluate the generation and never measure retrieval, so they cannot tell which half is broken.

    Prove it: build a RAG system where you can state recall@k for retrieval and faithfulness for generation as separate numbers on a golden set you built yourself. Then use those numbers to say which half to fix first.

    ~40–60h · the full deep dive is here

  4. The reasoning-and-acting loop, tool design, planning, memory, multi-step error recovery, sandboxing, and cost and latency control.

    Why it matters: agents compound errors. A 95% per-step success rate over ten steps leaves you at roughly 60% end-to-end, which is why impressive demos so often fail as products. The engineering answer is fewer steps, verification between them, and designing tools that are hard to misuse.

    Prove it: take an agent of yours and produce its per-step and end-to-end success rates on a fixed task set. Then cut the step count and show the end-to-end number move.

    ~30–50h · architectures and failure taxonomy

  5. Full fine-tuning, LoRA and QLoRA, dataset construction, the quality-over-quantity result, catastrophic forgetting, and evaluating a fine-tune honestly.

    Why it matters: fine-tuning is the most over-reached-for tool in the field. It teaches form, style and format reliably; it is a poor and expensive way to add facts, which is what retrieval is for. Teams fine-tune to fix a knowledge problem, get a model that is confidently wrong in the right tone, and conclude the technique does not work.

    Prove it: write the decision memo for a real case: what you tried first, why prompting and retrieval were insufficient, what the fine-tune costs to train and to serve, and the evaluation that would tell you it worked. Most such memos end with "do not fine-tune," and that is a successful outcome.

    ~25–40h · decision guide

  6. Input and output validation, prompt-injection defence, PII handling, refusal behaviour, rate limiting, fallbacks, and graceful degradation.

    Why it matters: any system that puts untrusted text into a model's context has an injection surface, and retrieval systems put untrusted text into context by design. Treat retrieved content as data, never as instructions, and assume any tool the model can call can be triggered by content it retrieves.

    Prove it: write an injection payload that makes your own RAG system ignore its system prompt. Then implement a mitigation and show the payload failing. If you have not attacked your own system, you do not know its behaviour.

    ~20–30h

[05]

Production and LLMOps

The stage that separates people who build demos from people who are trusted with revenue.

  1. Prefill against decode, KV cache memory arithmetic, continuous batching, quantisation, speculative decoding, and the throughput–latency trade-off.

    Why it matters: inference is where the money goes, for the entire life of the product. Prefill is compute-bound and decode is memory-bandwidth-bound, and confusing the two leads to optimising the wrong thing. Continuous batching is typically the largest single throughput win available on a shared endpoint.

    Prove it: compute the KV cache size for a given model, batch size and context length, and state the batch size at which you run out of memory. Then explain why time-to-first-token and inter-token latency need separate budgets.

    ~30–45h · serving economics

  2. Golden sets, task decomposition, programmatic checks before model-based ones, LLM-as-judge and its calibration, regression suites, and online evaluation.

    Why it matters: without evaluation you are not engineering, you are guessing with extra steps. This is the single highest-leverage investment in an LLM product and consistently the most deferred. An unjudged system cannot be improved deliberately, only changed.

    Prove it: build an eval suite that runs in CI, produces a number you trust, and has caught at least one regression before it shipped. Report your judge's agreement rate with human labels — an uncalibrated judge is a random number generator with good manners.

    ~30–50h · how to build one

  3. Tracing multi-step calls, logging prompts and completions with PII controls, token and latency and cost metrics, drift detection, and user feedback capture.

    Why it matters: traditional monitoring tells you the service returned 200. It cannot tell you the answers got worse. You need output-quality signals, and you need the trace to attribute a bad answer to the retrieval step, the prompt, or the model.

    Prove it: from a single production request id, reconstruct every model call, every retrieved document, the token counts, the latency breakdown and the cost. If any link is missing, your incident reviews are guesswork.

    ~25–40h

  4. Ingestion, incremental and idempotent indexing, backfills, schema evolution, deletion and the right to be forgotten, and data versioning.

    Why it matters: a RAG index is a derived dataset that must stay consistent with a source of truth that keeps changing. Full reindexing is affordable at prototype scale and ruinous later. Deletion is a legal requirement, not a feature, and it has to reach the index and every cache.

    Prove it: implement incremental indexing where re-running the pipeline on unchanged input is a no-op, and a delete propagates to the index and all caches within a stated time bound. Prove the no-op with a checksum, not by inspection.

    ~25–40h

  5. Token accounting per request, prompt caching, model routing by difficulty, context compression, batch processing, and knowing when self-hosting becomes cheaper than an API.

    Why it matters: LLM costs scale with usage, so success is what bankrupts you. Cost per successful outcome — not per token, not per request — is the number that governs the business, and it is the one nobody instruments by default.

    Prove it: compute your cost per successful user outcome, then halve it without a measurable quality regression on your eval suite. Routing easy cases to a cheaper model is usually the first and largest win.

    ~15–25h · worked arithmetic

  6. Shadow deployment, canaries, A/B tests, feature flags, model and prompt registries, and rollback procedures you have actually rehearsed.

    Why it matters: model changes are riskier than code changes because the blast radius is quality rather than availability — nothing errors, answers just get worse, and you find out from users. Shadow traffic lets you measure a new model on real inputs before it affects anyone.

    Prove it: roll back a model or prompt version in production in under five minutes, from a state where you did not know in advance which version was bad. Rehearse it; do not assume it.

    ~20–30h

[06]

Specialisation

Pick one and go deep. Breadth got you hired; depth is what gets you trusted with the hard problem.

  1. Candidate generation and ranking as separate stages, collaborative filtering, two-tower retrieval, learning to rank, cold start, feedback loops and position bias.

    Why it matters: recommenders are where the revenue is at most consumer companies, and they have a property almost nothing else does: the model's output changes the data it will be trained on next. Position bias means your logs record what users clicked given what you showed them, not what they preferred.

    Prove it: explain why offline ranking metrics routinely fail to predict online engagement, and describe a specific correction for position bias in your training data.

    ~60–100h

  2. Detection and segmentation, vision transformers, augmentation strategy, edge deployment and quantisation, annotation quality, and handling domain shift.

    Why it matters: vision systems fail on distribution shift you can see with your own eyes — a new camera, different lighting, a changed mounting angle — which makes it the best domain for learning what shift actually does to a model. Annotation quality also caps your ceiling: you cannot exceed your labels.

    Prove it: ship a model to a constrained device within a stated latency and memory budget, and quantify the accuracy you traded for it.

    ~60–100h

  3. Speech recognition, speaker diarisation, text to speech, streaming architectures, and latency budgets for conversational systems.

    Why it matters: voice is the strictest real-time constraint in applied ML. Conversation tolerates only a few hundred milliseconds of response delay before it feels broken, and that budget has to cover recognition, the model, and synthesis — so every component is designed around streaming rather than batch.

    Prove it: build a streaming pipeline and produce a latency budget per stage, showing where the time goes and which stage you would optimise first.

    ~50–80h

  4. Vision-language models, contrastive image–text training, document understanding, multimodal retrieval, and cross-modal evaluation.

    Why it matters: most enterprise documents are not plain text — they are tables, charts, diagrams and scans, and a pipeline that flattens them to text loses exactly the structure that carried the meaning. This is where a large amount of practical retrieval value currently sits unclaimed.

    Prove it: build retrieval over documents containing tables and figures where the answers depend on that structure, and show it beating a text-only pipeline on a set you built.

    ~50–80h

  5. Reading papers efficiently, reproducing results, ablation design, experiment tracking, and distributed training at scale.

    Why it matters: the ability to reproduce a paper is rarer than it should be and immediately valuable. Ablation design is the deeper skill: knowing which single thing to remove to establish that a result came from the mechanism claimed rather than from the extra compute.

    Prove it: reproduce a recent paper's headline result from the description alone, document every discrepancy you hit, and state which of them changed the conclusion.

    ~80–120h

  6. Feature stores, training orchestration, GPU scheduling and utilisation, model registries, multi-tenancy, and building the paved road other teams use.

    Why it matters: platform work multiplies every other engineer in the organisation. It is also the most reliable way to be genuinely senior without being the best modeller in the room: the constraint at most companies is iteration speed, and that is a platform problem.

    Prove it: measure the time from "I have an idea" to "I have a trained model with a metric" for your team, then halve it. Report the before and after with evidence.

    ~80–120h

What to read next

RAG, properly

The deep dive: chunking, indexes, hybrid retrieval, reranking, evaluation, and a failure-mode taxonomy with a diagnostic for each symptom.

read ~35m

Interview bank

122 questions across eight tracks with real answers, and — where it discriminates — what a weak answer sounds like versus a strong one.

read ~40m

Startup playbook

Build, buy or fine-tune. Unit economics with the arithmetic shown. Why "we use an LLM" is not a moat, and what is.

read ~25m