50 field notes

Look at your data. Actually look at it.Not df.describe() — open fifty rows and read them. Every experienced practitioner has a story about a month lost to something that was visible on inspection: a duplicated column, a placeholder date of 1970-01-01, labels shifted by one, an encoding that turned every apostrophe into mojibake. This is the highest-return hour in the whole job and almost nobody spends it.

Your label is the product, and it is probably wrong.Model architecture is a rounding error next to label quality. Before optimising anything, double-label a sample and compute agreement. That number is your ceiling: if two humans agree 85% of the time, a model scoring 90% is exploiting an artefact, not being clever.

Point-in-time correctness is not a detail.Every feature must use only information that existed at prediction time. This single discipline prevents more production failures than any modelling technique, and violating it produces a beautiful offline number and a flat A/B test.

Missing is a value, not an absence.A user who left a field blank is different from one who filled it in. Adding an explicit indicator column preserves that signal; silently imputing the mean destroys it and tells the model a confident lie.

Normalise before you deduplicate.Most 'near duplicates' in a real corpus are exact duplicates wearing different whitespace, case and boilerplate. Cheap string normalisation removes more duplication than any clever similarity method.

If your data pipeline is not idempotent, it is not finished.Rerunning on unchanged input must be a no-op, provable with a checksum rather than by inspection. Without it every backfill is a gamble and every failure is a full rebuild.

Always build the stupid baseline.A regularised linear model or a constant predictor takes an hour and gives every later number a denominator. 'We got 0.87 AUC' means nothing on its own; '0.87 against a 0.81 baseline' is a result.

On tabular data, beat gradient boosting before reaching for a network.The comparison is cheap and the answer is frequently that trees win at a fraction of the training and serving cost. If your tabular problem uses a neural network, be able to defend it with a measured comparison rather than a preference.

Overfit one batch before you train on everything.If the model cannot memorise eight examples, it has a bug, not a data problem. This test takes seconds, has a binary answer, and eliminates an enormous class of defects before you spend a day on a real run.

Check that the initial loss is what theory says.An n-class classifier should start near ln(n). If it does not, something is wrong before training has done anything — a doubled softmax, misaligned labels, a loss over the wrong axis.

Learning rate is the hyperparameter that matters. Tune it first, tune it properly.More results have been left on the table by an untuned learning rate than by any architectural choice. A range test costs one short run.

Ranking well and being calibrated are different skills.A model can order examples perfectly and still be systematically overconfident. The moment anything downstream treats the score as a probability — expected value, thresholds, budget pacing — calibration becomes the property that matters, and deep networks are poorly calibrated by default.

Complexity must earn its place with a measurement.Every added stage is a new failure mode, a new thing to monitor, and a new thing the next engineer must understand. If you cannot show the number it moved, remove it.

Read the failures. Fifty of them, categorised.The error distribution tells you what to fix, and it is almost never what you assumed. Teams that theorise about failures optimise the wrong stage for weeks; teams that read them fix the right one in a day.

Prompts are production code.Version them, review them, test them, and own them. A prompt edited by anyone, tested by nobody, and deployed outside the release path is a production dependency nobody controls — and it is a leading cause of 'it got worse and we did not deploy anything'.

Fine-tuning teaches form. Retrieval supplies facts.Confusing these is the most expensive mistake in applied LLM work. Fine-tuning to fix a knowledge gap produces a model that is confidently wrong in exactly the right tone.

Constrain the output rather than asking politely.Schema-constrained decoding never samples an invalid token. Asking for JSON and retrying on parse failure is strictly worse and fails at the worst times.

Tell the model what to do when it does not know.The single highest-value line in most system prompts. Without an explicit refusal instruction, an unanswerable question produces a fluent fabrication — which is the exact failure the system existed to prevent.

Retrieved text is data, never instructions.Any document in your corpus may be adversarial, and in RAG the attacker's delivery mechanism is simply getting a document indexed. Delimit it, say so in the system prompt, and never let it reach a tool-calling path unexamined.

Five good chunks beat thirty mediocre ones.A large context window is not an invitation to fill it. More context dilutes attention, costs proportionally more, and raises the chance of anchoring on a plausible irrelevant passage.

Log the transformed query, not just the original.If you rewrite or expand queries before retrieval, the rewritten version is invisible in your logs by default — and it is the first thing you will want when an answer is baffling.

Agents compound errors multiplicatively.Ninety-five percent per step over ten steps is about sixty percent end to end. The engineering answer is fewer steps and verification between them, not a better prompt.

Temperature zero is not deterministic in a batched server.Floating-point reduction order varies with batch composition, so identical inputs can produce different outputs. Design tests that tolerate this rather than assuming exact reproducibility.

A large share of 'the model is stupid' reports are tokenisation.Counting characters, arithmetic on long numbers, non-Latin scripts, and prompts with trailing whitespace all degrade for reasons that have nothing to do with reasoning.

Build the evaluation set before the feature.Fifty real examples and a script is one day of work, and it changes everything afterwards: you can refactor without fear, compare models honestly, and answer 'did that help' with evidence rather than impressions.

Measure retrieval and generation separately.A single end-to-end score averages two independent failure modes and tells you nothing actionable. The only useful question when a RAG answer is wrong is which half broke, and one number cannot answer it.

Programmatic checks before model-based ones.Schema validity, citation ids that exist, length bounds, required disclaimers. Free, deterministic, and they catch a surprising share of real defects without anyone judging anything.

An uncalibrated judge is a random number generator with good manners.Label fifty examples by hand, run your judge on them, and report the agreement. Without that number, every result your evaluation pipeline produces is decoration.

Include the questions your system should refuse.A system that always answers has not been tested, it has been flattered. Measuring correct abstention is measuring the failure mode users actually complain about.

A flaky quality gate gets disabled within two weeks.Pin dependencies, fix seeds, and make the evaluation set immutable and versioned. A non-deterministic CI check does not protect you; it trains the team to ignore red.

Benchmark scores are not production evidence.Public benchmarks are a shortlist, not an answer. The ordering frequently does not transfer to your corpus, your queries or your definition of good.

Check segments, not just the average.An overall improvement routinely conceals a serious regression for a minority of users, and that minority is often the one that complains loudest and churns first.

ML systems fail silently and stay up.The service returns 200, latency is fine, and the answers have quietly become worse. Every alert you have says green. This is the fundamental operational difference from ordinary software, and it is why output-quality monitoring is not optional.

Training-serving skew is the most common production defect.Log the actual feature vector used at inference, recompute it offline, and diff. Make that a scheduled job rather than a one-off investigation, because the divergence reappears every time someone touches either path.

Shadow deployment is the most valuable step teams skip.Real traffic, real volume, zero user risk. It catches the operational failures — memory growth, tail latency, malformed inputs — that offline evaluation structurally cannot.

A rollback procedure nobody has executed is a hypothesis.Rehearse it. Make the model version a config value so reverting is a flag flip rather than a redeploy, and time yourself doing it.

Deletion has to reach the caches too.A document erased from the source and still sitting in a retrieval cache is still being served. If deletion was not designed in, retrofitting it into three layers of caching is genuinely hard and the deadline is usually legal.

Denominate cost by successful outcome, not by request.A cheaper model that fails more often and triggers retries costs more. Cost per request cannot show you that; cost per outcome can, and it is the number that governs the business.

Idle accelerators are the most wasteful line in most ML budgets.A surprising share of GPU fleets are input-bound — the device waiting on the data loader. It is invisible without profiling and expensive with it.

Instrument freshness as a number.'Documents are searchable within five minutes' is a commitment you can alert on. 'We reindex nightly, probably' is not. Emit the age of the oldest un-indexed document.

One request id should reconstruct everything.Every model call, every retrieved document, token counts, latency per stage, cost. If any link is missing, your incident reviews are guesswork dressed as analysis.

Automatic retraining without an evaluation gate is automatic deployment of a worse model.The schedule is fine. The missing gate is what does the damage, and it has done real damage at real companies.

Iteration speed is the real constraint on model quality.The team that runs twenty experiments a week beats the team that runs three, regardless of who is cleverer. Anything that shortens the loop from idea to measured result is modelling work, even when it looks like plumbing.

Reproduce something in your first two weeks on a new team.Retrain an existing model and see whether you get the same number. Nothing else reveals the true state of reproducibility, data access and tooling that fast, and no document will tell you the truth.

Read the incident history before the architecture docs.It tells you what the system actually does under stress, which is different from what it was designed to do.

Kill your own project when it deserves it.Establishing early that the data cannot support the problem is a successful outcome, even though it does not feel like one. The expensive failure is spending a quarter hoping.

Describe what the model does mechanically, never what it 'understands'.Anthropomorphising sets expectations that reality will violate, and the disappointment lands on you. Stating the mechanism and the failure modes builds the kind of trust that survives a bad week.

Report your own regressions before anyone else notices.Trust with stakeholders is built far more by flagging a problem early than by announcing a win. It is also the only way to be believed the next time you say something is fine.

Fundamentals compound; releases do not.Attention, evaluation, distribution shift and cost structure have been the same problems for years. Time spent there pays for a decade. Tracking weekly model launches pays until the next one.

Propose the cheap decisive experiment instead of winning the argument.A week of work that settles a disagreement is worth more than a month of being right, and it converts opinions into evidence — including when the evidence goes against you.