Fine-tuning: mostly, a guide to not doing it

The question to ask first

Look at your actual failures — fifty of them, read individually — and sort them into two piles.

Pile one: the model does not know something. It gets your product names wrong, does not know your policies, invents details about your domain, cites things that do not exist. This is a knowledge problem.

Pile two: the model knows the content but produces it wrong. Wrong format, wrong length, wrong register, ignores a structural convention, will not consistently follow a schema, writes like a chatbot when it should write like a case note. This is a behaviour problem.

If pile one dominates, you want retrieval. If pile two dominates, fine-tuning is a real candidate. If both, do retrieval first, then re-read the errors — the remaining pile is usually much smaller than you expected.

What to try before this

Each step costs roughly an order of magnitude less than the next. Work up, not down.

1. A better prompt. Explicit structure, the task stated as a role rather than a request, the output format specified exactly, and the edge cases named. An afternoon. It resolves more cases than people expect, largely because most first prompts are underspecified rather than wrong.

2. Few-shot examples. Three to five examples in the prompt teach format extremely effectively — often as well as a fine-tune for formatting alone. Select examples that cover the variation you care about, including the awkward cases. If your examples are all easy, you have taught the model that the task is easy.

3. Constrained decoding. If the requirement is structural — valid JSON, a fixed schema, a restricted vocabulary — enforce it during generation. A grammar constraint never samples an invalid token, which is categorically stronger than any amount of training and instruction. Fine-tuning to improve schema adherence when constrained decoding would guarantee it is a common waste.

4. Decomposition. Split one hard call into two easy ones. A model that cannot reliably extract and classify in a single pass often does both perfectly in sequence.

5. A better model. Trivial to test, and frequently cheaper in total than a fine-tune of a smaller one once you count engineering time.

6. Now consider fine-tuning.

The reason for the order is not purity. It is that each of these is reversible in minutes and a fine-tune is an artefact you now own, must evaluate, must re-do when the base model is deprecated, and must keep a pipeline alive for.

Supervised fine-tuning

The mechanics are ordinary: continue training the model on your examples of input and desired output, with the standard next-token objective, at a much lower learning rate than pretraining used.

The detail that matters and is easy to get wrong is loss masking. You want the loss computed only on the response tokens, not on the prompt tokens. Training the model to predict the user's input is not the task and it dilutes the gradient toward whatever your prompts happen to look like.

# Mask the prompt: only the response contributes to the loss.
labels = input_ids.clone()
labels[:, :prompt_len] = -100   # the ignore index

# Also: match the base model's chat template EXACTLY.
# Different special tokens or spacing between training and serving
# is a real and silent failure — the model behaves as if it were
# given an input it never saw in training, because it was.

That second comment causes more confusing fine-tune failures than anything else on this page. The template is part of the interface, and a mismatch degrades quality in a way that looks like the fine-tune simply did not work.

Hyperparameters that actually matter: a learning rate one to two orders of magnitude below pretraining; very few epochs, since one to three is typical and more usually means memorisation; and a warmup, because a cold start at even a small learning rate can damage a pretrained model quickly.

LoRA and QLoRA

Full fine-tuning updates every parameter, which requires memory for the weights, their gradients, and the optimiser state — for Adam, roughly four times the model size beyond the weights themselves. For a large model that is more memory than most people have.

LoRA starts from an observation: the change a fine-tune makes to a weight matrix has much lower intrinsic rank than the matrix itself. So rather than learning the full update, learn a low-rank factorisation of it.

# W is frozen. Only A and B train.
# W: [d, k]   A: [d, r]   B: [r, k]   with r << min(d, k)

h = x @ W + (x @ A @ B) * (alpha / r)

# For d = k = 4096 and r = 16:
#   full update  = 16.8M parameters
#   LoRA update  =  0.13M parameters   (~0.8%)
# B is initialised to zero, so the adapter starts as a no-op
# and the model begins exactly as the base model.

Three consequences follow, and the third is the one that changes architecture decisions.

Memory drops enormously, because optimiser state is only needed for the adapter. Training is faster. And the adapter is tiny and separable — a few megabytes — so you can keep many task-specific adapters over one base model and swap them per request, rather than hosting many full models. For a multi-tenant product that is the difference between feasible and not.

QLoRA goes further: quantise the frozen base model to 4-bit while training the adapter in higher precision. The base is never updated, so quantisation error does not accumulate through training, and the memory saving is large enough to fine-tune substantial models on a single consumer GPU.

Practical notes: r between 8 and 64 covers most cases, and larger is not reliably better — it mostly increases the capacity to overfit. Apply adapters to the attention projections at minimum; including the feed-forward layers helps on harder adaptations at proportional cost. And alpha is a scaling factor conventionally set to twice r, which matters more than its obscurity suggests.

Full fine-tuning still wins when you are teaching a genuinely large behavioural shift, or adapting to a domain far from the pretraining distribution. For the format-and-style adaptations most products need, LoRA is usually indistinguishable in quality and vastly cheaper.

Preference tuning: DPO and RLHF

Supervised fine-tuning teaches the model to imitate good outputs. It has no way to express that one output is better than another, which is exactly the signal you have when a human compares two responses.

RLHF handles this in two stages. Train a reward model to predict which of two responses a human preferred, then optimise the policy against that reward with reinforcement learning, with a KL penalty holding it close to the reference model so it does not collapse into whatever degenerate output the reward model overrates.

It works and it is operationally demanding: four models in memory, sensitive hyperparameters, and genuine instability.

DPO observes that for this specific objective there is a closed-form relationship between the optimal policy and the implied reward, which lets you rewrite the whole thing as a classification loss directly on preference pairs.

# DPO: no reward model, no RL loop, no sampling during training.
# Raise the likelihood of the chosen response relative to the
# rejected one, measured against a frozen reference model.

loss = -log_sigmoid(
    beta * ((logp_chosen  - ref_logp_chosen)
          - (logp_rejected - ref_logp_rejected))
)

That is why it was adopted so quickly: it drops into an existing supervised pipeline. You need a reference model and preference pairs, and nothing else changes.

What you give up is real, though usually acceptable. A reward model can score arbitrary new outputs and be reused across projects. Online RL can explore beyond the preference dataset, while DPO is bounded by the pairs you collected. And DPO is sensitive to beta and can overfit to the preference data, drifting further from the reference than intended.

The practical sequence is almost always: SFT first to establish the behaviour, then preference tuning to refine it. Preference tuning on a base model that has not been taught the task is trying to refine something that does not exist yet.

The dataset is the project

The modelling is a few days. The data is the rest, and it determines the outcome almost entirely.

Quality dominates quantity, and the effect is not subtle. A carefully curated set of a thousand examples routinely beats fifty thousand scraped ones. This is genuinely counterintuitive to people arriving from classical ML, where more data is nearly always better. The difference is that a pretrained model already has the capability; you are selecting for a behaviour, not teaching a function from scratch, and every bad example teaches a bad behaviour with full force.

Consistency matters more than volume. If your examples disagree with each other — two of them format dates differently, three refuse a request the others answer — you are teaching the model to be inconsistent, and it will learn that faithfully. Write the annotation guideline first, then label, then audit a sample against the guideline.

Cover the distribution, including the awkward parts. A dataset of only clean, easy cases produces a model that handles clean, easy cases. Deliberately include the ambiguous inputs, the ones where the right answer is to ask a clarifying question, and the ones that should be refused.

Hold out a real test set before you start, split by whatever grouping matters — document, customer, time period — so that near-duplicates cannot straddle the boundary. And keep it untouched, because the temptation to look at it while iterating is strong and it destroys the only unbiased number you have.

On synthetic data: useful for augmentation and coverage of rare cases, dangerous as the bulk of a dataset. Generating training data with a model and training on it concentrates that model's biases and failure modes, and the resulting evaluation looks fine because your eval set is probably synthetic too. Use it to fill gaps you have identified, with human review, not to avoid the labelling work.

Catastrophic forgetting

Training on a narrow distribution degrades capabilities outside it. Fine-tune hard on legal document summarisation and the model gets worse at general conversation, at following unrelated instructions, and sometimes at basic reasoning.

It is not a bug; it is what optimisation does when you point it at a narrow objective. The model has finite capacity allocated by the gradient, and your gradient only cares about your task.

Mitigations, in order of practicality:

Train less. Fewer epochs, lower learning rate. Most forgetting comes from overtraining on a small set, and the cure is frequently just stopping earlier.

Use LoRA. Constraining the update to a low-rank adapter limits how far the model can move, which limits what it can lose. This is an underrated reason to prefer LoRA beyond the memory savings.

Mix in general data. Include a fraction of general instruction-following examples alongside your task data, so the gradient has a reason to preserve the general capability.

Measure it. This is the important one. Keep a small general-capability evaluation alongside your task evaluation and run both. Without it, forgetting is invisible during development and shows up as user complaints about behaviour you never tested — because you were only ever testing the thing you optimised.

Evaluating a fine-tune honestly

The comparison that means something is against the strongest alternative you could have shipped instead, not against a weak baseline.

Compare against a well-engineered prompt on the base model. Not a lazy prompt. If your fine-tune beats a bad prompt, you have learned nothing, and this is the most common way fine-tune results are inflated — usually unintentionally, because the prompt baseline was written in an hour and the fine-tune took three weeks.

Compare against few-shot prompting with examples drawn from the same training set. This is frequently the real competitor and it frequently wins on formatting tasks.

Compare against a larger base model without fine-tuning. Sometimes a bigger model with a good prompt beats your fine-tuned smaller one, at comparable cost and with none of the maintenance burden.

Then measure the things the task metric will not show you:

General capability, to catch forgetting. Refusal behaviour, because fine-tuning on task data frequently erodes safety training and this is a real risk rather than a theoretical one. Robustness to input variation, since a model trained on uniformly formatted inputs may be brittle to real ones. Calibration, if anything downstream uses confidence. And serving cost, including whether you can still batch efficiently with per-tenant adapters.

The costs nobody budgets

The training run is the cheap part and the only part that appears in most estimates.

Data labelling. Usually the dominant cost, usually underestimated by a large factor, and it recurs every time the task definition shifts.

The evaluation set. You need one regardless, but a fine-tune makes it mandatory and makes it need to be broader — task performance plus general capability plus safety.

Serving. A custom model may not fit the batching and caching your hosted endpoint gave you for free. Per-tenant adapters complicate batching further. This shows up as a bill, not as a project.

Base model deprecation. The model you fine-tuned will be superseded. When it is, you redo the fine-tune, re-evaluate, and re-deploy — and if your data pipeline was a notebook, you rebuild that too. Plan for it as a recurring obligation rather than a surprise.

Being one version behind, permanently. This is the strategic cost and it is the one that actually matters. Base models improve quickly, and each improvement is available to a prompting-based competitor immediately and to you after a re-training cycle. Over a couple of years that gap compounds.

None of which means do not fine-tune. It means fine-tune when the behaviour genuinely cannot be obtained otherwise, with a decision you can defend and an evaluation that would have told you not to.