The Actuary's Local LLM, Part 2: Fine-Tuning When Prompting Is Not Enough

Part 1 of this pair made the case for running a local model on hardware you control, and left one question open: what do you do when prompting is not enough? The answer is fine-tuning, and you should reach for it late, not early. Prompting is faster, reversible, and model-agnostic; exhaust it first.

Fine-tune when you hit one of three walls:

• Format discipline the prompt cannot quite enforce.
• Domain shorthand a general model only half-understands.
• Economics, because a small model fine-tuned on your narrow task can match a much larger prompted one while running faster on smaller hardware.

Everything below is measured, from one run on an ordinary consumer machine, and every script is in the companion repository.

A note on registers

This article borrows the four categories of authority introduced in our NIST AI RMF for Actuaries series: law (binding regulation or statute), supervisory expectation (regulator guidance, often “should” framing, used in examinations), professional standard (binding within a profession on its members), and author recommendation (our own practitioner judgement). The measured figures below are observations from a single run and are reported as such. The recommendation to trade auditability deliberately, and to version the base model, the dataset and the adapter together, is author recommendation resting on professional standard: the documentation obligations that follow model modification are named where they apply.

What fine-tuning does not do

Three misconceptions send teams to fine-tuning for the wrong reasons.

If your labels are inconsistent, training on them converts a prompt problem into a weights problem, which is strictly worse. A prompt can be edited in a text file and reviewed like any other change. An adapter has to be retrained and re-evaluated. In practice the commonest cause of a disappointing accuracy figure is an ambiguous rulebook rather than a weak model, so fix the labelling rules before you touch a training script.

What QLoRA actually does

Modern fine-tuning is not retraining. With QLoRA the base model stays frozen in 4-bit form and you train small low-rank adapter matrices alongside it. On this run that meant 33,030,144 trainable parameters, which is 0.82% of the 4.02 billion in the Qwen3 4B base, producing an adapter of 127MB and a merged 4-bit model of 2.4GB that Ollama serves like any other.

The training itself took 99 seconds across 60 optimiser steps, inside about 8GB of GPU memory. Ninety-nine seconds matters because the prevailing mental model of training is weeks and a cluster, and that model makes people dismiss the technique before costing it. On a narrow classification task with a few hundred examples, fine-tuning is a coffee break. The expensive part of this exercise was never the computation. It was writing the rulebook and labelling the data, which is precisely the part an actuary is qualified to do and cannot delegate.

The whole recipe is five commands:

pip install -r finetune/requirements.txt
python finetune/01_prepare_dataset.py
python finetune/02_finetune_qlora.py
ollama create claims-classifier -f finetune/outputs/gguf/Modelfile
python finetune/03_evaluate.py --models qwen3:8b claims-classifier

The task comes from general insurance: classifying free-text first-notification and adjuster notes into five reserving categories, motor bodily injury, motor property damage, escape of water, fire, and third-party liability injury. The labelling rules are reserving positions expressed as data rules. Injury outranks damage, because any injury, however minor, makes a motor claim a bodily injury claim, and bodily injury develops and reserves differently; misclassify it and the tail is understated. Origin governs, so water used to extinguish a fire is a fire loss. A fifth of the 200 synthetic notes are deliberately awkward on exactly these boundaries, and of the 200, 160 train the model and 40 are held out, stratified eight per class and never seen in training.

The result, measured

The evaluation puts the prompted 8B baseline and the fine-tuned 4B side by side on the same 40 unseen notes. The asymmetry is the point of the exercise. The baseline receives the full rulebook prompt; the fine-tuned model receives a one-line instruction, because the rules now live in the weights.

A draw, and the draw is the finding, because the two models are not paying the same price for it. The fine-tuned model is half the size, 2.5GB served against 5.2GB. Its system prompt is 74% shorter, saving 106 tokens on every single call. Neither model produced a single unparseable reply.

“The draw is the finding, because the two models are not paying the same price for it.”

Watch that invalid-output count as closely as the accuracy. It is the cheapest signal you have that something has changed underneath the model, and a forgiving parser, one that rescues labels from malformed replies, drives it to zero and destroys the signal along with the format discipline the fine-tune exists to deliver. The measurement discipline argued for in the confabulation piece is not paperwork; it is what tells you a model is broken while it is still cheap to find out.

Reading 95% honestly

A bare 95% invites more confidence than it has earned.

Forty notes is a small sample. The Wilson 95% interval around 38 out of 40 runs from 83.5% to 98.6%, more than fifteen percentage points wide. Both models landing on the same score is consistent with parity and is nowhere close to evidence of it, particularly since each made one error the other did not. The test set is also balanced at eight notes per class, which no real book is, and the notes are synthetic.

Quote the interval or quote nothing, and expect both accuracy and format-failure rates to move on real data.

The errors are more interesting than the score

The 8B model missed the boundary between a motor injury claim and a liability injury claim, in both directions. That is the genuinely ambiguous case in the dataset.

The fine-tuned model missed that boundary once too, but its other error was a fire loss classified as water damage, a failure of the origin-governs rule specifically. That is a diagnosis rather than a mystery, and it points at a targeted remedy. Write a dozen more extinguishment cases and retrain, rather than collecting more notes in general. When your evaluation is built on documented labelling rules, error analysis stops being an autopsy and becomes a work order.

Fine-tuning spends auditability

That 74% prompt reduction carries a trade that is not only economic.

When the rules live in a prompt, they are a text file. Readable by a reviewer, diffable, versionable, and changeable in a minute by someone who has never trained anything. When they live in the weights, they are free at inference and invisible to inspection. You cannot read an adapter to find out what it believes about sprinkler damage; you can only test it. Fine-tuning therefore buys speed and cost by spending auditability, and for regulated work that is a trade to make deliberately and document, not a free optimisation.

“You cannot read an adapter to find out what it believes; you can only test it.”

Once you have trained an adapter you are no longer merely a user of somebody else’s model; you are a modifier of one, and the documentation burden moves accordingly. An adapter is also meaningless in isolation. It reproduces only alongside the exact base model and the exact dataset that produced it, so those three artefacts have to be versioned together or the result is unreviewable. The repository’s model card template pins all three, and the bridge from ASOP 56 to NIST AI RMF shows where that evidence sits in a governance framework. Adopt the discipline on the first adapter you train rather than the tenth.

Built to be learned from

The repository is built to be learned from rather than merely cloned. Every numbered script pairs with an explainer written for actuaries rather than for developers: what the code does, why each design choice was made, how to read the output, and where it falls short. The fine-tuning explainer goes deepest: the hyperparameters and what each one trades, the measured training behaviour, and the failure modes the evaluation is designed to catch. Working through it beside the scripts is the fastest route from reading about the technique to holding a defensible opinion on it.

The dataset generator is committed alongside the data, so provenance is demonstrable rather than claimed.

The governance folder is where fine-tuning meets its obligations in running code. The machine-readable use case declaration that Part 1’s wrapper enforces carries a list of approved models with their digests, and enrolling a newly trained adapter on that list is a deliberate human act standing in for committee sign-off: new weights mean a new digest, and an unapproved digest is refused before any tokens are spent. The sign-off gate and the hash-chained audit log then apply to your fine-tuned model exactly as they do to the base model. Auditability spent in the weights is partly bought back in the surrounding system, and the governance README explains each mechanism and, just as usefully, names what it does not cover.

Try it on synthetic data first

The repository carries the full recipe as scripts and as a free-tier Colab notebook, with one caution that is Part 1’s thesis applied to training. Hosted notebooks are acceptable here only because every training example is synthetic. The moment the training set is real, training happens on hardware you control.

Start with the committed dataset and reproduce the result above. Then write the rulebook for a narrow task of your own, label a few hundred examples, and hold out a test set before you train anything. The rulebook is the work; the training is the coffee break.