A practitioner’s companion for local LLMs in actuarial work
Running, using, and fine-tuning local large language models for actuarial work. Everything here runs on your own hardware. Nothing you type, paste, or train on leaves your machine.
It comes in two parts. Part 1, which you are reading, gets your local LLM running. Part 2 fine-tunes it, for the day prompting is not enough.
A full Git repository accompanies both, and it carries everything in these articles in more detail: install scripts, examples you can run, and a full breakdown of what is happening at each step. Clone it and follow along.
In an earlier piece on the NIST Generative AI Profile I argued that confabulation is a property of large language models rather than a defect awaiting a patch, and that the answer is controls. Retrieval narrows what a model can draw on, citations make its claims checkable, validation catches the failures, and measurement tells you whether any of it is working. All of that concerns trusting the outputs.
This piece is about the other half of the trust problem, which is trusting the channel. A local LLM, an open-weight model running entirely on hardware you control, dissolves a category of risk that no vendor contract can fully discharge.
A note on registers
This article borrows the four categories of authority introduced in our NIST AI RMF for Actuaries series: law (binding regulation or statute), supervisory expectation (regulator guidance, often “should” framing, used in examinations), professional standard (binding within a profession on its members), and author recommendation (our own practitioner judgement). No framework in force anywhere requires a local deployment, and none prohibits a hosted one. The hardware tiers, model choices and setup steps below are practitioner judgement. The governance section at the end is where professional standards do the work, and they are named there.
Why some work should not leave the building
Some actuarial work should not leave organisational control, whatever a vendor’s terms say.
Work that belongs inside the boundary
- Claim narratives carry personal information.
- Experience-study commentary and draft reserving methodology are commercially sensitive.
- Some material sits under data-residency or client confidentiality obligations that no service agreement fully discharges.
For that work, “is the vendor trustworthy?” is the wrong question. The better one is “why is this leaving the building at all?”
A local model dissolves the question. The weights run on your machine, the prompt never crosses the network, and the trust boundary you have to defend is the one you already defend for every other confidential system. You also get cost predictability (hardware you own rather than tokens you meter), offline access, and something underrated: the learning value of seeing how these systems behave when you hold every dial.
“The trust boundary you have to defend is the one you already defend for every other confidential system.”
What a local LLM setup actually is
A local setup is four layers:
- A model. Open-weight families such as Qwen, Llama, Gemma, and DeepSeek, downloadable and licensed for local use.
- A runtime to execute it. Ollama, LM Studio, or llama.cpp.
- An interface. A chat window, a command line, or your own Python.
- The hardware underneath.
The install guide in the companion repository covers all four in detail, with hardware tiers and troubleshooting.
Hardware comes down to memory
There are two viable paths. Apple Silicon, where unified memory lets surprisingly large models run on a well-specified laptop, and discrete NVIDIA GPUs, where video memory is the constraint and the tooling ecosystem is deepest.
| Tier | Typical machine | Runs comfortably |
|---|---|---|
| Entry | 16GB laptop, no discrete GPU | 4B-class models, quantised. Fine for learning, at reduced speed |
| Anchor | Mac with 32 to 64GB unified memory, or a Windows or Linux machine with a 16 to 24GB GPU | 8B to 14B-class models, comfortably. Everything in the companion repository |
| Workstation | 96GB and above unified memory, or multi-GPU | 30B-class mixture-of-experts models and beyond |
The anchor tier is a believable corporate refresh, not an exotic rig. Three model dimensions determine what fits: parameter count, architecture (dense versus mixture-of-experts, the latter activating only a fraction of its weights per token), and quantisation, the storage precision of the weights. Four-bit quantisation cuts a model’s memory footprint to roughly a quarter of full precision with modest quality loss, and is the practical default. Choose by memory first, then by task.
One correction to the usual advice. The memory figure that matters is not what is installed, it is what is free. A routine desktop of browser tabs, a chat client, an office suite and video-call tooling can quietly take 7.7GB of an 8GB card, leaving 244MB; every model load then fails, and nothing in the error explains why. Treat video memory as a shared resource with other claimants, check it before you start, and read every figure in the table above as memory free, not memory fitted.
The thirty-minute setup
Install Ollama: a single installer on macOS, Windows, or Linux. Pull a model; I recommend Qwen3 8B, about 5GB at 4-bit, Apache 2.0 licensed, and a strong instruction-follower. Then ollama run qwen3:8b gives you a chat in your terminal. Unplug the network and ask it something, purely to enjoy the point.
Then comes the line that matters for practitioners. Ollama exposes an OpenAI-compatible API on localhost, which means the Python you would write against a cloud endpoint runs unchanged against your own machine. Change the base URL; keep everything else:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="qwen3:8b", temperature=0, seed=42,
messages=[{"role": "user",
"content": "Explain IBNR in three sentences."}])Temperature zero and a fixed seed reduce run-to-run variation. They do not eliminate it across hardware, runtimes, and model builds, which is why every example in the companion repository logs its calls. The log, not the settings, is what makes a run auditable.
The api_key is a placeholder: the client library refuses to be constructed without one, and Ollama ignores whatever you pass. There is no secret in this code path, nothing to rotate and nothing to leak into a repository. It is also a caution, because it means the local endpoint has no authentication of its own. Bind it to anything other than localhost and anyone who can reach the port can use your model. That is a network decision rather than an application one, and it is the one place where “it runs locally” can quietly stop being true.
“The log, not the settings, is what makes a run auditable.”
What you learn by holding every dial
When you run the model yourself you own every layer of it, and the layers stop being abstractions. You choose the quantisation and see the quality cost. You watch memory fill and learn what a context window physically is. You change a sampling temperature and watch determinism arrive.
Practitioners who have only ever called a hosted endpoint tend to reason about these systems as though they were oracles with a price list. Practitioners who have run one locally reason about them as software with resource constraints and failure modes, which is much closer to the truth and much more useful in a governance conversation.
Four scripts you can run this afternoon
The companion repository carries four worked examples, each paired with an explainer written for actuaries rather than for developers, covering the design choices, how to read the output, and where the approach falls short.
Example 01 is the bridge above: proof that cloud code runs unchanged against localhost.
Example 02 classifies free-text first-notification and adjuster notes into five reserving categories: motor bodily injury, motor property damage, escape of water, fire, and third-party liability injury. The pattern is the product: a strict output contract of exactly one label from a fixed list, validation in code that accepts the label or flags the response as invalid, and measurement against a labelled sample. The model proposes; your code disposes. The repository ships 200 synthetic notes, a fifth of them deliberately awkward: injury surfacing late in a damage-led motor note, flood water inside a motor claim, sprinkler water damage inside a fire loss. That is what real notes look like, and it is why the validation exists.
Example 03 answers questions over a term assurance valuation methodology document, entirely locally. It chunks the document by section, embeds the chunks with a local embedding model, retrieves the most relevant sections, and requires the model to answer only from the retrieved text, cite section numbers, and say plainly when the document does not cover the question. A closed world, checkable claims, and a legitimate exit instead of an invented answer. Resolving citations in code turns “is this true?” into the faster and more reliable question “does the citation support this?”, which surfaces confabulation directly. This is “ask questions of the confidential documents on your own machine”, in about 150 lines of Python with no framework.
Example 04 is built on the distinction between a system that logs and a system that can refuse: a log is written after the fact and asserts what happened, while a control is consulted before the fact and can stop the call. The wrapper does both. The model digest is pinned to an approved build and a mismatch is refused before any tokens are spent, output is required as structured JSON with every citation resolved in code, each audit record chains to the one before so a casually edited log stops verifying, and nothing is usable until a named reviewer signs off. It reads the repository’s machine-readable use case declaration at run time, so the governance position is enforced rather than asserted.
An 8B local model is not a frontier model, and it will not match one at open-ended reasoning. Computation does not belong to it either; that argument is made in full elsewhere. For narrow, well-specified language tasks the gap closes sharply. Those are precisely the tasks worth automating.
When a local LLM is the right tool, and when cloud is
The heuristic I use:
- Cloud for capability, on non-sensitive work.
- Local where confidentiality, cost predictability, or auditability dominates.
- Hybrid for much of the middle: draft locally, refine in the cloud on de-identified material.
The governance position does not change
Local deployment solves the channel problem, not the governance problem. Model governance obligations follow the model wherever it runs; for US actuaries, ASOP 56’s expectations around understanding, testing, and documenting the models you rely on do not care whose electricity the model uses. The bridge from ASOP 56 to NIST AI RMF covers that mapping in full.
The repository’s governance folder is the small, concrete version of the principle. A machine-readable use case declaration that example 04 enforces at run time, a prompt log schema that hashes every call, and a model card template. The declaration also names what it does not cover, which is the honest form of scope. None of it is paperwork for its own sake; all of it is what lets a reviewer reconstruct what happened.
Start on a quiet afternoon
Everything above is reproducible from github.com/globebyte/local-llm-actuary. Install, pull, one first query. Work on non-confidential examples first, verify outputs against source, and then decide what the trust boundary means for your own work.
When prompting stops being enough, Part 2 covers fine-tuning a small model on your own rulebook, with measured results and the auditability trade priced honestly.
Ready to explore confidential AI for your organisation?
Talk to our team about how Globebyte can help you evaluate, deploy, and govern local models for regulated work. From the trust boundary decision to the audit log that survives review.