It Can Do the Maths. It Cannot Show Its Working: Why Actuarial Computation Belongs in Code

A companion note to the series NIST AI RMF for Actuaries

Mathematical ability in these systems is real and was never programmed. It is also unauditable. Those two facts together, not either alone, are what determine how the technology should be used in regulated work.

That matters here because Part 4 of this series left a question open. It set out the controls for generative AI in actuarial work (retrieval against an authoritative corpus, structured prompting, eval suites, audit logs and human review), and it drew a hard line under code generation: every LLM-written artefact reviewed line by line, run against test data with known expected outputs, and treated as if a junior analyst had written it, because in effect that is exactly what has happened. What it did not answer is the prior question. Why route the computation through code at all, when the model can evidently do the arithmetic itself?

This article answers that question. The answer is not the one most people expect, because it does not rest on the model being poor at mathematics. It holds if the model is excellent at mathematics.

Three of the figures below are interactive, and they carry the argument rather than decorate it. The first compares the routes a number can take on its way into a report. The second prices the arithmetic. The third takes a small network apart, so you can break the mechanism and watch the answer fail.

A note on registers

As in Parts 1, 2, 3 and 4 of this series, this article distinguishes law, supervisory expectation, professional standard and author recommendation. This piece sits further towards the last of those than any other article in the series. No framework in force anywhere prohibits an actuary from asking a language model for a number. The position below is our own practitioner judgement about what makes actuarial work reviewable, and about what the professional standards already require of anything that supports an actuarial conclusion. Where a professional standard does the work, it is named.

The position in five claims

Claim 1: Capability

Large language models develop mathematical capability that was never programmed into them. Through training, they acquire emergent internal procedures: compressed representations that generalise, rather than stored answers retrieved from memory. The behavioural evidence is settled. In July 2025 a frontier model produced natural-language solutions to five of six International Mathematical Olympiad problems written after its training concluded, inside the time limit and without tools, and was officially graded at gold-medal standard. That rules out stored answers. What it does not reveal is mechanism. The evidence that genuine procedures form inside these systems comes from models small enough to take apart, one of which is dissected below.

Claim 2: Opacity

Those procedures, however, are not delivered with the answer. A model can print working, including equations, steps and an apparent method, but that text is one more output rather than a trace of the computation that produced it. Models asked to explain their arithmetic have been observed describing schoolbook carrying while mechanistically doing something else entirely. It can show a working. It cannot show its working.

Nor does inspection close the gap. The network dissected below was reverse-engineered precisely because it is a toy. No frontier model has been characterised at anything approaching the coverage certification requires, and boundary testing samples behaviour rather than bounding it, which an open-ended domain would need. A bounded, controlled workflow can be validated empirically. What no current method can validate is the unrestricted proposition: let the model do the mathematics directly, and trust the answer.

Claim 3: Recommendation

Our position is therefore that mathematical work is executed through generated code rather than by the model directly. Code is not automatically reproducible. Reproducibility is earned, through pinned dependencies, recorded environments and controlled execution, and it can be earned with discipline the profession already owns and can itself audit. More importantly, code is reviewable, testable and capable of being owned. A qualified individual can examine the basis, satisfy themselves it is correct, reproduce the run, and sign. No one can be accountable for a result that offers nothing to examine.

Claim 4: Durability

This recommendation does not rest on the model being poor at mathematics. It holds if the model is excellent at mathematics. The constraint is assurance rather than capability, which is why it will remain valid as capability continues to advance.

Claim 5: Efficiency

Nor does it rest on scarcity of capability elsewhere. Worked through a reasoning chain, a single addition consumes on the order of ten trillion primitive operations, against the handful its replacement instruction would cost. Configuration moves that figure, and sparsity, quantisation and smaller models claw back orders of magnitude. Nothing brings it within sight of the instruction, because a general-purpose generative network is the wrong execution engine for arithmetic the hardware already implements directly. Delegating computation is not a stopgap pending better models. It is the correct decomposition, and it would remain correct if reliability were solved tomorrow.

The argument does not depend on what the model cannot do. It depends on what it cannot show.

Where the number comes from

All three routes below can produce a correct answer. They differ entirely in what they leave behind for someone else to examine.

For actuarial work the second route is the default and the third is the exception worth knowing about. Reserving, pricing, capital projection and scheme valuation almost never offer a cheap check, because confirming that a reserve is appropriate is as hard as producing it, which is precisely why the profession built a review standard rather than a checking algorithm. Where a cheap and complete check does exist, such as a root that substitutes back, a factorisation that multiplies out, a duality certificate that bounds an optimum or a reconciliation that must foot, an unreliable generator is entirely legitimate and the gate does the assurance work. Everywhere else, the artefact is the assurance.

Why the standard is characterised error, not exactness

Almost nothing in production numerics is exact, and no engineer expects it to be. What every usable method supplies is a known bound, a known failure mode, and an error you can budget for. Judged on that standard rather than on exactness, the gap is not narrow.

Floating-point addition is not even associative, so the same valuation can differ across thread counts, and Monte Carlo is stochastic by design. Neither is disqualified, because in both cases the error is characterised under stated conditions and enters a budget. Nor is the model’s direct answer without signals: token confidence, agreement across repeated samples, and selective and conformal methods offering risk control under distributional assumptions. The difficulty is that those assumptions are exactly what deployment breaks, and none of it amounts to a validated bound on the answer in front of you. The disqualification is not inexactness. It is uncharacterised error.

That distinction is already the actuarial one. A stochastic capital model is accepted not because it is exact but because its error is characterised, its convergence is monitored, and the residual uncertainty is disclosed. ASOP 56 §3.6.2 asks for output validation appropriate to the model’s intended purpose, and ASOP 41 asks that the communication be sufficient for another qualified actuary to assess the work. A row without a bound, without a run-time signal and without a budget is not a row you can validate or disclose against. It is the only row in the table that fails on those terms, and it fails on them whether or not the number happens to be right.

What the computation actually costs

Set reliability aside entirely and assume a model that computes flawlessly. It should still not be doing the arithmetic, and the reason is built into the very hardware running it. A forward pass executes on floating-point multiply-add units, so the model performs trillions of exact additions in order to emulate one, unreliably. The adder is not nearby. It is the substrate.

These are illustrative operation counts rather than measurements of energy, latency or price. A dense forward pass costs roughly twice the active parameter count in floating-point operations, and arithmetic worked through a reasoning chain spends one such pass per token. Sparsity, quantisation and smaller models move the figure by orders of magnitude, as the chart itself shows. Nothing moves it within sight of the instruction, which is why delegating computation is an equilibrium rather than a workaround.

What this means in an actuarial function

The practical rule is a single sentence, and it is narrower than “do not use AI for the numbers”. The model may choose the method, write the code, interpret the output and draft the explanation. The number itself comes from the code. Everything the model is genuinely good at sits on the near side of that line, and the one thing it cannot evidence sits on the far side.

Read against the framework, the rule is mostly a MEASURE and MANAGE convenience. A generated-and-reviewed script gives the evidence pack something to contain: the artefact, its tests, its inputs, its environment, its reviewer and its run. A direct answer gives it a transcript, which records what was said rather than how it was computed. Against the professional standards the point is sharper still. ASOP 56 §3.4 asks the actuary to make a reasonable attempt at a basic understanding of a model developed by others. Sections 3.6.1 and 3.6.2 ask for model testing and output validation, and §3.7 asks for documentation. ASOP 41 asks that the communication let another qualified actuary assess the work. TAS 100 asks for the same substance in the UK vocabulary, and APS X2 v1.1 (effective 30 January 2026) governs the review of it. None of these standards mentions language models, and none of them needs to. Each of them presumes there is something to examine.

In our experience the following four things separate a team that can defend this from a team that cannot. None of them is exotic, and all of them are cheap if they are set up before the first production use rather than after the first challenge.

• The artefact, not the answer, is the deliverable. Store the script, its inputs, its outputs and its test results together. If the only surviving record of a calculation is a number in a memo and a chat transcript, the calculation has not been documented, whatever the transcript contains.
• Pin what makes it reproducible. Dependency versions, random seeds, data snapshots and environment. Reproducibility is earned rather than inherited, and it is the cheapest evidence in the pack to produce and the most expensive to reconstruct later.
• Test against known answers. Boundary cases, a hand-worked example, a prior period that must reconcile. LLM-written code that runs but is silently wrong is the dominant failure mode in this category, and it is the one a casual review will not catch.
• Name the reviewer, and review the method rather than the syntax. Specification error survives every route in the figure above. The risk moves from arithmetic to method selection, which is exactly where qualified judgement is needed and exactly where it cannot be automated away.

One caution, so that this is not read as more than it is. Nothing above bans a model from doing arithmetic in exploratory work, in scoping, in a sense-check, or anywhere else the output is not carrying an actuarial conclusion. The line is the same one the American Academy of Actuaries drew in its 2024 professionalism discussion paper: incidental use sits outside scope, and work supporting an actuarial conclusion sits inside it. Inside that boundary this is our recommendation. Outside it, the tool should be used freely.

Worked example: how a network invents arithmetic

Researchers trained a small transformer on modular addition and reverse-engineered what it had learned. It had not memorised a table, and it was not carrying digits. It had found a method. Here it is, on a twelve-hour clock rather than the 113 used in the original work.

On the real experiment

The original result used addition modulo 113 in a one-layer transformer. Applying a discrete Fourier transform to the trained embedding matrix showed the weight concentrated on roughly five frequencies out of 113. The network had chosen a small basis and discarded the rest. Predicted outputs matched the trigonometric formula. Ablating the key frequencies collapsed performance to chance, while ablating the others changed nothing.

On how far this generalises

This is a small model on a narrow task, reverse-engineered because it is small enough to be, and its very cleanliness is probably what makes it unrepresentative. Work on arithmetic in larger models suggests something considerably messier: not one unified algorithm but a collection of overlapping heuristic components, each firing on a different slice of the input, collectively producing behaviour that looks algorithmic without being invariant the way an algorithm is. Change the names and numbers in a word problem and accuracy moves, which a real procedure would not permit. What the result above establishes is only the existence claim, that training on next-token prediction can and does produce genuine algorithms rather than lookup tables. Later work found the same insight can be implemented more than one way, so this is a discoverable elegant method rather than the only possible one.

On the word “emergent”

Used here in the structural sense: arising from training without being designed or specified. Not in the contested sense of a capability appearing abruptly at a parameter threshold.

Sources

1. Google DeepMind, “Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad”, 21 July 2025. Reports in July 2026 that several frontier models achieved perfect scores at that year’s olympiad are credible but were not officially graded at the time of writing; this citation is the officially graded result.
2. Anthropic, “On the Biology of a Large Language Model”, Transformer Circuits Thread, 2025. Includes the finding that a model describes longhand carrying while mechanistically combining parallel approximate-magnitude and exact-digit pathways.
3. Nanda et al., “Progress Measures for Grokking via Mechanistic Interpretability”, ICLR 2023.
4. Nikankin et al., “Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics”, 2024.
5. Gond et al., “LLM-42: Enabling Determinism in LLM Inference with Verified Speculation”, arXiv:2601.17768, 2026.
6. He and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference”, 2025.
7. Horowitz, “Computing’s Energy Problem (and What We Can Do About It)”, ISSCC 2014. Cited for the per-operation energy scale of hardware arithmetic.
8. NIST, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), 26 July 2024. Cited for the confabulation risk category and its suggested actions.