Source-linked AI summary
Towards a Deterministic Math Solver for Clinical Language Models
Felipe Ocampo Osorio, Sebastián Andrés Cajas Ordoñez, Maximin Lange, Rafi Al Attrach, Sahil Kapadia, Zakaria Laouabdia Sellami, Angelo Antonio Talio, Leo Anthony Celi
TL;DR
Clinical language models can lose accuracy on calculator arithmetic, motivating alternatives to one validated function per calculator. This paper tests case-specific Python executed by a restricted local solver against direct arithmetic and a partial hand-written library. Program-Solve is not a reliable advantage at 7B but is one at 32B under matched access, while library coverage and formula validity remain decisive constraints.
Problem
Language models are unreliable at arithmetic for clinical calculators, while hand-written libraries require separate validated functions and leave unsupported cases unanswered.
Method
Program-Solve has the model write case-specific Python for a restricted local executor and is evaluated against direct arithmetic and a 22-calculator library.
Results
At matched formula, variable, and note access, Program-Solve is not reliably better at 7B but is better at 32B; its library advantage mainly comes from answering cases where the library abstains.
Takeaways & Limitations
A verified library is preferred where available, with program generation where it is not and explicit abstention where neither approach is trustworthy.
Takeaways & Limitations
The benchmark's gold answers validate neither program logic nor formula correctness, and an audit flagged 16 of 55 calculators with version, use, or coefficient concerns.
Abstract
from arXiv · showhide
Large language models are unreliable at arithmetic, which is a problem for clinical calculators where a single numerical error changes the recommendation. The standard response is to hardcode each calculator as a validated function, one at a time. We test an alternative: the model does not calculate. Instead, it writes case-specific Python that a restricted local executor runs as a deterministic solver, and the model's task reduces to deciding how to use it. We evaluate this Program-Solve interface on MedCalc-Bench Verified (1,100 cases, 55 calculators) against direct model arithmetic and a hand-written 22-calculator library, using Qwen2.5-7B and Qwen2.5-32B-AWQ, after auditing the benchmark's formulas against current clinical guidelines and flagging 16 of 55 with version, use or coefficient concerns. With formulas and gold variables supplied and both routes reading the whole note, handing off to the solver is not a reliable advantage at 7B (75.31% against 72.02%, a paired +3.29 points with a 95% calculator-cluster interval of [-3.49, 10.38]) but is one at 32B (90.53% against 83.47%, +7.05 [0.47, 14.60], clear of zero). The hand-written library is exact on its 440 supported cases but abstains elsewhere (40.0% overall). Adding an executor thus helps some open-weight models more than others even under matched formula, variable and note access, and is not a substitute for verified formulas or reliable variable extraction either way.
1 Introduction
Clinical calculator automation requires formula selection, variable extraction, and arithmetic, but language models often lose accuracy on these tasks. This paper evaluates replacing calculator-specific functions with case-specific programs executed by a restricted local solver.
- Clinical calculator automation combines formula selection, free-text variable extraction, and arithmetic, all of which can reduce model accuracy.Prior work reported incorrect answers in about one third of unaided ChatGPT trials across 48 calculation tasks.
- The standard remedy is a validated hand-written function for each calculator, but unsupported cases remain unanswerable.The evaluated 22-calculator library implements 440 of 1,100 cases, yielding 40.0% full-set accuracy under abstention.
- Program-Solve instead has the model write a short program from the case, formula, and variables for a restricted executor to run.The executor contains no calculator-specific functions or constants and returns the computed result.
- The study compares this general interface with direct arithmetic and a partial hand-written library across 55 calculators and 1,100 cases.The comparison also considers formula versioning, because benchmark formulas may differ from current clinical practice.
2 Method
The study evaluates two Qwen models on MedCalc-Bench Verified using matched calculator arms, a restricted executor, and calculator-cluster uncertainty intervals. The design separates formula access, variable access, note length, execution method, and library coverage.
- Data and models: The benchmark contains 1,100 cases across 55 calculators, evaluated with Qwen2.5-7B-Instruct and Qwen2.5-32B-Instruct-AWQ across five seeds.Cases were scored under benchmark-defined tolerance, with no test answer or explanation included in prompts.
- Arms and executor: Program-Solve receives the formula text and gold variables, writes a program, and uses an executor without calculator-specific functions; Blind Program-Solve extracts its own variables without those inputs.The executor uses a fresh subprocess, restricted builtins and imports, resource limits, and static rejection rules.
- Comparison design: Program-Solve and Open-book arithmetic are matched on formula access, variable access, and whole-note length, isolating the execution-method comparison.Other arms vary formula access, variable access, note budget, or use of the 22-calculator library.
- Statistics: Uncertainty is estimated primarily with a 10,000-draw calculator-cluster bootstrap, keeping seeds together because cases recur within calculators.Case bootstrap, exact McNemar tests, and sign-flip permutation tests are secondary analyses.
3 Results
Program-Solve is reliably better than matched Open-book arithmetic only for the 32B model, while library comparisons are dominated by coverage and hybrid routing can outperform individual arms. Removing formula and variable access causes large accuracy losses, and syntax additions provide limited evidence of improvement.
- 3.1 Matched execution and partial-library comparisons: +35.31 and +50.53 percentage points separated Program-Solve from Gold-Solve on the full set at 7B and 32B, mainly because the library abstained on 660 unsupported cases.On the 440 implemented cases, Gold-Solve was perfect while Program-Solve reached 84.20% and 98.64%.
- 3.1 Matched execution and partial-library comparisons: 81.64% and 91.07% were achieved by Gold-first hybrids, exceeding every single arm by using the exact library on its 440 cases and Program-Solve elsewhere.Open-book fallback hybrids reached 78.56% and 86.89%, while Extract-first hybrids performed substantially worse.
- 3.1 Matched execution and partial-library comparisons: 77.95% at 7B followed syntax additions, a +2.64-point change with an interval crossing zero; 32B moved only +0.44 points.The syntax lines were selected on the test split, making this an exploratory ablation.
- 3.2 Removing formula and gold-variable access: 28.04% at 7B and 44.71% at 32B were achieved by Blind Program-Solve, declines of 47.27 and 45.82 points from formula- and variable-supplied Program-Solve.Because formula and variable access changed together, the design cannot separate formula recall from variable extraction.
- 3.2 Removing formula and gold-variable access: Mistral’s code route trailed arithmetic by 7.6 points, while Phi-3.5’s led by 4.2 points, showing that execution effects varied across model families.Mistral’s Extract-Solve library reached 34.89%, above either of its Program-Solve arms.
4 Limitations
The study’s conclusions are bounded by limited model coverage, benchmark formula validity, and deployment settings that may not represent clinical use.
- Two Qwen checkpoints do not establish scaling, while Mistral-7B and Phi-3.5-mini move in opposite directions on the same comparison.
- A completeness audit found 28 of 55 calculator texts wrong or incomplete, including 10 that could not reproduce the benchmark number.
- The benchmark’s gold outputs validate neither program logic nor formula correctness; a literature audit flagged 16 calculators with version, use, or coefficient concerns.
- The evaluation covers English and US conventions, excluding Global South data, other languages, alternative laboratory units, and other date locales.
5 Conclusion
With formula, variable, and note access matched, program generation is not reliably better than arithmetic at 7B but is advantageous at 32B; verified libraries remain preferable where available.
- +7.1pp at 32B favored program generation, whereas its 7B advantage was not reliable under matched access.
- The program route’s advantage over the partial hand-written library mainly comes from answering cases where the library abstains; on shared cases, the library is more accurate.
- Mistral-7B and Phi-3.5-mini move in opposite directions, so the relevant pattern is not yet predictable from scale alone.
- The clinically defensible configuration is a verified library where available, program generation where it is not, and explicit abstention where neither is trusted.
A.1 Arm protocols and settings
The appendix defines single-call calculator arms that vary formula, variable, note, and execution access, with deterministic scoring and selected repair or voting variants.
- Each arm uses one chat request per turn, greedy decoding at temperature 0, and five seeds; sampled candidates use temperature 0.7.
- Note-only protocols vary note length and gold-variable access, while page and open-book arms add or withhold the supplied formula.
- Extract-Solve calls a hand-written calculator after extraction, whereas Formulate-Solve builds and exactly evaluates an expression tree, abstaining on invalid inputs or evaluation failures.
- Program-Solve writes code for an executor using supplied formula and variable information; blind variants add repair turns or multi-call voting after execution failures.
- Numeric answers use benchmark tolerances, while dates require the exact calendar day and gestational ages require exact week-and-day integers.
- Blind Program-Solve covered-subset results differ slightly across runs, with per-case answers agreeing about 95% at 7B and at least 99.8% at 32B.
A.2 Supplementary tables
The supplementary tables document benchmark coverage, solver and library comparisons, ablations, paired gaps, failure modes, reproducibility, and formula audits. They also clarify how abstention, execution outcomes, and clinical-formula concerns are counted.
- Coverage and reporting: The supplementary tables define the benchmark’s coverage and accuracy reporting, including full-set results, covered subsets, and seed variability.Table 3 reports mean accuracy and seed standard deviation; Figure 2 frames the 22-calculator library’s coverage against full-set accuracy.
- Ablations and comparisons: The ablation tables test voting, repair, generic syntax guidance, note-budget limits, and paired solver-versus-reference gaps on matched subsets.Table 4 covers voting and repair on 440 cases; Tables 5–7 and 20 define the corresponding ablation and paired-comparison designs.
- Outcome accounting: The library and program tables separate abstention, execution failures, unreadable returns, and valid scored outputs across cases and seeds.Tables 8 and 14 use a shared outcome taxonomy, while Table 21 combines library answers with Program-Solve or arithmetic fallbacks.