PaperlayerView exact revision on arXiv

Source-linked AI summary

Tool-Augmented Large Language Models for Reliable Medical Calculation from Patient Notes

Yang Zhong, Ling Luo, Xuanlyu Zhu, Yanhua Wang, Yuanyuan Sun, Jian Wang, Hongfei Lin

arXiv:2610.21477v1cs.AI

TL;DR

Medical calculation from patient notes requires extracting clinical variables and applying complex rules exactly, but direct prompting entangles extraction, arithmetic, and formatting. The paper retrieves calculator specifications, generates reusable executable tools, and uses constrained extraction before deterministic calculation. On MedCalc-Bench, it reports 86.73% final-answer accuracy, while remaining errors include implicit clinical information and tool-handling issues.

  • Problem

    Medical calculation from free-form notes requires extracting clinical variables and applying exact rules, boundaries, and output formats, but direct prompting entangles these demands.

    Source: §I · p. 1

  • Method

    The framework retrieves and structures calculator rules, synthesizes and validates executable tools, extracts schema-constrained inputs, and performs deterministic calculation.

    Source: §I · p. 2

  • Results

    86.73% final-answer accuracy and 97.47% parameter matching rate on 1,100 test cases, outperforming one-shot prompting baselines by 24.0 percentage points.

    Source: §I · p. 2

  • Takeaways & Limitations

    Grounding calculators in retrieved rules and separating extraction from execution improves transparency and helps mitigate formula hallucinations and arithmetic errors.

    Source: §I · p. 2, §VI · p. 8

    Errors persist when extracting implicit information such as medication-related conditions, negation, or temporality, and when generated tools use incorrect normalization, formula branches, or output formats.

    Source: §VI · p. 8

Abstract

from arXiv · show

Medical calculators translate raw patient observations into clinically actionable values and scores. However, applying these calculators directly to free-form patient notes requires precise variable extraction combined with the exact execution of complex formulas, scoring rules, and output conventions. Relying solely on direct large language model (LLM) prompting is unreliable in this context, as rule recall, parameter extraction, arithmetic reasoning, and formatting become entangled within a single generative step. To address these limitations, we propose a tool-augmented LLM framework for reliable medical calculation from patient notes. The framework retrieves and structures calculator specifications, generates reusable executable tools, and uses a constrained extractor to map each patient note to calculator-ready inputs. The tools then perform deterministic calculation, separating clinical language understanding from rule execution and arithmetic. Evaluated on the MedCalc-Bench dataset, our method achieves 86.73% final-answer accuracy, outperforming the one-shot prompting and supervised fine tuning-only baselines by 24.0 and 17.27 percentage points. Ablation and error analyses show that these performance gains are primarily driven by rule-grounded tool synthesis and schema-constrained extraction, whereas remaining failures are concentrated in clinical-semantic extraction and benchmark-specific output contracts. By grounding calculator logic in executable tools, the proposed approach alleviates arithmetic hallucinations and improves the computational reliability of the final clinical outputs.

I. INTRODUCTION

Medical calculation from free-form notes requires accurate clinical extraction alongside exact rule execution, while direct prompting can entangle these steps. The paper proposes a tool-augmented workflow and reports higher benchmark accuracy than one-shot prompting.

I. INTRODUCTION

  • Reliable calculation requires extracting numerical, categorical, Boolean, date, and unit information, then applying the correct rule, boundaries, and output format.These variables may be indirect, distributed across a note, negated, or implied; errors can change the result, so a trace makes failures inspectable.Source: §I · p. 1
  • Direct prompting can entangle formula recall, parameter extraction, arithmetic, and formatting, while fixed tool libraries limit coverage to calculators already implemented.The entanglement can obscure whether an error arose from clinical understanding, calculator implementation, or formatting.Source: §I · p. 1
  • The proposed workflow retrieves and structures calculator rules, synthesizes and validates reusable tools, extracts schema-constrained inputs, and executes calculations deterministically.This decoupled design is intended to mitigate formula hallucinations and arithmetic errors while supporting tool construction beyond predefined libraries.Source: §I · pp. 1–2
  • 86.73% final-answer accuracy and 97.47% parameter matching rate on 1,100 test cases, outperforming one-shot prompting baselines by 24.0 percentage points.The evaluation uses the corrected MedCalc-Bench benchmark with a unified split and official metrics.Source: §I · p. 2

II. RELATED WORK

The paper frames medical calculation as the joint task of extracting patient variables and exactly applying calculator rules. It contrasts existing LLM approaches with a workflow that synthesizes tools and separates extraction from deterministic execution.

A. Medical Calculation Benchmarks

  • MedCalc-Bench treats medical calculation as distinct from knowledge-oriented question answering because it tests both clinical extraction and exact execution of explicit calculator rules.The authors use a corrected benchmark release because audits found erroneous, ambiguous, or incomputable labels in the original data.Source: A. Medical Calculation Benchmarks · p. 2

B. LLMs for Medical Calculation

  • Prior LLM approaches use retrieval and prompting, supervised training, or tool calling, but may retain per-case execution loops or depend on prebuilt calculator libraries.Some approaches also couple parameter extraction and tool invocation, complicating separation of clinical interpretation errors from tool or formatting errors.Source: B. LLMs for Medical Calculation · p. 2
  • The proposed method synthesizes tools from retrieved rule evidence, validates and repairs them, and separates schema-constrained extraction from deterministic execution.Its trace records retrieved evidence, the generated tool, extracted parameters, tool input, and tool output to help localize errors.Source: B. LLMs for Medical Calculation · p. 2

III. METHOD

  • The method overview groups the workflow into rule-grounded specification retrieval, executable tool synthesis and validation, and parameter extraction with tool execution.Figure 1 illustrates the sequence using the Cockcroft–Gault creatinine-clearance calculator.Source: §III · p. 2, Figure 1 · p. 3

A. Task Definition

The task takes a patient note and calculation question as input and produces an answer consistent with a reference calculator and the benchmark’s output contract.

A. Task Definition

  • The input is a patient note paired with a calculation question, and the goal is an answer consistent with the reference calculator and benchmark-specific output contract.Source: A. Task Definition · p. 3
  • The workflow represents extracted patient information as a structured parameter object and uses calculator rules and a parameter schema to guide the process.The rule specification and schema describe the target calculator; the extracted object contains the note-derived inputs.Source: A. Task Definition · p. 3
  • The LLM organizes calculator rules, generates tools, and interprets clinical language, while a validated executable tool performs the final calculation.Source: A. Task Definition · p. 3

B. Rule-Grounded Calculator Specification Retrieval

The method retrieves calculator evidence and structures it into a specification and schema. It then generates reusable executable tools and validates them against examples before reuse.

B. Rule-Grounded Calculator Specification Retrieval

  • The system searches for the calculator corresponding to the question, reuses a cached specification when available, or retrieves calculator evidence and benchmark metadata.A rule-curation model turns the retrieved evidence into a structured calculator specification.Source: B. Rule-Grounded Calculator… · p. 3
  • The parameter schema specifies required inputs, value types, allowed units, and permitted categorical values; the specification is stored with provenance for reuse.Source: B. Rule-Grounded Calculator… · p. 3

C. Executable Calculator Tool Synthesis and Validation

  • A tool-writing model converts the structured specification and schema into Python code that validates inputs and implements formulas, scoring logic, branches, and boundary cases.The generated function returns an output consistent with the calculator’s expected contract.Source: C. Executable Calculator Tool… · p. 3
  • Candidate tools are tested on eight same-calculator training examples, with observed failures or mismatches returned for up to four repair rounds before caching.The eight examples are randomly sampled and reused across rounds; they do not guarantee branch or boundary coverage, and deterministic execution does not ensure a correct implementation.Source: C. Executable Calculator Tool… · pp. 3–4

D. Schema-Constrained Parameter Extraction and Tool Execution

The extraction stage uses calculator-specific schemas and examples to map each patient note into structured inputs, which a cached executable tool then calculates deterministically. Tools are generated once per calculator and reused, while extraction is repeated for each note.

D. Schema-Constrained Parameter Extraction and Tool Execution

  • A smaller Qwen3-8B model is fine-tuned for repeated extraction, while the tool-writing LLM handles calculator construction and validation; local deployment may suit data-governance constraints.Memory is described as auxiliary, storing specifications, schemas, tools, examples, traces, and errors to support reuse and auditability.Source: D. Schema-Constrained Parameter… · pp. 4–5
  • Before extraction, a retriever selects three same-calculator examples and supplies them with a schema covering the calculator’s required inputs.Inputs can include measurements, Boolean risk factors, categories, dates, gestational ages, time intervals, and values paired with units.Source: D. Schema-Constrained Parameter… · p. 4
  • The extractor receives the patient note, question, calculator, examples, and schema, then returns a constrained JSON object.The tools are generated once per calculator and reused; parameter extraction is the repeated inference-time task.Source: D. Schema-Constrained Parameter… · p. 4
  • The extractor returns a structured parameter object that the tool-serving interface sends to a cached calculator tool for deterministic execution; the LLM does not perform the final calculation.Each value may be a number, Boolean, category, date, time interval, or value-unit pair.Source: D. Schema-Constrained Parameter… · p. 4
  • Schemas are derived from calculator specifications and cover 328 required fields across 55 calculators; three demonstrations per calculator cover all fields without using test cases.The demonstrations comprise the public exemplar plus two greedily selected training examples.Source: D. Schema-Constrained Parameter… · pp. 4–5

IV. EXPERIMENTS

Experiments use a corrected MedCalc-Bench version with a unified split and official metrics. The evaluation reports end-to-end answer accuracy, parameter matching, and calculator-tool accuracy to assess different parts of the system.

A. Dataset and Experimental Settings

  • The evaluation uses MedCalc-Bench Verified v1.0.7, a corrected benchmark with 55 calculators, 10,538 training instances, and 1,100 test instances.The original v1.0 benchmark was not used because audits identified erroneous, ambiguous, and incomputable cases.Source: A. Dataset and Experimental Settings · p. 5
  • Final-answer accuracy measures correct test-case outputs, parameter matching rate measures extracted values matching gold annotations, and tool-level accuracy evaluates generated tools with gold parameters.The tool-level metric isolates calculator implementation quality from extraction quality.Source: A. Dataset and Experimental Settings · p. 5
  • The study compares prompting baselines, tool-writing systems, and extraction models under a unified data split and official metrics.GPT-5.5 generates calculator tools and Qwen3-8B is used for parameter extraction unless otherwise specified.Source: A. Dataset and Experimental Settings · p. 5

B. Main Results

On the MedCalc-Bench Verified test set, the method outperforms the strongest one-shot prompting and SFT-only baselines in the controlled comparison. Tool-quality analyses also find higher accuracy with rule-grounded retrieval and repair, while external benchmark rows remain contextual comparisons.

B. Main Results

  • The 0.49-percentage-point difference from an externally reported 86.24% does not establish a statistically significant advantage.External rows are contextual comparisons rather than part of the controlled evaluation.Source: B. Main Results · p. 5
  • 86.73% final-answer accuracy exceeds the strongest one-shot prompting baseline by 24.00 percentage points and the same-backbone SFT-only baseline by 17.27 percentage points.Table II reports the main results on the MedCalc-Bench Verified test set.Source: Table II · p. 5, B. Main Results · p. 5

C. Effect of Generated Tool Quality on Medical Calculation Performance

  • GPT-5.5 tools generated with rule-grounded retrieval and validation-and-repair achieve the highest tool-level and final-answer accuracy, with no runtime errors; removing retrieval and iterative repair reduces both accuracies by more than 10 percentage points.The final-answer comparison uses the same extraction model across tool sets.Source: C. Effect of Generated Tool Quality on… · pp. 5–6, Table III · p. 6
  • With GPT-5.5 and the Qwen3-8B-SFT extractor fixed, retrieval plus repair raises final accuracy from 73.27% to 86.73% (+13.46 percentage points).The comparison supports their joint contribution but does not separate retrieval from repair.Source: C. Effect of Generated Tool Quality on… · p. 6

D. Parameter-Extraction Model Comparison

With the GPT-5.5-generated tool set held fixed, the section compares extractor families and finds that Qwen3 performs best. Parameter matching generally tracks final-answer accuracy, but high parameter-level matching does not guarantee case-level correctness.

D. Parameter-Extraction Model Comparison

E. Category-Level Analysis

The category analysis distinguishes extraction challenges from post-extraction calculation and output-contract issues. Ablations show substantial performance drops when tool execution or calculator-specific schema and demonstrations are removed.

E. Category-Level Analysis

  • Physical-measurement and laboratory-test calculators performed strongest, while date and dosage results remained sensitive to boundaries, conversions, rounding, and output formatting.For measurement and laboratory calculators, inputs are often explicit and rules relatively direct; deterministic execution resolves most remaining computation after correct extraction.Source: E. Category-Level Analysis · p. 6, Figure 4 · p. 6
  • Risk, diagnosis, and severity calculators pose clinical-semantic extraction challenges, where one Boolean or categorical error can change the final score.The required judgments may involve implied comorbidities, symptom timing, disease categories, negation, and historical context.Source: E. Category-Level Analysis · p. 6
  • The gap between parameter matching rate and case-level accuracy alone cannot identify whether failures arise in the generated tool or output contract.The analysis motivates case-level inspection of boundary handling, unit conversion, rounding, and benchmark-specific output conventions.Source: E. Category-Level Analysis · pp. 6–7

F. Ablation Analysis

  • Removing tool-augmented execution caused the largest ablation drop, at 17.27 percentage points; removing schema and few-shot examples reduced performance by 7.55 percentage points.The No tool condition instead has the fine-tuned model generate the final answer directly; the No schema/few-shot condition removes calculator schemas and demonstrations.Source: F. Ablation Analysis · p. 7, Table IV · p. 7
  • Removing calculator-specific schemas and demonstrations substantially reduced performance, while removing extractor fine-tuning caused a moderate decrease.The authors suggest schema constraints structure extraction, while fine-tuning helps with missing data, implied conditions, and consistent categorical labels.Source: F. Ablation Analysis · p. 7, Table IV · p. 7

G. Construction Usage and Inference Time

The section describes tool-construction usage and inference-time measurements, then examines representative cases and the distribution of remaining errors. Errors arise both from clinical-semantic extraction and from tool rules or benchmark output requirements.

G. Construction Usage and Inference Time

V. CASE STUDY AND ERROR ANALYSIS

  • The HEART Score case missed hypercholesterolemia inferred from atorvastatin, while the steroid-conversion case misnormalized hydrocortisone as cortisone.In the latter case, the extractor identified the inputs correctly, but substring matching selected the wrong source-steroid equivalence and produced 4.89 mg instead of the benchmark answer of 6.029 mg.Source: §V · p. 7, Table VII · p. 8
  • Of 146 incorrect answers, 80 were extraction errors and 66 were generated-tool errors.Generated-tool errors include cases with correctly extracted fields but incorrect tools, rounding, units, date boundaries, or answer strings that violate the benchmark contract.Source: §V · pp. 7–8, Figure 5 · p. 8
  • Extraction errors clustered in clinical-semantic judgments, while generated-tool errors more often involved formula branches, normalization, rounding, units, date boundaries, or answer representation.The error categories distinguish incorrect required fields from incorrect final results despite correctly extracted fields.Source: §V · pp. 7–8
  • Among seven calculators with the largest error counts, Estimated Gestational Age errors chiefly involved dates and output contracts, whereas Steroid Conversion errors chiefly involved tool rules.Wells PE, FeverPAIN, the Charlson Comorbidity Index, and the HEART Score were primarily affected by extraction errors involving symptoms, timing, risk factors, or implied history.Source: §V · p. 8, Figure 5 · p. 8

VI. CONCLUSION

The conclusion presents a framework that retrieves and validates calculator rules, extracts structured inputs, and delegates calculation to executable tools. It reports improved benchmark performance while noting persistent extraction and tool-rule errors and limits to the evaluation scope.

VI. CONCLUSION

  • The framework separates rule retrieval, executable tool generation, schema-constrained extraction, and deterministic tool execution.Retrieved evidence and execution feedback are used to ground and validate calculator tools.Source: §VI · p. 8
  • The method substantially outperformed prompting and SFT-based baselines on the corrected MedCalc-Bench Verified test set.The conclusion reports the comparison without giving numeric values in these passages.Source: §VI · p. 8
  • Remaining failures include implicit information, negation, and temporality in extraction, as well as incorrect normalization, formula branches, or output formats in generated tools.The authors identify reinforcement learning and intelligent-agent verification, boundary testing, and clarification as future directions.Source: §VI · p. 8
  • Evaluation used new patient notes for the same 55 calculators used during tool construction; generalization to unseen calculators and clinical deployment remain untested.Source: §VI · p. 8

Related papers

Loading 2610.21477v1…