PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Medical Calculation of LLMs with Embedded Coding

Tianshi Ming, Yingying Zhang, Xian Wu

arXiv:2609.31908v1cs.AIcs.CL

TL;DR

LLMs remain unreliable on exact medical calculations, which support high-stakes clinical decisions. MedCode trains them to generate code executed by a deterministic interpreter, with datasets and preference optimization tailored to medical calculations. Embedded code generation improves accuracy by 20–30 absolute percentage points, while wDPO adds gains on selected tasks.

  • Problem

    LLMs remain unreliable at exact medical calculation, despite strong performance on medical examinations and question answering.

    Source: §1 · p. 1

  • Method

    MedCode trains LLMs to generate embedded executable code, using SFT and preference data and wDPO to emphasize difficult-to-distinguish pairs.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    Embedded code generation improves calculation accuracy by 20–30 absolute percentage points, and wDPO provides additional gains on selected tasks.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Across the evaluated models and datasets, embedded code generation substantially improves medical calculation accuracy.

    Source: §6 · p. 8

    MedCode relies on predefined calculators, and performance on longer workflows, multiple patients, or incomplete and conflicting records remains untested.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) perform well on medical examinations and question-answering benchmarks, but remain unreliable on medical calculation tasks that require exact numerical outputs. These calculations support high-stakes decisions such as medication dosing, organ-function assessment, and prognostic scoring, for which even small errors can have serious clinical consequences. We introduce MedCode, a framework that improves medical calculation by training LLMs to generate embedded executable code. Given a clinical context, the model identifies the relevant calculator, extracts its input variables, and produces a script that delegates arithmetic operations to a deterministic interpreter. Executing the script returns the calculated value together with an explanation and the appropriate unit. We construct supervised fine-tuning (SFT) and preference datasets from the MedCalc benchmark and additionally curate a dataset for calculation tasks in Intensive Care Unit (ICU) scenarios. We further propose weighted Direct Preference Optimization (wDPO), which adaptively emphasizes preference pairs that are difficult for the model to distinguish. Experiments with LLaMA3-8B, Qwen2.5-7B, and Mistral-7B show absolute accuracy gains of 20--30 percentage points, demonstrating the effectiveness of embedded code generation for medical calculation.

1 Introduction

MedCode addresses LLMs’ persistent difficulty with reliable medical calculation by generating executable code for deterministic arithmetic. The paper also introduces preference-learning methods and datasets for benchmark and ICU calculation tasks.

  • LLMs can perform well on medical examinations yet remain unreliable at numerical calculation needed for precise clinical outputs.The paper highlights ICU calculations such as GFR, where calculator selection and variable extraction matter.Source: §1 · p. 1
  • MedCode separates clinical interpretation from arithmetic: the model generates an executable script, and a deterministic interpreter calculates and explains the result with units.The script is generated in one LLM pass, followed by execution without another model call.Source: §1 · p. 1, Figure 1 · p. 2
  • The authors convert MedCalc examples into SFT scripts, pair erroneous generated scripts with correct references, and curate clinician-authored synthetic ICU tasks.The ICU tasks require extracting, filtering, and counting longitudinal measurements.Source: §1 · p. 2
  • wDPO weights preference pairs by how difficult the model finds them to distinguish, emphasizing ambiguous pairs during training.The paper introduces this alongside an error-driven procedure for constructing preference pairs.Source: §1 · p. 2
  • MedCode improves medical-calculation accuracy by 20–30 absolute percentage points through embedded code generation.Source: §1 · p. 2

2 Related Work

Medical calculation benchmarks require open-ended numerical outputs beyond the multiple-choice format common in medical QA. MedCode combines domain-specific fine-tuning with executable code generation, distinguishing it from general code-assisted and multi-stage tool-use approaches.

2.1 Medical Large Language Model

  • MedCalc-Bench evaluates open-ended equation- and rule-based calculator tasks, unlike common medical benchmarks built primarily around multiple-choice questions.It reports low accuracy for both open- and closed-source models.Source: §2.1 · p. 2

2.2 Coding Generation

  • MedCode trains models to emit a complete executable script in one pass, whereas some code-assisted and agent-based approaches use multiple inference stages.Medical calculation programs must encode the right calculator, units, coefficients, and input variables.Source: §2.2 · p. 3

3 Dataset Generation

The paper builds embedded-code and preference-training data from MedCalc-Bench, then constructs a synthetic ICU dataset for longitudinal calculation. Its data-generation methods target calculator execution, plausible script errors, and extraction and counting over clinical records.

3.1 Embedded Coding Dataset Generation based on MedCalc

  • The MedCalc-Bench code dataset combines calculator logic in comments, executable formulas with extracted values and unit conversions, and printed results with explanations and units.Source: §3.1 · p. 3

3.2 Preference Dataset Generation Based on MedCalc

  • Preference pairs are created by executing SFT-generated scripts and pairing incorrect or non-executable outputs with the correct reference script.Sampling across temperatures is used to expose a broader range of failure modes.Source: §3.2 · p. 3

3.3 ICU-Dataset

  • The ICU dataset tests threshold-based extraction and frequency counting over longitudinal EHR entries, with clinician writing and verification, using 5,000 training examples and 1,000 test examples.It covers 7 clinical categories and 16 medical metrics.Source: §3.3 · p. 4

4 Preference Training Strategy

Because SFT models still make subtle script errors, MedCode applies preference optimization to distinguish reference scripts from erroneous alternatives. Its wDPO objective emphasizes difficult-to-separate pairs while retaining direct supervision from preferred scripts.

4 Preference Training Strategy

  • SFT models can still produce subtle script errors, motivating preference optimization to distinguish reference scripts from erroneous alternatives.Source: §4 · p. 4
  • Preference pairs are constructed by pairing erroneous scripts produced by the fine-tuned model with reference scripts, then SFT is followed by wDPO.Source: Figure 2 · p. 5

4.1 Weighted Direct Preference Optimization

  • Standard DPO treats every preference pair equally, although errors can range from conspicuous missing steps to subtle coefficient or unit-conversion differences.Source: §4.1 · p. 5
  • Pairs with similar preferred- and dispreferred-script probabilities receive larger adaptive weights, while pairs with a clear probability gap receive smaller weights.Source: §4.1 · p. 5
  • The complete wDPO objective combines adaptive DPO with an adaptive SFT term that gives greater weight to preferred-response likelihood when w is small.Source: §4.1 · pp. 5–6

5 Experiments

On MedCalc-Bench, embedded-code SFT substantially outperformed prompting and direct-answer SFT, with further gains from wDPO; ICU results also favored code-based calculation. Ablations found that wDPO and adaptive regularization improved aggregate or selected results, while few-shot gains were modest.

5.1 Experiment Settings

  • Experiments evaluated LLaMA3-8B, Mistral-7B, Qwen2.5-7B, and Baichuan2-7B, with GPT-3.5 and GPT-4 as prompted baselines and prompting and training-based variants on MedCalc-Bench.MedCalc-Bench includes four equation-based and three rule-based task categories; its training split has 9,766 examples.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 Main Results

  • 54.39 average accuracy for LLaMA3-8B SFT Code and 57.92 for Mistral-7B exceeded one-shot CoT and direct-answer SFT; wDPO added 0.95 and 1.81 points, respectively.Both SFT-Code models also exceeded the reported GPT-4 averages.Source: §5.2 · p. 6, Table 2 · p. 7
  • wDPO gains were task-specific: laboratory accuracy rose by 5.52 points for LLaMA3-8B and 5.83 for Mistral-7B, while LLaMA3-8B dosage and diagnosis rose by 22.50 and 8.33 points.Several other categories remained unchanged or decreased; the findings use approximately 1,500 mined preference pairs.Source: §5.2 · p. 6

5.3 Evaluation on ICU dataset

  • On the ICU dataset, LLaMA3-8B code SFT raised extraction accuracy from 52.39 to 82.10 and frequency-count accuracy from 75.10 to 97.60; wDPO raised them further to 83.20 and 98.00.Qwen2.5-7B showed the same overall pattern, particularly for frequency counting; DeepSeek-R1 scored 85.87 and 99.53 on the two measures.Source: §5.3 · p. 6, Table 3 · p. 7, Table 4 · p. 7

5.4 Ablation Study on Adaptive SFT

  • Adaptive regularization gave wDPO its highest equation-based accuracy (72.26) and overall average (55.34), while fixed coefficient 1.0 gave the highest rule-based score (29.40).Adaptive regularization had the best aggregate result, but its advantage was not uniform across task types.Source: §5.4 · pp. 6–7, Table 7 · p. 8

5.5 Comparison with zero and few-shot

  • Few-shot code generation raised the average score from 33.37 with zero examples to 34.56 with one and 36.77 with three, a modest improvement relative to training-based code generation.The passage attributes the main benefit to task-specific training rather than additional inference-time examples.Source: §5.5 · p. 7, Table 5 · p. 8

5.6 Comparison with Other Preference Optimization Approaches

  • wDPO averaged 55.34, ahead of rDPO (50.84), ROPO (54.25), and WPO (52.33), and exceeded the SFT-Code baseline of 54.39 by 0.95 points.The methods shared the same LLaMA3-8B-Instruct SFT-Code checkpoint and preference data; wDPO also had the highest aggregate score over four equation-based categories.Source: §5.6 · p. 8, Table 6 · p. 8

6 Conclusion

MedCode improves medical calculation by having fine-tuned models generate executable scripts that instantiate calculators from extracted values, with a deterministic interpreter performing arithmetic. The paper constructs code-based MedCalc-Bench and longitudinal ICU data, and reports substantial accuracy improvements from embedded code generation, with wDPO gains on selected tasks.

  • MedCode has the model generate an executable script instantiating the relevant medical calculator, then uses a deterministic interpreter to return the result, explanation, and unit.Source: §6 · p. 8
  • The paper constructs an executable-code version of MedCalc-Bench and a synthetic dataset for longitudinal ICU calculation.Source: §6 · p. 8

Limitations

MedCode may not generalize beyond predefined calculators or training data, and its error mining and evaluation cover limited failure modes and task settings.

  • MedCode may not generalize to unseen formulas or clinical rules; error mining may miss rare or complex failures, and longer workflows or incomplete records remain untested.The experiments focus on short, well-specified calculations.Source: Limitations · p. 8

Ethics

The ICU dataset uses synthetic records authored and verified by clinicians, and the work is not validated for autonomous clinical decision-making.

  • The ICU dataset contains no real patient records or personally identifiable information, and the datasets and models are not validated for autonomous clinical decision-making.The synthetic records were manually authored and verified by experienced clinicians.Source: Ethics · p. 9

A Coding Scripts Design details for MedCalc-based Dataset

The MedCalc-based scripts encode equation- and rule-based calculators, with verified templates spanning 41 calculators. Training and inference settings and prompts follow established implementations and adaptations.

A Coding Scripts Design details for MedCalc-based Dataset

  • Scripts represent equation-based tasks with compact or explicit multi-step calculations and rule-based tasks with scored criteria and model-interpreted categorical inputs.Dosage conversions define constants, and date-based calculators use datetime operations.Source: A Coding Scripts Design details for… · p. 11
  • The manually verified dataset contains 9,766 training and 1,048 test examples across 41 calculators: four equation-based and three rule-based task types.Source: A Coding Scripts Design details for… · p. 11

B Hyperparameter and Prompts Settings

  • Training uses LLaMA-Factory defaults as a basis; inference follows MedCalc-Bench settings, and prompts adapt its released code-generation templates.The paper reports training and inference settings in Tables 9 and 10.Source: B Hyperparameter and Prompts Settings · p. 11
  • Generating scripts at multiple temperatures yields approximately 1,500 preference pairs, with k = 1,000 chosen to balance inference cost and mined-error diversity.Source: B Hyperparameter and Prompts Settings · p. 11

C Case Study of Minor Error And Substantial Error

The case study reports fewer minor-error scripts but more substantial-error scripts after wDPO, which weights preference pairs by model probability gaps. The execution procedure preprocesses and runs generated scripts.

C Case Study of Minor Error And Substantial Error

  • wDPO weights preference pairs by model probability gaps without explicit error-category labels; minor errors affect local elements, while substantial errors alter steps or cause execution failures.Source: C Case Study of Minor Error And… · p. 11
  • Minor-error cases fell from 360 to 312 after wDPO, while substantial-error cases rose from 87 to 144, so total failures did not uniformly decline.Source: C Case Study of Minor Error And… · p. 11

D Coding Execution Implementation

E Case Study

The case study illustrates embedded-code prompts and scripts for medical calculations using an MDRD GFR example and an ICU blood-pressure counting task.

E Case Study

  • The coding inference prompt instructs the model to write code for equations, plug in patient-note values, and print outputs rather than compute a numerical answer itself.The output is specified as a JSON dictionary containing reasoning and code.Source: E Case Study · p. 14
  • The example script reports an MDRD GFR of 49.677 mL/min/1.73 m² for a 73-year-old patient with creatinine 1.4 mg/dL.It prints the equation, patient values, coefficients, and result.Source: E Case Study · p. 16
  • The ICU example identifies six blood-pressure readings at or below 137.2 mmHg in the timestamped entries.The listed qualifying readings include systolic, diastolic, invasive, and non-invasive measurements.Source: E Case Study · p. 17

Related papers

Loading 2609.31908v1…