PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrated Evidence Inference With a Small, Permissively Licensed Open Model

Johann Emmanuel Li

arXiv:2610.07019v2cs.CLcs.LG

TL;DR

Reviewers need a model’s answer and a measure of how far to trust it. This paper fine-tunes and openly releases a small model for evidence inference using permissively licensed articles. On EI’s test split, it met pre-registered criteria for calibration and comparisons, with low error among its most confident answers.

  • Problem

    Reviewers need calibrated probabilities so they can accept confident answers and check uncertain ones by hand.

    Source: §1 · p. 1

  • Method

    The paper fine-tunes Qwen3-4B-Base for three-label evidence inference using only training articles under CC BY, CC0, or the public domain.

    Source: §1 · pp. 1–2

  • Results

    Expected calibration error was 0.0168 against the 0.05 limit, and the 80 percent most confident answers had an error rate of 0.0256.

    Source: §11 · p. 12

  • Takeaways & Limitations

    The release makes the accuracy cost of permissive-license filtering explicit while providing an openly licensed model.

    Source: §1 · p. 2, §11 · p. 12

    The trial evaluation covered one English trial comparison with articles cut to 6,144 tokens; other questions, wordings, and document types were untested, and calibration under distribution shift was unmeasured.

    Source: §10 · p. 11

Abstract

from arXiv · show

Systematic reviewers who use a model to read randomized trials need each answer and a measure of how far to trust it. If the model is calibrated, so that its stated probabilities match how often it is right, a reviewer can accept its confident answers and check the rest by hand. This paper fine-tunes Qwen3-4B-Base with low-rank adapters and a classification head for evidence inference (Lehman et al., 2019; DeYoung et al., 2020): given a trial's article and a prompt that names an intervention, a comparator and an outcome, classify the result as a significant increase, a significant decrease or no significant difference. The articles of Evidence Inference 2.0 come from the open-access subset of PubMed Central, which includes articles under non-commercial and no-derivatives licenses, so the released model was fine-tuned only on the 1,431 of 2,657 training articles under CC BY, CC0 or the public domain; it is released under the Apache License 2.0 and runs on a laptop. On the test split, its expected calibration error is 0.017 and its log loss is lower than that of Gemma 4 31B-it given the same input zero-shot, meeting criteria registered on the Open Science Framework before testing. Its macro-F1, 0.925, is above every published result on that split that I found, though those systems used other inputs, scoring rules and prompt filters, and the test labels are public. Accepting its 80 percent most confident answers leaves an error rate of 2.6 percent among them. Context ablations show that the answers depend on the article, but the title alone carries some signal, which an informative title or recall of the trial could explain.

1 Introduction

The paper presents a calibrated, openly released small model for classifying randomized-trial results, addressing reviewers’ need for both answers and a measure of how much to trust them. Its introduction frames the work around licensed training data, pre-registered evaluation, and tests of article dependence and selective answering.

1 Introduction

  • Small models can cost less per question and run locally, while a review of clinical language models up to 4 billion parameters found none of 11 eligible studies reported scalar calibration error.Calibration matters because reviewers can accept confident answers and check uncertain ones by hand.Source: §1 · p. 1
  • The paper releases a calibrated 4-billion-parameter model fine-tuned only on training articles under CC BY, CC0, or the public domain.It is released under Apache License 2.0 and predicts significant increase, significant decrease, or no significant difference from an article and an intervention–comparator–outcome prompt.Source: §1 · pp. 1–2
  • The pre-registered test evaluation met all four criteria, while the released model’s macro-F1 exceeded published results on the split in a non-controlled comparison.The comparisons varied in inputs, scoring rules, and prompt filters.Source: §1 · p. 2
  • License filtering reduced accuracy under the same training configuration, making the release’s permissive-license constraint an explicit trade-off.The released model used only permissively licensed training articles.Source: §1 · p. 2
  • The paper tests article dependence with no-article and title-only ablations, tests comparator sensitivity by swapping intervention and comparator, and measures selective answering.These analyses ask how many confident answers could be accepted without checking them.Source: §1 · pp. 1–2

2 Related work

  • Prior EI work includes prompt-only classification, evidence-span systems, and long-document models, but published results use different scoring rules and task inputs.This model returns no evidence span; later work also recasts EI annotations as extraction tasks.Source: §2 · p. 2

3 Data and licenses

The study uses Evidence Inference 2.0 and applies a restrictive article-level license rule to determine which training material can be used for the released model. The rule excludes licenses that permit commercial use but restrict derivatives or impose other conditions.

3 Data and licenses

  • Evidence Inference 2.0 contains 2,657 training articles and 10,046 prompts, with labels assigned by the most common accepted annotation and ties resolved by the smallest label code.After stated exclusions, its validation split has 340 articles and 1,237 prompts, and its test split has 333 articles and 1,218 prompts.Source: §3 · p. 3
  • The license rule accepts CC BY, CC0, or public-domain articles only when no statement restricts commercial use, derivatives, or licensing of resulting work.Where license statements conflict, the more restrictive terms govern; CC BY-SA is excluded despite appearing in PMC’s commercial-use group.Source: §3 · p. 3
  • The rule classified 1,431 of 2,657 training articles as permissively licensed, so the released model’s EI fine-tuning used only that subset.The paper describes whether weights trained on the other articles may be openly released as legally unsettled; the full-data model remains private.Source: §3 · p. 3
  • The passage-selection BM25 statistics were counted over all training articles, and the initialization model’s training-abstract licenses were not checked individually.The BM25 statistics contain single words and counts, not article sentences.Source: §3 · p. 3

4 The model

The model uses a frozen Qwen3-4B-Base backbone with LoRA adapters and a classification head, and the released configuration was selected using validation data. The evaluation was pre-registered and compared the models on four criteria using the test split.

4 The model

  • The EI model uses frozen Qwen3-4B-Base weights, LoRA adapters on every decoder linear layer, and a classification head that scores three classes.The released model was initialized from PubMedQA weights and trained for two epochs on 5,279 permissively licensed prompts with 6,144-token inputs; its three runs were combined into one model soup.Source: §4 · p. 4
  • Only the 6,144-token input change passed the fixed validation-based selection rule among the tested configuration changes.The rule required a log-loss improvement beyond the baseline’s seed variation and that each candidate run beat the same-seed baseline.Source: §4 · p. 4
  • Temperature scaling was selected by out-of-fold validation log loss and fitted on all 654 permissively licensed validation prompts.The chosen temperature was 1.340; the released model answers every question itself, with no deferral.Source: §4 · p. 4

5 Pre-registered evaluation

  • The released-model evaluation was the second use of EI’s test split, and three design choices drew on the full-data model’s test results.The released model’s training configuration and calibration were chosen on validation data only; its candidate and calibration were frozen before its test predictions began.Source: §5 · p. 5
  • The evaluation table covers four pre-registered criteria on 1,218 test prompts in 333 articles, comparing the full-data model with the released model.The criteria include ECE and log-loss differences with 95 percent intervals; Gemma 4 31B-it reads the same selected input as each model.Source: Table 1 · p. 6

6 Results

The released model met all four pre-registered criteria, with low calibration error, lower log loss than the class-prior and Gemma 4 31B-it references, and macro-F1 of 0.9248. Results also compare it with published systems, examine calibration and context variants, and check whether the release reproduces on a laptop.

Main results

  • 0.0168 ECE met the 0.05 limit, while the released model’s log loss was lower than the class-prior baseline by −0.8603 and Gemma 4 31B-it by −0.1829.Its macro-F1 was 0.9248 against Gemma 4 31B-it’s 0.8668; these results met all four registered criteria.Source: §6 · p. 6, Table 2 · p. 7

Published results

  • 0.9248 macro-F1 exceeded the best strictly scored published full-text result, 0.780, and the best leniently scored result, 0.8154, but this was not a controlled comparison.Published systems used different inputs, scoring and counting; test-label exposure during pretraining is also possible.Source: §6 · p. 6, Table 3 · p. 7

Calibration

  • 966 prompts in the top probability bin had 0.9741 correct top answers against mean probability 0.9840; no class-level miscalibration was detected by slope and intercept.The bin contained probabilities above 0.9333; the reliability diagram reports exploratory intervals.Source: §6 · p. 7, Figure 1 · p. 8

Reproducibility check

  • All three models passed the laptop reproducibility check, but the EI check used articles already cut to 6,144 tokens and did not test whole-article passage selection.Across 60 randomly drawn EI prompts, the mean absolute difference was 0.0028 and no top answer changed.Source: §6 · p. 8

7 Does the model use the article?

Pre-registered context tests examined whether predictions depend on article text and whether they respond to reversed comparisons. No-article performance supported article dependence, while title-only performance retained a signal that could reflect title content or trial recall.

No article

  • 0.4384 released-model macro-F1 without the article was below the pre-specified 0.60 threshold, supporting article-text dependence but not excluding test-set recall.The same interpretation applied to the full-data model, whose no-article macro-F1 was 0.4289.Source: §7 · p. 9, Table 4 · p. 9

Title only

  • 0.0939 was the released model’s title-only macro-F1 increase over no article on shared prompts, consistent with title information or trial recall, which the test cannot distinguish.The full-data model’s increase was 0.1024; the intervals for both exceeded 0.05.Source: §7 · p. 9, Table 4 · p. 9

Intervention and comparator swapped

  • 0.6652 of the released model’s direction answers flipped after intervention and comparator were swapped; the pre-specified interpretation called both models’ swapped-label accuracy partial.The released model’s accuracy against swapped labels was 0.7808; only 0.2162 of 37 direction answers flipped for time-point or baseline comparators.Source: §7 · p. 9, Table 4 · p. 9
  • Swap augmentation of all training prompts failed model selection on validation log loss, while swap test-time augmentation worsened test log loss.The authors suggest testing comparator-specific swapped training prompts or a consistency loss; validation selection did not measure swap consistency.Source: §7 · p. 10

8 The cost of license filtering

Comparing permissively licensed training data with all training data showed an accuracy cost, while log-loss differences depended on the model configuration and calibration prompts.

The cost of license filtering

  • 0.0123 lower accuracy and 0.0170 higher log loss were observed for the permissively licensed baseline configuration versus the full-data model.The baseline used 1,431 permissively licensed training articles and 654 validation prompts, versus 2,657 articles and 1,237 prompts for the full-data model; both chose temperature scaling.Source: §8 · p. 10
  • 0.0140 lower accuracy was observed for the released model versus the full-data model, while no log-loss difference was detected.The released model also used the longer input; its log-loss difference was 0.0169 (−0.0095 to 0.0397).Source: §8 · p. 10

9 Selective prediction and deferral

Selective prediction quantifies how many answers a reviewer could accept without checking them, while deferral tests whether sending uncertain answers to Gemma 4 31B-it helps. The findings are bounded by a single English trial-comparison task and several evaluation and model-scope limitations.

9 Selective prediction and deferral

  • At 80% coverage, the released model’s error rate was 0.0256; the 35% most confident answers contained no errors.The risk-coverage analysis had AURC 0.0128 (0.0082 to 0.0182).Source: §9 · p. 10
  • At 10% deferral, max-probability deferral reached 0.9335 combined accuracy, versus 0.9269 for the released model and 0.8752 for Gemma 4 31B-it alone.No interval was reported for the cascade minus the released model alone, so no difference from that model alone is claimed.Source: §9 · p. 10, Table 6 · p. 11
  • The released model was wrong where Gemma 4 31B-it was right on 37 prompts and right where it was wrong on 100, bounding possible gains from deferral.The validation-learned rule did not differ from max-probability deferral at any tested rate.Source: §9 · p. 11

10 Limitations

  • The trial evaluation covers one English comparison per article, with articles truncated to 6,144 tokens; calibration under shifts to other questions, wordings, or documents is unmeasured.Source: §10 · p. 11
  • The EI test split had already been used for full-data criteria, secondary analyses, and context ablations before the released model’s criteria were evaluated.Source: §10 · p. 11
  • Qwen3 and Gemma 4 pretraining data are unpublished and may include PMC articles, EI’s repository, PubMedQA, or readmission data, potentially favoring both models.Source: §10 · p. 11
  • The released model’s EI fine-tuning used permissively licensed articles, but its training-passage word statistics covered all training articles and its PubMedQA initialization abstracts were not individually license-checked.Source: §10 · pp. 11–12
  • The baseline comparison omits fine-tuned encoders and few-shot or reasoning-enabled large models, and Fiorillo returns no evidence span and is not for clinical use.Source: §10 · p. 12

11 Discussion

The discussion reports calibrated probabilities and selective accuracy for the released model, while qualifying how much its accuracy can be attributed to reading the article. It also reports the accuracy cost of restricting fine-tuning to permissively licensed articles.

11 Discussion

  • Expected calibration error was 0.0168 against the pre-registered limit of 0.05, and accepting the 80% most confident answers left an error rate of 0.0256.The model met criteria fixed in advance on EI’s test split; these figures hold as far as that split represents a reviewer’s trials.Source: §11 · p. 12
  • The answers depend on the article, but the title adds information, and the ablations cannot distinguish an informative title from recall of the trial.The paper identifies post-release trials, membership-inference tests, counterfactual evidence-span edits, and a model that follows swapped comparisons as results that would change this reading.Source: §11 · p. 12
  • Fine-tuning only on permissively licensed articles made the same configuration less accurate, an explicit cost of the release’s license filtering.The NOTICE file credits each of the 1,610 permissively licensed training and validation articles with its identifier, license, and citation.Source: §11 · p. 12

Statements

The statements disclose a potential conflict, funding and ethics information, release and code details, and the author’s use of AI tools.

Statements

  • The author states they may later build a company on the model.Source: Statements · p. 12
  • The work had no external funding, used public de-identified data and published articles, and recruited or contacted no person.Source: Statements · p. 12
  • Fiorillo v0.5 is released under Apache License 2.0 with a model card, inference script, and article-crediting NOTICE file.Source: Statements · pp. 12–13
  • Claude wrote much of the code, ran analyses with the author, and drafted text; the author reviewed the work and takes responsibility.The statement says AI tools are not authors.Source: Statements · p. 13

A Every run and model

The appendix reports released-model training runs and model variants, then maps registered analyses and documents changes to the analysis plan. Its run comparison identifies longer inputs as the only candidate configuration that passed.

A Every run and model

  • The longer-input configuration was the only candidate to pass, with mean validation log loss 0.1834 versus 0.2116 for the baseline’s first two runs.Its pre-registered log-loss difference was −0.0286 (−0.0496 to −0.0097); the other three candidates had no detected difference.Source: A Every run and model · p. 16, Table 7 · p. 17
  • The third longer-input run was exploratory: it was calibrated after training, and its calibration entered no frozen model or pre-registered comparison.Its weights were included in the released soup, s9_ei_wavg.Source: A Every run and model · p. 16, Table 7 · p. 17

B The pre-registered analyses and the deviations

C Per-class results and calibration

The released model’s per-class calibration intervals include their ideal values, while the section also reports per-class F1 comparisons and GPU cost measures. The supplied passages do not provide the underlying per-class F1 or cost values.

C Per-class results and calibration

D The cost per 1,000 questions

  • Table 12 reports GPU time per question and cost per 1,000 questions at a rented-GPU rate of $3.52 per hour.The PubMedQA time for Gemma 4 31B-it is an item-count allocation that the text says overstates its time.Source: Table 12 · p. 20

E The PubMedQA and readmission models

The release includes PubMedQA and hospital-readmission models alongside the evidence-inference model. PubMedQA results include a detected calibration issue for one class, while the readmission ensemble’s log loss was below the class-prior baseline’s.

PubMedQA

  • The PubMedQA model uses Qwen3-4B-Base with a classification head and initializes the evidence-inference model.It was trained on PubMedQA abstracts under the MIT License; licenses were not individually checked, and unlabeled abstracts received pseudo-labels from an earlier model.Source: E The PubMedQA and readmission models · pp. 20–21

Readmission

  • 0.8275 log loss versus the class-prior baseline’s 0.8861 and 0.6622 AUROC for readmission within 30 days were reported on 6,998 encounters.The released model is the earlier weighted XGBoost–CatBoost ensemble; the new ensemble missed the pre-set 0.002 out-of-fold log-loss improvement threshold, and TabICLv2 was not trained after a version-check error. The test split was used in every version, so results are not confirmatory.Source: E The PubMedQA and readmission models · p. 21, Table 15 · p. 22
Loading 2610.07019v2…