PaperlayerView exact revision on arXiv

Source-linked AI summary

ChestPheNoT: Deployable, Auditable Label-Status-Evidence Extraction from Radiology Reports

Kai Yu, Chenyu Zhu, Zaifu Zhan, Meijia Song, Min Zeng, Xiaoyi Chen, Mingquan Lin, Rui Zhang

arXiv:2609.31629v1cs.CLcs.AIcs.LG

TL;DR

Radiology reports contain findings useful for clinical analytics, but practical extraction must account for restricted API use, scarce expert annotations, and the need for auditable evidence. CHESTPHENOT trains compact models on hybrid silver supervision to jointly predict findings, status, and evidence. The 3B model remains below CheXbert in-distribution but significantly surpasses it on cross-institution detection, supporting compact local models as a practical option under distribution shift.

  • Problem

    Radiology reports support clinical analytics, but unstructured text, restricted third-party API use, scarce expert annotations, and labelers without evidence spans complicate practical extraction.

    Source: §I · p. 1

  • Method

    CHESTPHENOT fine-tunes compact Qwen2.5 models on hybrid CheXbert+72B silver labels and refines them with GRPO to jointly return finding labels, status, and verbatim evidence.

    Source: §I · p. 1, B. Hybrid Silver Data Construction · p. 2

  • Results

    The 3B model remains below CheXbert on in-distribution mimic-685 but significantly exceeds it on nih-200 by +2.0 present-F1 (95% paired CI [+0.5, +3.7]).

    Source: A. Detection: In-Distribution vs.… · p. 4

  • Takeaways & Limitations

    Compact task-specific models support local inference while remaining competitive with established labelers under distribution shift and producing explicit supporting evidence.

    Source: §VI · p. 6

    Evidence relevance is evaluated with a single LLM judge and only on nih-200; independent radiologist adjudication is needed.

    Source: §VI · p. 6

Abstract

from arXiv · show

Structured phenotype extraction from radiology reports supports cohort construction, quality auditing, and clinical analytics, but practical deployment requires local inference and auditable predictions, while expert annotations remain scarce. Conventional labelers provide structured findings and assertion states but no supporting evidence, while API-hosted large language models may be unsuitable when clinical text cannot leave institutional infrastructure. We present CHESTPHENOT, a compact 0.5-3B language model that jointly extracts finding labels, three-class status (present/absent/uncertain), and verbatim supporting evidence spans. CHESTPHENOT is trained using hybrid CheXbert+72B silver supervision followed by supervised fine-tuning and lightweight GRPO refinement. Across three human-annotated gold sets spanning in-distribution, cross-taxonomy, and cross-institution evaluation, the 3B model remains below its CheXbert silver teacher in distribution but is competitive under distribution shift, significantly surpassing CheXbert on cross-institution detection (+2.0 F1). Task-specific training also enables the 3B model to match or exceed substantially larger prompted models on most detection and status comparisons. For evidence-grounded extraction, over 99% of final evidence spans are locatable in the source report, and the 3B model achieves 47.5 auditable-F1, outperforming Qwen2.5-7B one-shot prompting by 7.6 points and approaching Qwen2.5-72B. These results demonstrate that locally deployable models can provide competitive and directly auditable radiology-report extraction without relying on external inference APIs. Code and the full extraction/judge prompts will be made available at https://github.com/yukkai/ChestPheNoT.

I. INTRODUCTION

CHESTPHENOT addresses the need for local, auditable radiology-report extraction when API use is constrained and expert annotations are scarce. It jointly predicts findings, status, and verbatim evidence, with evaluation spanning multiple gold sets and an evidence-quality measure.

I. INTRODUCTION

  • Practical extraction must balance local deployment, traceable predictions, and scarce expert annotations; conventional labelers omit evidence, while API-hosted LLMs raise privacy, cost, and grounding concerns.These constraints leave a gap between local deployment, competitive extraction, and auditability.Source: §I · p. 1
  • CHESTPHENOT is a compact 0.5–3B Qwen2.5-based extractor that runs locally and returns finding labels, three-class status, and verbatim report evidence.It targets thirteen chest findings and is trained with hybrid CheXbert+72B silver labels followed by supervised fine-tuning and GRPO.Source: §I · p. 1
  • The evaluation spans in-distribution, cross-taxonomy, and cross-institution gold sets, with comparisons against established labelers and one-shot models up to 72B.It uses paired bootstrap confidence intervals.Source: §I · p. 1
  • Evidence-quality evaluation separates locatability from semantic relevance and combines them with prediction correctness in auditable-F1.Source: §I · pp. 1–2

II. RELATED WORK

Prior report labelers and clinical extraction systems vary in taxonomy, evidence output, and deployment footprint. CHESTPHENOT combines fixed-taxonomy labels and status with locatable evidence, while evaluating evidence relevance separately from locatability.

Report labelers

  • CheXpert and NegBio are rule-based labelers, CheXbert uses BERT with rule-based and expert supervision, and CheX-GPT trains a lightweight BERT labeler from LLM-generated labels.These systems provide structured labels and assertion states but not supporting evidence spans.Source: §II · p. 2
  • MAPLEZ offers richer annotations with a locally executable 70B-class LLM, but its reported footprint places it outside the compact tier and it emits no evidence spans.Its reported ≈150 GB VRAM footprint and two-A100 inference use are outside that tier; its categorical F1 is reported under a different scoring protocol.Source: §II · p. 2
  • RadGraph extracts span-level entities, relations, and certainty but does not normalize them to a fixed finding taxonomy with per-finding status.CHESTPHENOT instead jointly predicts findings, status, and locatable evidence within a fixed taxonomy.Source: §II · p. 2

LLMs for clinical extraction

  • Clinical LLMs can be unsuitable when text cannot be transmitted to external APIs; this work fine-tunes 0.5B–3B models for local extraction with status and evidence output.Smaller task- or domain-specialized models can also match substantially larger models on clinical NLP tasks.Source: §II · p. 2

Reinforcement learning and evidence evaluation

  • CHESTPHENOT’s GRPO reward uses label–status agreement and output validity, not evidence quality, which is evaluated independently.Evidence grounding is assessed for both source-report locatability and whether an evidence-only LLM judge considers evidence sufficient; the judge is a proxy, not clinical ground truth.Source: §II · p. 2

III. METHOD

The method specifies a per-finding JSON output contract and constructs silver training examples by combining CheXbert labels and statuses with filtered evidence proposals from an on-premise 72B model.

A. Extraction Schema

  • For each identified finding among the 13 standard CheXpert observations excluding No Finding, the model emits at most one label, status, and evidence record in a JSON array, or an empty array.The intended contract requires non-empty source-copied evidence for every emitted record, making compliant predictions directly auditable.Source: A. Extraction Schema · p. 2

B. Hybrid Silver Data Construction

  • Silver data uses 22k MIMIC-CXR reports, with CheXbert supplying labels and statuses and an on-premise Qwen2.5-72B-Instruct proposing evidence spans for nonblank findings.Only candidate spans that are exact report substrings survive, and a finding is retained only when at least one span survives; no manual annotation is used.Source: B. Hybrid Silver Data Construction · p. 2

C. Model Training

Training applies supervised fine-tuning followed by GRPO, whose reward emphasizes label–status agreement and valid output structure. At inference, predictions and evidence are aggregated across sampled generations and checked against the source report.

C. Model Training

  • Qwen2.5-Instruct models at 0.5B, 1.5B, and 3B are supervised-fine-tuned on the silver corpus, then refined with GRPO.Source: C. Model Training · p. 3
  • GRPO reward evaluates label–status agreement using the average of precision and recall, with empty predictions and targets handled explicitly.A format reward requires parseable JSON records with a valid label, status, and non-empty evidence; empty predictions against non-empty targets receive no reward or format credit.Source: C. Model Training · p. 3
  • At inference, five generations are sampled per report, and label–status pairs appearing in at least three are retained; evidence is aggregated, matched verbatim to the report, and unlocatable spans discarded.Matched spans are expanded to enclosing sentences, deduplicated, and ordered by cross-generation frequency; rare majority-supported findings without locatable evidence are emitted with a non-locatable flag.Source: C. Model Training · p. 3

A. Datasets

The evaluation spans three human-annotated gold sets across two taxonomies and two institutions, with defined scoring metrics and a locally deployable model tier.

A. Datasets

  • The three gold sets cover in-distribution MIMIC-CXR reports, cross-taxonomy MIMIC-CXR reports, and cross-institution, cross-taxonomy NIH reports.mimic-685 uses the native CheXpert taxonomy; mimic-350 and nih-200 use MAPLEZ merging rules and 12 scored classes.Source: A. Datasets · p. 3

B. Baselines and Metrics

  • Evaluation uses present-only micro-F1, three-class status macro-F1, evidence faithfulness, and auditable-F1, with 95% CIs from 1,000 report-level bootstrap resamples.Status is scored only on gold-annotated cells; auditable-F1 credits findings only when status is correct and evidence is locatable and judged relevant.Source: B. Baselines and Metrics · p. 3

C. Implementation Details

  • The deployable tier is defined as ≤8B by bf16 weight footprint; the 0.5B–3B models require approximately 1–6 GB and fit a 24 GB single-GPU budget.70B-class models require multi-GPU serving; experiments used four NVIDIA A100 80 GB GPUs.Source: C. Implementation Details · p. 3

D. Evidence Relevance Judge and Data Use

A third-party API judge assesses evidence relevance only on de-identified NIH reports, while MIMIC-CXR remains on-premise. On detection, the 3B model significantly exceeds CheXbert on NIH-200, but not on mimic-350.

D. Evidence Relevance Judge and Data Use

  • Evidence relevance is judged through a third-party API using only predicted label, status, and evidence; evaluation is limited to publicly distributed, de-identified nih-200 reports.MIMIC-CXR remains entirely on-premise, including silver-data construction with the 72B teacher.Source: D. Evidence Relevance Judge and Data Use · p. 3

A. Detection: In-Distribution vs. Out-of-Distribution

  • +2.0 present-F1 (95% paired CI [+0.5, +3.7]) is the 3B model’s significant detection advantage over CheXbert on nih-200; its +1.2 on mimic-350 is not significant.Across scales, models remain below CheXbert in-distribution on mimic-685, with the gap narrowing from −5.4 at 0.5B to −3.9 at 3B.Source: Table I · p. 4, A. Detection: In-Distribution vs.… · p. 4

B. Three-Class Status Classification

Three-class status performance varies by distribution and class: the 3B model trails CheXbert in-distribution but is not significantly different from it out-of-distribution, where uncertain is the weakest axis.

B. Three-Class Status Classification

C. Effect of the Reinforcement-Learning Stage

Task-specific training lets a compact model compete with substantially larger prompted models, while GRPO’s reported effects are small in-distribution and concentrated under shift.

C. Effect of the Reinforcement-Learning Stage

  • +1.5 present-F1 is the significant 3B GRPO step on mimic-350 (CI [+0.5, +2.7]); on nih-200, its present-F1 CI is [−0.1, +4.6].GRPO adds essentially nothing in-distribution; the step-level tests are exploratory and each configuration reflects a single training run.Source: C. Effect of the… · pp. 4–5, Table II · p. 5

D. Trained Small vs. Prompted Large Models

  • The trained 3B model exceeds one-shot Qwen2.5-72B on three-class status across all three sets, with paired CIs excluding zero (+4.0 to +9.6), and is competitive with 70B-class models on detection.It exceeds both 70B-class models on mimic-685 and nih-200 detection, trailing on mimic-350; one-shot prompting at small scale performs far below same-size trained models.Source: D. Trained Small vs. Prompted Large… · p. 5
  • CheXbert, CheXpert, and NegBio provide no evidence spans, so they are not auditable under the auditable-F1 metric.Source: D. Trained Small vs. Prompted Large… · p. 5

E. Evidence: Faithfulness and Auditability

Over 99% of standardized final evidence spans are locatable, and CHESTPHENOT reaches 47.5 auditable macro-F1, close to Qwen2.5-72B. Uncertain findings remain the weakest evidence category, with relevance evaluation limited to one dataset and one LLM judge.

E. Evidence: Faithfulness and Auditability

  • Over 99% of final evidence spans are locatable after standardization, although exact raw string matches vary with report formatting.On nih-200, 95.9% of non-exact raw spans match after punctuation-spacing normalization, and 0.7% cannot be aligned; raw MIMIC spans are 99.9% verbatim.Source: E. Evidence: Faithfulness and… · p. 5
  • Auditable-F1 counts a finding only when its status is correct, its evidence is locatable, and the judge accepts the span.On nih-200, the evidence-only judge accepts 89/96/42% of final spans for present/absent/uncertain findings.Source: E. Evidence: Faithfulness and… · p. 5
  • 47.5 auditable macro-F1 beats Qwen2.5-7B one-shot by 7.6 points (39.9) and is within 0.7 F1 points of Qwen2.5-72B (48.2).The 72B model recovers more absent gold cells (Rec 33.1 vs. 27.1) but at lower precision (68.4 vs. 89.7).Source: E. Evidence: Faithfulness and… · pp. 5–6
  • Per-span evidence acceptance changes little from SFT to SFT+GRPO: 83.4 vs. 82.4%, and 85.0 vs. 85.1% on 565 shared findings.The reward is based on label and status rather than evidence relevance.Source: E. Evidence: Faithfulness and… · p. 6
  • Uncertain findings are weakest (F1 18.2; 42% acceptance), partly reflecting a mismatch between gold conventions and the judge’s evidence definition.About two thirds of uncertain evidence cites stability language; uncertainty is also the rarest silver signal (9.8%), and its gold prevalence rises from 13% on mimic-685 to 26% on nih-200. Relevance evaluation uses one set (n=200) and one LLM judge; radiologist adjudication remains future work.Source: E. Evidence: Faithfulness and… · p. 6

VI. DISCUSSION AND CONCLUSION

CHESTPHENOT offers compact local inference with explicit evidence and remains competitive under distribution shift. The discussion qualifies that result with an in-distribution gap, weak uncertainty performance, and limits in supervision and evidence adjudication.

VI. DISCUSSION AND CONCLUSION

  • Compact 0.5B–3B models support local inference, remain competitive with established labelers under distribution shift, and produce explicit supporting evidence.The authors describe task-specific compact models as a practical path toward locally deployable, evidence-grounded clinical NLP.Source: §VI · p. 6
  • Under the same MAPLEZ merge on nih-200, trained 3B surpasses CheXbert by +2.0, while one-shot Qwen2.5-72B trails CheXbert (81.4 vs. 82.6).The cross-taxonomy advantage partly reflects the merge: one-shot Qwen2.5-72B also exceeds CheXbert on mimic-350 (82.4 vs. 80.1), so output-format flexibility alone does not explain the cross-institution gain.Source: §VI · p. 6
  • All model scales remain below CheXbert on in-distribution MIMIC, and uncertainty is the weakest status and evidence category.Evidence relevance is evaluated using one LLM judge on nih-200; radiologist adjudication is needed, and CheXbert-derived silver supervision carries inherited errors and biases.Source: §VI · p. 6
Loading 2609.31629v1…