PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

Ignacy Kolton, Wenxuan Li, Zongwei Zhou, Jarosław B. Ćwikła, Arkadiusz Sitek, Pedro R. A. S. Bassi, Szymon Płotka

arXiv:2610.30980v1cs.CVcs.LG

TL;DR

Radiology reports must state which diagnoses are present or absent, but conventional training objectives are misaligned with this diagnostic requirement. DLoss adds asymmetric sequence-level supervision, and evaluations across six 3D MedVLMs report improved diagnostic faithfulness with competitive language-generation performance.

  • Problem

    Radiology reports must state which diagnoses are present or absent, and missing actionable diagnoses or adding unsupported differentials can affect staging and treatment.

    Source: §1 · pp. 1, 3

  • Method

    DLoss asymmetrically pools decoder token probabilities to encourage supported diagnoses at least once and penalize probability spikes for unsupported diagnoses across report positions.

    Source: §1 · p. 3

  • Results

    Across six 3D MedVLMs, DLoss adds mean 4.05 GREEN and 3.82 RadGraph-F1 points on Merlin, and 4.28 GREEN and 2.96 RadGraph-F1 points externally, with consistent language-metric gains.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The lightweight objective improves six report-generation methods across six evaluation metrics while maintaining competitive language-generation performance.

    Source: §5 · p. 9

    Targets derive from training reports, and evaluations remain within abdominal CT using automatic metrics; 2D X-rays and radiologist judgments remain untested.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize text agreement over diagnostically critical terms. As a result, models can miss key diagnoses and introduce wrong ones. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To encode these requirements, we introduce Diagnostic Loss (DLoss), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DLoss separately aggregates probability mass over vocabularies associated with diagnoses supported by the reference report and with unsupported alternatives. Using asymmetric pooling across token positions in the report, it encourages supported diagnoses to receive sufficient probability in at least one position while suppressing spurious probability spikes for unsupported diagnoses across all positions during training. The loss can be applied to existing 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We train and evaluate DLoss on 50,000 abdominal CT scans across several 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Across these settings, DLoss improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, showing the value of asymmetric sequence-level supervision for radiology report generation. The code for DLoss is available at https://github.com/gmum/DLoss.

1 INTRODUCTION

Radiology reports must name supported diagnoses and avoid unsupported ones, but token-level training does not explicitly supervise this two-sided diagnostic obligation. DLoss adds asymmetric sequence-level supervision and is evaluated across six 3D MedVLMs on in- and out-of-distribution abdominal CT benchmarks.

1 INTRODUCTION

  • Token-level maximum-likelihood training treats diagnostic and narrative tokens alike, leaving little sequence-level pressure to name supported diagnoses and avoid unsupported ones.Fluent reports can still omit, weaken, or introduce diagnoses, errors with greater clinical consequence than stylistic errors.Source: §1 · pp. 2–3
  • DLoss directly supervises decoder probabilities, rewarding supported diagnostic targets somewhere in the report while penalizing probability spikes for unsupported targets across positions.The loss is differentiable and architecture-agnostic, and can be integrated without changing the underlying backbone, tokenizer, or report-level supervision.Source: §1 · p. 3
  • Concept-level diagnostic targets map associated words and phrases into each decoder’s existing tokenizer, enabling supervision across tokenizers without a task-specific prediction head.The objective does not alter the inference procedure.Source: §1 · p. 3
  • The evaluation uses 50,000 abdominal CT scans across several 3D MedVLMs and in- and out-of-distribution report-generation benchmarks.The in-distribution experiments use Merlin; out-of-distribution performance is evaluated on a private in-house abdominal CT dataset.Source: §1 · p. 3
  • Across six 3D MedVLMs, DLoss yields mean gains of 4.05 GREEN points and 3.82 RadGraph-F1 points on Merlin, with positive language-metric gains on both test sets.On the external test set, mean gains are 4.28 GREEN points and 2.96 RadGraph-F1 points.Source: §1 · p. 3

2 RELATED WORK

Prior work spans medical report-generation architectures, clinical evaluation metrics, and training objectives that incorporate clinical information. DLoss complements architectural advances by changing decoder-distribution supervision and differs from existing objective modifications in its target construction and pooling across positions.

MedVLMs for Radiology Report Generation

  • For volumetric CT, recent report-generation models include fVLM, RadFM, HLIP, BTB3D, SPECTRE, and Merlin; DLoss changes supervision rather than the vision or language architecture.Earlier work includes CNN–RNN and hierarchical recurrent models, Transformer generators, pretrained vision–language representations, and generative VLMs.Source: §2 · p. 4

Clinical Supervision and Training Objectives

  • Clinical fidelity measures such as RadGraph-F1 and GREEN assess diagnostic errors that lexical metrics may miss, while prior training approaches include semantic rewards, token reweighting, disease-aware generation, and preference learning.RadGraph-F1 extracts entity-relation knowledge graphs, and GREEN uses domain-tuned models.Source: §2 · p. 4
  • Unlike SAT-RRG and unlikelihood training, DLoss defines supported and unsupported targets from each reference and pools their probabilities across report positions.Supported-target pooling allows probability at any position; unsupported-target pooling limits probability spikes throughout the sequence.Source: §2 · p. 4

3 METHODOLOGY

DLoss adds diagnostic-target supervision to autoregressive report training without changing the model architecture or inference procedure. It uses organ-level target sets and asymmetric pooling so supported diagnoses receive probability somewhere in the report while unsupported targets remain unlikely across positions.

3.1 PRELIMINARIES

  • DLoss operates on decoder probabilities already computed during training, complementing token-level cross-entropy with explicit supervision for diagnostic targets.The decoder predicts report tokens from image-derived inputs and a textual prompt, conditioned on the reference prefix.Source: §3.1 · pp. 4–5
  • For 3D CT, each study–report pair is decomposed into anatomy-level image–text pairs, with organ regions segmented and findings parsed into organ-level reports.Targets are scoped to each organ-level pair because applying all organ targets to a full report could give opposing signals for the same word.Source: §3.1 · p. 5
  • Each diagnostic concept maps to tokenizer-specific vocabulary indices representing lexical realizations, including synonyms gathered from organ-specific training reports through semantic tagging.The lexical sets are corpus-grounded and do not cover every possible paraphrase; the method provides token-level rather than exact phrase-level supervision.Source: §3.1 · p. 5

3.2 DIAGNOSTIC LOSS

  • Different pooling operators encode the two diagnostic requirements: noisy-OR rewards supported targets receiving probability at any valid position, while temperature-controlled log-sum-exp suppresses unsupported-target peaks.Higher temperature concentrates unsupported-target pooling increasingly on the position with the largest probability.Source: §3.2 · pp. 5–6
  • Non-negative coefficients λ+ and λ− independently weight supported- and unsupported-target components, and sample weight s_i scales the sample-level diagnostic loss.The target-level weights are set to one in all reported experiments.Source: §3.2 · pp. 6–7
  • The weighted diagnostic loss is added to cross-entropy during training; target sets are discarded at inference, so decoding remains unchanged and adds no inference-time computation.Numerical stability is addressed by clamping logarithm arguments away from zero in the implementation.Source: §3.2 · pp. 6–7

4 EXPERIMENTS

Across six 3D MedVLMs, DLoss improves diagnostic metrics on internal and external benchmarks, with gains in most model–metric–test comparisons. Experiments also examine training cost, qualitative report differences, and DLoss component and pooling variants.

4.1 EXPRIMENTAL SETUP

  • Experiments compare six MedVLMs fine-tuned with and without DLoss under identical conditions, using Merlin and CancerVerse for training and Merlin and an in-house dataset for evaluation.The Merlin test set provides in-distribution assessment and the in-house set provides out-of-distribution evaluation; GREEN and RadGraph-F1 assess diagnostic accuracy, while BLEU, ROUGE-L, METEOR, and CIDEr assess surface-level overlap.Source: §4.1 · pp. 7–8

4.2 EXPERIMENTAL RESULTS

  • GREEN and F1-RG rise by 4.05 and 3.82 points on Merlin, and by 4.28 and 2.96 points externally; 71 of 72 comparisons improve.Every external comparison improves; the sole exception is a 0.01-point F1-RG decrease for Merlin.Source: Table 1 · p. 8, §4.2 · p. 8
  • Across four 3D MedVLMs, the largest measured step-time increase is 4.4%, peak allocated and reserved memory rise by at most 1.6% and 2.9%, and DLoss adds no inference-time computation.Table 2 profiles matched batches on a single GH200 GPU; step times average 12 fresh-process runs per condition across six blocks, and memory averages three runs per condition.Source: §4.2 · p. 8, Table 2 · p. 8
  • Across four CT studies, qualitative examples show DLoss mitigating baseline false positives and false negatives while retaining findings aligned with the reference.Figure 2 pairs representative CT slices with reference, baseline, and baseline + DLoss excerpts by organ; green marks reference-consistent text and red marks unsupported baseline statements.Source: §4.2 · pp. 8–9, Figure 2 · p. 9

4.3 ABLATION STUDIES

  • On fVLM with seed 13, full DLoss improves BLEU-4, GREEN, and F1-RG over its matched CE baseline, while single-target and alternative-pooling variants lower all three scores.The ablation uses the full Merlin test set; U-mean produces the largest declines.Source: §4.3 · p. 9

5 CONCLUSIONS

DLoss is presented as a lightweight, plug-and-play objective that improves diagnostic faithfulness while retaining competitive language-generation performance. The conclusion notes that its evaluation remains limited to abdominal CT and automatic metrics.

5 CONCLUSIONS

  • Table 3 presents component and pooling ablations on fVLM, with percentage changes relative to the CE baseline.Source: Table 3 · p. 9
  • DLoss improves six state-of-the-art report-generation methods across six evaluation metrics while maintaining competitive language-generation performance and enhancing diagnostic faithfulness.Source: §5 · p. 9
  • The controlled diagnostic target space is derived from training reports, so direct supervision covers only concepts and expressions represented there.Source: §5 · p. 9
  • Experiments remain within abdominal CT, and reported scores rely on automatic metrics; testing on 2D X-rays and against radiologist judgments is proposed.Source: §5 · pp. 9–10

A DATA

The data section identifies abdominal CT datasets used for training and testing and directs readers to a table summarizing their composition and splits.

A DATA

  • The study evaluates Diagnostic Loss on abdominal CT data and summarizes the training and test datasets in Table 4.Source: A DATA · p. 16
  • The dataset descriptions are separated between the training data in Section A.1 and test data in Section A.2.Source: A DATA · p. 16
  • Table 4 reports scan, exam, and patient counts alongside training and test splits for the 3D datasets.Source: Table 4 · p. 16

A.1 TRAINING SET

The appendix describes abdominal CT training and test data and illustrates report preprocessing, diagnostic target construction, and the loss’s sequence-level supervision.

A.1.1 MERLIN

  • The Merlin training split contains 15,309 CT scans, with 5,060 scans reserved for validation from a 20,369-scan set.Source: §A.1.1 · p. 16

A.1.2 CANCERVERSE

  • The training set comprises 24,168 multi-phase CT scans from 13,863 patients1, and all cases are used for training; each CT exam is paired with its corresponding radiology report.Source: §A.1.2 · p. 16

A.2.1 MERLIN

  • Merlin’s official test set contains 5,125 contrast-enhanced CT scans.Source: §A.2.1 · p. 16

A.2.2 IN-HOUSE

  • The dataset comprises 2,706 multi-center, contrast-enhanced CT scans from 1,948 patients, each paired with its radiology report; scans were acquired across 300 sites within a nationwide cohort.Source: §A.2.2 · p. 16

B REPORT PREPROCESSING

  • Figure 3 illustrates Qwen3 converting a free-text CT report into cleaned text and findings for 11 organ groups.Source: Figure 3 · p. 17

B.1 REPORT PREPROCESSING EXAMPLE

  • Figure 4 shows raw report text, cleaned current-study text, selected organ-specific outputs, and binary organ labels; the complete output covers all 11 organ groups.Source: Figure 4 · p. 18, §B.1 · p. 18

C DIAGNOSTIC SUPERVISION CONSTRUCTION

D DIAGNOSTIC LOSS ILLUSTRATIONS

E.1 IMPLEMENTATION DETAILS

Implementation details describe shared training configurations for CE and DLoss runs, while the metric appendix explains the lexical overlap measures used to assess generated reports.

E.1 IMPLEMENTATION DETAILS

  • During report-generation training, visual features are fixed while decoder LoRA parameters and the visual-to-text projector or adapter are trained.Source: §E.1 · p. 21
  • For each backbone, CE and DLoss runs use the same decoder, tokenizer, visual initialization, and pooled Merlin–CancerVerse training data.Source: §E.1 · p. 21
  • The full decoder weights remain frozen, and both objectives use the same backbone-specific LoRA configuration and trainable projector or adapter.Source: §E.1 · p. 21
  • Main-experiment DLoss coefficients were selected by a small grid search, and all runs use τ = 8.Source: §E.1 · p. 21, Table 5 · p. 21

E.2.1 LEXICAL METRICS

  • BLEU-4 combines clipped n-gram precisions through a geometric mean and brevity penalty, which reduces scores for outputs shorter than the reference.Source: §E.2.1 · pp. 21–22
  • ROUGE-L measures overlap using the longest common subsequence, preserving word order without requiring adjacent matches.Source: §E.2.1 · p. 22
  • METEOR aligns unigrams through exact matches, stems, and synonyms, then weights recall more than precision and penalizes fragmented matching chunks.Source: §E.2.1 · p. 22
  • CIDEr weights n-grams by their informativeness in the evaluation collection and averages cosine-similarity scores across the first four n-gram orders.Source: §E.2.1 · pp. 22–23

E.2.2 CLINICAL-CONTENT METRICS

Clinical-content metrics compare report meaning rather than surface wording. The section describes RadGraph’s structured entity-and-relation matching and GREEN’s structured error evaluation, while noting that neither establishes clinical safety on its own.

Clinical-content metrics

  • These metrics compare report meaning rather than only surface wording, but are automatic evaluations, not organ-specific radiologist reviews or evidence of clinical safety.The experiments compute RadGraph and GREEN with RadEval.Source: §E.2.2 · p. 23
  • RadGraph compares extracted clinical structures in generated and reference reports, including observations, anatomy, assertion status, and relations.The reported complete F1 variant scores agreement on entities and relations; empty reports or graphs contribute zero.Source: §E.2.2 · p. 23
  • GREEN uses a language-model evaluator to identify report errors, including false or missing findings, wrong locations or severity, and unsupported comparisons.Its score aggregates matched findings and clinically significant errors; experiments use RadEval’s local GREEN-RadLLaMA2-7b evaluator.Source: §E.2.2 · pp. 23–24

F ROBUSTNESS ACROSS TRAINING SEEDS

Nine matched BTB3D seed runs on Merlin test whether DLoss gains persist across training runs and resampled studies. Clinical and lexical metrics show positive average effects, though results vary across runs and do not establish robustness beyond this protocol.

F ROBUSTNESS ACROSS TRAINING SEEDS

  • The paired comparison evaluated CE and DLoss BTB3D runs across nine matched seeds on 5,125 Merlin studies and 56,375 organ-specific report rows.All rows, including empty generations, remained in metric denominators; each seed pair used the same test rows.Source: F ROBUSTNESS ACROSS TRAINING SEEDS · p. 25
  • The analysis averaged within-seed DLoss-minus-CE differences and used 5,000-resample bootstrap intervals, with a second bootstrap sampling studies while retaining their organ rows.Pairing preserves comparison within a training seed, and study-level resampling respects clustering of organ rows.Source: F ROBUSTNESS ACROSS TRAINING SEEDS · p. 25
  • GREEN and RadGraph Complete had positive mean differences with both bootstrap intervals above zero; each improved in eight of nine seed pairs.BLEU-4 improved in seven of nine pairs, so positive mean effects did not mean every trained model improved.Source: Table 6 · p. 25, F ROBUSTNESS ACROSS TRAINING SEEDS · p. 25

F.1 VARIATION ACROSS INDIVIDUAL RUNS

  • Table 7 displays within-seed effects for representative metrics, making run-to-run variation visible alongside average effects and intervals.Intervals are unadjusted for multiple metrics and apply to this BTB3D-on-Merlin protocol, not other backbones or the external test cohort.Source: §F.1 · p. 26, Table 7 · p. 26

G ABLATION STUDY DETAILS

The fVLM ablations compare DLoss components and alternative pooling choices against a matched cross-entropy baseline. Full DLoss improves all three reported metrics in this run, while component-only and alternative-pooling variants score lower.

G ABLATION STUDY DETAILS

  • Full DLoss improves all three reported metrics over the matched CE run, while either target-loss component alone lowers them.The ablation uses fVLM, seed 13, and the full Merlin test set; percentage changes use the matched CE run as baseline.Source: G ABLATION STUDY DETAILS · p. 26
  • Supported-target maximum pooling discards evidence beyond the strongest position, while mean pooling can dilute evidence localized to a few positions; both score below noisy-OR here.U-mean has the largest declines, consistent with penalizing high unsupported-target probability even when it occurs at one position.Source: G ABLATION STUDY DETAILS · p. 26
  • The DLoss run generated 1,423 empty findings (2.52%), versus 48 (0.09%) for CE, and both counts remain in metric denominators.Source: G ABLATION STUDY DETAILS · p. 26
Loading 2610.30980v1…