PaperlayerView exact revision on arXiv

Source-linked AI summary

Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation

Janhavi Prabhu, Sahil, Akshay V, Shivam Shukla, Manoj Tadepalli, Preetham Putha

arXiv:2609.29156v1cs.CVcs.AI

TL;DR

Long-tailed chest X-ray classification needs representations that capture common and rare findings, but controlled evidence comparing radiology-native autoregressive and medical contrastive pretraining remains limited. Med-AR uses heterogeneous radiology supervision and is evaluated for discrimination and selective prediction. It improves discrimination across public benchmarks, while selective-prediction results vary by dataset.

  • Problem

    It remains unclear whether radiology-native autoregressive pretraining transfers better than medical contrastive pretraining for long-tailed CXR classification under controlled conditions, despite the importance of rare findings and uncertainty-aware evaluation.

    Source: §1 · pp. 2–3

  • Method

    Med-AR compares autoregressive and contrastive InternViT pretraining with a shared ML-Decoder and uses structured reports, abnormality-focused targets, region annotations, and LLM-expanded labels.

    Source: §1 · p. 3

  • Results

    Across three public benchmarks, Med-AR improves discrimination over Med-CLIP across prevalence groups; Med-AR-8B leads on MIMIC-CXR and CheXpert, Med-AR-2B on PadChest, and EAURC results differ by dataset.

    Source: §5.6 · p. 31

  • Takeaways & Limitations

    Discrimination and confidence-based risk ordering are distinct evaluation axes, since stronger discrimination does not guarantee better selective prediction.

    Source: §6 · p. 33

    The comparison controls downstream architecture but does not isolate next-token prediction, because supervision, initialization, and compute may differ and no matched-supervision ablation is available.

    Source: §5.7 · p. 32

Abstract

from arXiv · show

Long-tailed chest X-ray classification requires visual representations that capture both common abnormalities and subtle, infrequent findings. We propose Med-AR-8B and Med-AR-2B, two radiology-native autoregressive vision-language models pretrained with structured reports, abnormality-focused text, and region annotations. We evaluate the transfer of their visual encoders to multi-label classification against contrastive, self-supervised, and supervised pretrained encoders, including Med-CLIP, CheXFound, EVA-Base, ARK, and BioViL-T, using a common ML-Decoder classification head. To assess fine-grained recognition, we also construct LLM-expanded, report-derived label sets for MIMIC-CXR and CheXpert. Across PadChest, MIMIC-CXR, and CheXpert, Med-AR-8B outperforms Med-CLIP in mean AUROC and AUPRC for head, medium, and tail findings. On MIMIC-CXR, it increases tail-label mean AUPRC from 0.1033 to 0.1441. Med-AR-2B achieves the strongest discrimination results on PadChest. Across the broader encoder comparison, a Med-AR variant achieves the highest mean AUROC and AUPRC in every reported prevalence group on each public dataset. Both Med-AR variants also achieve lower excess area under the risk-coverage curve than Med-CLIP on all three public datasets, indicating improved selective-prediction performance under the evaluated protocol. Internal results are metric-dependent, with Med-CLIP retaining advantages in overall and tail AUPRC and in selective prediction. These findings establish Med-AR as a strong pretraining recipe for long-tailed chest X-ray classification on the evaluated public benchmarks and demonstrate the value of assessing discrimination and selective prediction together.

1 Introduction

Long-tailed chest X-ray classification must capture rare, subtle findings, yet evidence for autoregressive pretraining under controlled comparisons remains limited. Med-AR combines heterogeneous radiology supervision and is evaluated for both discrimination and selective prediction.

1 Introduction

  • Rare CXR findings can be subtle, localized, and dependent on anatomical context, so robust long-tail recognition requires local visual detail alongside clinically meaningful global context.Aggregate metrics can conceal poor performance on rare conditions, where reliable recognition may be especially important.Source: §1 · p. 2
  • Whether radiology-native autoregressive pretraining transfers better than medical contrastive pretraining under controlled conditions remains unclear, and prior evaluations emphasize aggregate discrimination over long-tail and uncertainty behavior.Comparisons often vary in architecture, pretraining corpus, optimization, and downstream protocol, complicating isolation of the pretraining objective.Source: §1 · p. 3
  • Across an internal dataset, MIMIC-CXR, CheXpert, and PadChest, autoregressive pretraining achieves stronger downstream discrimination on public benchmarks and lower EAURC on public datasets, while contrastive pretraining has lower internal EAURC.Source: §1 · p. 3
  • The study compares contrastive and autoregressive pretraining with independently fine-tuned InternViT encoders and a shared ML-Decoder, using structured reports, abnormality-focused targets, and region annotations.LLM-expanded label sets and risk-coverage metrics extend evaluation to fine-grained recognition and uncertainty-aware performance.Source: §1 · p. 3

2 Related Work

Prior work addresses long-tailed CXR classification mainly through downstream imbalance methods, while vision-language research explores contrastive and autoregressive supervision. Uncertainty-aware evaluation remains comparatively underexplored in this setting.

2.1 Long-tailed chest X-ray classification

  • Long-tail CXR methods mainly modify downstream training, while rare findings can require localized evidence and anatomical context beyond coarse global features.Report supervision has consequently drawn interest as a source of clinically meaningful context for fine-grained findings.Source: §2.1 · p. 4

2.2 Contrastive pretraining

  • Contrastive methods align reports and images; GLoRIA, MedKLIP, and dense vision-language approaches add local or knowledge-guided structure relevant to subtle findings.Large language models have also been used to extract structured disease information from reports.Source: §2.2 · p. 4, §2.1 · p. 4

2.3 Autoregressive pretraining

  • Autoregressive pretraining supports diverse textual supervision, including reports, structured findings, and region descriptions, but its advantage over medical contrastive pretraining for long-tail CXR classification remains insufficiently established under controlled comparisons.Prior comparisons often vary in architecture, optimization, and downstream evaluation protocol.Source: §2.3 · p. 5

2.4 Uncertainty-aware evaluation in medical imaging

  • Risk-coverage analysis assesses whether confidence ranks errors appropriately for possible clinical deferral, yet uncertainty-aware evaluation remains underexplored in long-tailed medical vision-language studies.Rare findings may be more difficult to recognize and more prone to overconfident failures.Source: §2.4 · pp. 5–6

3 Methodology

The methodology compares contrastive and autoregressive pretraining for long-tailed chest X-ray classification under a shared downstream architecture. It evaluates transferred encoders with multi-dataset frequency-group analysis, including report-derived expanded labels.

3.1 Overview

  • Both pretraining recipes use InternViT-family encoders and the same ML-Decoder downstream head, controlling downstream architecture but not isolating objective from supervision or optimization.The autoregressive recipe additionally uses structured and region-level targets.Source: §3.1 · pp. 6, 8

3.2 Shared Vision Backbone Architecture

  • InternViT-300M is the shared transformer image backbone, designed for high-resolution visual understanding across both pretraining paradigms.The autoregressive encoder operates within InternVL3; the contrastive encoder is trained with the MedCLIP-style recipe.Source: §3.2 · p. 8

3.3 Autoregressive Vision-Language Pretraining

  • Autoregressive pretraining conditions report, summary, abnormality, and region-text generation on X-rays; Med-AR-2B and Med-AR-8B vary decoder scale while transferring only the encoder.The decoder is trained through conditional next-token prediction, then discarded before downstream classification.Source: §3.3 · pp. 8–9

3.4 MedCLIP-Style Contrastive Pretraining

  • The MedCLIP-style baseline uses semantic image-report matching, including non-paired reports with related medical labels; the comparison tests complete pretraining recipes, not objective effects alone.Its semantic-matching targets come from report-derived labels.Source: §3.4 · pp. 9–10

3.5 Downstream Fine-Tuning with ML-Decoder

  • ML-Decoder uses learnable queries to predict abnormality labels from encoder features, supporting large label spaces while modeling label dependencies and co-occurrence.Its logits are converted to class probabilities with sigmoid activation.Source: §3.5 · p. 10

3.6 Loss Function for Long-Tailed Multi-Label Classification

  • Class-specific weighting increases rare-positive-class contributions while preserving negative examples; the objective addresses inter-class imbalance by up-weighting rare classes and intra-class imbalance by down-weighting easy negatives, using γ+=1, γ−=4, and probability margin m=0.05.Source: §3.6 · pp. 10–11

3.7 Long-tailed evaluation protocol

  • Evaluation covers an internal dataset and three public datasets, with dataset-specific frequency groups and LLM-expanded MIMIC-CXR and CheXpert labels probing rare-label transfer.Head, medium, and tail groups are based on positive training counts, with thresholds derived from the 33rd and 67th percentiles.Source: §3.7 · pp. 11–12

4 Experiments

The experiments prepare four CXR datasets, expand report-derived labels for MIMIC-CXR and CheXpert, and evaluate extraction concordance, implementation choices, and classification metrics. The evaluation also uses paired-label tests to assess differences between pretraining paradigms.

4.1 Dataset Preparation

  • Autoregressive pretraining uses 1.4 million internal CXRs with reports, structured across 15 anatomical subheadings plus findings, impression, summary, reasoning, and prominence sections, and includes expert bounding boxes for about 20 findings.Qwen3 generates structured report-derived targets; dedicated prompt templates support consistent JSON outputs across sources.Source: Pre-training · pp. 12–13
  • The four datasets are trained and evaluated separately, with models tested on each dataset’s own held-out split; the combined splits contain 1,872,398 training and 297,608 test images.The datasets are Internal, PadChest, CheXpert, and MIMIC-CXR.Source: Finetuning · p. 13

4.2 LLM-based Label Extraction and Concordance Analysis

  • For MIMIC-CXR and CheXpert, an LLM converts report text into structured presence-or-absence labels, restricts positives to documented findings, and maps variable terms to canonical labels.An other_findings field captures abnormalities outside the predefined vocabulary.Source: §4.2 · p. 17
  • Across 1,000 reports and seven LLMs, mean pairwise Cohen’s κ was 0.83 by individual label and 0.87 by parent group; parent-group Fleiss’ κ was 0.79–0.98, versus 0.56 for other_findings.The authors attribute the higher parent-group agreement to differences in label granularity and the lower other_findings agreement to its heterogeneous vocabulary.Source: §4.2 · p. 17
  • Residual extraction disagreement often reflected synonym routing: nodules replaced a missed solitary_pulmonary_nodule in 96% of cases, and pleural_effusion replaced a missed effusion in 95%.Artifacts were an exception, with model-specific over-extraction described as annotation noise.Source: §4.2 · p. 18
  • Expanded labels are report-derived weak supervision, not independently validated image-level ground truth; without expert reference labels, agreement does not establish clinical precision or recall.Shared model errors may remain undetected, and parent-group agreement may conceal fine-grained label errors.Source: §4.2 · p. 18

4.3 Implementation

  • The implementation uses AdamW for MedCLIP pretraining and fine-tunes the encoders with ML-Decoder, asymmetric loss, AdamW, and a OneCycle scheduler.The reported fine-tuning batch size is 2304, maximum learning rate is 5e-05, and both MedCLIP and Med-AR training use 8 H100 GPUs.Source: §4.3 · pp. 18–19

4.4 Evaluation Metrics and Statistical Analysis

  • Performance is assessed with per-label AUROC, AUPRC, sensitivity, specificity, and EAURC, averaged overall and across head, medium, and tail subsets.EAURC evaluates confidence-based ranking of prediction difficulty; lower values indicate lower excess selective risk under the evaluated confidence ordering, not calibration or deployment readiness.Source: §4.4 · p. 19

5 Results and Discussion

Across the three public benchmarks, Med-AR variants generally outperform Med-CLIP on discrimination, while internal results are mixed and uncertainty ranking does not always track discrimination. The broader encoder comparison supports Med-AR transfer under a common downstream head, but the experiments do not isolate the pretraining objective as the cause.

5.1 Main quantitative comparison

  • Across public datasets, preferred variants differ: Med-AR-2B beats Med-CLIP across PadChest label groups (overall AUROC 0.8657→0.8800; AUPRC 0.4310→0.4595), while Med-AR-8B leads MIMIC-CXR and CheXpert.On CheXpert, Med-AR-8B raises overall AUROC/AUPRC from 0.8585/0.2142 for MedCLIP to 0.8759/0.2391; Med-AR-2B does not surpass Med-CLIP.Source: §5.1 · p. 20

5.2 Statistical significance across labels

  • Wilcoxon tests support nominally significant gains for both variants across PadChest and MIMIC-CXR; on CheXpert, gains are broad for Med-AR-8B but absent for Med-AR-2B.Internal results are less uniform: Med-AR-8B comparisons are non-significant, while Med-AR-2B gains are concentrated in AUROC and higher-prevalence groups.Source: §5.2 · pp. 22, 24

5.3 Label-wise AUROC differences

  • Label-wise AUROC differences favor autoregressive pretraining broadly on public datasets, while Internal results are heterogeneous, especially among tail labels.The figures mark statistically significant per-label differences under DeLong’s test.Source: §5.3 · p. 25, Figure 7 · p. 26, Figure 8 · p. 27

5.4 Uncertainty-aware evaluation

  • Med-AR improves EAURC over Med-CLIP on all public benchmarks, but Med-CLIP leads on Internal; CheXpert shows that discrimination and confidence-based risk ordering can diverge.On CheXpert, Med-AR-2B has lower AUROC and AUPRC than Med-CLIP yet the best EAURC across prevalence groups.Source: §5.4 · pp. 26–28, Table 5 · p. 28

5.5 Comparison of pretrained encoders

  • Under the common ML-Decoder setup, Med-AR variants lead the broader encoder comparison across prevalence groups on CheXpert, MIMIC-CXR, and PadChest.Med-AR-8B leads on CheXpert and MIMIC-CXR, while Med-AR-2B leads on PadChest; this compares full pretrained systems rather than isolating components.Source: §5.5 · pp. 29, 31, Table 6 · p. 30

5.6 Discussion

  • Med-AR-8B leads MIMIC-CXR and CheXpert while Med-AR-2B leads PadChest; Internal results are mixed, and discrimination gains need not improve risk ordering.Tail-label gains are consistent with a benefit from structured, region-aware supervision, but do not demonstrate the proposed visual-grounding mechanism.Source: §5.6 · pp. 31–32

5.7 Limitations

  • The controlled downstream architecture does not isolate the pretraining objective, because recipes may differ in supervision, initialization, and compute.The expanded MIMIC-CXR and CheXpert labels are report-derived; inter-model concordance does not establish expert-level correctness, and unmentioned findings may be negatives.Source: §5.7 · p. 32
  • Separate dataset fine-tuning is not unseen-institution deployment; EAURC evaluates selective prediction under this protocol, not calibration or clinical utility.The statistical analyses also do not quantify training-seed variability or establish equivalence for non-significant comparisons.Source: §5.7 · p. 32

6 Conclusion and Future Work

On the public benchmarks, Med-AR improves discrimination over Med-CLIP across prevalence groups, while internal results and selective prediction are more mixed. The authors conclude that discrimination and uncertainty behavior should be assessed together and identify component-wise ablations and other extensions as future work.

6 Conclusion and Future Work

  • The study compares contrastive and autoregressive pretraining using a shared InternViT vision architecture and common ML-Decoder head across an internal benchmark and three public datasets.The comparison controls the downstream architecture while evaluating the pretraining recipes.Source: §6 · pp. 32–33
  • Across the three public benchmarks, Med-AR-8B improves AUROC and AUPRC over Med-CLIP in every prevalence group, while Med-AR-2B performs best on PadChest.The encoder comparison favors Med-AR under the evaluated setup, but separate contributions of the objective, supervision, and decoder scale remain unresolved.Source: §6 · p. 33
  • On the internal benchmark, Med-AR-2B has a modest AUROC edge with partial statistical support, mixed AUPRC results, and worse selective-prediction behavior than Med-CLIP.Across CheXpert and the internal dataset, discrimination and confidence-based risk ordering can diverge; on CheXpert, Med-AR-2B has the best EAURC across prevalence groups despite lower AUROC and AUPRC than Med-CLIP.Source: §6 · p. 33
  • Future work includes component-wise ablations, rare-label learning strategies, calibration methods, broader label-expansion evaluation, and assessment in clinical workflows.The authors identify disentangling the autoregressive objective from richer supervision as an open question.Source: §6 · p. 33
  • The authors position radiology-native autoregressive pretraining as a strong foundation for long-tailed CXR classification on public benchmarks, particularly for underrepresented findings.They recommend joint assessment of discrimination and uncertainty behavior because the best configuration can differ by dataset and evaluation axis.Source: §6 · pp. 33–34

7 Statements and Declarations

The statements report author contributions, planned release of derived annotations subject to dataset access and licensing, and no relevant competing interests.

7 Statements and Declarations

  • All authors contributed to study conception and design, with individual roles spanning training, methodology, dataset preparation, implementation, oversight, and manuscript revision.All authors reviewed and approved the final manuscript.Source: Author Contributions · p. 34
  • The authors plan to release derived MIMIC-CXR and CheXpert annotations and split identifiers after publication, subject to source-dataset access and licensing requirements.Original images and reports remain subject to their respective access conditions.Source: Data Availability · p. 34
  • The authors declare no competing interests relevant to the paper.Source: Data Availability · p. 34

Supplementary Material

The supplement documents report-derived targets, label construction and concordance checks, operating-point metrics, and top-2 performance summaries across public CXR datasets.

Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification

  • The supplement presents per-label results by dataset and marks tail and medium-frequency labels, with bold and underlining indicating the highest and second-highest displayed values rather than significance tests.Its sections cover target preparation, cross-dataset summaries, and per-label metrics for the internal dataset, CheXpert, MIMIC-CXR, and PadChest.Source: Med-AR: Autoregressive Vision-Language… · p. 39

S1.1 Pretraining Target Structure

  • Pretraining targets combine report sections and 15 anatomical subheadings with finding-specific fields, report-derived prominence scores, and separate expert box supervision for approximately 20 findings.Non-mention is mapped to negation, which may introduce false-negative supervision; reasoning and prominence fields are synthetic targets, not independent expert image annotations.Source: S1.1 Pretraining Target Structure · p. 39

S1.2 LLM-Based Label Expansion Pipeline

  • Label construction parses hierarchical JSON into parent-grouped child-tag presence values; unparsed responses are missing, not negative.Only binary presence is used, while tags in a successfully parsed report's vocabulary but not emitted are treated as implicitly absent.Source: S1.2 LLM-Based Label Expansion Pipeline · p. 39
  • The open-ended vocabulary produced 5,586 distinct keys across seven models, but only 193 canonical tags were emitted by all seven; much variation reflected lexical fragmentation.Normalization standardizes formatting and maps equivalent clinical concepts while retaining clinically related but distinct findings as separate labels.Source: S1.2 LLM-Based Label Expansion Pipeline · pp. 39, 42
  • Parent-level mapping increased mean pairwise Cohen’s κ from 0.83 to 0.87, and many child-label discrepancies reflected synonymous or hierarchical routing rather than demonstrated clinical correctness.The supplement highlights substitutions such as nodules for solitary_pulmonary_nodule and pleural_effusion for effusion, while keeping potentially distinct categories separate.Source: S1.2 LLM-Based Label Expansion Pipeline · p. 40, Figure S1 · p. 40
  • Artifacts was treated as a model-dependent catch-all rather than merged with co-occurring clinical findings, and taxonomy auditing considered vocabulary coverage, prevalence, agreement, and single-model calls.Only 92 reports reached majority consensus for artifacts although at least one model emitted it in 228 reports.Source: S1.2 LLM-Based Label Expansion Pipeline · p. 42, Figure S4 · p. 42

S2.2 Top-2 Win Rate Analysis for Public Datasets

S2.1 Operating-Point Metrics

  • Public-dataset sensitivity, specificity, and Youden’s J use per-label thresholds selected by maximizing J on the validation set, not clinically validated deployment thresholds.Source: S2.1 Operating-Point Metrics · p. 43

S3 Internal Dataset: Per-Label Results

The internal per-label supplement reports validation-set metrics for 40 labels and describes variation across labels and metrics, with neither recipe uniformly superior.

S3 Internal Dataset: Per-Label Results

  • The table preserves validation-set AUROC, AUPRC, sensitivity, specificity, and Youden’s J results for 40 labels.Tail abnormalities are marked with an asterisk and medium-prevalence abnormalities with a dagger.Source: S3 Internal Dataset: Per-Label Results · p. 45, Table S1 · p. 45
  • Neither pretraining recipe is uniformly superior across the reported internal validation-set labels and metrics.Source: S3 Internal Dataset: Per-Label Results · p. 45
  • Sensitivity, specificity, and Youden’s J thresholds are selected per abnormality by maximizing J = Sens+Spec−1; the table marks each row’s best and second-best value for each metric.Source: Table S1 · p. 45

S4 CheXpert: Per-Label Results

The section lists CheXpert per-label results for 115 labels across five classification metrics.

S4 CheXpert: Per-Label Results

S4.1 AUROC

S5 MIMIC-CXR: Per-Label Results

The section lists available MIMIC-CXR per-label results, covering 108 of the 110 labels in the evaluated vocabulary.

S5 MIMIC-CXR: Per-Label Results

  • The available listings cover 108 of 110 labels, with AUROC, AUPRC, sensitivity, specificity, and Youden’s J reported in that order.Results for the remaining two labels are not included in these listings.Source: S5 MIMIC-CXR: Per-Label Results · p. 61

S5.1 AUROC

S6 PadChest: Per-Label Results

The section lists PadChest per-label results for 29 labels across five classification metrics.

S6 PadChest: Per-Label Results

S6.1 AUROC

Loading 2609.29156v1…