Source-linked AI summary
Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reports
Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, Yizhou Yu
TL;DR
Radiograph pre-training must learn transferable representations without the costly structured-label workflow, while self-supervised alternatives remain behind label-supervised pre-training. REFERS uses free-text radiology reports for cross-supervision and study-level multi-view representation learning, and it outperforms major alternatives under limited supervision. The paper therefore presents REFERS as a potential replacement for canonical pre-training, while noting that its assumptions include the usefulness of report reasoning and stable pre-trained representations.
Problem
Supervised radiograph pre-training requires complex, labor-intensive two-stage annotation, while self-supervised representations retain performance gaps versus label-supervised pre-training.
Method
REFERS cross-supervises radiographs and free-text reports, using report-based study-level signals and multi-view representation learning without structured labels during pre-training.
Results
REFERS outperforms self-supervised learning and natural-image transfer by substantial margins, with about 2.5% improvements when fewer than 10k training images are used and 90% fewer annotated target-domain data needed on NIH ChestX-ray at a 10% label ratio.
Takeaways & Limitations
REFERS has the potential to replace canonical pre-training methodologies for radiograph analysis under limited supervision.
Takeaways & Limitations
REFERS assumes that abstract, complex reasoning sentences in radiology reports provide sufficient information for transferable visual-feature learning and that pre-trained representations are relatively stable.
Abstract
from arXiv · showhide
Pre-training lays the foundation for recent successes in radiograph analysis supported by deep learning. It learns transferable image representations by conducting large-scale fully-supervised or self-supervised learning on a source domain. However, supervised pre-training requires a complex and labor intensive two-stage human-assisted annotation process while self-supervised learning cannot compete with the supervised paradigm. To tackle these issues, we propose a cross-supervised methodology named REviewing FreE-text Reports for Supervision (REFERS), which acquires free supervision signals from original radiology reports accompanying the radiographs. The proposed approach employs a vision transformer and is designed to learn joint representations from multiple views within every patient study. REFERS outperforms its transfer learning and self-supervised learning counterparts on 4 well-known X-ray datasets under extremely limited supervision. Moreover, REFERS even surpasses methods based on a source domain of radiographs with human-assisted structured labels. Thus REFERS has the potential to replace canonical pre-training methodologies.
1 Introduction
REFERS addresses limitations of natural-image transfer, self-supervised learning, and labor-intensive structured-label extraction by learning radiograph representations directly from free-text reports. It cross-supervises radiographs and reports, fuses multiple study views, and improves transfer under limited supervision.
- Motivation and contribution: Structured-label extraction requires complex, labor-intensive human assistance, and mistakes in individual steps or tools can produce disastrous annotation results.Annotators must define alternate spellings, synonyms, and abbreviations for each target label.
- Motivation and contribution: REFERS learns transferable radiograph representations directly from accompanying free-text radiology reports rather than requiring structured labels during pre-training.The method automatically cross-checks radiographs and reports to define supervision signals.
- Method: REFERS uses a radiograph transformer backbone and report transformer to learn joint representations through report generation and study-report consistency reinforcement.A contrastive loss reinforces consistency between radiograph-study and report representations.
- Method: An attention-based view fusion module processes all radiographs in each patient study simultaneously, preserving both study-level and image-level information.Traditional paradigms using a single image address only image-level information.
2 Results
Across NIH ChestX-ray, VinBigData, COVID-19, and Shenzhen Tuberculosis evaluations, REFERS generally performs best when target-domain supervision is limited. It exceeds self-supervised, natural-image transfer, and in several comparisons structured-label pre-training baselines.
- NIH ChestX-ray: REFERS achieves the highest AUC on all 14 NIH ChestX-ray classes across different amounts of fine-tuning data.Its largest advantages over self-supervised and ImageNet-based baselines occur with 0.8k training images, corresponding to a 1% label ratio.
3 Discussion
REFERS shows strong transfer performance under limited target-domain supervision, including advantages over self-supervised, natural-image, and structured-label pre-training. Its reported benefits extend across datasets, disease categories, annotation scales, and interpretability analyses.
- Transferability: REFERS outperforms self-supervised learning and natural-image transfer learning across four X-ray datasets, with advantages that persist across different amounts of annotated training data.The discussion reports consistent cross-dataset improvements and significant differences across varying label quantities.
- Supervision source: REFERS surpasses human-assisted structured-label pre-training, including LSP, and the discussion attributes its small-data advantage to richer supervision from raw radiology reports.The authors contrast free-text reports with structured labels and state that reports provide additional information for learning transferable radiograph representations.
- Annotation efficiency: REFERS reduces target-domain annotation needs: on NIH ChestX-ray, it needs 90% fewer annotated data to match Model Genesis and ImageNet pre-training, and on VinBigData, 10% data yields better results than those baselines at 100%.These comparisons are presented as evidence of high-quality pre-trained representations for fine-tuning with limited annotations.
- Clinical tasks: REFERS improves performance on both elusive and regular abnormalities, including emphysema, mass, atelectasis, and pneumothorax, using small numbers of target-domain radiographs.Reported gains include 5% on emphysema and mass, and over 7% and 9% on atelectasis and pneumothorax under especially limited supervision.
- Interpretability: REFERS produces attention regions that mostly achieve IoU above 0.5 with radiologists’ lesion boxes and can identify both large and small lesions.The discussion presents CAM-based localization as evidence that the learned representations can support lesion-focused analysis.
4 Methods
REFERS uses free-text radiology reports to provide cross-supervision without structured labels, while jointly learning representations from multiple radiographs in each patient study. Its evaluations and ablations examine performance under limited supervision and the contributions of report-related tasks, view fusion, and transformer architecture.
- Data and supervision: The pre-training corpus contains 217k patient studies organized from radiographs and associated reports.
- Data and supervision: REFERS uses radiology reports as freely available supervision, avoiding the two-stage human effort required to obtain structured labels.Reports contain Findings and Impressions sections; the pre-training data retain these sections after screening out reports with fewer than three tokens.
- Evaluation design: REFERS is evaluated against self-supervised, ImageNet-based, and label-supervised pre-training methods using matched transformer-based backbones for fairness.
- Cross-supervised learning: REFERS acquires cross-supervision through report generation and study-report representation consistency reinforcement.
- Ablation findings: Replacing the radiograph transformer with ResNet-101 lowers COVID-19 performance by about 7 percents, while the transformer-like architecture is reported as more effective with limited annotations.
- Ablation findings: Removing the two report-related tasks reduces performance by 2 percents, while removing either task alone changes performance by about 1 percent.
Data Availability
The paper provides source links for the VinBigData Chest X-ray Abnormalities Detection and COVID-19 Image Data Collection datasets.
- The VinBigData Chest X-ray Abnormalities Detection dataset is linked through its Kaggle competition page.
- The COVID-19 Image Data Collection dataset is linked through its GitHub repository.
Author Contributions Statement
The authors divide responsibility across idea conception, experiment design and execution, manuscript preparation, and analysis of data and results.
- H.Z. and Y.Y. conceived the idea and designed the experiments.
- H.Z., X.C. and Y.Z. implemented and performed the experiments, while H.Z. and Y.Y. wrote the manuscript.
- All authors analyzed the data and experimental results and commented on the manuscript.
Tables
The tables compare REFERS with self-supervised, transfer-learning, and human-assisted label-supervised methods, and report an ablation study of its modules.
- Table 1: Table 1 compares REFERS with self-supervised learning and transfer-learning baselines.
- Table 2: Table 2 compares methods using human-assisted structured labels, with REFERS reported alongside label-supervised pre-training.
- Table conventions: The tables use AUC as the evaluation metric, and the reported comparisons include NIH, VinBigData, Shenzhen Tuberculosis, and COVID-19 tasks.
- Table 3: Table 3 removes or replaces individual REFERS modules to assess their contributions.
Figures
The figures present REFERS as a multi-view radiograph framework supervised by radiology reports, then compare it with self-supervised, transfer-learning, and label-supervised pre-training under varying annotation levels.
- Figure 1: REFERS forwards each patient study's radiographs through a radiograph transformer, fuses view representations, and applies report generation and study-report consistency supervision.The design includes attention-based view fusion and two supervision tasks involving visual and textual study features.
- Figure 2: REFERS achieves the highest AUC on all 14 classes across different amounts of training data during fine-tuning.The comparison evaluates performance while changing the amount of annotated training data.
- Figure 2: Figure 2 varies the percentage of annotated target-domain training data to evaluate performance under limited supervision.The experiments report the percentage of annotated training data used during fine-tuning.
- Figure 3: The visualizations compare original radiographs with REFERS attention maps, showing radiologist lesion boxes alongside predicted lesion regions.Darker fuchsia indicates higher disease confidence, while green boxes mark predicted lesion areas obtained using a fixed confidence threshold.
- Figures 4 and 6: Figures 4 and 6 compare REFERS with self-supervised and transfer-learning baselines using AUC, with baselines sharing the same transformer-based backbone.The figure descriptions specify AUC as the evaluation metric and indicate that best results are bolded.
- Figures 5 and 7: Figures 5 and 7 compare REFERS with label-supervised pre-training using the same transformer-based backbone and AUC evaluation.The comparisons are conducted on NIH ChestX-ray and VinBigData datasets, respectively.