Source-linked AI summary
CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y. Ng, Matthew P. Lungren
TL;DR
Radiology report labeling supports large-scale medical imaging research, but existing rule-based systems miss linguistic complexity while expert annotations are limited. CheXbert combines rule-based labels with expert annotations using biomedical BERT and backtranslation, outperforming the previous best labeler by 0.055 F1 on MIMIC-CXR.
Problem
Existing radiology report labelers rely on feature engineering or expert annotations and may miss the complexity, ambiguity, and subtlety of clinical language.
Method
CheXbert pretrains biomedical BERT on existing labeler outputs, then fine-tunes it on a small expert-annotated corpus augmented with backtranslation.
Results
0.055 F1 improvement (95% CI 0.039, 0.070) over the previous best labeler on MIMIC-CXR, placing CheXbert 0.007 F1 from a radiologist benchmark.
Takeaways & Limitations
Combining scalable labeler outputs with higher-quality expert annotations provides a broadly useful approach to accurate medical report labeling.
Takeaways & Limitations
The task covers only 14 labeled observations and does not test performance on rarer conditions.
Abstract
from arXiv · showhide
The extraction of labels from radiology text reports enables large-scale training of medical imaging models. Existing approaches to report labeling typically rely either on sophisticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT-based approach to medical image report labeling that exploits both the scale of available rule-based systems and the quality of expert annotations. We demonstrate superior performance of a biomedically pretrained BERT model first trained on annotations of a rule-based labeler and then finetuned on a small set of expert annotations augmented with automated backtranslation. We find that our final model, CheXbert, is able to outperform the previous best rules-based labeler with statistical significance, setting a new SOTA for report labeling on one of the largest datasets of chest x-rays.
1 Introduction
CheXbert combines rule-based labeler outputs with expert annotations to improve automated chest-x-ray report labeling. It outperforms the previous best reported labeler on MIMIC-CXR and approaches radiologist benchmark performance.
- Radiology report labeling supports clinical applications, including large-scale training of medical imaging models, while chest-x-ray datasets commonly use engineered rule-based methods.
- Transformer-based labeling reduces feature engineering but requires considerable expert annotation and does not exploit existing feature-engineered labelers.
- CheXbert first trains biomedically pretrained BERT on an existing labeler’s outputs, then fine-tunes it on expert annotations augmented with automated backtranslation.
- CheXbert applies this method to chest-x-ray radiology report labeling.
- 0.055 (95% CI 0.039, 0.070) improvement on the F1 metric over the previous best reported labeler was achieved on the external MIMIC-CXR dataset.CheXbert was only 0.007 F1 away from a radiologist performance benchmark.
2 Related Work
Prior radiology-report labeling methods span feature-engineered rule systems, expert-supervised deep learning, and newer Transformer models. Related work also explores weak supervision to reduce reliance on costly expert annotations.
- Rule-based labeling: Many radiology-report labeling systems use controlled vocabularies and grammatical rules to extract and classify findings from free text.These approaches rely on substantial medical-domain feature engineering.
- Rule-based labeling: The CheXpert labeler improves chest x-ray classification over NegBio through controlled mention extraction and enhanced uncertainty and negation rules.It has been used to generate labels for the CheXpert and MIMIC-CXR datasets.
- Expert-supervised learning: Expert-supervised deep learning approaches are constrained by radiologist time and expertise, which limit training-set size.Reported approaches include CNNs, recurrent and convolutional networks with attention, and models trained on physician-labeled reports.
- Transformer-based labeling: Transformer-based models, including BERT and XLNet, have also been applied to radiology report labeling.These models represent a newer direction beyond earlier rule-based and conventional neural approaches.
- Weak supervision: Weak-supervision methods seek to reduce expert annotation requirements by using heuristic labeling functions to programmatically label medical-report training data.Data programming has been successfully applied to medical report labeling.
3 Methods
The study frames report labeling as multi-class extraction across 14 observations, using modified BERT models trained on large chest-x-ray datasets and evaluated with observation-averaged weighted F1. The method uses expert-annotated subsets for evaluation and reports 95% bootstrap confidence intervals.
- Task definition: The labeler predicts blank, positive, negative, or uncertain for 13 observations, while No Finding has only blank and positive classes.The task extracts clinically important observations from free-text radiology reports.
- Datasets: The study uses CheXpert’s 224,316 images and MIMIC-CXR’s 377,110 images, with reports labeled for the same 14 observations by the CheXpert labeler.CheXpert supports model development, while the MIMIC-CXR test set supports evaluation.
- Model architecture: Models use modified BERT-base with 14 linear heads, tokenize each report, cap inputs at 512 tokens, and feed the CLS hidden state to every head.The heads cover 12 abnormalities, medical support devices, and No Finding.
- Optimization: Training fine-tunes all BERT layers with cross-entropy loss and Adam at a learning rate of 2 × 10−5, using batch size 18 on 3 TITAN-XP GPUs.Losses across the 14 observations are summed, and the checkpoint with the highest average dev-set performance is saved.
- Evaluation: Evaluation averages weighted-F1 scores across positive, negative, and uncertainty extraction tasks and then across all 14 observations.The weighted-F1 score weights each task’s F1 by the support for its class of interest.
- Evaluation: The reported F1 confidence intervals are 95% two-sided intervals estimated with 1000 nonparametric percentile-bootstrap replicates.The bootstrap procedure follows Efron and Tibshirani (1986).
4 Experiments
Experiments show that hybrid training and backtranslation improve report-labeling performance, with CheXbert significantly outperforming the CheXpert labeler while approaching the radiologist benchmark. BlueBERT initialization and backtranslation produce the strongest results among evaluated models.
- Training strategies: 0.775 F1 was achieved by T-hybrid, exceeding T-auto at 0.755 and T-rad at 0.705.T.cls-rad and T.token-rad achieved 0.286 and 0.396, respectively.
- Biomedical pretraining: 0.741 F1 was achieved by Tblue-rad, exceeding T-rad at 0.705, whereas Tbio-rad and Tclinical-rad achieved 0.616 and 0.677.The lower biomedical-pretraining results may reflect differences in vocabulary, sequence length, stopping procedure, embedding dimensions, and other configurations.
- Backtranslation: 0.798 F1 was achieved by Tblue-hybrid-bt, while Tblue-rad-bt achieved 0.770 and exceeded the CheXpert labeler.Backtranslation augmented the manual training set to twice its original size.
- Benchmark comparison: p < 0.001: CheXbert significantly outperformed the CheXpert labeler, which achieved an F1 of 0.743.Tblue-rad-bt, trained only on manually labeled reports, performed at least as well as the CheXpert labeler.
- Benchmark comparison: 0.055 F1 points was CheXbert’s overall statistically significant improvement over CheXpert, with significant gains in 9 of 14 medical conditions.The largest improvements were for Pneumonia [0.151 (0.093, 0.206)] and Fracture [0.120 (0.019, 0.236)].
- Inference times: 18 minutes was CheXbert’s inference time for 190,460 report impressions using a single TITAN-XP GPU, versus approximately 36 hours for CheXpert on one CPU core.On a system with 32GB RAM and 1 CPU core, CheXbert required approximately 3.7 hours.
5 Analysis
The analysis shows that T-auto can generalize beyond CheXpert’s rules, while CheXbert handles clinical findings, negation, and uncertainty that both earlier labelers miss. Backtranslation expands phrasing and vocabulary but can introduce errors, despite physician validation of semantic preservation.
- T-auto generalization: T-auto correctly labels uncertainty and negative fracture mentions that CheXpert’s rules miss.It detects “cannot be entirely excluded” as uncertainty and labels fracture negative in “no evidence of pneumothorax or bony fracture.”
- CheXbert improvements: CheXbert identifies positive cardiomegaly and enlarged cardiomediastinum when T-auto and CheXpert output blank or fail to detect them.Examples include “mildly enlarged heart” and “Right hilum appears slightly more prominent.”
- CheXbert improvements: CheXbert correctly resolves nuanced negation and uncertainty that CheXpert and T-auto mislabel.It labels cardiomegaly negative despite “still within normal range,” recognizes negation from “interval removal,” and labels lung opacity positive rather than uncertain when opacities are “suspicious for aspiration.”
- Backtranslation analysis: Backtranslation frequently rephrases radiology text and paraphrases medical vocabulary into possible semantic equivalents, but it can introduce errors such as “costophrine.”Examples include rephrasing “redemonstration” and substituting “cutaneous” with “skin” or “osseous” with “bone” or “bony.”
- Backtranslation analysis: A physician validated that the backtranslation outputs used correct radiology language and maintained the original reports’ semantics.The validation results are provided in Table A1 of the Appendix.
6 Limitations
The study’s hybrid and automatic approaches depend on an existing labeler, while CheXbert is constrained by input length and the set of labeled observations. Evaluation is also limited by incomplete testing of rare conditions and ground-truth labeling details reported only partially here.
- Hybrid and automatic approaches require an already-existing labeler.
- 512 tokens is the report labeler’s maximum input size, although only 3 of 190,460 CheXpert report impressions exceeded that length.The authors note that the limit may be extended to handle longer medical or radiology reports.
- The task covers only 14 labeled observations and does not test the model’s ability to label rarer conditions.CheXbert can mark No Finding as blank, potentially indicating another condition when the other 13 conditions are also blank.
- Ground-truth labels for the MIMIC-CXR test set were determined by a single board-certified ra.
7 Conclusion
The study proposes CheXbert, which combines rule-based report labels, biomedical BERT pretraining, expert annotations, and backtranslation to improve radiology report labeling. The authors expect this approach to generalize across medical domains and support more accurate, robust medical imaging models.
- Method: CheXbert combines existing report labelers with hand annotations through biomedical BERT pretraining followed by fine-tuning on backtranslation-augmented manual annotations.The method uses rule-based labels at scale and expert annotations for quality.
- Broader applicability: The authors expect this training method to be broadly useful where expert labels are limited but feature-engineered labelers can generate labels at scale.It is designed to extract highly accurate labels by leveraging both sources.
- Downstream impact: Accurate report labels can enable downstream tasks, including developing more accurate and robust medical imaging models for clinical deployment.The passage specifically identifies medical imaging model development as an important downstream application.
A Physician validation of backtranslation quality
A physician evaluated backtranslation quality on 100 randomly sampled CheXpert manual-set reports, assessing radiology-language validity and preservation of report semantics. Scores from 1 to 3 captured the severity of language deviations and semantic changes.
- Validation setup: 100 randomly sampled CheXpert manual-set reports were evaluated by a physician alongside their backtranslations.The physician read each original report and its backtranslation.
- Validation setup: The validation assessed whether backtranslations used valid radiology language and maintained the original report’s semantics.These criteria addressed potential noise introduced by backtranslation.
- Scoring criteria: Radiology-language quality used a 3-point scale: 3 indicated near-perfect language, 2 minor deviations, and 1 a major deviation.The scale measured deviations from valid radiology language.
- Scoring criteria: Semantic preservation used a 3-point scale: 3 indicated full preservation, 2 minor semantic errors, and 1 major change or loss of semantic information.Scores compared each backtranslation with its original report.
B Additional results
Additional results describe the deduplicated CheXpert dataset, dev-set construction, and qualitative cases in which CheXbert resolves uncertainty, negation, semantic variation, and unsupported mentions better than CheXpert and T-auto. Backtranslation and model-comparison analyses are also reported alongside these examples.
- Dataset and evaluation: 190,460 reports remain after removing duplicate patient reports from CheXpert, with labels supplied by the CheXpert labeler.The table reports class prevalences for each medical condition in this deduplicated set.
- Dataset and evaluation: Dev-set F1 scores are reported for all models using task-specific sampling from the CheXpert manual set or CheXpert dataset.Backtranslation-based models use augmented versions of 250 randomly sampled manual-set reports; T-auto uses a random 15% CheXpert split.
- Qualitative examples: CheXbert correctly handles uncertainty and negation, labeling unchanged cardiomegaly and ambiguous atelectasis as uncertain while CheXpert and T-auto assign positive labels.The examples cite “unchanged” cardiac silhouette and difficulty distinguishing atelectasis from pneumonia as cues for uncertainty.
- Qualitative examples: CheXbert correctly labels cardiomegaly, edema, and pneumonia as negative when surrounding report language indicates normality, absence, or resolution.The examples contrast CheXbert’s negative labels with positive or uncertain labels from CheXpert and T-auto.
- Qualitative examples: CheXbert identifies support-device mentions and complex negation that rules-based systems miss, labeling removed devices and emphysema-related pneumothorax as negative while detecting a jugular introducer as positive.CheXpert and T-auto respectively miss negation around PICC lines, omit the jugular introducer, and incorrectly label pneumothorax positive.
- Backtranslation: Backtranslation introduces several lexical errors, including changes to “chest,” “tracheostomy tube,” “atelectasis,” and “cardiomediastinal.”The examples also note semantically similar but suboptimal substitutions such as “skin clamps” for “cutaneous staples.”