Source-linked AI summary
RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, Pranav Rajpurkar
TL;DR
Radiology reports contain valuable clinical information, but existing extraction schemas and dense annotations are limited. RadGraph introduces a radiologist-annotated entity-and-relation dataset and benchmark model, achieving relation-extraction micro F1 scores of 0.82 and 0.73 on MIMIC-CXR and CheXpert test sets. It also releases large-scale automatically generated annotations linked to chest radiographs.
Problem
Existing radiology-report extraction is constrained by narrow schemas and limited densely annotated datasets, despite the value of structured clinical information for downstream applications.
Method
RadGraph defines a clinical entity-and-relation schema, releases radiologist-annotated development and test datasets, trains RadGraph Benchmark, and generates inference annotations.
Results
RadGraph Benchmark achieves relation-extraction micro F1 scores of 0.82 on MIMIC-CXR and 0.73 on CheXpert test sets.
Takeaways & Limitations
The released resources support research in medical NLP, computer vision, and multimodal learning linked to chest radiographs.
Takeaways & Limitations
The annotations are limited to chest X-ray reports from MIMIC-CXR and CheXpert, both collected at U.S. hospitals.
Abstract
from arXiv · showhide
Extracting structured clinical information from free-text radiology reports can enable the use of radiology report information for a variety of critical healthcare applications. In our work, we present RadGraph, a dataset of entities and relations in full-text chest X-ray radiology reports based on a novel information extraction schema we designed to structure radiology reports. We release a development dataset, which contains board-certified radiologist annotations for 500 radiology reports from the MIMIC-CXR dataset (14,579 entities and 10,889 relations), and a test dataset, which contains two independent sets of board-certified radiologist annotations for 100 radiology reports split equally across the MIMIC-CXR and CheXpert datasets. Using these datasets, we train and test a deep learning model, RadGraph Benchmark, that achieves a micro F1 of 0.82 and 0.73 on relation extraction on the MIMIC-CXR and CheXpert test sets respectively. Additionally, we release an inference dataset, which contains annotations automatically generated by RadGraph Benchmark across 220,763 MIMIC-CXR reports (around 6 million entities and 4 million relations) and 500 CheXpert reports (13,783 entities and 9,908 relations) with mappings to associated chest radiographs. Our freely available dataset can facilitate a wide range of research in medical natural language processing, as well as computer vision and multi-modal learning when linked to chest radiographs.
1 Introduction
Radiology reports contain clinically important information, but their free-text form and natural-language complexity hinder structured use. RadGraph addresses limitations in extraction schemas and densely annotated datasets with a new schema, radiologist-labeled datasets, benchmark models, and large-scale inference annotations.
- Free-text radiology reports contain critical patient-health information, but their unstructured and ambiguous language complicates clinical research and downstream applications.
- Automated extraction can support large-scale medical imaging model training and disease surveillance.
- Existing extraction efforts are limited by narrow information schemas and the scarcity of densely annotated datasets requiring expert time.
- RadGraph introduces a novel schema, radiologist-annotated development and test datasets, benchmarked models, and an inference dataset.
- 14,579 entities and 10,889 relations were annotated across 500 MIMIC-CXR development reports.
- RadGraph Benchmark achieves micro F1 scores of 0.94/0.91 for named entity recognition and 0.82/0.73 for relation extraction on MIMIC-CXR/CheXpert test sets.
2 Related work
Prior radiology-report resources range from automated labels for common conditions to finer-grained entity and relation annotations. RadGraph targets broader, dense coverage of report information while connecting radiology-specific extraction to established NLP approaches.
- MIMIC-CXR and CheXpert use automated labelers to extract a restricted set of common medical conditions from radiology reports.
- Existing radiology datasets provide entity annotations, radiographic findings, diagnoses, anatomic locations, or spatial-role-based labels.
- RadGraph uses dense annotations for both entities and relations to cover a broader range of radiology text.
- Pipeline approaches train named entity recognition and relation extraction separately, whereas joint approaches model both subtasks simultaneously.
3 Information extraction schema
RadGraph structures radiology reports with entities centered on anatomy and observations, plus three directed relation types. The schema is designed for clinically relevant coverage while keeping annotation relatively consistent and manageable.
- The schema was adapted from prior work and refined through radiologist feedback to balance broad clinical coverage with annotation ease and consistency.
- Entities: The schema defines four entity categories: Anatomy and three Observation states—Definitely Present, Uncertain, and Definitely Absent.
- Entities: Anatomy denotes anatomical body parts, while Observations denote visual features, pathophysiologic processes, or diagnostic disease classifications.
- Relations: Relations are directed edges between entities, with three types: Suggestive Of, Located At, and Modify.
- Relations: Suggestive Of links observations when one implies the presence of another, and Located At links an Observation to related Anatomy.
- Relations: Modify links Observation or Anatomy entities when the first changes the scope or degree of the second.
4 Dataset
RadGraph provides radiology reports with entity and relation annotations from board-certified radiologists, plus large-scale automatically annotated inference data. The datasets include development, test, and demographic statistics, while their use is bounded by de-identification, distribution-shift concerns, and documented scope limitations.
- Dataset overview: RadGraph releases development and test datasets annotated by board-certified radiologists, alongside an inference dataset annotated by RadGraph Benchmark.The development and test datasets contain human annotations, whereas the inference dataset is automatically annotated by the benchmark model.
- Development and test datasets: 500 MIMIC-CXR reports form the development dataset, with 14,579 entities and 10,889 relations, and non-overlapping train and dev patients.The dev split comprises 15% of the development dataset.
- Development and test datasets: The test dataset contains 50 MIMIC-CXR and 50 CheXpert reports, each independently annotated by two board-certified radiologists.It was designed to test generalization across institutions and produced averages of 2,766 entities and 2,009 relations per annotator.
- Inference dataset: RadGraph Benchmark automatically annotates 220,763 MIMIC-CXR reports and 500 CheXpert reports for the inference dataset.The selected MIMIC-CXR reports contain 6,161,934 entities and 4,409,026 relations; the CheXpert reports contain 13,783 entities and 9,908 relations.
- Data usage and ethics: The dataset links full-text reports, schema-based knowledge graphs, and associated chest radiographs for NLP and multimodal research.The released data and model also support population-level analysis and AI-assisted diagnosis use cases.
- Data usage and ethics: The datasets are de-identified under HIPAA, but users must account for distribution shifts and scope limitations when applying models elsewhere.The schema omits clinical context such as Comparison or History sections, and the annotations are limited to chest X-ray reports from two US hospitals.
5 Benchmarks
The benchmark evaluates pipeline and joint entity-relation extraction approaches using transformer and biomedical-pretrained models, against a radiologist benchmark. DYGIE++ with PubMedBERT performs best among the learned approaches, while remaining below the radiologist benchmark on both test sets.
- The benchmark includes pipeline and joint extraction architectures, with BERT and four biomedical pretrained model initializations evaluated across the task.
- The evaluation reports micro and macro F1, using exact entity spans and types for entity recognition and exact entity pairs, relation types, spans, and types for relation extraction.
- DYGIE++ and PURE outperform the Baseline but underperform the radiologist benchmark on both MIMIC-CXR and CheXpert relation-extraction test sets.
- DYGIE++ with PubMedBERT achieves the highest micro and macro F1 on MIMIC-CXR and the highest macro F1 on CheXpert for relation extraction.
- 0.82/0.73 micro F1 is achieved by RadGraph Benchmark for relation extraction on MIMIC-CXR/CheXpert, compared with 0.95/0.75 for the human benchmark.
- 0.94/0.91 micro F1 is achieved by RadGraph Benchmark for named entity recognition on MIMIC-CXR/CheXpert, compared with 0.99/0.93 for the human benchmark.
6 Analysis
The analysis examines annotation agreement and disagreement across radiologists and between radiologists and RadGraph Benchmark, identifying ambiguity in uncertainty, relation structure, entity granularity, and challenging cases.
- Agreement: 0.974 and 0.829 were the Cohen’s Kappa scores for named entity recognition on the MIMIC-CXR and CheXpert test sets respectively.For relation extraction, the corresponding scores were 0.841 and 0.397.
- Agreement: CheXpert showed lower agreement than MIMIC-CXR, possibly because of different ICU-patient proportions and denser annotations concentrated in fewer sentences.The analysis notes that denser annotations are more likely to contain layered relations with ambiguities.
- Annotation disagreements: Annotators disagreed over relation direction, implantable-device labeling, uncertainty levels, entity granularity, and cases without one mandated answer.Examples include modifying “opacity,” interpreting “pleural” in “pleural tube,” labeling “distal tip,” segmenting “cephalad portion,” and annotating “right greater than left pleural effusions.”
- Model disagreements: RadGraph Benchmark shared many disagreement patterns with radiologists, including ambiguity in observation uncertainty and errors on rarer words such as “fat pad”.For “with suggestion of osteopenia,” the radiologist used Observation: Definitely Present while the model used Observation: Uncertain.
7 Conclusion
RadGraph introduces a schema, radiologist-annotated datasets, a benchmark model, and large inference datasets for structuring full-text radiology reports. The authors propose these resources for research spanning medical NLP, computer vision, and multimodal learning.
- Contributions: RadGraph introduces a novel schema and radiologist-annotated datasets for extracting clinical entities and relations from full-text radiology reports.The development dataset contains 14,579 entities and 10,889 relations from 500 reports.
- Contributions: The RadGraph Benchmark achieves relation-extraction micro F1 scores of 0.82 on MIMIC-CXR and 0.73 on CheXpert.These results are reported for the respective test sets.
- Contributions: The released inference datasets contain automatically generated annotations for 220,763 MIMIC-CXR reports and 500 CheXpert reports, linked to associated chest radiographs.The datasets contain around 6 million entities and 4 million relations for MIMIC-CXR, and 13,783 entities and 9,908 relations for CheXpert.
- Implications: The authors hope RadGraph will facilitate research in natural language processing, computer vision, and multimodal learning across medical domains.The proposed resources structure clinically relevant information from unstructured text and can be linked to chest radiographs.