Source-linked AI summary
TorchXRayVision: A library of chest X-ray datasets and models
Joseph Paul Cohen, Joseph D. Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Matteo Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, Hadrien Bertrand
TL;DR
TorchXRayVision addresses the difficulty of comparing chest X-ray research by providing common interfaces and preprocessing across datasets and models. It combines reusable datasets, pretrained classifiers, feature extractors, distribution-shift tools, and an autoencoder to support baselines, transfer learning, and model analysis. Its scope includes publicly available datasets and models with differing label coverage and preprocessing assumptions.
Problem
Differences in dataset organization, processing, and training procedures make chest X-ray results difficult to compare and baselines strenuous to establish.
Method
The library standardizes chest X-ray datasets and models through common interfaces, preprocessing components, pretrained models, feature extraction, and tools for distribution-shift studies.
Results
TorchXRayVision provides datasets that can be swapped, merged, and filtered, pretrained models for baselines and feature extraction, and an autoencoder trained across four datasets.
Takeaways & Limitations
The library supports reproducible baseline experiments, transfer learning, semi-supervised learning, and studies of model failures and out-of-distribution generalization.
Takeaways & Limitations
Image preprocessing avoids arbitrarily increasing contrast because doing so could remove information from the specific image.
Abstract
from arXiv · showhide
TorchXRayVision is an open source software library for working with chest X-ray datasets and deep learning models. It provides a common interface and common pre-processing chain for a wide set of publicly available chest X-ray datasets. In addition, a number of classification and representation learning models with different architectures, trained on different data combinations, are available through the library to serve as baselines or feature extractors.
1 Introduction
TorchXRayVision addresses reproducibility and comparison challenges in chest X-ray research by providing reusable infrastructure that separates datasets from models. Its common interfaces, preprocessing, pretrained models, and distribution-shift utilities support evaluation, development, and failure analysis.
- Motivation: Chest X-ray research is difficult to compare because dataset organization, processing, and training differences can substantially affect results.The resulting need to repeatedly reimplement dataloaders makes establishing meaningful baselines strenuous.
- Framework: TorchXRayVision provides a reusable framework for reproducible research and consistent baseline experiments.Its design separates datasets and models through a clear interface.
- Framework: Common interfaces, preprocessing and postprocessing components, torchvision compatibility, and downloadable pretrained models support interchangeable experiments.Datasets can be swapped during model training and evaluation, while models can provide baselines or feature vectors.
- Use cases: The library supports model development through pretrained models used for baseline comparisons, feature extraction, transfer learning, and related chest X-ray tasks.Reported applications include severity scoring, clinical trajectory prediction, few-shot transfer learning, and multimodal models.
- Use cases: Its datasets and pretrained models enable studies of model failures, out-of-distribution generalization, covariate shifts, concept shifts, shortcut learning, and saliency-map failures.Utilities can produce datasets with covariate shifts and spurious correlations.
- Design: An object-oriented design represents datasets, models, pretrained weights, and utilities as swappable experiment components.The package is distributed in Python and follows PyTorch and torchvision conventions for adoption.
2 Models
TorchXRayVision supplies interchangeable pretrained classifiers, baselines, feature extractors, calibration utilities, and an autoencoder through a common model interface. Models support multi-dataset training, pathology-aligned outputs, downstream feature use, and unsupervised latent representations.
- Classifiers: Core and baseline classifiers share interfaces, input scaling, and resizing behavior, allowing models trained with different weights to be swapped in experiments.Core classifiers are trained for the library, while baseline classifiers are adapted from other papers.
- Classifiers: Pretrained model weights encode architecture, training dataset, and resolution, while models target 18 independent pathology outputs when labels are available.Unavailable pathology labels may return NaN, and weights marked “all” use as many available datasets as possible.
- Classifiers: Core classifiers use augmentation including rotations up to 45 degrees, translations up to 15%, and scaling up to 10%.The stated purpose of augmentation is to improve generalization.
- Baseline Classifiers: Two external baselines are included: a JFHealthcare model trained on CheXpert and the official CheXpert model.Both adhere to the common interface and can be swapped with core models.
- Classifier Interface: Classifier outputs are aligned with pathology names through each model’s model.pathologies field, and selected outputs remain differentiable for embedding in larger networks.The available pathology list depends on the loaded weights.
- Feature Extraction: Pretrained models expose feature vectors for semi-supervised learning and transfer learning, with feature size depending on architecture and input image size.Some models also provide features2 for extraction at another point in the computation graph; Figure 1 visualizes representations with UMAP.
- Autoencoders: The library provides a pretrained autoencoder trained on PadChest, NIH, CheXpert, and MIMIC for unsupervised representation extraction and image reconstruction.It supplies a fixed latent representation and a decoder rather than relying on supervised labels.
3 Datasets
The library integrates chest X-ray datasets through standardized interfaces, metadata access, download documentation, and PyTorch-compatible transforms.
- Dataset classes use image and metadata paths, with some metadata files packaged so only the image path is required.
- Documentation provides citation information and download links, with some licensed datasets available through Academic Torrents.
- Dataset objects support standard PyTorch transforms, although tensor shape handling may require care.The dataloader returns tensors shaped [1,RES,RES].
- Tables report total image counts and available label counts for each dataset.
3.1 Image pre-processing
Image preprocessing maps pixel values using the image encoding range rather than per-image extrema, while dataset tables distinguish total from usable frontal images and summarize labels.
- Image pre-processing: Pixel values are expected between [-1024,1024], with preprocessing based on the possible encoding range rather than each image’s minimum and maximum.For 16-bit images, the possible range is mapped instead of the observed pixel extrema.
- Image pre-processing: The preprocessing avoids arbitrary contrast increases because they could remove information.
- Dataset counts: Table 1 reports total images and usable frontal images, requiring readable images, necessary metadata, and specified frontal views.
- Label counts: Table 2 summarizes available labels and positive examples across datasets, excluding COVID-19 datasets because of their many unique pathologies.
3.2 Dataset common fields
Datasets expose common pathology, label, metadata, patient, timing, and view fields that remain aligned when datasets are merged or subsetted.
- Common dataset fields are preserved by SubsetDataset and MergeDataset operations.
- The pathologies list names label columns, while labels stores per-item values of 1, 0, or NaN for each pathology or attribute.
- The csv field stores aligned metadata rows, including catch-all associated data whose meanings are defined by each dataset’s reference publication.
- Patient IDs support preventing patient overlap or observing patients over time; offset_day_int records relative image timing, and view records the acquisition projection.
3.3 Dataset tools
TorchXRayVision provides tools to relabel, filter, merge, subset, inspect, and select chest X-ray dataset objects while preserving core dataset fields.
- Relabelling datasets: relabel_dataset adds, removes, or reorders labels to match a requested pathology order, inserting NaN for unavailable pathologies.The operation directly changes the dataset and returns no value.
- Filtering based on views: View filtering selects standardized PA, AP, or AP Supine projections when view metadata is available.Users are advised to inspect unique available views before filtering.
- Filtering patients: The unique_patients option restricts a dataset to one image per patient when patient IDs are provided.
- Inspecting datasets: Printing a dataset or calling totals() reports counts for available labels and their classes.
- Merging datasets: MergeDataset combines datasets in order while maintaining labels, metadata, and pathology fields.
- Taking a subset of a dataset: SubsetDataset creates indexed subsets while preserving labels, metadata, and pathology fields, including subsets selected by metadata criteria.An example selects female patients using the PatientSex_DICOM field.
- Taking a subset of a dataset: Figure 2 shows that mask information across datasets ranges from bounding boxes to more exact segmentations.
3.4 Pathology and semantic masks
TorchXRayVision supports pathology and semantic masks for segmentation and attribution validation, with explicit controls for returning available masks and metadata indicating mask availability.
- Pathology and semantic masks support segmentation and validation of whether models attribute importance to the correct regions.
- Masks are disabled by default and must be requested with pathology_masks=True and/or semantic_masks=True when constructing a dataset.
- Pathology masks correspond to dataset.pathologies, whereas semantic masks represent unrelated regions such as lung segmentations.
- Multiple masks for one pathology are merged with a logical OR, and augmentations apply the same seeded transformation to the masks.
- The has_masks CSV column identifies whether any masks exist for a sample; an example contains 6,012 masked and 20,672 unmasked samples.
3.5 Distribution shift tools
TorchXRayVision provides CovariateDataset to construct controlled distribution shifts by varying the mixture of source datasets while balancing target labels. Its visualizations show how sampling across AP and PA views changes perceived class differences.
- The target label remains balanced 50/50 in each split while the ratio of examples from the origin datasets varies.
- CovariateDataset introduces covariate shift by correlating dataset-specific nuisance features with output labels through a specified source-dataset ratio.
- Using different ratios for training and testing has been shown to decrease generalization performance and exacerbate incorrect feature attribution.
- Changing the ratio between AP-only and PA-only COVID-19 datasets alters the mean class-difference images and their deviations from balanced sampling.
4 Transfer Learning Example
The transfer-learning example fine-tunes a TorchXRayVision pre-trained model on a dataset in the library’s format. It applies standard XRV image transforms, replaces the classifier, and trains on a selected pathology label.
- The example fine-tunes a pre-trained model using a dataset in TorchXRayVision format.
- XRV transforms center-crop and resize input images to 224 pixels.
- A DenseNet model with densenet121-res224-all weights is loaded, its classifier is reinitialized as a one-output linear layer, and calibration thresholds are disabled.
- The training loop computes outputs from batch images, selects the Lung Opacity targets, backpropagates the loss, and updates the optimizer.