PaperlayerView exact revision on arXiv

Source-linked AI summary

ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Dataset

Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S. Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, Henning Müller, Peter A. Horn, Felix Nensa, Christoph M. Friedrich

arXiv:2405.10004v2eess.IVcs.CVcs.LG

TL;DR

Medical image analysis uses multimodal datasets for model development and validation, and ROCOv2 updates the original ROCO dataset with radiological images, captions, and medical concepts. The dataset adds manually curated image concepts and reports that baseline models perform better on ROCOv2 than on the ImageCLEF dataset while matching ImageCLEF challenge participants.

  • Problem

    ROCOv2 updates the original ROCO dataset with radiological images and associated medical concepts and captions.

    Source: Background & Summary · p. 2

  • Method

    The authors combined PMC-derived image-caption pairs with MedCAT-extracted concepts and manually curated modality concepts for all images plus anatomy and directionality concepts for X-rays.

    Source: Background & Summary · p. 2, Caption Processing and Concept… · p. 5

  • Results

    Baseline models achieved results similar to ImageCLEF challenge participants on the ImageCLEF dataset while performing better on ROCOv2.

    Source: Technical Validation · p. 6

  • Takeaways & Limitations

    ROCOv2 can support radiological-image pre-training, structured reporting, multi-label medical concept classification, caption prediction, and evaluation of deep-learning models for multi-task learning.

    Source: Usage Notes · p. 15

    Faulty or fuzzy original captions remain a limitation of the ground truth, including modality confusion and ambiguous image descriptions.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Automated medical image analysis systems often require large amounts of training data with high quality labels, which are difficult and time consuming to generate. This paper introduces Radiology Object in COntext version 2 (ROCOv2), a multimodal dataset consisting of radiological images and associated medical concepts and captions extracted from the PMC Open Access subset. It is an updated version of the ROCO dataset published in 2018, and adds 35,705 new images added to PMC since 2018. It further provides manually curated concepts for imaging modalities with additional anatomical and directional concepts for X-rays. The dataset consists of 79,789 images and has been used, with minor modifications, in the concept detection and caption prediction tasks of ImageCLEFmedical Caption 2023. The dataset is suitable for training image annotation models based on image-caption pairs, or for multi-label image classification using Unified Medical Language System (UMLS) concepts provided with each image. In addition, it can serve for pre-training of medical domain models, and evaluation of deep learning models for multi-task learning.

Background & Summary

ROCOv2 updates the original ROCO dataset with radiological images, captions, and medical concepts, adding manually curated concepts and improved concept extraction. Its multimodal image-caption and UMLS concept pairs support several medical-imaging research uses.

Background & Summary

  • Manually curated modality concepts cover all images, while anatomy and directionality concepts were added for X-rays; concept extraction also uses MedCAT and improved filtering.MedCAT uses a newer UMLS version and word embeddings rather than QuickUMLS’s direct dictionary matches.Source: Background & Summary · p. 2
  • ROCOv2 contains 79,789 radiological images with associated captions and medical concepts.The dataset excludes out-of-class images and includes images from CC BY-licensed articles, including CC BY-NC but excluding CC BY-ND and CC BY-SA.Source: Background & Summary · p. 2
  • The dataset can support image-caption model training, medical image retrieval, pre-training, and multi-label classification using provided UMLS concepts.The paper also describes potential extensions to multimodal retrieval-augmented generation and medical reporting.Source: Background & Summary · pp. 1–2
  • Images and captions were extracted from openly available PMC Open Access publications with CC BY licenses.The paper states that dataset sources were openly available in the PMC Open Access Subset at publication time.Source: Background & Summary · p. 2

Related Work

Related datasets provide medical image-text pairs, but differ from ROCOv2 in anatomical and modality coverage, filtering, and distribution permissions. ROCOv2 is presented as a manually validated, radiology-focused resource with permissive licensing.

Related Work

  • ROCO has supported ImageCLEFmedical Caption tasks through 2023 and has also been used to fine-tune medical-domain CLIP for visual question answering.The cited CLIP study reported that PubMedCLIP outperformed general CLIP on two MedVQA benchmarks.Source: Related Work · p. 2
  • Unlike chest-X-ray-focused MIMIC-CXR, Open-I, and PADCHEST, ROCOv2 covers a wider range of anatomical regions, medical concepts, and modalities.MIMIC-CXR, Open-I, and PADCHEST focus on chest X-rays.Source: Related Work · pp. 2–3
  • ROCOv2 excludes images with CC BY-ND and CC BY-SA licenses, which can prohibit distribution of processed images or require sharing under the same license.MedICaT and PMC-OA include images under these licenses; ROCOv2 reports manual validation as a filtering step.Source: Related Work · p. 3
  • PMC-VQA generates question-answer pairs from PMC-OA image-caption pairs, but its authors report no manual or license-based filtering.PMC-VQA includes 226,946 question-answer pairs and 149,075 images.Source: Related Work · p. 3
  • ROCOv2 aims to address limits in existing datasets by combining manual validation, broad radiological coverage, and permissively licensed images that can be distributed.The cited limitations include restricted anatomical focus, license restrictions, or lack of manual validation.Source: Related Work · p. 3

Methods

The authors assembled ROCOv2 from PMC images, applying image, license, duplicate, and caption filters before creating dataset splits. Caption-derived concepts were extracted and filtered, then combined with manually created and validated image concepts.

Methods

  • The workflow downloaded 4,798,923 PMC archives, then applied binary classifiers to extracted images to filter compound and non-radiological images.The classifiers had reported accuracies of about 90% and 98.6%, respectively; subsequent license and duplicate filtering reduced the corpus to 119,140 images.Source: Dataset Creation · pp. 3–4
  • New validation and test sets combined 21,545 images from 2021–2022 with previously used ROCO and ImageCLEF images, using stratified sampling based on manually curated concepts.Concepts absent from the training set were removed from validation and test sets.Source: Dataset Creation · p. 4
  • The final dataset contains 79,789 images from 44,975 articles, split into 59,958 training, 9,904 validation, and 9,927 test images.After filtering for valid captions, the authors removed 1,528 non-English captions and very short captions; they also report removing approximately 1,500 non-radiological images relative to the ImageCLEFmedical Caption 2023 dataset.Source: Dataset Creation · p. 4
  • Caption preprocessing excluded non-English captions identified with fastText above a 45% confidence threshold and removed URLs, LaTeX-only captions, and captions with minimal information.Captions identified as non-English below 45% confidence were retained under the assumption that they were likely English.Source: Caption Processing and Concept… · pp. 4–5
  • MedCAT extracted caption concepts, mapped SNOMED CT IDs to UMLS CUIs and semantic types, and applied a frequency cutoff retaining concepts occurring more than 10 times.Further filtering focused on visually observable concepts and manually excluded concepts not directly associated with image content, including some temporal or qualitative qualifiers.Source: Caption Processing and Concept… · p. 5
  • Manually curated concepts cover seven imaging modalities and add X-ray body regions and projection directions; annotators classified subsets, models pseudo-labeled remaining images, and curators corrected errors.A radiologist validated representative subsets; directional concepts were introduced for experimental purposes only, and manually curated concepts took priority when combined with automatic concepts.Source: Caption Processing and Concept… · p. 5

Data Records

ROCOv2 data records provide the images, captions, and concepts in training, validation, and test splits, alongside mapping and license information.

Data Records

  • The dataset provides JPEG images, caption CSVs, and concept CSVs for the training, validation, and test splits.Caption files pair image filenames with captions; concept files pair image filenames with semicolon-separated CUIs.Source: Data Records · pp. 5–6
  • A separate concepts file contains manually curated concepts only, while the combined concept files include manually curated and automatically generated concepts.The concept mapping file provides CUIs and their canonical names.Source: Data Records · p. 6
  • License information records each image filename, PMCID, CC BY attribution string, and PMC article link.Source: Data Records · p. 6

Technical Validation

Baseline models performed similarly on the ImageCLEF dataset and better on ROCOv2, while physician agreement was high for manually curated modality and X-ray body-region concepts but moderate for directionality.

Technical Validation

  • Baseline models performed similarly on ImageCLEF and better on ROCOv2, which the authors interpret as evidence of improved dataset quality and suitability for both subtasks.The paper also notes that baseline results and several years of challenge scores support ROCOv2 for concept detection and caption prediction.Source: Technical Validation · p. 6
  • 0.5215 and 0.5925 were the final EfficientNetv2-s sample-averaged F1-scores on the ImageCLEFmedical Caption and ROCOv2 test sets, respectively.The corresponding secondary sample-averaged F1-scores were 0.9407 and 0.9430.Source: Concept Detection · p. 8
  • The caption-prediction baseline achieved BERTScores of 0.6217 on ImageCLEFmedical Caption and 0.6241 on ROCOv2.Its ROUGE scores were 0.2318 and 0.2325 on those datasets, respectively.Source: Caption Prediction · p. 9
  • Captions averaged about 21 words, 3.36 UMLS concepts, and ranged from 1 to 848 words and up to 28 concepts; articles averaged 1.76 image-caption pairs.Table 4 provides descriptive statistics for the dataset.Source: Dataset Analysis · p. 9, Table 4 · p. 9
  • Physician agreement was almost perfect for modality (κ = 0.928) and X-ray body regions (κ = 0.886), but moderate for directionality (κ = 0.557).The authors characterize the directional labels as experimental.Source: Annotator-Radiologist Evaluation of… · p. 11

Limitations

ROCOv2 inherits source-archive and caption-quality constraints, while rare modalities and manually curated concepts also have specific coverage and labeling limits.

Source and captions

  • PMC Open Access sourcing introduces image-selection bias and archive quality issues, including rare image-caption mix-ups that may be difficult to correct.Archive images may be lower quality than those in published articles.Source: Limitations · p. 14
  • Faulty or vague original captions serve as ground truth and can contain modality confusion or lack detail and context, leaving some images without sufficient concepts.Examples include CT labeled as MRI or PET/CT and absent modality or region information.Source: Limitations · pp. 14–15

Dataset composition

  • PET and combined modalities are relatively rare, reflecting their rarity in publications and requiring consideration when using the dataset for rare-modality work.Source: Limitations · p. 15

Concepts and labels

  • Semantic types for image-identifiable concepts were selected by consent and best effort, not by dedicated medical personnel, because a well-defined process and resources were lacking.Source: Limitations · p. 15
  • Including images with only one concept may leave insufficient complexity for comprehensive model training, so users are advised to consider their selection criteria.Source: Limitations · p. 15
  • Manually created modality, X-ray body-region, and directionality concepts have validation limitations, including nontrivial projection distinctions and anatomical boundaries.Directionality was reduced to four classes and may be difficult to determine without context; some regions could not be assigned under IRMA.Source: Limitations · p. 15

Usage Notes

ROCOv2 images are distributed as they appear in PMC and may need preprocessing; the dataset is proposed for several model-development and evaluation uses.

Image preparation

  • Images are provided as they appear in the PMC Open Access Subset and must be resized or cropped before machine-learning use.Their quality may differ from the images published in the journal.Source: Usage Notes · p. 15

Potential applications

  • Proposed uses include radiological-image pretraining, structured-reporting systems, multi-label concept classification, caption prediction, and multi-task model evaluation.Source: Usage Notes · p. 15

Resources

  • Example scripts for baseline models and evaluation are referenced in the next section.Source: Usage Notes · p. 15

Code availability

The repository provides code for filtering extracted images, training baseline models, and processing the ImageCLEFmedical Caption 2023 tasks.

Image filtering

  • The “roco-2018” folder contains scripts and models from the original pipeline to filter PMC images for non-compound radiological figures.Source: Code availability · p. 15

Baseline models

  • The “baseline” folder contains code for training concept-detection and caption-prediction baseline models described in Technical Validation.Source: Code availability · p. 15

ImageCLEF scripts

  • The “ImageCLEF” folder contains preprocessing and evaluation scripts for the ImageCLEFmedical Caption 2023 challenge tasks.Source: Code availability · p. 16

Supplementary Information

The supplementary material reports dataset distributions and provides distilled annotation guidance for modalities, X-ray body regions, and projection directions.

Publication and license distributions

  • Publication counts rose from fewer than 15 annually in 1987–2002 to 9,341 articles in 2021; excluded CC BY-ND and CC BY-SA licenses peaked in 2017.Source: Figure S1 · p. 20, Figure S2 · p. 20

Image characteristics

Dataset distributions

  • The publication-type table covers distinct PMC articles with available standardized types, while the modality table reports image-caption pair distributions across seven modality classes.Source: Table S1 · p. 23, Table S2 · p. 24
  • Tables list manually created X-ray body-part and directionality label distributions, journal distributions, non-English captions, and frequent CUIs by modality.Source: Table S3 · p. 24, Table S4 · p. 24, Table S5 · p. 25

Annotation guidance

Loading 2405.10004v2…