PaperlayerView exact revision on arXiv

Source-linked AI summary

The Brain Tumor Segmentation (BraTS-METS) Challenge 2023: Brain Metastasis Segmentation on Pre-treatment MRI

Ahmed W. Moawad, Anastasia Janas, Ujjwal Baid, Divya Ramakrishnan, Rachit Saluja, Nader Ashraf, Nazanin Maleki, Leon Jekel, Nikolay Yordanov, Pascal Fehringer, Athanasios Gkampenis, Raisa Amiruddin, Amirreza Manteghinejad, Maruf Adewole, Jake Albrecht, Udunna Anazodo, Sanjay Aneja, Syed Muhammad Anwar, Timothy Bergquist, Veronica Chiang, Verena Chung, Gian Marco Conte, Farouk Dako, James Eddy, Ivan Ezhov, Nastaran Khalili, Keyvan Farahani, Juan Eugenio Iglesias, Zhifan Jiang, Elaine Johanson, Anahita Fathi Kazerooni, Florian Kofler, Kiril Krantchev, Dominic LaBella, Koen Van Leemput, Hongwei Bran Li, Marius George Linguraru, Xinyang Liu, Zeke Meier, Bjoern H Menze, Harrison Moy, Klara Osenberg, Marie Piraud, Zachary Reitman, Russell Takeshi Shinohara, Chunhao Wang, Benedikt Wiestler, Walter Wiggins, Umber Shafique, Klara Willms, Arman Avesta, Khaled Bousabarah, Satrajit Chakrabarty, Nicolo Gennaro, Wolfgang Holler, Manpreet Kaur, Pamela LaMontagne, MingDe Lin, Jan Lost, Daniel S. Marcus, Ryan Maresca, Sarah Merkaj, Gabriel Cassinelli Pedersen, Marc von Reppert, Aristeidis Sotiras, Oleg Teytelboym, Niklas Tillmans, Malte Westerhoff, Ayda Youssef, Devon Godfrey, Scott Floyd, Andreas Rauschecker, Javier Villanueva-Meyer, Irada Pfluger, Jaeyoung Cho, Martin Bendszus, Gianluca Brugnara, Justin Cramer, Gloria J. Guzman Perez-Carillo, Derek R. Johnson, Anthony Kam, Benjamin Yin Ming Kwan, Lillian Lai, Neil U. Lall, Fatima Memon, Mark Krycia, Satya Narayana Patro, Bojan Petrovic, Tiffany Y. So, Gerard Thompson, Lei Wu, E. Brooke Schrickel, Anu Bansal, Frederik Barkhof, Cristina Besada, Sammy Chu, Jason Druzgal, Alexandru Dusoi, Luciano Farage, Fabricio Feltrin, Amy Fong, Steve H. Fung, R. Ian Gray, Ichiro Ikuta, Michael Iv, Alida A. Postma, Amit Mahajan, David Joyner, Chase Krumpelman, Laurent Letourneau-Guillon, Christie M. Lincoln, Mate E. Maros, Elka Miller, Fanny Moron, Esther A. Nimchinsky, Ozkan Ozsarlak, Uresh Patel, Saurabh Rohatgi, Atin Saha, Anousheh Sayah, Eric D. Schwartz, Robert Shih, Mark S. Shiroishi, Juan E. Small, Manoj Tanwar, Jewels Valerie, Brent D. Weinberg, Matthew L. White, Robert Young, Vahe M. Zohrabian, Aynur Azizova, Melanie Maria Theresa Bruseler, Mohanad Ghonim, Mohamed Ghonim, Abdullah Okar, Luca Pasquini, Yasaman Sharifi, Gagandeep Singh, Nico Sollmann, Theodora Soumala, Mahsa Taherzadeh, Philipp Vollmuth, Martha Foltyn-Dumitru, Ajay Malhotra, Aly H. Abayazeed, Francesco Dellepiane, Philipp Lohmann, Victor M. Perez-Garcia, Hesham Elhalawani, Maria Correia de Verdier, Sanaria Al-Rubaiey, Rui Duarte Armindo, Kholod Ashraf, Moamen M. Asla, Mohamed Badawy, Jeroen Bisschop, Nima Broomand Lomer, Jan Bukatz, Jim Chen, Petra Cimflova, Felix Corr, Alexis Crawley, Lisa Deptula, Tasneem Elakhdar, Islam H. Shawali, Shahriar Faghani, Alexandra Frick, Vaibhav Gulati, Muhammad Ammar Haider, Fatima Hierro, Rasmus Holmboe Dahl, Sarah Maria Jacobs, Kuang-chun Jim Hsieh, Sedat G. Kandemirli, Katharina Kersting, Laura Kida, Sofia Kollia, Ioannis Koukoulithras, Xiao Li, Ahmed Abouelatta, Aya Mansour, Ruxandra-Catrinel Maria-Zamfirescu, Marcela Marsiglia, Yohana Sarahi Mateo-Camacho, Mark McArthur, Olivia McDonnell, Maire McHugh, Mana Moassefi, Samah Mostafa Morsi, Alexander Munteanu, Khanak K. Nandolia, Syed Raza Naqvi, Yalda Nikanpour, Mostafa Alnoury, Abdullah Mohamed Aly Nouh, Francesca Pappafava, Markand D. Patel, Samantha Petrucci, Eric Rawie, Scott Raymond, Borna Roohani, Sadeq Sabouhi, Laura M. Sanchez-Garcia, Zoe Shaked, Pokhraj P. Suthar, Talissa Altes, Edvin Isufi, Yaseen Dhemesh, Jaime Gass, Jonathan Thacker, Abdul Rahman Tarabishy, Benjamin Turner, Sebastiano Vacca, George K. Vilanilam, Daniel Warren, David Weiss, Fikadu Worede, Sara Yousry, Wondwossen Lerebo, Alejandro Aristizabal, Alexandros Karargyris, Hasan Kassem, Sarthak Pati, Micah Sheller, Katherine E. Link, Evan Calabrese, Nourel hoda Tahon, Ayman Nada, Yuri S. Velichko, Spyridon Bakas, Jeffrey D. Rudie, Mariam Aboian

arXiv:2306.00838v3q-bio.OTeess.IV

TL;DR

Brain metastasis segmentation needs diverse, well-annotated imaging data to support evaluation across clinical settings. BraTS-METS 2023 curated and annotated a multi-institutional dataset and benchmarked challenge algorithms; the discussion reports leading-team Dice scores of 0.60–0.65 across tumor entities while identifying small lesions as a continuing challenge.

  • Problem

    Brain-metastasis algorithms depend on diverse, high-quality annotated imaging datasets, and small-lesion detection remains a challenge for current models.

    Source: §1 · p. 7, §6 · p. 19

  • Method

    BraTS-METS 2023 curated multi-institutional MRI data and annotations and evaluated challenge algorithms using lesion-wise segmentation metrics.

    Source: §1 · p. 7, §7 · p. 23

  • Results

    0.60 to 0.65 mean lesion-wise Dice score across different tumor entities for Team NVAUTO, the highest-scoring team.

    Source: §6 · p. 20

  • Takeaways & Limitations

    The challenge advanced development of models intended to be robust and adaptable across varied clinical environments.

    Source: §7 · p. 24

    Precise detection of small metastases remains a principal challenge; improving sensitivity may require larger sample sizes and lesion-wise-focused architectures or loss functions.

    Source: §6 · p. 19

Abstract

from arXiv · show

The translation of AI-generated brain metastases (BM) segmentation into clinical practice relies heavily on diverse, high-quality annotated medical imaging datasets. The BraTS-METS 2023 challenge has gained momentum for testing and benchmarking algorithms using rigorously annotated internationally compiled real-world datasets. This study presents the results of the segmentation challenge and characterizes the challenging cases that impacted the performance of the winning algorithms. Untreated brain metastases on standard anatomic MRI sequences (T1, T2, FLAIR, T1PG) from eight contributed international datasets were annotated in stepwise method: published UNET algorithms, student, neuroradiologist, final approver neuroradiologist. Segmentations were ranked based on lesion-wise Dice and Hausdorff distance (HD95) scores. False positives (FP) and false negatives (FN) were rigorously penalized, receiving a score of 0 for Dice and a fixed penalty of 374 for HD95. Eight datasets comprising 1303 studies were annotated, with 402 studies (3076 lesions) released on Synapse as publicly available datasets to challenge competitors. Additionally, 31 studies (139 lesions) were held out for validation, and 59 studies (218 lesions) were used for testing. Segmentation accuracy was measured as rank across subjects, with the winning team achieving a LesionWise mean score of 7.9. Common errors among the leading teams included false negatives for small lesions and misregistration of masks in space.The BraTS-METS 2023 challenge successfully curated well-annotated, diverse datasets and identified common errors, facilitating the translation of BM segmentation across varied clinical environments and providing personalized volumetric reports to patients undergoing BM treatment.

1. Introduction

Brain metastasis assessment can be laborious, while algorithms depend on diverse, high-quality annotated imaging data. BraTS-METS 2023 focused on pre-treatment metastasis segmentation and outlined a path toward collaborative research and clinical translation.

1. Introduction

  • Reviewing multiple prior scans to track metastasis changes can be laborious, motivating automated volumetric assessment and lesion organization for more personalized, time-efficient evaluation.The introduction describes AI-based volumetric assessments as a potential source of personalized treatment-response reports.Source: §1 · p. 7
  • Machine-learning performance depends on annotated imaging data, and limited large-scale resources have left many researchers reliant on smaller local datasets that constrain cross-institutional generalizability.The passage presents multi-center medical-image challenges as platforms for development, testing, and benchmarking.Source: §1 · p. 7
  • Brain metastases vary in lesion size, shape, and location and change over time or with treatment, creating a need for adaptable multi-lesion and longitudinal assessment.The introduction links these complexities to the need for sophisticated machine-learning approaches.Source: §1 · p. 7
  • BraTS-METS 2023 set out to segment pre-treatment metastases on T1, T1 post-contrast, T2, and FLAIR MRI, with the resulting standardized algorithm made openly accessible.The stated aim was to support integration into clinical and research protocols across institutions.Source: §1 · p. 7
  • The challenge proposed an annotation-to-consortium trajectory: segment pre-treatment metastases, delineate enhancing tumor, edema, and necrotic portions, then foster collaborative research and clinical translation.The consortium was also intended to support community education.Source: §1 · p. 7

2. Background

Conventional brain-metastasis assessment relies on qualitative review and lesion measurements, but irregular lesions and changing scans complicate longitudinal evaluation. The challenge vision broadens from pre-treatment segmentation toward future datasets and consortium research.

2. Background

  • RANO-BM clinical-trial criteria primarily measure a lesion’s longest diameter, while conventional evaluation also uses qualitative size and number changes and manual two-dimensional measurements.These criteria may not fully capture metastasis dynamics and morphology, particularly with heterogeneous, irregular growth.Source: §2 · pp. 7–8
  • The BraTS-METS vision begins with pre-treatment segmentation and plans to expand toward post-treatment cases, hemorrhagic tumor components, and non-skull-stripped images.The planned expansion is intended to help evaluate dural-based and osseous metastases and support an inter-institutional consortium.Source: Figure 1 · p. 8
  • MRI can detect lesions as small as 1–2 mm, but inter-rater variability below 5 mm has kept consensus measurable-disease criteria at a 10 mm minimum; practice has also shifted toward a 5 mm threshold.The passage notes that improved reproducibility from algorithm-based measurements could support future reassessment of standardized criteria.Source: §2 · p. 8
  • Automated segmentation across serial pre- and post-treatment scans could complement clinicians’ tracking of disease progression, treatment response, and new lesions.The passage frames automation as a way to improve the precision and efficiency of clinical assessment.Source: §2 · p. 8
  • Routine clinical implementation faces time demands and variation across temporal scans from differences in imaging equipment and radiologist interpretation.The passage identifies these as sources of acquisition heterogeneity and inter-reader variability.Source: §2 · p. 8

3. Related Works

Prior work explores automated metastasis segmentation using varied MRI inputs and segmentation strategies, while the paper identifies inconsistent public datasets and the absence of a comprehensive benchmark as gaps.

3. Related Works

  • Prior studies describe automated BM segmentation with T1 contrast-enhanced MRI, core-sequence selection, multi-phase delayed-enhanced MRI, and 2.5D or 3D methods on multinational data.The cited work reports that delayed-enhanced images improved segmentation and classification accuracy, while multinational data supported robustness and adaptability.Source: §3 · p. 9
  • Reported approaches address different needs, including post-radiosurgery treatment assessment, standardized imaging protocols, and adaptation to complex metastasis presentations.The cited studies include self-supervised learning work on data-domain effects and methods aimed at reducing false positives.Source: §3 · p. 9
  • Accurately identifying and localizing 1–2 mm metastases matters for treatment planning, but existing glioma-focused algorithms perform less well on smaller lesions than on larger ones.The passage notes that gross total volume may be prognostically important, while validated volumetric segmentation tools are lacking.Source: §3 · p. 9
  • Public BM datasets differ in their inclusion criteria and quality, and the paper identifies a lack of comprehensive public data and universally accepted metastasis-specific AI tools.The authors describe standardized benchmarking as a way to compare and refine segmentation models.Source: §3 · p. 9

4. Materials & Methods

BraTS-METS assembled untreated metastasis MRI data from varied institutions, standardized image preparation, and assigned three tumor subregion labels. Its methods specify eligibility, challenge allocation, and label definitions.

4.1 Data

  • The dataset included retrospectively collected scans from diverse institutions with untreated metastases and T1 pre-contrast, T1 post-contrast, T2, and FLAIR sequences; prior treatment changes, missing sequences, and technically unacceptable scans were excluded.The dataset was centrally curated; post-treatment cases were reserved for BraTS-METS 2024.Source: §4.1 · pp. 9–10
  • Challenge allocation followed a 70% training, 10% validation, and 20% testing split, with labels supplied only for training and validation and test cases kept hidden.Additional public or private data were restricted during ranking; external data could be used for publication if transparently disclosed.Source: §4.1 · p. 10

4.2 Imaging Data Description

  • All scans underwent DICOM-to-NIfTI conversion, registration to the SRI24 template, resampling to 1 mm³ isotropic resolution, and skull stripping.The pipeline was made available through CaPTk and FeTS; conversion removed DICOM-header protected health information, and skull stripping mitigated facial reconstruction or recognition.Source: §4.2 · pp. 10–11

4.3 Tumor Labels

  • Tumor annotations comprised enhancing tumor (ET, label 3), surrounding non-enhancing FLAIR hyperintensity (SNFH, label 2), and non-enhancing tumor core (NETC, label 1).ET is the T1Gd-enhancing portion; NETC is the presumed necrotic core; SNFH includes infiltrative non-enhancing tumor and peritumoral vasogenic edema.Source: §4.3 · p. 11

4.4 Tumor Annotation Protocol

The protocol defines tumor subregions on structural mpMRI, then uses a staged annotation pipeline with model pre-segmentations to support consistent ground-truth labeling.

4.4 Tumor Annotation Protocol

  • WT combines labels 1, 2, and 3, while ET is treated separately because edema can overlap between multiple lesions.The protocol describes WT as including tumor core and surrounding edematous or invaded tissue.Source: §4.4 · p. 11

4.5 Annotation Pipeline

  • The annotation pipeline has five stages: pre-segmentation, annotation refinement, technical QC, initial approval, and final approval.The pipeline is designed to support uniform imaging and tumor labeling.Source: §4.5 · p. 11

4.6 Pre-segmentation

  • Three nnU-Net approaches generated pre-segmentations, with label-specific fusion: STAPLE for SNFH and minority voting for ET.The models were trained on UCSF-BMSR, AURORA, and Heidelberg datasets; NETC was produced only by the UCSF-BMSR model.Source: §4.6 · p. 11

4.7 Annotation Refinement and Initial Approval

Student annotations were refined and reviewed through multiple neuroradiologist checks and technical QC; the process also addressed recurring segmentation errors.

4.7 Annotation Refinement and Initial Approval

  • More than 150 student annotators and volunteer neuroradiologists refined model-assisted annotations, with 52 board-certified neuroradiologists reviewing completed cases.Incomplete cases were returned for correction, and students received group reviews, lectures, and expert guidance.Source: §4.7 · p. 13

4.8 Annotation Final Approval

  • A second neuroradiologist reviewed each refined case, and a final expert review checked consistency across the dataset.Discrepancies were corrected by the second approver without returning cases to trainees.Source: §4.8 · p. 13

4.9 Common Errors of Automated Segmentations

  • Common errors included missed small metastases, white-matter changes from microvascular disease, and intrinsic T1 hyperintensity in non-enhancing lesions.Attendings manually segmented missed small metastases, modified peritumoral edema masks, and removed intrinsic T1-hyperintense voxels from ET masks.Source: §4.9 · pp. 13–14

4.10 Performance Evaluation Framework

The framework ranked teams using lesion-wise Dice and HD95 across regions and test subjects, penalizing false detections and comparing relative rankings statistically.

Performance evaluation

  • Containerized submissions were run on contributing sites' evaluation data through MedPerf, which calculated metrics before Synapse ranked teams.Participants could use the GaNDLF baseline framework.Source: §4.10 · p. 14
  • Teams were ranked using lesion-wise Dice and HD95 across ET, TC, and WT, with false positives and false negatives receiving Dice 0 and an HD95 penalty of 374.Scores were averaged by case within each tissue category.Source: §4.10 · p. 14
  • The final ranking score averaged each team's subject-level cumulative ranks across test subjects, based on 354 rankings per team.The 354 rankings came from 59 subjects, three regions, and two metrics.Source: §4.10 · p. 15
  • Pairwise team rankings were tested with 100,000 permutations, with p-values reflecting how often randomized rank differences exceeded the observed difference.The results were reported in an upper-triangular matrix.Source: §4.10 · p. 15

4.11 Analysis

The analysis evaluates individual lesions while accounting for lesion connectivity and small-volume annotation uncertainty; the dataset section notes institutional contributions at different approval stages.

4.11 Analysis

  • The framework measures lesion-wise Dice and HD95 across ET, TC, and WT, avoiding reliance on whole-image scores that can favor large lesions.Predicted connected components were compared individually with ground truth after dilation, while the ground-truth labels remained unchanged.Source: §4.11 · p. 15
  • Lesions below 2 voxels (2 mm3) were excluded from evaluation to avoid penalizing models for stray ground-truth voxels or unrelated small lesions.A one-voxel 3D dilation factor was chosen to avoid combining small metastases.Source: §4.11 · p. 15

4.12 Dataset

  • Datasets came from multiple institutions and were at different stages of annotation and approval.Source: §4.12 · p. 15

5. Results

BraTS-METS 2023 reports challenge results across diverse dataset groups, with NVAUTO ranking first and leading performance across tumor entities and lesion sizes. The results also show that smaller lesions remain especially challenging for segmentation algorithms.

5.1 Dataset Sources

  • The challenge reviewed 1303 cases from eight institutions; reported allocations included 402 training cases, 31 validation cases, and 59 testing cases.An additional 474 UCSF and Stanford cases were optional training data because they lacked original T2 images.Source: §5.1 · p. 16, Table 2 · p. 16

5.2 Lesion Characteristics

5.3 Performance Analysis

  • NVAUTO ranked first, with an average rank across subjects of 7.9 and a PatientWise mean of 0.38; SY ranked second with a PatientWise mean of 0.41.Table 4 presents the relative team rankings, with lower scores indicating better performance.Source: §5.3 · p. 16, Table 4 · p. 17
  • NVAUTO, SY, and blackbean had higher median patient-wise accuracy and relatively narrow IQRs, while DeepRadOnc had a wider IQR.Figure 5 shows the distribution of patient-wise LesionWise rankings, where lower is better.Source: §5.3 · pp. 16–17, Figure 5 · p. 17

5.4 Detailed Performance by Tumor Entities

  • NVAUTO ranked first across ET, TC, and WT, with mean Dice scores of 0.60, 0.65, and 0.62, respectively.SY and blackbean shared second rank for ET, with a mean Dice score of 0.57.Source: §5.4 · p. 17
  • NVAUTO led lesion detection, sensitivity, and PPV across ET, TC, and WT, while blackbean and SY also showed high sensitivity and detection rates.NVAUTO’s detection rates were 76%, 78%, and 80%, and its sensitivity was 90%, 91%, and 90%, for ET, TC, and WT, respectively; its PPV was 82%, 84%, and 84%.Source: §5.4 · p. 17

5.5 Algorithm Sensitivity to Lesion Size

  • Segmentation performance declined as lesion size decreased, with teams struggling particularly on tumors under 100 mm^3; NVAUTO consistently outperformed its counterparts.Source: §5.5 · p. 17

6. Discussion

NVAUTO led the challenge across tumor entities, but small-lesion detection remained a major weakness. Dataset heterogeneity, annotation demands, and image-registration issues shaped the challenge’s practical limits and data-construction work.

6. Discussion

  • Small-metastasis detection remained the principal challenge, despite NVAUTO’s strong performance across lesion sizes and the need for improved lesionwise sensitivity.The discussion notes that current loss functions are optimized for voxelwise rather than lesionwise performance, and suggests larger sample sizes or novel architectures or loss functions.Source: §6 · pp. 19–21
  • The challenge metric penalized false positives and false negatives, yielding low Dice coefficients while prioritizing algorithms suited to diverse clinical practices.Source: §6 · p. 19
  • NVAUTO achieved the highest mean lesion-wise Dice scores, 0.60 to 0.65 across tumor entities, while SY and blackbean performed closely.The discussion also calls for further precision, particularly for smaller and more challenging lesions.Source: §6 · p. 20
  • Ground-truth annotation was labor-intensive for variable, often numerous lesions, prompting educational annotation and iterative expert review.The discussion describes student training and expert sessions alongside revisions that surfaced overlooked small lesions and segmentation inaccuracies.Source: §6 · pp. 22–23
  • Dataset heterogeneity and sequence misregistration complicated segmentation; inadequate skull stripping could remove metastases and hinder dural-lesion differentiation.The discussion also notes that skull stripping limits evaluation of calvarial osseous metastases.Source: §6 · p. 23
  • The diversity of contributed data may benefit algorithm generalization, while open science requires safeguards for sensitive patient data.Source: §6 · pp. 21–23

7. Conclusion

The inaugural BraTS-METS challenge addressed technical and practical challenges in creating datasets, reference-standard annotations, and assessment metrics for brain metastasis segmentation. It highlighted the need to detect small lesions while balancing sensitivity and false-positive reduction.

7. Conclusion

  • The inaugural BraTS-METS challenge addressed technical and practical challenges in establishing datasets, high-quality reference-standard annotations, and assessment metrics for BM segmentation.Source: §7 · p. 23
  • Small lesions can be overlooked because of human error or imaging limitations, making their detection a critical need for segmentation algorithms.Source: §7 · p. 24
  • Algorithms must balance detection sensitivity with minimizing false positives that can disrupt clinical workflows.Source: §7 · p. 24
  • The challenge used multi-institutional datasets to advance robust, adaptable models for varied clinical environments and practical medical use.Source: §7 · p. 24

Funding

The authors report partial support from the National Cancer Institute and institutional radiology departments.

Funding

  • The research was partly supported by NCI/NIH awards U01CA242871 and R21CA259964, Yale Department of Radiology, and CHOP Department of Radiology.Source: Funding · p. 24

Ethical Standards

The authors state that the work followed applicable ethical standards and laws governing research involving animals or human subjects.

Ethical Standards

  • The work followed applicable ethical standards and laws and regulations concerning animal or human subjects.Source: Ethical Standards · p. 24

Related papers

Loading 2306.00838v3…