Source-linked AI summary
The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification
Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C. Kitamura, Sarthak Pati, Luciano M. Prevedello, Jeffrey D. Rudie, Chiharu Sako, Russell T. Shinohara, Timothy Bergquist, Rong Chai, James Eddy, Julia Elliott, Walter Reade, Thomas Schaffter, Thomas Yu, Jiaxin Zheng, Ahmed W. Moawad, Luiz Otavio Coelho, Olivia McDonnell, Elka Miller, Fanny E. Moron, Mark C. Oswood, Robert Y. Shih, Loizos Siakallis, Yulia Bronstein, James R. Mason, Anthony F. Miller, Gagandeep Choudhary, Aanchal Agarwal, Cristina H. Besada, Jamal J. Derakhshan, Mariana C. Diogo, Daniel D. Do-Dai, Luciano Farage, John L. Go, Mohiuddin Hadi, Virginia B. Hill, Michael Iv, David Joyner, Christie Lincoln, Eyal Lotan, Asako Miyakoshi, Mariana Sanchez-Montano, Jaya Nath, Xuan V. Nguyen, Manal Nicolas-Jilwan, Johanna Ortiz Jimenez, Kerem Ozturk, Bojan D. Petrovic, Chintan Shah, Lubdha M. Shah, Manas Sharma, Onur Simsek, Achint K. Singh, Salil Soman, Volodymyr Statsevych, Brent D. Weinberg, Robert J. Young, Ichiro Ikuta, Amit K. Agarwal, Sword C. Cambron, Richard Silbergleit, Alexandru Dusoi, Alida A. Postma, Laurent Letourneau-Guillon, Gloria J. Guzman Perez-Carrillo, Atin Saha, Neetu Soni, Greg Zaharchuk, Vahe M. Zohrabian, Yingming Chen, Milos M. Cekic, Akm Rahman, Juan E. Small, Varun Sethi, Christos Davatzikos, John Mongan, Christopher Hess, Soonmee Cha, Javier Villanueva-Meyer, John B. Freymann, Justin S. Kirby, Benedikt Wiestler, Priscila Crivellaro, Rivka R. Colen, Aikaterini Kotrotsou, Daniel Marcus, Mikhail Milchenko, Arash Nazeri, Hassan Fathallah-Shaykh, Roland Wiest, Andras Jakab, Marc-Andre Weber, Abhishek Mahajan, Bjoern Menze, Adam E. Flanders, Spyridon Bakas
TL;DR
BraTS 2021 addresses automated glioma sub-region segmentation and MGMT promoter methylation prediction from pre-operative multi-institutional mpMRI. It defines benchmark tasks, curated annotations, and evaluation procedures, with testing conducted through hidden cohorts and challenge platforms. The challenge’s scope is broad, while annotation and MGMT-labeling limitations constrain interpretation.
Problem
Glioma sub-region delineation is laborious and subjective, while MGMT promoter methylation status is clinically relevant for prognosis and chemotherapy response.
Method
BraTS 2021 benchmarks segmentation of enhancing tumor, tumor core, and whole tumor alongside binary MGMT methylation classification using curated multi-institutional pre-operative mpMRI.
Results
The challenge evaluates participating algorithms through hidden testing, using Sage Bionetworks Synapse for Task 1 and Kaggle for Task 2.
Takeaways & Limitations
BraTS 2021 provides a common benchmark for comparing automated tumor compartmentalization and imaging-based molecular characterization methods.
Takeaways & Limitations
Tumor annotations lack measurable inter-rater agreement, MGMT labels are binary despite institution-specific assays and thresholds, and some scans contain unannotated non-glioma abnormalities.
Abstract
from arXiv · showhide
The BraTS 2021 challenge celebrates its 10th anniversary and is jointly organized by the Radiological Society of North America (RSNA), the American Society of Neuroradiology (ASNR), and the Medical Image Computing and Computer Assisted Interventions (MICCAI) society. Since its inception, BraTS has been focusing on being a common benchmarking venue for brain glioma segmentation algorithms, with well-curated multi-institutional multi-parametric magnetic resonance imaging (mpMRI) data. Gliomas are the most common primary malignancies of the central nervous system, with varying degrees of aggressiveness and prognosis. The RSNA-ASNR-MICCAI BraTS 2021 challenge targets the evaluation of computational algorithms assessing the same tumor compartmentalization, as well as the underlying tumor's molecular characterization, in pre-operative baseline mpMRI data from 2,040 patients. Specifically, the two tasks that BraTS 2021 focuses on are: a) the segmentation of the histologically distinct brain tumor sub-regions, and b) the classification of the tumor's O[6]-methylguanine-DNA methyltransferase (MGMT) promoter methylation status. The performance evaluation of all participating algorithms in BraTS 2021 will be conducted through the Sage Bionetworks Synapse platform (Task 1) and Kaggle (Task 2), concluding in distributing to the top ranked participants monetary awards of $60,000 collectively.
1 Introduction
BraTS 2021 addresses the clinical need for automated glioma sub-region segmentation and pre-operative MGMT promoter methylation prediction using multi-institutional mpMRI. These tasks matter because gliomas are aggressive and heterogeneous, while manual delineation is burdensome and MGMT status can influence treatment decisions.
- Clinical motivation: Glioblastoma and WHO Grade 4 astrocytoma are aggressive CNS malignancies with highly heterogeneous appearance, shape, and histology.Average prognosis is 14 months after standard treatment and 4 months untreated.
- Clinical motivation: Manual MRI delineation of tumor sub-regions is tedious, time-consuming, subjective, and impractical for numerous patients.The paper identifies automated deterministic segmentation as an unmet need.
- Clinical motivation: MGMT promoter methylation is a favorable prognostic factor and predictor of chemotherapy response in newly diagnosed glioblastoma.Determining methylation status can influence treatment decision making.
- Challenge scope: BraTS 2021 evaluates automated tumor sub-region segmentation and MGMT methylation classification from pre-operative baseline, multi-institutional mpMRI scans.The challenge distinguishes MGMT-methylated from unmethylated tumors.
2.1 Data
BraTS 2021 provides heterogeneous, retrospectively collected multi-institutional mpMRI data with standardized preprocessing, curated tumor annotations, binary MGMT labels, and hidden testing. Its annotation protocol combines algorithmic initialization with expert refinement while acknowledging biologic and segmentation limitations.
- Data: The dataset grew from 660 to 2,000 cases and comprises scans acquired across institutions with different equipment and imaging protocols.The resulting image quality reflects diverse clinical practice.
- Data: Data are divided into training, validation, and hidden testing cohorts, with ground-truth labels provided only for training.Participants submit validation results online, while final methods are evaluated on an undisclosed out-of-distribution cohort.
- Data: The mpMRI data include native T1, post-contrast T1Gd, T2, and T2-FLAIR volumes acquired using varied scanners and protocols.All scans undergo conversion, co-registration, isotropic resampling, and skull-stripping.
- Annotation protocol: Task 1 annotations were initialized by STAPLE fusion of nnU-Net, DeepScan, and DeepMedic segmentations, then manually refined and approved by neuroradiologists.The annotated regions include enhancing tumor, edema/invasion, and necrotic core.
- Annotation protocol: BraTS evaluates enhancing tumor, tumor core, and whole tumor, although these image-based sub-regions do not represent strict biologic entities.Alternative delineation criteria can produce slightly different sub-regions.
- Annotation protocol: Common automated-segmentation errors include labeling blood products or choroid plexus as edema, vessels as tumor regions, and white-matter hyperintensities as tumor.These errors motivate careful interpretation of automated labels.
2.2 Challenge Tasks
BraTS 2021 contains two complementary tasks: segmenting heterogeneous glioblastoma sub-regions and predicting MGMT promoter methylation status from imaging-derived features. Participants may focus on either task or both.
- Task 1: Tumor sub-region segmentation: Task 1 evaluates state-of-the-art methods for segmenting intrinsically heterogeneous glioblastoma sub-regions in multi-institutional mpMRI.The evaluated regions are enhancing tumor, tumor core, and whole tumor.
- Task 2: Radiogenomic classification: Task 2 evaluates methods that predict MGMT promoter methylation status at pre-operative baseline using quantitative imaging phenomic features and machine learning.Methylated cases are labeled 1 and unmethylated cases 0.
- Challenge participation: Participants may choose to focus on only one task or address both tasks.
2.3 Performance Evaluation
BraTS 2021 evaluates segmentation and MGMT classification methods using held-out testing data, task-specific metrics, and ranking procedures designed to assess performance and statistical differences between teams.
- Testing uses a cohort excluded from training and validation to evaluate the generalizability of submitted methods on testing out of distribution data.The test dataset remains hidden, and participants submit containerized methods for final evaluation.
- Segmentation performance uses Dice similarity coefficient, 95% Hausdorff distance, Sensitivity, and Specificity to assess tumor-region overlap, distance, and potential over- or undersegmentation.
- BraTS ranking aggregates team-level rankings across testing subjects, tumor regions, and evaluation measures into a final ranking score.In BraTS 2020, this produced 996 individual rankings per team before final-score calculation.
- Permutation testing repeats randomizations 100,000 times to assess whether pairwise ranking differences exceed those expected by chance.
- Top validation teams are invited to present at MICCAI 2021, while the final three testing-data teams are invited to RSNA 2021 and receive monetary awards.
- Task 2 ranks MGMT methylation classifiers primarily by AUC, alongside accuracy, FScore (Beta), and Matthew’s Correlation Coefficient.AUC measures discriminatory capacity across thresholds but does not guarantee calibration or have straightforward clinical meaning.
2.4 Participation Timeline
BraTS 2021 releases data sequentially for training, validation, and final testing, allowing preliminary evaluation while preserving an unseen test set for final ranking.
- The training release includes imaging data and corresponding ground-truth labels for method design and training.
- Validation data are released within three weeks without ground truth, enabling preliminary results on unseen cases and multiple online submissions.
- Final evaluation ranks all participants on the same hidden testing data after they upload containerized methods, with $60,000 in total monetary prizes.
3 Discussion
BraTS 2021 provides a large, curated, multi-institutional benchmark for computational neuro-oncology, while its annotation, molecular-label, and disease-scope limitations motivate future expansion.
- BraTS 2021 presents a curated multi-label mpMRI dataset with refined tumor annotations for approximately 2,000 cases and evaluation through dedicated platforms.
- The dataset is intended to serve as a common benchmark for computational neuro-oncology beyond the specific 2021 tasks.
- Single-annotator tumor segmentations approved through iterative review prevent assessment of potential inter-rater agreement.
- Institution-specific MGMT testing methods and thresholds led to providing only binary methylation labels rather than continuous values.
- Some MRI cases contain abnormalities beyond gliomas, but these were excluded from annotation because the challenge focused on gliomas.
- Future directions include broader brain-abnormality coverage, postoperative scans, and a resection-cavity label for treatment-response and disease-progression assessment.
Funding
The publication reports partial support from NCI and NINDS programs of the NIH and disclaims that its content represents official government views or endorsement.
- Research support came partly from NCI and NINDS programs of the NIH under the listed award and contract numbers.
- The publication disclaims official NIH, RSNA R&E Foundation, Department of Health and Human Services views, and endorsement of named products or organizations.