PaperlayerView exact revision on arXiv

Source-linked AI summary

VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images

Anjany Sekuboyina, Malek E. Husseini, Amirhossein Bayat, Maximilian Löffler, Hans Liebl, Hongwei Li, Giles Tetteh, Jan Kukačka, Christian Payer, Darko Štern, Martin Urschler, Maodong Chen, Dalong Cheng, Nikolas Lessmann, Yujin Hu, Tianfu Wang, Dong Yang, Daguang Xu, Felix Ambellan, Tamaz Amiranashvili, Moritz Ehlke, Hans Lamecker, Sebastian Lehnert, Marilia Lirio, Nicolás Pérez de Olaguer, Heiko Ramm, Manish Sahu, Alexander Tack, Stefan Zachow, Tao Jiang, Xinjun Ma, Christoph Angerman, Xin Wang, Kevin Brown, Alexandre Kirszenberg, Élodie Puybareau, Di Chen, Yiwei Bai, Brandon H. Rapazzo, Timyoas Yeah, Amber Zhang, Shangliang Xu, Feng Hou, Zhiqiang He, Chan Zeng, Zheng Xiangshang, Xu Liming, Tucker J. Netherton, Raymond P. Mumme, Laurence E. Court, Zixun Huang, Chenhang He, Li-Wen Wang, Sai Ho Ling, Lê Duy Huynh, Nicolas Boutry, Roman Jakubicek, Jiri Chmelik, Supriti Mulay, Mohanasankar Sivaprakasam, Johannes C. Paetzold, Suprosanna Shit, Ivan Ezhov, Benedikt Wiestler, Ben Glocker, Alexander Valentinitsch, Markus Rempfler, Björn H. Menze, Jan S. Kirschke

arXiv:2001.09193v6cs.CVeess.IV

TL;DR

Automated spine processing needs reliable vertebral labelling and segmentation, but anatomical and acquisition variability and scarce public annotations hinder algorithm development. This work presents and evaluates the VerSe benchmark across challenge datasets and performance settings. The reported top methods achieve high identification and Dice scores, while performance depends on correctly identifying vertebrae with rare anatomical variations.

  • Problem

    Automated vertebral labelling and segmentation are important for spine analysis but are challenged by variable fields of view, anatomy, acquisition settings, and scarce public annotations.

  • Method

    The study benchmarks challenge algorithms on the VerSe datasets and examines performance across vertebra-, scan-, and field-of-view levels, including cross-iteration evaluation.

  • Results

    VerSe’19 achieved 94.25% identification rate and 89.80% Dice, while VerSe’20 achieved 96.6% identification rate and 91.72% Dice.

  • Takeaways & Limitations

    Algorithm performance hinges on correctly identifying vertebrae in cases involving rare anatomical variations.

  • Takeaways & Limitations

    The study lacks inter-rater annotations, and adding them was delegated to future challenge iterations.

Abstract

from arXiv · show

Vertebral labelling and segmentation are two fundamental tasks in an automated spine processing pipeline. Reliable and accurate processing of spine images is expected to benefit clinical decision-support systems for diagnosis, surgery planning, and population-based analysis on spine and bone health. However, designing automated algorithms for spine processing is challenging predominantly due to considerable variations in anatomy and acquisition protocols and due to a severe shortage of publicly available data. Addressing these limitations, the Large Scale Vertebrae Segmentation Challenge (VerSe) was organised in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in 2019 and 2020, with a call for algorithms towards labelling and segmentation of vertebrae. Two datasets containing a total of 374 multi-detector CT scans from 355 patients were prepared and 4505 vertebrae have individually been annotated at voxel-level by a human-machine hybrid algorithm (https://osf.io/nqjyw/, https://osf.io/t98fz/). A total of 25 algorithms were benchmarked on these datasets. In this work, we present the the results of this evaluation and further investigate the performance-variation at vertebra-level, scan-level, and at different fields-of-view. We also evaluate the generalisability of the approaches to an implicit domain shift in data by evaluating the top performing algorithms of one challenge iteration on data from the other iteration. The principal takeaway from VerSe: the performance of an algorithm in labelling and segmenting a spine scan hinges on its ability to correctly identify vertebrae in cases of rare anatomical variations. The content and code concerning VerSe can be accessed at: https://github.com/anjany/verse.

1. Introduction

Automated vertebral labelling and segmentation support spine analysis but remain difficult because of limited annotated data, anatomical and acquisition variability, and complex vertebral morphology. VerSe addresses these gaps with a public multi-detector CT benchmark containing voxel- and vertebral-level annotations.

  • Vertebral labelling and segmentation support diagnosis, spinal deformity assessment, biomechanical modelling, and surgical planning.
  • Manual vertebral segmentation is impractical for large datasets because each scan may contain about 25 structures of approximately 10^4 voxels.
  • Automated processing must handle variable fields of view, large scans, correlated adjacent-vertebra shapes, noise, scanner settings, implants, fractures, cement, and transitional vertebrae.
  • Existing publicly available datasets and algorithms were limited by single-centre collection, scarce high-quality annotations, or insufficient validation.
  • VerSe provides 374 multi-detector CT scans with vertebral centroids and voxel-level segmentation masks as a public benchmark for spine-processing algorithms.
  • The work analyses top-performing challenge algorithms and evaluates vertebral labelling and segmentation across the VerSe challenge iterations.

2. Materials and challenge setup

VerSe comprises two challenge iterations built from diverse multi-detector CT data with coordinate and voxel-level annotations, standardized evaluation metrics, and public training and test resources.

  • Data and annotations: 374 CT scans from 355 patients were anonymized and collected across multiple sites and scanner manufacturers.The dataset includes scanners from GE, Siemens, Phillips, and Toshiba.
  • Data and annotations: The dataset spans cervical, thoraco-lumbar, and cervico-thoraco-lumbar fields of view, with fractures, implants, and foreign materials represented.The composition was designed to resemble a typical clinical distribution in field of view, scan settings, and findings.
  • Data and annotations: Annotations contain vertebral-centroid coordinates and voxel-level segmentation masks for selected vertebrae, excluding partially visible vertebrae at scan boundaries.Twenty-six vertebrae were considered, including transitional T13 and L6 labels.
  • Data and annotations: Annotations were generated by automated initialization, iterative refinement by trained medical students and radiologists, and final approval by one radiologist.The annotation process combined human and machine contributions.
  • Challenge setup: The two challenge iterations used training, Public test, and Hidden test phases, with hidden evaluation preventing test-data fine-tuning.Participants submitted predictions for Public scans and containerized code for Hidden scans.
  • Challenge setup: All 374 scans and annotations, together with processing and evaluation scripts, were made publicly available.The resources are distributed through the VerSe repositories and OSF datasets.
  • Evaluation metrics: Labelling was evaluated with identification rate and mean localisation distance, while segmentation used Dice and Hausdorff distance.Identification requires a nearest predicted landmark within 20 mm; metrics were generally averaged over scans or vertebrae as specified.
  • Evaluation metrics: For predictions missing vertebrae, mean localisation and Hausdorff distance ignore those vertebrae, while identification rate and segmentation-related metrics penalize missing detections.This convention preserves performance estimates for correctly predicted vertebrae but does not remove the missing-vertebra penalty entirely.

3. Methods

The paper presents Anduin, a semi-automated interactive framework developed to generate voxel-level vertebral annotations at scale.

  • Methods: The framework supports scalable annotation generation by combining automated processing with medical-expert interaction.The paper introduces Anduin before describing selected participating algorithms.
  • Methods: Anduin is an in-house semi-automated, interactive processing tool used to generate initial annotations for more than 4000 vertebrae.The framework is organized as a three-stage pipeline.
  • Methods: Its three stages form the basis for subsequent spine localisation, vertebral labelling, and vertebral segmentation.The detailed stage descriptions are provided in the following methods sections.

1) Spine detection,

The benchmarked approaches use staged spine localisation, vertebral localisation and identification, and segmentation, with several methods adding interaction, iterative processing, or global label consistency.

  • Anduin: Anduin combines low-resolution spinal-centreline heatmaps, vertebral centroid predictions, and 1 mm vertebral patches for segmentation.The segmentation predictions are resampled and translated back to form a multi-label result.
  • Anduin: Anduin’s detection and labelling stages allow users to alter predicted bounding boxes and vertebral centroids, enabling accurate annotations with minimal human effort.A web version was made publicly available.
  • Participating methods: The benchmark included algorithms from multiple teams and summarized their methods in Table 3 across the two challenge iterations.The paper presents the best and second-best-performing teams for each iteration before discussing remaining algorithms.
  • Payer C.: Payer’s approach localises the spine, identifies vertebral centres, and segments each located vertebra before merging individual masks.The pipeline uses a U-Net variant for spine localisation, SpatialConfiguration-Net for vertebral localisation and identification, and a U-Net for binary segmentation.
  • Lessmann et al.: Lessmann’s method iteratively moves a region of interest over detected vertebral fragments, using a memory mask to avoid already segmented vertebrae.A global maximum-likelihood model subsequently resolves duplicate labels or gaps.

Net and U-Net [VerSe‘20]

For VerSe‘20, the approach retained the three-stage pipeline but replaced its post-processing with graph-based candidate selection to handle greater anatomical variation.

  • VerSe‘20 changes: The VerSe‘20 system preserves spine localisation, vertebral localisation and identification, and binary segmentation from the earlier approach.The main change occurs after localisation and identification.
  • Graph-based post-processing: For each vertebra, multiple location candidates are generated and selected by maximizing an MRF-based graph objective.Unary terms score candidate heatmap responses, while pairwise terms model spatial relationships between successive vertebrae.
  • Graph-based post-processing: The unary bias is 2.0 and the weighting factor λ is 0.2, encouraging vertebra detection while increasing reliance on directional information.The pairwise terms penalize deviations from average ground-truth vertebral displacement vectors.
  • Graph-based post-processing: The graph is solved by finding the shortest negative path from a virtual start vertex to a virtual end vertex.The formulation represents possible vertebral sequences through graph vertices and edges.
  • Implementation: Mixed-precision networks reduce training memory through 16-bit intermediate outputs while retaining 32-bit network weights.The authors report that network accuracy stays high under this representation.

4. Experiments

The benchmark evaluates vertebral labelling and segmentation across two VerSe iterations, multiple metrics, and scan, vertebra, region, field-of-view, and cross-iteration analyses. Performance is generally strong but varies substantially with rare anatomy, landmarks, and dataset distribution.

  • Overall evaluation: 25 algorithms were evaluated on vertebral labelling and segmentation across the VerSe’19 and VerSe‘20 challenge iterations.
  • Overall performance: 96.9% identification rate and 93.0% Dice were the highest VerSe‘19 Public-phase results, while VerSe‘19 Hidden results reached 94.3% and 89.8%.
  • Overall performance: 95.6% identification rate and 91.7% Dice were the highest VerSe‘20 Public-phase results, compared with 96.6% and 91.2% in Hidden.
  • Overall performance: At least four VerSe‘19 methods and seven VerSe‘20 teams achieved a median identification rate of 100%.
  • Cross-iteration comparison: Mean performance for all teams dropped in VerSe‘20, whereas Top-5 performance remained relatively consistent; identification rate and Dice increased from 2019 to 2020 for both groups.
  • Scan-level evaluation: Scan-level failures concentrated on anomalies at the lumbar-sacral junction, with absent L5 or transitional L6 commonly producing incorrect labels.Experts can use the last rib attached to T12, whereas algorithms often relied on the lumbar-sacral junction.
  • Vertebrae-wise evaluation: Transitional vertebrae caused sharp vertebra-level performance drops: no VerSe‘19 method identified L6, while almost all VerSe‘20 methods identified part of it.
  • Field-of-view evaluation: For similar fields of view, VerSe‘19 performance dropped sharply when L5 or C1 was absent, indicating reliance on the cranium and sacrum.

5. Discussion

The benchmark compares algorithm designs across processing dimensionality, staging, and model type, while emphasizing performance on rare anatomical variations and evaluation limitations. Data-driven deep-learning approaches generally outperform hybrid methods, but no single-stage or multi-stage strategy dominates, and robust handling of transitional vertebrae remains decisive.

  • Deep learning vs. hybrid: Data-driven deep-learning approaches show clear superiority over hybrid methods in this benchmark.The authors relate this result to the size of the VerSe dataset.
  • 3D patch-based vs. 2D slice-wise: 3D processing is generally preferable to naive 2D slice-wise segmentation because it preserves crucial 3D context for the spine.The trade-off is that 2D processing offers a larger field of view and greater memory efficiency, while missing vertebrae across slices can make labelling noisy.
  • Single model vs. multi-staged: Neither single-stage nor multi-stage approaches yields a clear winner; both categories occupy the upper regions of the leaderboards.Single-stage models may require many large-patch inferences, whereas multi-stage pipelines may compound errors from preliminary stages.
  • Anatomical variations: Rare anatomical variations expose major performance differences: algorithms frequently failed on L6, while some methods identified T13 despite its rarity.Xiangshang Z. was the only approach reported to identify all T13 instances in both test phases, and increased anomalous cases in VerSe’20 enabled some L6 detections and segmentations.
  • Limitations: The study’s main annotation limitation is the lack of inter-rater annotations because voxel-level annotation was labor-intensive and partly machine-assisted.Future challenge iterations could incorporate multiple raters, inter-rater variability studies, uncertainty prediction, and annotator-bias learning.
  • Evaluation metrics: A one-label shift can penalize current metrics more than label mixing, producing unusable masks and contributing to a drastic public-to-hidden performance drop.The authors therefore identify better domain-specific evaluation metrics, especially differentiable variants for neural-network optimisation, as an area of interest.

6. Conclusions

VerSe benchmarks vertebral labelling and segmentation across two challenge iterations and analyzes performance across anatomy, scans, and fields of view. The results emphasize rare anatomical variations, three-dimensional processing, and the role of sufficient data and post-processing.

  • Challenge scope: 374 CT scans from 355 patients supported evaluation of 25 algorithms for vertebral labelling and segmentation.The work describes the challenge setup, participating algorithms, baseline, and comparative benchmarking.
  • Benchmark results: 94.25% identification rate and 89.80% Dice were achieved by Payer C. in VerSe’19, while Chen D. achieved 96.6% and 91.72% in VerSe’20.These figures are reported on the respective Hidden test sets.
  • Benchmark results: Performance increased on average from VerSe‘19 to VerSe‘20 despite the latter data being more multi-centred and anomalous.The reported trend concerns the average performance of the algorithms.
  • Design observations: Spine processing was better approached in 3D, either with large patches or an appropriately designed sequence of stages.This is one of the paper’s stated key observations.
  • Design observations: Transitional vertebrae such as T13 and L6 could be handled efficiently given sufficient data and post-processing.The conclusion identifies rare anatomical variations as central to algorithm performance.
  • Future directions: Future benchmarks should consider multi-rater annotations, inter-rater variability, runtime objectives, spine-centred measures, and sacrum modelling.The authors identify these as future directions or considerations for benchmarking.

xxx

The cited work spans automated vertebral detection, identification, localization, and segmentation using deep learning, deformable or model-based methods, and contextual reasoning.

  • Prior approaches: Prior work includes automated vertebra localization, identification, and segmentation in CT and MR images.The cited studies address these tasks using varied imaging modalities and problem formulations.
  • Deep learning: Deep-learning approaches include 3D U-Net, fully convolutional networks, iterative convolutional networks, Mask R-CNN, and convolutional neural networks.These methods appear across the cited literature on vertebral processing and volumetric segmentation.
  • Structured methods: Other cited approaches use articulated spine models, landmark-assisted deformable models, manifold embeddings, and higher-order Markov random fields.These references represent model-based and structured methods for spine analysis.

Appendix A.1. Statistical Tests & Points

The appendix ranks participants through pairwise statistical comparisons of performance measures, with special handling for missing predictions and differences between challenge iterations.

  • Measures: Four performance measures were reported: id.rate, dmean, Dice, and HD.These measures were used in the technical report’s evaluation framework.
  • Missing predictions: HD and dmean were undefined for missing vertebral predictions and were excluded from VerSe’20 ranking to avoid substitution-induced bias.For VerSe‘19, missing predictions were substituted with fixed maximum distances of 1000 mm for dmean and 100 mm for HD.
  • Statistical comparison: Pairwise Wilcoxon signed-rank tests used a p−value threshold of 0.001 to identify significant performance differences.A point was assigned to the better team when the difference was statistically significant.
  • Point calculation: Points from pairwise comparisons were aggregated by team and normalised to scores between 0 and 1 for each measure.The normalised scores were then combined across measures for challenge ranking.
  • Reported outputs: The resulting points and pairwise significance matrices were reported in Tables A.8a–A.8b and Figures A.11–A.12.The figures show p-values and their binarised versions for the performance measures.

Appendix A.2. Final Ranking: Combining all the scores

The final ranking combines normalised performance scores across evaluation stages using task- and dataset-specific weights. Segmentation receives greater weight than labelling, while Hidden data receives greater weight than Public data.

  • Ranking construction: Normalised points across performance measures were combined across multiple evaluation stages to produce the final ranking.Figure A.13 illustrates the multi-stage combination procedure.
  • Measure weights: dmean and HD were weighted at a 1:2 ratio relative to id.rate and Dice, except in VerSe‘20.This weighting was intended to reduce the contribution of selected upper bounds when predictions were missing.
  • Dataset weights: Hidden evaluation received twice the weight of Public evaluation because it used a completely hidden dataset.The weighting was intended to reduce opportunities for over-fitting or retraining on the test set.
  • Task weights: The segmentation task received twice the weight of the labelling task because labelling can possibly be a consequence of segmentation.This weighting reflected the challenge’s final goal.

Appendix B. Description of Anduin

Anduin is a three-stage framework for predicting voxel-level vertebral segmentations from spine CT scans. Its stages detect the spine, label vertebrae, and segment individual vertebrae.

  • Anduin assists VerSe ground-truth creation and is described as a framework for accurate voxel-level vertebral segmentation.The framework divides the task into spine detection, vertebrae labelling, and vertebrae segmentation.
  • The three subtasks’ network architectures, loss functions, and training and inference details are elaborated in the appendix.
  • The appendix identifies submitted approaches and point-count comparisons for VerSe‘19 and VerSe‘20.

Appendix B.1. Notation.

Anduin represents a CT scan and its vertebral annotations using centroids, task-specific ground truths, and multi-label segmentation masks.

  • The input CT scan is represented as x ∈R^h×w×d, with h, w, and d denoting scan height, width, and depth.
  • Vertebral centroids μ_i ∈R^3 are available for i ∈{1, 2, . . . N} and support construction of detection and labelling ground truths.
  • The multi-label segmentation masks are denoted by y_s ∈Z^h×w×d.

Appendix B.2. Spine Detection

Anduin detects the spine, labels vertebrae, and segments each vertebra through sequential, resolution-specific modules operating on focused CT data.

  • Spine Detection: The spine detector is a parametrically light 3D FCN operating at 4 mm isotropic resolution and regressing Gaussian heatmaps at vertebral locations.The heatmap is generated at 1 mm with σ = 8, then downsampled to 4 mm.
  • Spine Detection: Spine detection optimizes a combination of ℓ2 and binary cross-entropy losses for foreground and background prediction.The ground truth concatenates a Gaussian location map with a background channel, while softmax and cross-entropy define the classification term.
  • Vertebrae Labelling: The labelling module adapts Btrfly net to sagittal and coronal 2D MIPs extracted from a spine-focused region at 2 mm isotropic resolution.Spine-focused cropping reduces rib and pelvic-bone occlusions and makes the labelling input more uniform.
  • Vertebrae Labelling: A median-frequency weight map gives greater loss weight to less frequent vertebral classes during sagittal and coronal labelling.
  • Vertebral Segmentation: After labelling, each vertebra is segmented from a centroid-centered patch using a 1 mm U-Net with a vertebra-of-interest arm.The VOI arm processes a Gaussian heatmap centered at the predicted vertebral location and shares features with the image encoder.
  • Inference: Inference takes a 3D MDCT spine scan and outputs vertebral centroids and segmentation masks, with interaction points for correcting predictions.

tification

The appendix surveys multi-stage vertebral labelling and segmentation systems, including projection-based, template-based, attention-based, and joint instance-segmentation approaches.

  • Multi-stage approaches: Several approaches first obtain rough spine or vertebral regions, then refine voxel-level segmentation within local regions of interest.
  • Template-based approaches: Template-based systems identify vertebrae by fitting manually generated anatomical templates and enforcing consistent vertebral ordering.
  • Joint approaches: One highlighted approach jointly solves identification and segmentation using a keypoint-based 3D anchor-free instance-segmentation framework.The text presents this combination as a first to the participant’s knowledge.
  • Attention-based approaches: A2UNet introduces attention and aggregation mechanisms into a 3D U-Net to enhance feature representation for both tasks.
  • Limitations: An extended 3D Mask R-CNN requires significant memory, limiting the number of regions of interest passed to its second stage.
Loading 2001.09193v6…