PaperlayerView exact revision on arXiv

Source-linked AI summary

Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth

Vanya V. Valindria, Ioannis Lavdas, Wenjia Bai, Konstantinos Kamnitsas, Eric O. Aboagye, Andrea G. Rockall, Daniel Rueckert, Ben Glocker

arXiv:1702.03407v1cs.CV

TL;DR

The paper addresses how to assess segmentation quality and detect failures on new clinical images when ground-truth references are unavailable. It introduces RCA, which trains a reverse classifier from a predicted segmentation and evaluates it on reference images with ground truth. The results indicate that RCA can predict individual segmentation quality, with strongest support for overlap-based metrics and some support for RVD.

  • Problem

    After deployment, segmentation performance and failures on new data are difficult to assess because ground-truth references are unavailable.

  • Method

    RCA trains a reverse classifier using a new image’s predicted segmentation as pseudo ground truth, then evaluates it on reference images with available ground truth.

  • Results

    RCA can predict individual segmentation quality without ground truth; performance is strongest for overlap-based metrics, extends to some extent to RVD, and is inaccurate for distance-based metrics in the current setting.

  • Takeaways & Limitations

    RCA is suited for identifying failed segmentations in clinical pipelines and selecting high-quality segmentations in large-scale imaging studies.

  • Takeaways & Limitations

    RCA assumes a suitable relationship between predicted and real scores and requires a reference database that captures relevant image variability.

Abstract

from arXiv · show

When integrating computational tools such as automatic segmentation into clinical practice, it is of utmost importance to be able to assess the level of accuracy on new data, and in particular, to detect when an automatic method fails. However, this is difficult to achieve due to absence of ground truth. Segmentation accuracy on clinical data might be different from what is found through cross-validation because validation data is often used during incremental method development, which can lead to overfitting and unrealistic performance expectations. Before deployment, performance is quantified using different metrics, for which the predicted segmentation is compared to a reference segmentation, often obtained manually by an expert. But little is known about the real performance after deployment when a reference is unavailable. In this paper, we introduce the concept of reverse classification accuracy (RCA) as a framework for predicting the performance of a segmentation method on new data. In RCA we take the predicted segmentation from a new image to train a reverse classifier which is evaluated on a set of reference images with available ground truth. The hypothesis is that if the predicted segmentation is of good quality, then the reverse classifier will perform well on at least some of the reference images. We validate our approach on multi-organ segmentation with different classifiers and segmentation methods. Our results indicate that it is indeed possible to predict the quality of individual segmentations, in the absence of ground truth. Thus, RCA is ideal for integration into automatic processing pipelines in clinical routine and as part of large-scale image analysis studies.

I. INTRODUCTION

Clinical segmentation requires reliable quality assessment, but deployment data usually lacks ground-truth references needed for conventional evaluation. This gap is especially consequential when failures must be detected automatically across large imaging cohorts.

  • I. INTRODUCTION: Standard evaluation compares predicted segmentations with annotated references using overlap, volume-difference, and surface-distance metrics.Dice’s similarity coefficient is identified as a commonly used overlap-based measure.
  • I. INTRODUCTION: After deployment, segmentation performance on new clinical data cannot be measured with traditional metrics when ground truth is unavailable.The paper frames failure detection without manual reference segmentations as its central introduction problem.
  • I. INTRODUCTION: This limitation matters for automated pipelines because segmentation failures may go unnoticed when visual quality control is infeasible.The concern is particularly relevant to large-scale studies processing thousands of images for subsequent population analysis.
  • I. INTRODUCTION: Existing approaches include contextual, unsupervised, multi-reference, generative, and learned methods, but their applicability to general medical segmentation or direct score prediction is limited.The cited approaches differ in whether they require suitable features, thresholds, multiple references, or specialized task assumptions.

B. Contribution

The paper introduces reverse classification accuracy as a per-image framework for estimating segmentation quality without ground truth. It evaluates the idea across segmentation methods and metrics for potential use in automated clinical and large-scale analysis pipelines.

  • B. Contribution: Reverse classification accuracy (RCA) assesses an individual segmentation without ground truth by training a classifier on its predicted labels as pseudo ground truth.The classifier is then evaluated on reference images with available ground-truth segmentations.
  • B. Contribution: The study tests RCA across different segmentation methods, RCA classifiers, and segmentation metrics in a multi-organ segmentation task.The evaluation examines how well performance can be predicted for each individual case.
  • B. Contribution: The reported results indicate that segmentation performance can be predicted at least to some extent without ground truth, supporting RCA’s use in automated pipelines and large-scale imaging studies.The paper positions RCA as a tool for clinical routine and population-scale image analysis.

II. REVERSE CLASSIFICATION ACCURACY

RCA learns reverse classifiers from a new image’s predicted segmentation and implements them with several voxel-wise or atlas-based approaches. The paper compares Atlas Forests, CNNs, and atlas-based label propagation as realizations of this framework.

  • II. REVERSE CLASSIFICATION ACCURACY: RCA trains a reverse classifier on one image using that image’s predicted segmentation as pseudo ground truth.The framework is designed to use the resulting classifier to assess the original segmentation’s quality.
  • II. REVERSE CLASSIFICATION ACCURACY: Atlas Forests encode individual image-segmentation pairs with Random Forests and can operate without image registration in this RCA implementation.The experiments use forests with maximum depth 30 and 50 trees.
  • II. REVERSE CLASSIFICATION ACCURACY: The study also uses a 3D DeepMedic convolutional neural network as a reverse classifier for voxel-wise prediction.Its dual-pathway architecture supports multi-scale image context, while training on a single image constrains its use as an RCA classifier.
  • II. REVERSE CLASSIFICATION ACCURACY: Atlas-based label propagation provides a third RCA implementation, with the target image and its predicted segmentation serving as a single atlas.The approach uses non-rigid registration before propagating labels.

B. Predicting Segmentation Accuracy

RCA applies a classifier trained from a new image’s predicted segmentation to reference images with ground truth, then uses the best reference-image score as a proxy for the unknown accuracy. The approach relies on suitable reference databases and works best for some metrics and settings.

  • B. Predicting Segmentation Accuracy: RCA applies a classifier trained on a predicted segmentation to reference images, compares the resulting segmentations with their ground truth, and uses those scores to estimate the original accuracy.The framework assumes scores on reference images correlate with the unavailable score for the new image.
  • B. Predicting Segmentation Accuracy: RCA requires a reference database containing multiple segmented images whose ground-truth labels capture expected variability.An existing training or cross-validation database may be reused for this purpose.
  • B. Predicting Segmentation Accuracy: The proxy score is the maximum evaluation metric obtained across the reference images, because the reverse classifier may work well only on images similar to the target.For metrics where lower values indicate better quality, the maximum is replaced by a minimum.
  • B. Predicting Segmentation Accuracy: The method assumes that a reverse classifier trained on a good segmentation performs well on at least some reference images, whereas a poor segmentation produces poor performance across the references.The latter expectation is explicitly stated for bad-quality predictions.

C. Summary

RCA estimates the quality of a predicted segmentation without ground truth by training an image-specific reverse classifier, evaluating it on reference cases, and computing a proxy score.

  • RCA trains a classifier on an image and its predicted segmentation, then evaluates that classifier on reference images with available ground truth.
  • The framework computes a segmentation-quality proxy from the reverse classifier's outputs and can threshold it to flag poor cases or identify high-quality cases.

III. EXPERIMENTAL VALIDATION

The framework is validated on whole-body MRI using a multi-organ segmentation task and three different segmentation methods.

  • The validation evaluates RCA on whole-body MRI across Random Forests, 3D CNNs, and probabilistic multi-atlas label propagation.

A. Experimental Setting

The experimental setting evaluates DSC prediction across 525 structures produced by three segmentation methods and assessed with three RCA classifier variants.

  • The study uses 3-fold cross-validation to segment 525 structures across 15 organs and 35 subjects with Random Forests, CNNs, and Multi-Atlas.
  • RCA prediction accuracy is assessed using DSC correlation, mean absolute error, and classification accuracy.
  • The study also groups segmentations into bad, medium, and good quality using DSC ranges [0.0, 0.6), [0.6, 0.8), and [0.8, 1.0], respectively.

C. Results for Predicting Dice’s Similarity Coefficients

RCA predicts individual segmentation quality most accurately with Single-Atlas, which achieves strong correlations, low DSC errors, and good three-category classification accuracy.

  • Single-Atlas achieves correlations above 0.95, MAEs between 0.05 and 0.07, and classification accuracies between 81% and 89% across the three segmentation methods.
  • Atlas Forests retain correlation above 0.82 but show larger errors, with MAEs between 0.12 and 0.17 and classification accuracies from 0.62% to 0.78%.
  • Figure 3 illustrates liver-quality prediction using a Random Forest segmentation and 24 reference-database segmentations with their DSC variation.
  • The examples support using the maximum score across the reference database as a proxy for segmentation quality.
  • Zero-Dice structures mainly affect CNN-based results, while their effect on Atlas Forests and Single-Atlas accuracies is very small.

D. Detecting Segmentation Failure

RCA can identify failed segmentations by predicting low real DSC scores, particularly with Atlas Forests and Single-Atlas classifiers.

  • D. Detecting Segmentation Failure: Atlas Forests and Single-Atlas produce high correlations, low MAEs, and very good classification accuracies for detecting segmentation failures.Low real DSC scores are correctly predicted, and failed segmentations are identified in the degraded Random Forest experiment.
  • D. Detecting Segmentation Failure: The failure-detection experiment evaluates three RCA classifiers on substantially degraded Random Forest segmentations caused by limiting test-time tree depth to 8.Results are summarized in Table II.
  • D. Detecting Segmentation Failure: RCA performs reasonably well on major organs such as liver, heart, and lungs, where the standard segmentations contain no failure cases.The degraded experiment is used to demonstrate that RCA can nevertheless predict failures in these structures.
  • D. Detecting Segmentation Failure: Constrained CNNs are less suitable for failure prediction because they suffer from many zero predictions.Atlas Forests and Single-Atlas instead provide accurate predictions in this experiment.
  • D. Detecting Segmentation Failure: The results suggest that Atlas Forests and Single-Atlas can support automatic quality control in large-scale studies.This is especially relevant when failed segmentations must be detected without feasible visual quality control.

E. Results for Predicting Different Segmentation Metrics

RCA predicts overlap-based segmentation metrics and relative volume difference effectively, but the current setting performs poorly for distance-based metrics such as ASD.

  • E. Results for Predicting Different Segmentation Metrics: Predictions for Jaccard index, precision, and recall show good correlation with real values, low MAEs, and high accuracies, although precision is less accurate than recall.Jaccard predictions are expected to resemble DSC because the two metrics are directly related.
  • E. Results for Predicting Different Segmentation Metrics: Figure 2 compares predicted and real DSC across three segmentation methods and three RCA classifiers using scatter-plot rows and columns.Single-atlas label propagation gives high correlation and low prediction errors across all 15 anatomical structures.
  • E. Results for Predicting Different Segmentation Metrics: RCA predicts overlap-based scores well, whereas distance-based metrics cannot be accurately predicted with the current setting.The experiment uses Random Forests for segmentation and Atlas Forests for RCA, with results summarized in Table III.
  • E. Results for Predicting Different Segmentation Metrics: Relative volume difference predictions achieve high classification accuracy of 0.68%, comparable to overlap-based scores.RVD is defined as the absolute volume difference divided by the reference volume, with perfect segmentation yielding zero.
  • E. Results for Predicting Different Segmentation Metrics: Average surface distance predictions are not convincing, with low correlation, high MAE, and low classification accuracy.Distance-based errors are unbounded, requiring thresholding in the evaluation.

IV. DISCUSSION AND CONCLUSION

RCA can predict individual segmentation quality without ground truth, with Single-Atlas label propagation providing the strongest overall predictions. The framework remains subject to classifier choice, calibration assumptions, overfitting risk, and uncertainty about whether overlap metrics fully capture quality.

  • RCA can predict segmentation quality without ground truth, but the most suitable RCA classifier may depend on the application.Atlas Forests and Single-Atlas label propagation performed accurately across different segmentation methods, whereas constrained CNNs worked well mainly for major organs.
  • The framework assumes predicted and real scores have a near-identity linear relationship, while calibration can improve Atlas Forest and constrained-CNN results but risks overfitting.Calibration may fail to generalize because it learns the relationship on available training data.
  • The strongest predictions were obtained for overlap measures such as DSC and Jaccard Index, but whether these metrics fully capture segmentation quality remains debatable.The discussion specifically identifies this as a limitation of the evaluated quality measures.
  • RCA can identify failed segmentations and support automatic selection of high-quality cases in clinical pipelines and large-scale imaging studies.The framework is presented as useful when manual visual quality control is infeasible, including large datasets such as UK Biobank imaging data.
Loading 1702.03407v1…