PaperlayerView exact revision on arXiv

Source-linked AI summary

Finally Outshining the Random Baseline: A Simple and Effective Solution for Active Learning in 3D Biomedical Imaging

Carsten T. Lüth, Jeremias Traub, Kim-Celine Kahl, Till J. Bungert, Lukas Klein, Lars Krämer, Paul F. Jäger, Klaus Maier-Hein, Fabian Isensee

arXiv:2601.13677v1cs.CV

TL;DR

3D biomedical segmentation requires costly expert annotation, yet active learning has not consistently outperformed improved random baselines. The paper introduces ClaSP PE, combining class-stratified querying with scheduled power-noising, and reports stronger segmentation quality without sacrificing annotation efficiency across benchmark and rollout evaluations.

  • Problem

    Active learning has not consistently outperformed improved random baselines for 3D biomedical image segmentation, despite the high cost of volumetric annotation.

  • Method

    ClaSP PE combines class-stratified uncertainty sampling with exponentially scheduled log-scale power-noising to improve class representation and query diversity.

  • Results

    ClaSP PE is reported as the only AL method improving over improved random strategies in average Final Dice and AUBC while retaining annotation efficiency.

  • Takeaways & Limitations

    Rollout evaluation on four additional datasets used deployment guidelines without manual adaptations, supporting ClaSP PE’s practical generalization beyond the benchmark.

  • Takeaways & Limitations

    The evaluation may overfit benchmark-specific design decisions and does not establish that results extend to boundary-based metrics or tasks focused on specific classes.

Abstract

from arXiv · show

Active learning (AL) has the potential to drastically reduce annotation costs in 3D biomedical image segmentation, where expert labeling of volumetric data is both time-consuming and expensive. Yet, existing AL methods are unable to consistently outperform improved random sampling baselines adapted to 3D data, leaving the field without a reliable solution. We introduce Class-stratified Scheduled Power Predictive Entropy (ClaSP PE), a simple and effective query strategy that addresses two key limitations of standard uncertainty-based AL methods: class imbalance and redundancy in early selections. ClaSP PE combines class-stratified querying to ensure coverage of underrepresented structures and log-scale power noising with a decaying schedule to enforce query diversity in early-stage AL and encourage exploitation later. In our evaluation on 24 experimental settings using four 3D biomedical datasets within the comprehensive nnActive benchmark, ClaSP PE is the only method that generally outperforms improved random baselines in terms of both segmentation quality with statistically significant gains, whilst remaining annotation efficient. Furthermore, we explicitly simulate the real-world application by testing our method on four previously unseen datasets without manual adaptation, where all experiment parameters are set according to predefined guidelines. The results confirm that ClaSP PE robustly generalizes to novel tasks without requiring dataset-specific tuning. Within the nnActive framework, we present compelling evidence that an AL method can consistently outperform random baselines adapted to 3D segmentation, in terms of both performance and annotation efficiency in a realistic, close-to-production scenario. Our open-source implementation and clear deployment guidelines make it readily applicable in practice. Code is at https://github.com/MIC-DKFZ/nnActive.

1 Introduction

3D biomedical segmentation makes annotation costly, while active learning has not consistently surpassed improved random baselines. ClaSP PE addresses this gap with class-stratified uncertainty querying and scheduled score perturbation, supported by broad benchmark and rollout evaluations.

  • Specialized expertise and detailed volumetric masks make 3D biomedical annotation time-consuming and expensive.
  • Active learning aims to reduce annotation costs by querying informative data points, but its effectiveness for 3D biomedical segmentation remains largely unproven.
  • Existing active learning methods do not consistently outperform random sampling, including improved random strategies adapted to 3D biomedical imaging.
  • ClaSP PE combines class-specific uncertainty stratification with exponentially scheduled power-noising to address class imbalance and low query diversity.
  • ClaSP PE is reported to outperform standard and improved random baselines in segmentation quality without sacrificing annotation efficiency.
  • The evaluation covers 24 benchmark settings across four datasets and four additional rollout datasets configured without manual adaptations.

2 Challenges of Active Learning for 3D Biomedical Image Segmentation

3D biomedical active learning must account for large, imbalanced volumes and the practical design of annotation queries. The nnActive framework uses partial annotations and 3D patches, but overlapping patch candidates create computational and representational challenges.

  • Segmentation on 3D biomedical images: 3D biomedical volumes can exceed 500 × 500 × 500 voxels and often contain dominant background regions, homogeneous structures, and strong volumetric differences.
  • Evaluation of Active Learning Methods: The nnActive evaluation addresses restricted settings, failure to train with partial annotations, unsuitable random baselines, and voxel-based annotation-cost measurement.
  • Query Design: 3D query patches with partial loss balance annotation efficiency and informativeness while allowing query sizes to reflect structures of interest.
  • Query Design: Overlapping 3D patch candidates increase algorithmic complexity and hinder representation-based and sophisticated uncertainty-based query methods under runtime and memory constraints.

3 Method

ClaSP PE modifies predictive-entropy top-k querying in two ways: class-stratified selection improves class coverage, while scheduled log-scale power-noising promotes early diversity and later exploitation.

  • ClaSP PE balances informativeness, class representation, and query diversity by modifying standard uncertainty-based top-k sampling.
  • Class Stratified Sampling: Class-stratified sampling selects equal numbers of patches per predicted class, with 66% of the budget assigned to this subset.
  • Class Stratified Sampling: Predicted class probabilities rank patches within each class, allowing underrepresented classes to contribute equally without additional label information.
  • An Exponential Scheduler for Score Perturbation via Log-scale Power Noising: Log-scale power-noising adds Gumbel perturbations to patch scores before top-k selection, increasing query diversity during early active-learning cycles.
  • An Exponential Scheduler for Score Perturbation via Log-scale Power Noising: An exponential schedule decreases perturbation strength over cycles, shifting selection from exploration toward exploitation.
  • Predictive Entropy supplies the uncertainty scores, while the implementation fixes β0 = 1 and βmax = 100 without additional tuning.

4 Experimental Results on the nnActive Benchmark

Across 24 nnActive settings, ClaSP PE generally provides the strongest segmentation performance while retaining high annotation efficiency. Its advantages are clearest in aggregate comparisons, though performance varies by dataset, label regime, and training duration.

  • Benchmark and setup: Over 1,000 training runs spanning 24 settings, ClaSP PE was evaluated across four datasets, three Label Regimes, and two query patch sizes.The benchmark included dedicated ablations and used four seeds with a fixed test split, 200 training epochs, and five AL cycles.
  • Aggregate performance: ClaSP PE achieved the best overall AUBC and Final Dice rankings while remaining among the most annotation-efficient methods by FG-Eff.It did not always achieve the top FG-Eff, reflecting a trade-off between segmentation quality and annotation efficiency.
  • Statistical comparisons: ClaSP PE formed a distinct top-performing Nemenyi group for AUBC and Final Dice and consistently separated from random and uncertainty-based baselines across all three metrics.The statistical groups used the conservative Nemenyi post-hoc test at p = 0.05.
  • Aggregate performance: ClaSP PE was the only AL method improving over improved random strategies on average Final Dice and AUBC across all settings.The comparison included standard random, Random 33% and 66% FG, and five uncertainty-based query methods.
  • Dataset variation: Absolute performance differences were generally small for ACDC and Hippocampus and often fell within their error bars.The authors therefore emphasize broad evaluation across datasets and label regimes to reveal overall trends.
  • Loss scenarios: In roughly 20% of pairwise comparisons, Random 66% FG surpassed ClaSP PE, concentrated almost exclusively in AMOS Low-Label settings.These settings are challenging because AMOS has many classes and a constrained annotation budget.
  • Training duration and AMOS: Increasing training from 200 to 500 epochs improved ClaSP PE’s win-to-lose ratio against random baselines, with High-Label AMOS consistently favoring ClaSP PE over Random 66% FG.Low-Label AMOS remained dominated by losses, while Medium- and High-Label regimes showed clearer gains against Random 33% FG.
  • Component ablations: Class-stratified querying improved standard Predictive Entropy across all metrics, while gradually decayed power noising produced the best overall trade-off.The selected configuration combines 66% stratified querying with gradually decayed power noising; power noising improved FG-Eff but reduced AUBC and Final Dice when applied without the schedule.

5 Simulating Real-World Active Learning in a Roll-Out Study

The roll-out study evaluates ClaSP PE on four previously unseen biomedical datasets without dataset-specific fine-tuning. ClaSP PE generally matches or exceeds the baselines across metrics and shows the strongest aggregated performance against random strategies.

  • Experimental setup: Four previously unseen datasets were evaluated in a plug-and-play roll-out without dataset-specific fine-tuning.The study followed predefined deployment decisions and introduced systematic query-patch sizing and normalized per-class query budgets.
  • Compared methods: The comparison included ClaSP PE, Predictive Entropy, uniform random sampling, and foreground-aware Random 66% FG.Random 66% FG is described as a stronger baseline incorporating foreground-aware sampling.
  • Results: ClaSP PE performed on par with or better than all baselines across roll-out datasets and metrics while maintaining or exceeding annotation efficiency.The reported metrics were AUBC, Final Dice, and FG-Eff.
  • Results: On the MAMA MIA dataset, ClaSP PE performed substantially better on a complex task with many redundant structures.The paper contrasts this with cases where random sampling showed high FG-Eff without corresponding segmentation gains.
  • Results: ClaSP PE had the best aggregated roll-out performance, winning in over 50% of cases and never being significantly outperformed by Random or Random 66% FG.It significantly outperformed Predictive Entropy in 25% of cases and was significantly outperformed in 5%.

6 Limitations

The paper identifies limitations involving benchmark-specific overfitting, metric and model dependence, economic trade-offs, and the task-dependent economics of annotation techniques.

  • Scope and evaluation: The evaluation may overfit to benchmark-specific design decisions, although the dual benchmark and roll-out evaluation mitigates rather than eliminates this concern.The authors also caution that results based on average Dice may not extend to boundary-based metrics or settings focused on specific classes.
  • Method dependence: ClaSP PE depends on the underlying model’s predictive capacity, making stratified querying less effective when initial segmentation quality is insufficient.The authors note that deployment guidelines and pretrained models may mitigate this risk.
  • Economic trade-offs: Active learning involves an economic trade-off between reduced annotation cost and additional computational overhead.The paper frames annotation savings and computation as competing components of practical deployment cost.
  • Annotation design: No consensus exists on the most economical annotation technique because the advantages and drawbacks of sparse annotations, superpixels or voxels, and scribbles depend on the task and available tooling.The evaluation uses 3D patches, which can support several annotation processes and targeted structure-level annotation.

7 Conclusion

ClaSP PE is presented as a lightweight, practical active-learning query method with open-source implementation and deployment guidance. It combines class-stratified uncertainty sampling with scheduled score perturbation to improve coverage and query diversity.

  • 7 Conclusion: ClaSP PE is described as a practical query method with computational cost comparable to standard top-k selection.The authors position it for integration into existing active-learning frameworks and real-world annotation workflows.
  • 7 Conclusion: The method is available as open-source code and is accompanied by guidelines intended to reduce experimental overhead and support reproducible comparisons.The implementation is also described as suitable for integration into real-world annotation workflows.
  • B ClaSP PE Algorithm: ClaSP PE combines class-stratified scores with standard uncertainty scores so that only a fraction of queries is stratified while the remainder remains sensitive to highly uncertain examples.Class-specific scores are formed by weighting uncertainty with predicted class probabilities, then queries are selected separately by class and by standard uncertainty.
  • B ClaSP PE Algorithm: Scheduled power noising perturbs query scores more strongly early in active learning and gradually reduces the perturbation toward later cycles.The schedule uses an initial and final value over the active-learning cycles.
  • B ClaSP PE Algorithm: The implementation provides both abstract pseudocode and a detailed algorithm covering uncertainty computation, patch aggregation, overlap checks, and stratified query construction.The detailed algorithm initializes class-specific query sets, ranks candidate patches, and enforces overlap constraints.

C Dataset Details

The study documents dataset configurations for the nnActive benchmark and roll-out study, along with preprocessing and implementation details for patch-based active learning. Standard uncertainty methods are applied after ensemble predictions and patch-level aggregation.

  • C Dataset Details: Dataset characteristics for the nnActive benchmark and roll-out study are provided separately, with all images resampled to median dataset spacing.Further segmentation-task details are referenced in the dataset tables.
  • C Dataset Details: The MAMA MIA dataset uses a subtraction image formed by subtracting the precontrast image from the first available post-contrast image.
  • C Dataset Details: The implementation stores queried patch metadata in nnU-Net dataset structures to preserve compatibility with the nnU-Net training and data-management pipeline.
  • C Dataset Details: Standard top-k uncertainty methods compute uncertainty maps after ensemble members predict each image, aggregate scores to patches, and select non-overlapping patches.The selection procedure ranks patch uncertainties in descending order while checking overlap with already selected or labeled regions.

D.1 Evaluation Metrics

The evaluation uses complementary metrics for final segmentation quality, performance across the active-learning trajectory, pairwise statistical comparisons, and annotation efficiency. Experimental settings include standardized training, ensemble, budget, and hyperparameter choices.

  • D.1 Evaluation Metrics: Final Dice measures segmentation performance after the full annotation budget and emphasizes later active-learning stages.
  • D.1 Evaluation Metrics: AUBC measures performance across the active-learning trajectory as the normalized area under the Mean Dice curve.Higher AUBC values indicate better performance.
  • D.1 Evaluation Metrics: PPM quantifies how often methods significantly outperform one another across datasets and label regimes using pairwise two-sided t-tests at α = 0.05.Rows represent fractions of wins and columns represent fractions of losses, expressed as percentages.
  • D.1 Evaluation Metrics: FG-Eff measures how quickly a method reaches full-data performance as a function of annotated foreground voxels.Higher γ indicates faster convergence with less annotation, but the metric can be misleading when foreground querying is extremely small.
  • D.1 Evaluation Metrics: FG-Eff is directly interpretable only when methods share the same model, training regime, and annotation budgets.
  • D.2 Experiment Details: The experiments use nnU-Net with five-model ensembles, complete retraining at each active-learning loop, and standardized dataset-related parameters.Unless otherwise stated, models are trained for 200 epochs.
  • D.2 Experiment Details: ClaSP PE compares stratified fractions of 33% and 66%, with β0 = 1 and βmax = 100 selected without further method-parameter tuning.

E.1 Results aggregated over Main Benchmark and Patch× 1 2 Setting

Across 24 configurations spanning four datasets, three label regimes, and two patch sizes, ClaSP PE generally ranks strongest against competing query methods and random baselines. Statistical analyses indicate that its observed separations are exploratory and conservative tests may understate practical differences.

  • E.1 Evaluation Results: 24 configurations cover four datasets, three label regimes, and two query patch sizes in the aggregated analysis.The analysis uses nine methods and paired performance outcomes across configurations.
  • E.1 Statistical Analysis: The Friedman tests across nine methods and 24 configurations are significant at p = 0.05, followed by Nemenyi post-hoc comparisons of average ranks.
  • E.1 Statistical Analysis: Nemenyi significance groups are exploratory rather than definitive because the test is conservative with many methods and a relatively small sample size.The authors state that observed significant separations likely understate rather than overstate the true differences.
  • E.1 Evaluation Results: Fine-grained tables report AUBC, Final Dice, and FG-Eff with standard deviations, while PPM figures summarize overall and dataset-specific comparisons.AUBC and Final Dice are multiplied by 100 for readability and are directly comparable within each label regime and dataset.
  • E.4 AMOS Ablation: On AMOS, longer training increases ClaSP PE’s Final Dice advantage over Random 66% FG but also increases queried foreground.For adrenal-gland classes with lower-label disadvantages, additional queries reduce or eliminate the performance difference by 500 epochs.
  • E.6 PPM Comparisons: ClaSP PE is the best-performing method in the mean-row trend for p-values 0.05 and 0.02, followed by several uncertainty methods and improved random baselines.The unadapted Random strategy is reported as least performant in this comparison.
  • E.6 PPM Comparisons: After Holm-Bonferroni correction, ClaSP PE remains best-performing, while Predictive Entropy is the only other active-learning method reported to outperform random baselines.

G Roll-Out Results

The roll-out study reports fine-grained results in Table 9, with performance metrics presented separately by label regime and dataset. AUBC and Final Dice are scaled by 100 for readability, while direct comparisons are restricted within each label-regime–dataset pair.

  • The corresponding roll-out-study details are provided in Table 9, with the PPM shown in Figure 6.
  • Table 9 reports fine-grained roll-out results using AUBC, Final Dice, and FG-Eff across label regimes and datasets.The table states that higher values are better and provides dataset characteristics separately in Table 4.
  • AUBC and Final Dice are reported with a factor of ×100 for improved readability.
  • AUBC, Final, and FG-Eff can only be directly compared within the same label regime and dataset.

H Limitations

The reported limitations concern benchmark and model dependence, economic trade-offs, practical comparisons with complex baselines, hyperparameter choices, and metric coverage. These constraints bound how broadly the empirical findings and deployment conclusions should be interpreted.

  • ClaSP PE’s development on nnActive creates a benchmark-overfitting risk, although roll-out performance against strong alternatives provides evidence of generalization.
  • The empirical findings are specific to nnActive, nnU-Net, and 3D-patch queries, so results may differ with other architectures, learning approaches, or query designs.
  • ClaSP PE depends on sufficiently accurate multi-class predictions because those predictions underpin stratified query selection.Limited initial labels in the AMOS low-label regime reduced segmentation quality and the effectiveness of stratification.
  • AL introduces an economic trade-off between computational cost and expected annotation-effort reduction, so adoption depends on annotation costs and available resources.
  • More sophisticated AL baselines could potentially yield similar gains, but adapting them to practical 3D biomedical segmentation remains unresolved.
  • The method’s stratification ratio and power-noising schedule require careful hyperparameter balancing despite validation across benchmark and held-out roll-out datasets.
  • The evaluation uses mean Dice and does not establish generalization to metrics that weight classes differently.

I Qualitative Results

Qualitative analyses compare patches selected by Predictive Entropy and ClaSP PE after the first AL loop across four benchmark datasets. They illustrate greater diversity, more foreground coverage, and class-wise localization of uncertainty under ClaSP PE.

  • ACDC: After the first AL loop, ClaSP PE produced more diverse queries and queried more foreground than Predictive Entropy on ACDC.
  • Hippocampus: On Hippocampus, ClaSP PE mitigated over-focus on the prominent posterior hippocampus class compared with Predictive Entropy.
  • KiTS: On KiTS, ClaSP PE mitigated over-focus on the tumor class compared with Predictive Entropy.
  • AMOS: On AMOS, the qualitative comparison examined whether stratification reduced over-focus on prominent classes during early querying.
  • Stratification mechanism: Class-wise stratification shifts high-uncertainty regions toward areas where predictions indicate the corresponding classes are present.
  • Stratification mechanism: Figure 21 visualizes class probabilities and weighted entropy maps, with colormaps rescaled per image and high values clipped at the 98% quantile.
Loading 2601.13677v1…