Source-linked AI summary
MedSegBenchmarker: A Raw-Count-First Framework for Controlled 2D Medical Image Segmentation Benchmarks
Vanessa Borst, Lukas Horn, Daniel Grillmeyer, Thomas Prantl, Samuel Kounev
TL;DR
Reliable MIS benchmarking is difficult because reported comparisons depend on heterogeneous data and methodological choices whose effects are often not retained for inspection. MEDSEGBENCHMARKER addresses this with a configuration-driven 2D framework that preserves raw sample–class evidence and evaluation context. Its case study shows that aggregation and resolution choices can change conclusions, motivating explicit, reproducible benchmark conditions.
Problem
MIS comparisons lack consistently controlled and reproducible evaluation across datasets, protocols, aggregation choices, and computational conditions.
Method
MEDSEGBENCHMARKER integrates controlled configurations, data diagnostics and splitting, resumable training, search, cross-validation, checkpoint evaluation, and raw sample–class count export.
Results
Aggregation changed the leading architecture in several dataset–resolution cells, while higher resolution produced dataset- and architecture-dependent gains and losses.
Takeaways & Limitations
Benchmark conclusions should treat aggregation and resolution as explicit evaluation choices and retain elementary evidence for recomputation and qualification.
Takeaways & Limitations
The toolbox is scoped to supervised 2D semantic segmentation and does not replace complete dataset-version or containerized environment manifests.
Abstract
from arXiv · showhide
Despite rapid advances in MIS, fair and reproducible comparisons of segmentation models remain challenging due to heterogeneous datasets, inconsistent evaluation protocols, and rapidly evolving architectures. In particular, comparisons often implicitly assume that model rankings are invariant to data partitioning, preprocessing, metric aggregation, uncertainty estimation, and computational constraints. The lack of extensible and unified evaluation frameworks further limits systematic investigation of new models, datasets, and training paradigms. We present MEDSEGBENCHMARKER (MSB), a configuration-driven framework for controlled benchmarking of 2D MIS. It integrates duplicate and near-duplicate image detection, group-aware data splitting, YAML study specifications, resumable training, hyperparameter optimization, cross-validation, and checkpoint-based evaluation. Rather than retaining only aggregate performance measures, MSB exports sample- and class-level pixel counts and predictions together with the evaluation context. These elementary artifacts enable post-hoc analyses without repeated inference. We demonstrate MSB in a case study involving three heterogeneous 2D datasets and multiple MIS and general-purpose vision models evaluated at 256- and 512-pixel input resolutions. Reaggregation of identical predictions changes the top-ranked architecture in three of six dataset-resolution settings, despite high rank correlations between aggregation strategies. Increasing input resolution produces model- and dataset-dependent performance gains and losses that must be considered alongside empirically measured inference complexity. These results show that seemingly minor choices in evaluation and experimental setup can affect benchmark conclusions. MSB, available at GitHub, provides a practical and extensible basis for making benchmark conditions and evaluation choices explicit and reproducible.
1. Introduction
MEDSEGBENCHMARKER addresses the difficulty of making 2D MIS comparisons controlled and inspectable by recording experimental conditions and retaining raw evaluation evidence. Its case-study design supports post-hoc examination of how aggregation, resolution, and other choices affect benchmark conclusions.
- Reported segmentation scores depend on dataset curation, patient separation, resolution, augmentation, optimization, checkpoint selection, and metric aggregation.
- Rare or heterogeneous targets make aggregation consequential because global pixel pooling, image-level averaging, and class-macro scores answer different questions.
- Reproducibility failures and methodological pitfalls, including data leakage, can produce substantially overoptimistic performance claims and risk unreliable high-stakes deployments.
- MEDSEGBENCHMARKER provides configuration-driven controlled 2D MIS benchmarking with duplicate diagnostics, group-aware splitting, YAML specifications, resumable execution, search, cross-validation, and checkpoint evaluation.
- The framework exports sample–class confusion counts with evaluation context, enabling aggregation, uncertainty estimation, figures, and tables without repeated inference.
2. Related Work
Existing medical-imaging frameworks, benchmarks, datasets, and general segmentation libraries provide complementary components but do not primarily target controlled, extensible comparisons across heterogeneous MIS architectures. MSB fills this benchmark-control role with explicit execution, raw-count export, diagnostics, and efficiency profiling.
- Existing resources span medical-imaging components, curated benchmark tasks, datasets, and general-purpose segmentation toolkits.
- MONAI is the closest comparator in scope, but it is a component framework rather than a benchmark-control layer.
- MSB provides declared grid or Hyperband execution, checkpoint-backed raw-count export, duplicate diagnostics, and separate efficiency profiling.
- Benchmark datasets and studies define valuable tasks, data, and reference settings, but their primary focus is not general-purpose benchmarking capabilities.
- MSB supplies study-level control for declared 2D MIS comparisons while retaining architectures through adapters and general-purpose framework components.
3. MedSegBenchmarker Toolbox
The toolbox organizes supervised 2D MIS studies around explicit configurations, persisted execution state, checkpoint-backed evaluation, and reusable raw-count artifacts. Its benchmark contract records the methodological choices that define a comparison while preserving modularity and traceability.
- Study-specific fragmentation can obscure how splits, checkpoints, thresholds, and aggregation rules produce reported results.
- Benchmark Contract: The benchmark contract records dataset, split, preprocessing, model, optimization, inference, evaluation, aggregation, and uncertainty choices.
- MEDSEGBENCHMARKER connects dataset preparation, YAML configuration, training or search, cross-validation, checkpoint evaluation, and post-hoc reporting.
- Study Specification and Configuration: Configurations declare architectures, transforms, optimizers, losses, checkpoint policies, and evaluation settings while allowing legitimate model accommodations alongside common controls.
- The toolbox emphasizes modularity, transparency, reproducibility, extensibility, and traceability through interfaces, YAML, seeds, deterministic options, hashes, and checkpoint references.
TP FP FN
MEDSEGBENCHMARKER supports reproducible, checkpoint-backed 2D MIS studies with configurable execution and preserved raw evaluation artifacts. Its raw-count-first outputs retain per-image, per-class evidence for flexible post-hoc metric analysis without repeated inference.
- Study Configuration: Layered YAML specifications compose global, dataset-specific, and architecture-specific settings for controlled comparisons.The framework also supports duplicate and near-duplicate detection and dataset specifications containing images, masks, classes, and patient IDs.
- Study Execution: The execution layer supports single runs, grid search, Hyperband, and K-fold cross-validation with persisted fold indices and resumable checkpoints.Checkpoint state includes model, optimizer, scheduler, gradient-scaler, early-stopping, metric-tracking, and random-number-generator states.
- Checkpoint-backed Raw-Count Evaluation: Raw-count evaluation records TP, FP, FN, and TN for each image–class pair, alongside prediction and evaluation metadata.Records also include intersection, reference-mask, predicted-mask, and union counts, plus dataset, split, fold, run, model, checkpoint, image, and class identifiers.
- Checkpoint-backed Raw-Count Evaluation: DSC and IoU can be recomputed by pooling retained counts under different image-, class-, foreground-, or fold-wise summaries.The stored counts support DSC = 2TP/(2TP + FP + FN) and IoU = TP/(TP + FP + FN).
- Post-hoc Analysis: Post-hoc analyses can validate, consolidate, and reaggregate retained artifacts without reloading checkpoints or repeating inference.Checkpoint access is reserved for analyses requiring model forward passes, such as Grad-CAM, complexity profiling, and external-dataset evaluation.
4. Case Study
The case study evaluates controlled 2D MIS comparisons across three heterogeneous datasets, multiple architectures, two input resolutions, and several post-hoc analyses. Results show that aggregation, resolution, and computational efficiency can materially alter how model comparisons are interpreted.
- Case Study Setup: Three heterogeneous datasets—ISIC’18, NeoPolyp, and CAMUS—were evaluated after duplicate assessment, with dataset sizes of 3,565, 945, and 1,996 images.The datasets cover dermoscopic lesions, endoscopic polyps, and echocardiographic structures, respectively.
- Case Study Setup: The protocol compared 256 × 256 and 512 × 512 inputs using hyperparameter selection followed by five-fold cross-validation.CENet* exceeded available GPU memory at the larger resolution, leaving matched artifacts for ten architectures at both resolutions.
- Aggregation Sensitivity: Reaggregation changed the leading architecture in several dataset–resolution cells despite high rank correlations of ρ = 0.93–1.00.Global-micro and image-macro Dice selected different winners for NeoPolyp at 256 × 256 and ISIC’18 at both resolutions.
- Input-Resolution Sensitivity: +3.46 pp was the largest reported resolution gain for SU-Mamba on NeoPolyp, while U-Net lost −2.08 pp there.CAMUS showed mostly small gains, whereas ISIC’18 included losses for InternImage and U-Net.
- Inference Efficiency: At 256 × 256, models spanned 9.36–59.89 GMACs and 4.09–74.92 ms per image; at 512 × 512, they spanned 37.43–238.43 GMACs and 13.17–57.46 ms.These measurements expose deployment trade-offs beyond parameter counts alone.
- External Transfer: External Kvasir-SEG transfer produced strict-majority ensemble DSC ranges of 0.689–0.862 at 256 × 256 and 0.687–0.861 at larger input size.Resolution effects differed by architecture, improving some models while reducing others.
5. Discussion
Benchmark conclusions depend on aggregation rules, input resolution, and measured computational conditions rather than isolated scores. MSB preserves evaluation evidence and protocol context, but remains scoped and requires ongoing study-specific maintenance.
- Evaluation claims: Aggregation and resolution choices define the evaluation question, with reaggregation changing leading architectures and higher resolution producing non-uniform accuracy and computational effects.Inference records also show that parameter count alone does not characterize efficiency; latency and memory depend on configuration.
- Scope and limitations: The toolbox is scoped to supervised 2D semantic segmentation rather than 3D, instance, detection, prompt-based, or deployment workflows.New architectures and datasets still require adapters or conversion to the index.csv schema.
- Traceability and reproducibility: Configuration hashes and checkpoints improve traceability but do not replace complete dataset-version or containerized environment manifests.Duplicate diagnostics support review but do not automatically prevent leakage or substitute for patient-, study-, or site-level metadata.
- Extensibility: Post-hoc analyses currently require study-specific maintenance when models, datasets, or resolutions change, despite the underlying modular structure.The framework is not yet fully plug-and-play, and deterministic execution remains best effort across hardware and software environments.
6. Conclusion
MEDSEGBENCHMARKER is a configuration-driven framework for controlled 2D medical image segmentation benchmarks. Its case study shows that rankings and accuracy–efficiency trade-offs depend on evaluation aggregation, input resolution, dataset, and architecture, while retained evidence supports later inspection and recomputation.
- Framework: MEDSEGBENCHMARKER links duplicate-aware preparation, layered experiment specifications, resumable execution, checkpoint-backed evaluation, and raw-count-first post-hoc analysis.The framework is implemented as a configuration-driven PyTorch framework for controlled 2D MIS benchmarks.
- Case study: Architecture rankings can depend on the aggregation rule, while increasing input resolution produces dataset- and architecture-specific accuracy–efficiency trade-offs.These findings are stated as the central case-study conclusions.
- Implication: MSB preserves elementary evidence and declared conditions needed to inspect, recompute, and qualify benchmark claims rather than treating results as standalone scores.This supports post-hoc examination of the resulting evaluation claims.