A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

Yang Song; Pengbo Sun; Shichang Feng; Ye Zhu; Xin Xu; Ziran Wang

Source-linked AI summary

A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

Yang Song, Pengbo Sun, Shichang Feng, Ye Zhu, Xin Xu, Ziran Wang

arXiv:2608.28063v1Aug 28, 2026cs.CV

TL;DR

Motivated complexity in uncertainty-aware multi-organ ultrasound classifiers lacks controlled evidence that each component improves the trained system. This paper audits Full-EDL against simpler alternatives using matched seeds, capacity- and optimisation-aware comparisons, symmetric calibration, retraining, and separate OOD decisions. Full-EDL showed no established macro-F1 gain, while Simple-CE+TS was selected for the evaluated in-distribution objective but could not support an unconditional OOD-safety claim.

Problem
The paper addresses whether added architectural complexity in uncertainty-aware multi-organ ultrasound classification earns retention under controlled system-level comparison.
Method
The audit compares Full-EDL and simpler candidates using matched seeds, frozen image-level partitions, capacity-aware evaluation, calibration, functional counterfactuals, module deletion, retraining, and separate OOD vetoes.
Results
Full-EDL showed no established macro-F1 gain; Simple-CE met the TS-NLL criterion on both datasets, had favourable selective-risk ordering, and its raw calibration advantage did not persist after scaling or replication.
Takeaways & Limitations
Simple-CE+TS was selected for in-distribution classification, calibrated probabilities, and selective prediction, while Full-EDL remained the maximal reference and OOD reliability was treated separately.
Takeaways & Limitations
Claims are limited to frozen image-level partitions without verifiable patient, study, or video identifiers, and the jointly deleted chain cannot identify individual component effects.

Abstract

from arXiv · show

Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) objectives to address heterogeneous anatomy and acquisition. Yet a plausible design rationale does not by itself establish that an added component improves the trained system. We contribute a controlled complexity-audit framework, applied to the deployment decision between the maximal evidential candidate Full-EDL and simpler alternatives. Six candidates were evaluated on the primary dataset and three in an internal replication, using ten matched seeds, frozen image-level partitions, capacity- and optimisation-aware comparisons, symmetric temperature scaling, paired decision rules, and a separate out-of-distribution (OOD) veto. Retaining Full-EDL did not establish a reliable macro-F1 gain on either dataset, while the simplified alternatives remained inconclusive under the non-inferiority margin. Simple cross-entropy with temperature scaling (Simple-CE+TS) met the calibrated negative log-likelihood criterion on both datasets and showed favourable selective-risk ordering. The raw calibration advantage of evidential training disappeared after temperature scaling and did not recur on the second dataset. The gate had negligible observable influence at the audited checkpoints, and deleting the Full-only chain revealed no stable task or calibrated-loss benefit. Simple-CE nevertheless triggered the OOD veto against the fetal probe but not the lung probe, precluding an unconditional OOD-safety claim. We therefore selected Simple-CE+TS for the evaluated in-distribution objective while retaining Full-EDL as the maximal reference. Components should earn retention through functional and retraining-based evidence, and calibration and distribution-shift reliability should be evaluated separately.

1. Introduction

Multi-organ ultrasound demands evaluation beyond mean classification because anatomy, acquisition, image quality, and class composition vary substantially. This study asks whether motivated architectural complexity earns retention under controlled system-level comparison.

Ultrasound supports portable, real-time, radiation-free imaging for point-of-care and resource-constrained applications.
Multi-organ deployment exposes classifiers to variation in anatomy, acquisition source, image quality, and class composition.
Attention, expert routing, fallback paths, reliability gates, and EDL offer distinct motivations for handling heterogeneous patterns and representing uncertainty.
The central question is whether each added complexity layer earns retention under controlled comparison rather than whether another motivated component can be added.
The audit evaluates Full-EDL against simpler structures using counterfactuals, module deletion, capacity-matched retraining, calibration, and paired multi-seed comparisons.

2. Related work

Prior work motivates compact, multi-organ, expert, and evidential ultrasound systems, but component plausibility does not establish engineering value. The paper frames model selection as evidence-based comparison across performance, calibration, selectivity, and shift reliability.

Deep-learning ultrasound systems span fetal, breast, and other anatomical applications, including ensembles, multiscale integration, graph features, representation learning, and transfer across organs.
The study shifts attention from proposing another high-scoring architecture to testing whether added complexity is justified under matched optimisation, capacity, calibration, and stochastic evaluation.
Mixture-of-experts models use input-dependent routing to support local specialisation or conditional computation, including medical systems for heterogeneous domains and tasks.
EDL represents class evidence with Dirichlet parameters, while temperature scaling separately rescales trained logits using held-out calibration data.
Repeated seeds, prespecified margins, ablations, implementation considerations, and operating context are combined to distinguish evidence for retention from inconclusive or non-inferiority claims.

3. Materials and methods

The audit compares shared-backbone candidates across structural complexity and training objectives, using common prediction rules, matched evaluation, and functional and physical structural tests. Full combines attention, evidential probing, routed experts, a fallback branch, and a reliability gate.

All candidates use an ImageNet-pretrained MobileNetV3-Small backbone and produce ten-class logits, probabilities, and predictions from a common softmax-based evaluation.
The audit evaluates task performance, calibrated loss, selective-risk ordering, OOD separation, structural complexity, and computational cost without collapsing them into one score.
Simple uses one residual bottleneck, whereas Expert-only retains routing and experts but deletes NALA, the probe, fallback, and reliability gate.
Dataset 2 evaluates six candidates and Dataset 1 replicates three candidates under ten matched seeds and frozen image-level partitions.
Simple’s width-1280 design approximately matches Full’s capacity, with 2.411 million versus 2.438 million parameters and 127.2 million versus 129.9 million MACs.
The jointly deleted Expert-only chain cannot isolate the individual effects of NALA, the probe, shared fallback, or the reliability gate.
Full combines low-level spatial reliability weighting, an auxiliary evidential probe, sparse expert routing, shared fallback, and a parameter-free sample-level gate.
The gate combines routing confidence, probe uncertainty, and router uncertainty factors, while its components are not assumed statistically independent.

3.5. Learning objectives

The candidates use either class-weighted cross-entropy or class-weighted EDL objectives, with architecture-specific auxiliary terms retained only where applicable. Temperature scaling is fitted after checkpoint selection rather than trained jointly with either objective.

Slim-CE, Simple-CE, and Full-CE use class-weighted cross-entropy with effective-number weighting and β=0.999.
Simple-CE and Slim-CE use only the primary cross-entropy loss, while Full-CE retains architecture-specific auxiliary terms.
Temperature scaling is excluded from training objectives and fitted only after checkpoint selection.
Simple-EDL, Expert-only-EDL, and Full-EDL use a class-weighted evidential objective based on evidence, Dirichlet concentration, and total strength.
EDL uncertainty is defined internally as K/S_n for K=10 classes, while cross-objective evaluation uses common softmax-based conventions.
Expert-only-EDL adds router load balancing, and Full variants additionally retain an auxiliary probe loss with coefficient 0.05.

3.6. Datasets, partitions, and leakage controls

The study used two image-level, multi-organ ultrasound collections with frozen class-stratified partitions and a primary audit plus internal replication. Leakage controls prevented image-file overlap, but unavailable patient and study identifiers limited dependence assessment.

Dataset 1 contained 16,520 images from three organ groups, while Dataset 2 contained 4,403 images from five organ groups.
The OOD pairings used a lung probe for Dataset 1 and a fetal-ultrasound probe containing 1,669 images for Dataset 2.
All four partitions were mutually disjoint at the image-file level, and the restricted boundary analysis used Dataset 2-trained Full-EDL checkpoints on 648 Dataset 1 breast images without fine-tuning or temperature refitting.
Dataset 2 was the primary controlled-audit dataset, and Dataset 1 provided an internal replication under a different organ and source composition.
The fixed image-level partition assigned 65% of images to training, 10% to validation, 5% to calibration, and 20% to frozen replication testing.
Because patient, study, and video identifiers were unavailable, unknown within-patient or within-study dependence could not be excluded.The authors therefore limited claims to frozen image-level partitions and identified patient-aware and source-held-out evaluation as future priorities.

3.7. Training and optimiser verification

Training was standardized across candidates through fixed preprocessing, matched seeds, controlled initialization, and verified module-specific optimisation settings. The protocol retained all prespecified runs rather than selecting seeds post hoc.

Images were resized to 224 × 224 pixels and normalized with ImageNet statistics; training augmentation used horizontal flips and rotations within ±10°.
Models were trained for at most 30 epochs with batch size 16 under ten matched seeds, using the same seed set for every primary candidate and Slim-CE.
All ten prespecified seeds were retained, including low-performing runs, so reported variation reflected the complete matched-seed experiment.
AdamW used weight decay 10^-4, with learning rates of 10^-4 for the pretrained backbone, 3 × 10^-4 for non-router task modules, and 5 × 10^-5 for routers after two frozen epochs.
Every candidate shared the ImageNet-pretrained MobileNetV3-Small backbone and a controlled task-specific initialization scale of γ = 4.0.

3.8. Calibration, metrics, and OOD scores

Calibration was performed symmetrically for CE- and EDL-trained checkpoints using held-out calibration data, while task, calibration, and OOD performance were assessed with separate metrics and score definitions.

One positive temperature T was fitted per model and seed on the dedicated 5% calibration partition by minimizing calibration-set NLL.
Temperature search evaluated 300 logarithmically spaced values over [0.25, 4.0], expanding to [0.05, 10.0] when the optimum reached an endpoint.
Validation and frozen replication-test observations were excluded from temperature fitting, and both raw and temperature-scaled outputs were retained.
Macro-F1 was the primary classification metric, while NLL, 15-bin ECE, and multiclass Brier score assessed probabilistic calibration.
OOD separation used raw 1 − MSP from unscaled softmax probabilities, with OOD observations treated as the positive class and higher AUROC indicating better separation.
All candidates used the same softmax-based OOD score definitions, and OOD observations were excluded from training, checkpoint selection, and temperature fitting.

3.9. Statistical analysis and decision rules

The audit used paired ten-seed comparisons against Full-EDL with prespecified margins for macro-F1, calibrated NLL, and OOD safety. Formal labels were kept separate, and bootstrap sensitivity analysis did not replace matched-seed inference.

Candidate-minus-Full-EDL differences were paired across ten matched seeds and analyzed separately within each dataset using two-sided 90% paired t intervals.
The macro-F1 non-inferiority margin was −0.01, with PASS requiring the lower interval bound to exceed −0.01 and INFERIOR requiring the upper bound below it.
A macro-F1 PASS indicated criterion attainment under the matched-seed design, not equality, equivalence, or superiority.
The TS-NLL margin was +0.02, with PASS requiring the upper interval bound below +0.02 and FAIL requiring the lower bound above it.
The OOD veto triggered when the lower paired-interval bound was ≤ −0.03 relative to Full-EDL; NOT_VETOED did not establish OOD safety or equivalence.
Image-level stratified bootstrap sensitivity analysis neither replaced matched-seed analysis nor determined formal labels, and it could not capture unknown within-patient or within-study dependence.

4. Experiments and results

Across two datasets, Full-EDL did not establish a reliable macro-F1 advantage, while Simple-CE consistently showed stronger calibrated-loss and selective-risk results. However, OOD safety, raw calibration, and gate effects were dataset- or checkpoint-dependent, and retraining found no stable benefit from the Full-only chain.

Dataset 2 primary audit: All five Dataset 2 candidate-minus-Full-EDL macro-F1 comparisons were INCONCLUSIVE, with no established gain or alternative non-inferiority under the −0.01 margin.Full-EDL’s mean macro-F1 was 0.790 ± 0.017, while candidate means ranged from 0.778 to 0.790.
Dataset 2 primary audit: Simple-CE achieved a TS-NLL of 0.376 ± 0.019 versus 0.427 ± 0.024 for Full-EDL, receiving a PASS decision.All three CE candidates passed the calibrated-loss criterion, whereas EDL candidates were inconclusive or only narrowly passed.
Selective prediction: Simple-CE had lower mean raw-MSP AURC than Full-EDL on both datasets, indicating more favourable selective-risk ordering.On Dataset 2, AURC was 0.051 ± 0.007 for Simple-CE versus 0.083 ± 0.011 for Full-EDL; Dataset 1 showed the same ordering.
OOD evaluation: Simple-CE triggered the Dataset 2 fetal-probe OOD veto but was NOT_VETOED on the Dataset 1 lung probe, so OOD safety did not transfer across probes.The Dataset 2 paired difference was −0.0234 [−0.0429, −0.0039], while Dataset 1’s was +0.0126 [−0.0295, +0.0547].
Functional gate audit: Changing inference batch size caused no predicted-class flips and altered maximum softmax probability by at most approximately 0.0034 at the audited checkpoints.The batch-relative F3 audit therefore found little observable sensitivity, but behaviour beyond the tested models, datasets, and batch compositions remained open.
Functional gate audit: Shared-only and expert-only inference disagreed on only 1.0% of images, while observed gating improved none of the recorded means over forced expert-only inference.Expert-only inference also had lower ECE, NLL, and Brier score and higher entropy-based OOD AUROC than observed gated inference.
Retraining ablation: Removing the Full-only component chain yielded an inconclusive macro-F1 difference but passed TS-NLL, revealing no stable task or calibrated-loss benefit.Expert-only-EDL achieved 0.785 ± 0.013 macro-F1 versus 0.790 ± 0.017 for Full-EDL; joint deletion prevented isolating individual components.
Calibration: The raw ECE advantage of EDL on Dataset 2 disappeared after temperature scaling and did not recur in Dataset 1.On Dataset 1, Simple-CE had the lowest raw ECE and retained the lowest mean after scaling.

5. Discussion

The audit found that several reliability and evidential components did not provide stable system-level benefits, while calibration, selective prediction, and OOD detection produced distinct conclusions. These findings support endpoint-specific model selection, bounded by image-level data and protocol limitations.

Calibration audit: Full-EDL’s raw calibration advantage appeared only on Dataset 2 and reversed after matched temperature scaling.Raw ECE and post-hoc calibrated performance answer different questions about native confidence and correctable confidence mismatch.
Reliability endpoints: Calibration, selective risk, and OOD detection need separate acceptance criteria because they can rank models differently.The study treats calibrated probabilities and referral ordering as in-distribution objectives, while OOD-sensitive use requires its own validated decision.
Recurring findings: Both multi-organ datasets showed no established Full-EDL macro-F1 gain, but Simple-CE+TS remained competitive for calibrated loss and selective risk.The raw-ECE advantage did not recur on Dataset 1, whereas the calibrated-loss and selective-risk pattern recurred across datasets.
Limitations and future work: The image-level public-source datasets could contain unknown patient or study dependence, so the repeated-seed intervals are not clinical replication estimates.Source identity was also coupled to organ and class composition, limiting source-held-out inference.
Limitations and future work: Conclusions are bounded to the evaluated compact backbone, ultrasound classification task, transfer protocol, two OOD probes, and limited score conventions.The restricted breast test used Full-EDL only without target adaptation or target-domain calibration.

6. Conclusion

The controlled audit did not establish a macro-F1 benefit for Full-EDL and selected Simple-CE with temperature scaling for the evaluated in-distribution objectives. Full-EDL remains the maximal reference because the F1 comparison was inconclusive and an OOD veto prevented unconditional replacement.

Conclusion: Full-EDL showed no established macro-F1 gain across matched optimisation, capacity-aware comparison, symmetric calibration, and repeated seeds.The comparison remained inconclusive rather than establishing superiority for either candidate.
Conclusion: The audited reliability gate had negligible observable influence at the evaluated checkpoints.This finding concerns fixed audited checkpoints, not a causal account of parameter evolution.
Conclusion: Simple-CE with temperature scaling was selected for in-distribution classification, calibrated probabilities, and selective prediction.The choice did not establish unconditional OOD safety.
Conclusion: The Dataset 2 OOD veto ruled out an unconditional replacement, so Full-EDL remains the maximal reference.The conclusion separates the evaluated in-distribution choice from distribution-shift reliability.
Conclusion: Components should earn retention through functional and retraining-based evidence, with distribution-shift reliability validated separately from in-distribution performance.The conclusion also supports symmetric calibration opportunities for probabilistic objectives.

A. Architectural details of the maximal candidate

The maximal Full-EDL candidate combines a compact backbone with spatial reliability weighting and an adaptive evidential expert branch. The diagrams document this reference design’s implementation structure, including residual modulation and uncertainty-aware routing.

Reference design: The documented diagrams describe implementation details for Full-EDL, which serves as the maximal audited reference design.They do not by themselves establish that the added components improve system performance.
Full-EDL architecture: Full-EDL uses MobileNetV3-Small low- and high-level stages, with NALA weighting before the adaptive branch reaches the high-level feature.The resulting representation is pooled globally before evidential output.
NALA spatial-reliability module: NALA multiplies attention interactions with a learned suppression map to form a spatial-reliability map that modulates only the positive residual contribution.The identity path remains intact, and the suppression map is an internal weighting signal rather than verified physical noise.
UGMoE routing and gating: The adaptive branch combines an evidential probe, sparse top-2 routing across four experts, a shared fallback, and a parameter-free reliability gate.The gate uses routing-confidence, probe-uncertainty, and router-uncertainty factors before adding the adaptive output through a positive residual scale.