PaperlayerView exact revision on arXiv

Source-linked AI summary

OPERA: Offline Policy-guided Expert Routing and Adaptation for Universal Biomedical Image Analysis

Zihan Li, Feiyang Liu, Dandan Shan, Ruibo Wang, Qingqi Hong

arXiv:2607.25108v1cs.CVcs.AIcs.LGeess.IV

TL;DR

Biomedical models often degrade under clinical distribution shifts, making repeated adaptation costly. OPERA learns expert-routing weights offline and adapts predictions at test time without retraining, consistently outperforming strong baselines across nine biomedical imaging benchmarks.

  • Problem

    Biomedical models often degrade across clinical domains, while adapting specialized or general-purpose models requires costly retraining or additional labeled data.

  • Method

    OPERA coordinates heterogeneous pretrained experts using offline validation-based weight learning, confidence calibration, and gradient-free test-time adaptive routing.

  • Results

    Across nine benchmarks, OPERA consistently outperformed strong single-model and ensemble baselines in classification and segmentation, including 3.35% AUC over RET-CLIP on RFMiD.

  • Takeaways & Limitations

    OPERA provides a deployable biomedical imaging predictor that combines complementary expert behavior without training parameter updates during inference.

  • Takeaways & Limitations

    Ensembling increases inference cost and latency, and OPERA requires a labeled validation set to initialize model-level weights.

Abstract

from arXiv · show

Biomedical image analysis spans diverse modalities and tasks, yet real-world deployment is hindered by severe distribution shifts across scanners, protocols, and patient populations. High-performing models consequently require repeated domain-specific fine-tuning, which is a costly cycle that becomes impractical when labels are scarce or privacy constraints limit data sharing. We propose OPERA (Offline Policy-guided Expert Routing and Adaptation), a multi-agent ensemble framework that addresses this deployment bottleneck by treating expert weight assignment as an offline policy learning problem: a routing policy is learned from a small validation set without gradient updates to any expert agent, then deployed with test-time adaptation to handle distribution shift. OPERA coordinates heterogeneous specialist agents through complementary mechanisms. The expert profiling module learns selection policies offline, enabling informed allocation of expertise. Each agent undergoes confidence calibration through temperature adjustment, ensuring more reliable probabilistic outputs. OPERA also incorporates distribution aware adaptation, where class weights are dynamically adjusted at the batch level using statistics derived from unlabeled test data. Instance level routing assigns each sample to the most suitable expert by leveraging inter model agreement and predictive entropy. We evaluate OPERA on 9 datasets covering fundus photography, chest X-ray, CT, MRI, and multimodal diagnostic benchmarks, comparing against 30+ baselines across classification, segmentation, and multimodal settings. OPERA consistently improves performance and calibration quality, demonstrating that offline policy-guided expert agents coordination is a practical path to deployable biomedical AI without retraining. Code is on \href{https://github.com/HUANGLIZI/OPERA}{GitHub}.

1 Introduction

Biomedical AI deployment is hindered by distribution shifts, costly domain-specific adaptation, and limited target-domain annotations, while foundation models demand substantial pretraining or fine-tuning. OPERA addresses this gap by coordinating diverse expert agents through offline policy optimization, confidence calibration, and backpropagation-free, distribution-aware test-time adaptation.

  • Deployment challenge: Specialized medical models can perform strongly yet remain brittle under distribution shifts, requiring costly retraining for each clinical environment.Foundation models offer potential generalization but typically require large-scale pretraining and task-specific fine-tuning, creating substantial resource demands.
  • Deployment challenge: Limited or unavailable target-domain annotations make domain adaptation difficult, and general-purpose segmentation models show limited zero-shot performance on medical images.Adapting such models commonly requires additional labeled data and optimization.
  • Motivation: Ensemble learning can improve robustness by combining diverse predictors to reduce correlated errors and increase reliability, but standard ensembles are often expensive to train and deploy.
  • OPERA: OPERA frames ensemble weight learning as offline policy optimization, using validation-set routing, confidence calibration, and distribution-aware test-time adaptation without deployment backpropagation.The framework coordinates expert agents through a coordinator that assigns routing weights.

2 Related Works

Prior work establishes ensembles as effective when constituent models provide diverse, weakly correlated errors, while offline policy learning avoids risky online exploration in medical imaging. Foundation-model adaptation remains costly and annotation-dependent, motivating OPERA’s label-free, retraining-free test-time adjustment of model contributions.

  • Ensemble Learning: Ensembles often improve accuracy and reliability over individual models, with effectiveness depending on constituent-model diversity and weakly correlated errors.These benefits have been reported across diverse medical imaging benchmarks.
  • Offline Policy Learning: Offline policy learning uses fixed historical datasets instead of active environmental interaction, making it suitable where online exploration poses safety and regulatory risks.Distributional shift remains a central challenge when proposed actions fall outside the behavioral training distribution.
  • Medical-Imaging Adaptation: Foundation-model adaptation commonly requires massive self-supervised pretraining, expensive fine-tuning, or target-domain annotations despite PEFT methods such as LoRA and adapters.These requirements limit adaptation efficiency for new medical-imaging domains.
  • OPERA: OPERA dynamically adjusts model contributions during inference without labeled data or model retraining, distinguishing it from fine-tuning and existing TTA methods.Its test-time adaptation mechanisms are designed for deployment under changing data distributions.

3 Methods

OPERA coordinates frozen specialist agents through offline-learned model- and class-level expertise, then adapts their calibrated predictions at deployment. Its dynamic fusion combines distribution-aware class weighting with instance-level routing based on agreement and uncertainty.

  • Expert coordination: OPERA treats pretrained experts as specialist agents and learns offline which agents to trust by disease class and input instance instead of uniformly averaging predictions.The framework uses multi-level weighting to reflect individual model strengths and input characteristics.
  • Offline expert profiling: The Expert Profiling Module learns model- and class-level routing weights from labeled validation data using only forward passes through frozen experts, without gradient updates.Model-level weights derive from average AUC across classes, while class-level weights favor experts with higher disease-specific AUC.
  • Confidence calibration: Per-Expert Confidence Calibration independently adjusts each model’s probability sharpness through temperature scaling while preserving prediction rank ordering.The calibrated outputs support more reliable confidence estimates for ensemble combination.
  • Deployment-time adaptation: Distribution-Aware Adaptation updates class-level attention weights during inference from unlabeled test-batch confidence statistics while keeping model-level weights fixed.The method blends validation-tuned and confidence-based weights gradually, using α=0.1 and β=0.1, without retraining.
  • Instance-level routing: Instance-level Expert Routing assigns sample-specific weights using inter-model agreement and prediction uncertainty, enabling hierarchical use of overall, disease-specific, and sample-specific expertise.For segmentation, each pixel is treated as an instance; routing uses calibrated predictions and conditional weighting strategies based on agreement, entropy, and prediction extremity.

4 Experiments

OPERA is evaluated across diverse biomedical imaging tasks and datasets, consistently achieving leading performance in classification and segmentation, including under limited annotation. Ablations attribute these gains to distribution-aware adaptation, calibrated routing, and complementary expert composition.

  • Ophthalmic classification: OPERA consistently achieves top performance across AUC, ACC, and mAP on RFMiD and OIA-DDR under linear probing and full fine-tuning.The evaluation uses a CLIP-style vision-language setup with heterogeneous CLIP vision encoders.
  • Classification across modalities: 83.05% AUC, 92.64% ACC, and 30.58% mAP are achieved on Chest X-Ray14, while OrganSMNIST reaches 98.83% AUC, 84.31% ACC, and 80.12% mAP.On Chest X-Ray14, OPERA surpasses Ark+ at 82.42% AUC and ELF at 82.34% AUC.
  • COVID-19 segmentation: 82.12% Dice and 73.26% Jaccard are achieved on QaTa-COV19, outperforming the strongest competing method, STPNet.The gain is attributed to heterogeneous backbones reducing typical over- and under-segmentation.
  • Limited-annotation segmentation: With 20% labeled data, OPERA consistently outperforms recent semi-supervised baselines on LA-MRI and Pancreas-CT segmentation.On LA-MRI, OPERA achieves 92.71% Dice and 85.62% Jaccard, improving over SASNet under the same labeling condition.
  • Multimodal diagnosis: OPERA applies probabilistic fusion to multimodal LLMs by extracting token-level log-probabilities for four candidate answers on a ten-disease ophthalmic benchmark.The framework aggregates expert-agent outputs under the same multiple-choice prompting protocol.
  • Ablation studies: Adding DAA, PECC, and IER improves performance, raising QaTa-COV19 Dice from 80.52% to 82.12% and MosMedData+ Dice from 75.94% to 77.03%.Using all three experts consistently performs best, while balanced blend factors α=0.10 and β=0.10 outperform skewed or aggressive blending.

5 Conclusion · A Theoretical Analysis of OPERA · A.1 Bias-Variance Decomposition for Weighted Ensembles

OPERA is presented as an offline-calibrated, zero-retraining framework that coordinates heterogeneous pre-trained agents through validation-driven weighting, confidence calibration, and test-time adaptive fusion. The theoretical analysis explains ensemble error through bias, variance, and covariance, motivating diversity-aware routing and moderate weight concentration.

  • 5 Conclusion: OPERA composes heterogeneous pre-trained agents at inference time without training-parameter updates.Its coordination framework uses offline calibration and aims to produce a single deployable predictor.
  • 5 Conclusion: Validation-driven weight initialization, confidence calibration, and test-time adaptive fusion convert complementary expert behaviors into one predictor.The framework is evaluated across nine benchmarks spanning fundus image and chest imaging domains, although the supplied passage is truncated afterward.
  • A Theoretical Analysis of OPERA: The theoretical section justifies OPERA’s multi-level weighting, confidence calibration, test-time adaptation, and fixed routing policy learned from a static validation set.The analysis draws on classical bias-variance decomposition, expert diversity theory, and calibration-aware fusion principles.
  • A.1 Bias-Variance Decomposition for Weighted Ensembles: OPERA’s fixed pre-trained experts are deterministic functions of x, so variance and covariance describe variability across the input distribution rather than model stochasticity.The probabilistic perspective samples test instances (x,y) ∼D from the data distribution D.
  • A.1 Bias-Variance Decomposition for Weighted Ensembles: The weighted ensemble’s expected squared error decomposes into squared bias and a variance-covariance structure under expectations over (x,y) ∼D.The derivation uses the true conditional mean μ(x)=E[y|x], with the cross-term vanishing because E[y−μ(x)|x]=0.
  • A.1 Bias-Variance Decomposition for Weighted Ensembles: Uniform weights minimize the weighted individual-variance term when experts have equal variances, while negatively correlated or uncorrelated errors further reduce ensemble variance.Architecturally diverse ViT, ResNet, and DenseNet experts are intended to promote error decorrelation.
  • A.1 Bias-Variance Decomposition for Weighted Ensembles: Temperature-scaled softmax with τ=10.0 moderately concentrates weights on higher-performing models while preserving diversity for variance reduction.The analysis frames weight selection as a bias-variance trade-off: concentration may reduce bias but increases effective variance.

A.2 Expert Diversity and Error Decorrelation … A.6 Hierarchical Weight Combination

OPERA combines diverse experts through calibrated, distribution-aware, and instance-specific weighting. Its hierarchical fusion integrates model quality, class expertise, and instance reliability while reducing the influence of unreliable predictions.

  • A.2 Expert Diversity and Error Decorrelation: Higher diversity lowers ensemble error when individual model accuracy is maintained, motivating OPERA’s use of ViT, ResNet50, and DenseNet121 to induce decorrelated errors.The experts use attention-based, residual, and dense connections, respectively.
  • A.3 Calibration Theory and Optimal Fusion: Temperature scaling adjusts prediction sharpness, with T>1 softening overconfident outputs and T<1 sharpening underconfident outputs before fusion.PECC assigns T_m=1.5 when s_m>0.4 and T_m=0.7 when s_m<0.1.
  • A.3 Calibration Theory and Optimal Fusion: EPM weights classes according to discriminative performance: models with higher class-specific AUC receive proportionally higher class-level weights.The temperature-scaled softmax approximates the Bayes-optimal weighting scheme described for calibrated models.
  • A.4 Test-Time Adaptation: Convergence and Consistency: DAA converges to a stationary weight distribution when confidence scores are bounded and α, β∈(0,1) with α+β<1.With α=β=0.1, validation-based weights retain major influence while test-time confidence contributes minor adaptation.
  • A.4 Test-Time Adaptation: Convergence and Consistency: Under covariate shift, confidence-based adaptation provides consistent weight estimates when confidence scores are monotonically related to model accuracy.This supports adapting class-level weights using unlabeled test data while preserving validation-based knowledge.
  • A.5 Instance-Level Weighting: Uncertainty Quantification: IER uses inverse-entropy weighting as an empirical heuristic in high-disagreement cases rather than claiming theoretical optimality.The formulation exp(γ/h_i,m) uses γ=2.0 and is intended to distinguish confidence levels, while acknowledging that calibrated models can still be confidently wrong.
  • A.5 Instance-Level Weighting: Uncertainty Quantification: IER applies entropy-based weighting only when a_i<0.60 or σ^2_i>0.12, otherwise defaulting to validation-derived weights grounded in empirical AUC performance.This conditional design limits the influence of overconfident but incorrect predictions.
  • A.6 Hierarchical Weight Combination: OPERA’s multiplicative hierarchical weights combine model-level quality, class-level affinity, and instance-level suitability, downweighting failed high-entropy predictions to approximately zero.The instance-level factor acts as a safety mechanism for out-of-distribution and challenging medical-imaging cases.

A.7 Computational Complexity Analysis · B Multiple-choice Evaluation Benchmark Details

OPERA adds a quadratic-in-expert-count inference overhead dominated by inter-model agreement, while expert forward passes remain the primary computational cost. With three experts, fusion is minimal and supports practical, latency-sensitive deployment, including clinical settings.

  • A.7 Computational Complexity Analysis: The computational overhead analysis frames OPERA’s additional inference operations as negligible relative to expert-agent forward-pass cost.This supports OPERA’s practicality for deployment settings where inference latency is a concern.
  • A.7 Computational Complexity Analysis: OPERA’s inference-time overhead is O(M^2BC) for M experts, C classes, and batch size B.This follows from PECC, DAA, and weight combination each costing O(MBC), while IER agreement costs O(M^2BC).
  • A.7 Computational Complexity Analysis: IER agreement computation costs O(M^2BC), whereas PECC scaling, DAA confidence computation, and weight combination each cost O(MBC).The quadratic term arises specifically from inter-model agreement computation.
  • A.7 Computational Complexity Analysis: With M = 3 experts, the quadratic term is small and ensemble-fusion overhead is minimal.The main cost remains running the experts’ forward passes, which can be parallelized across GPUs.
  • A.7 Computational Complexity Analysis: Running multiple expert agents sequentially increases latency proportionally, while modern GPUs can support parallel execution.Parallelizing expert forward passes helps preserve deployment efficiency when inference latency matters.
  • A.7 Computational Complexity Analysis: Fusion operations contribute less than 1% of total inference time, so they are not a practical bottleneck for clinical deployment.The cited fusion operations are PECC, DAA, IER, and weight combination.

B.1 Benchmark Source and Labels · B.2 Multiple-choice Construction

The benchmark comprises 845 color fundus photographs annotated across 10 disease categories. Each image is converted into a four-option multiple-choice question using its ground-truth label and three unique distractors.

  • B.1 Benchmark Source and Labels: 845 color fundus photographs form the comprehensive fundus image evaluation benchmark.The benchmark is extracted from [37].
  • B.1 Benchmark Source and Labels: The benchmark includes 10 disease categories.The categories are AMD, CSR, DR, DME, glaucoma, cataract, media haze, myopia, retinitis, and tessellation.
  • B.1 Benchmark Source and Labels: The images are annotated with disease labels for evaluation.The benchmark’s label distribution is reported in table 11.
  • B.2 Multiple-choice Construction: Each image and question is represented as a multiple-choice task with four answer options.This construction recasts the fundus evaluation task as multiple choice.
  • B.2 Multiple-choice Construction: The correct answer is always the image’s ground-truth disease label.The ground-truth label determines the correct option for each question.
  • B.2 Multiple-choice Construction: Three distractors are randomly sampled without replacement from disease labels other than the ground-truth label.Excluding the correct label prevents it from appearing among the distractors.
  • B.2 Multiple-choice Construction: The construction ensures that no duplicate answer options occur within a question.Distractor sampling without replacement guarantees option uniqueness.

C Detailed Experimental Setup … D.2 Visualization Results on COVID-CT (MosMedData+)

The experiments use a reproducible PyTorch-based setup with frozen experts, validation-only ensemble-weight tuning, and unsupervised test-time adaptive weighting. Qualitative results on QaTa-COV19 and MosMedData+ show OPERA producing smoother, more complete, and more faithful infection segmentations than several specialist baselines.

  • C.1 Hardware and Software Environment: Experiments run on NVIDIA RTX A6000 GPUs using PyTorch 1.13, CUDA, timm, and scikit-learn, with random seed 42 fixed throughout.cuDNN benchmarking is enabled for GPU performance optimization.
  • C.2 Image Preprocessing and Hyperparameters: Classification inputs are resized to 224×224, normalized with ImageNet statistics, and evaluated without augmentation.Validation weight tuning optionally uses RandAugment with magnitude 6 and standard deviation 0.5.
  • C.3 Weight Tuning of Expert Profiling Module: Ensemble weights are tuned on validation data with Adam at learning rate 0.01 for 20 epochs, maximizing average AUC while expert parameters remain frozen.Early stopping is used, and only cross-expert combination weights are trainable.
  • C.4 Adaptive Weighting Strategies of Test-time Modules: High agreement and low variance retain base model-level weights, using thresholds of agreement > 0.90 and variance < 0.03.This is one of the unsupervised adaptive weighting strategies applied during inference.
  • C.4 Adaptive Weighting Strategies of Test-time Modules: Disagreement triggers inverse-entropy weighting, intermediate cases use 70% extremity and 30% base weighting, and temperature scaling calibrates prediction sharpness.Temperature values are T=1.5 for sharpness > 0.4 and T=0.7 for sharpness < 0.1.
  • D More Visualization Results: Additional visualization results compare OPERA with specialist models across COVID-Xray and COVID-CT segmentation cases.The examples emphasize boundary quality, lesion completeness, and resistance to false-positive artifacts.
  • D.1 Visualization Results on COVID-Xray (QaTa-COV19): On QaTa-COV19, OPERA produces smoother, more accurate infection contours with fewer false positives than RAD-DINO and MedSAM.The examples include both scattered-boundary artifacts and bilateral lung infections.
  • D.2 Visualization Results on COVID-CT (MosMedData+): On MosMedData+, OPERA more faithfully captures lesion shape and extent than nnUNet, RAD-DINO, and MedSAM in thin and bilateral lesion cases.The baselines show under-segmentation, discontinuous predictions, or over-segmentation artifacts.

D.3 Visualization Results on LA-MRI Dataset and Pancreas-CT Dataset · E Comparison of the Same-expert-pool

On LA-MRI and Pancreas-CT, OPERA improves qualitative segmentation under 20% annotation by preserving boundaries, reducing discontinuities, suppressing spurious foreground, and recovering more complete pancreas structures. In same-expert-pool comparisons, OPERA surpasses Test-Time Ensemble by +5.17% AUC on OIA-DDR and +1.09% Dice on QaTa-COV19.

  • D.3 Visualization Results on LA-MRI Dataset and Pancreas-CT Dataset: Under 20% labeled data, OPERA preserves thin atrial boundaries on LA-MRI.The comparison is presented as an additional qualitative result in Fig. 9.
  • D.3 Visualization Results on LA-MRI Dataset and Pancreas-CT Dataset: OPERA reduces local discontinuities associated with noisy pseudo-labels.The reported improvements highlight geometric consistency with limited annotations.
  • D.3 Visualization Results on LA-MRI Dataset and Pancreas-CT Dataset: OPERA suppresses spurious foreground regions in MRI and CT segmentation.This behavior is reported in qualitative results on the LA-MRI and Pancreas-CT datasets.
  • D.3 Visualization Results on LA-MRI Dataset and Pancreas-CT Dataset: OPERA recovers more complete pancreas structures under ambiguous boundaries and noisy contexts.The passage characterizes this as improved robustness in MRI and CT.
  • E Comparison of the Same-expert-pool: Same-expert-pool comparisons evaluate competing methods on OPERA’s frozen three-expert pool.The compared methods include temperature-scaled or validation-weighted averaging, stacking, MoE gating, BMA, and TTA.
  • E Comparison of the Same-expert-pool: +5.17% AUC is OPERA’s margin over Test-Time Ensemble on OIA-DDR (cls.).The same-pool comparison is designed to rule out gains from simply using more models.
  • E Comparison of the Same-expert-pool: +1.09% Dice is OPERA’s margin over Test-Time Ensemble on QaTa-COV19 (seg.).Test-Time Ensemble is identified as the strongest same-pool baseline.

F Discussion of Fully Unsupervised Mode

OPERA supports deployment without labeled target-domain data through source-domain transfer or fully unsupervised initialization. In fully unsupervised evaluation, OPERA* remains competitive with strong methods while using test-time adaptive fusion and adjustment.

  • Source-domain transfer: OPERA transfers source-domain expert-profiling weights to related target domains within the same modality.The relative expertise of expert agents often generalizes across datasets within a modality.
  • Fully unsupervised operation: Fully unsupervised OPERA initializes model weights and class attention uniformly, then relies on PECC, DAA, and IER for test-time adaptation.The initialization uses w_model,m = 1/M and W_class,m,c = 1/M.
  • Unsupervised evaluation: 83.18% AUC on OIA-DDR under linear probing and 86.30% under full fine-tuning enable OPERA* to outperform individual experts and remain competitive with RETFound (85.96%) and FLAIR (85.54%).OPERA* bypasses validation-based EPM initialization and uses uniform weights.
  • Deployment overhead: OPERA’s one-time validation-based calibration requires less overhead than approaches requiring massive pretraining, target-domain samples, retraining, or joint expert optimization.The calibration can leverage small labeled subsets or existing annotated datasets from similar domains.

G More Discussion

OPERA achieves broad, state-of-the-art biomedical imaging performance through offline-calibrated, zero-retraining expert coordination across modalities and tasks. Its main tradeoff is higher inference cost, while offline calibration remains lightweight and cacheable.

  • Generalization and positioning: OPERA achieves state-of-the-art performance across diverse biomedical imaging tasks without retraining, rivaling or surpassing specialized models through expert diversity.The framework targets distribution shifts and unseen clinical domains across fundus photography, chest X-rays, CT scans, and structured multimodal diagnostic queries.
  • Classification results: 89.47% AUC and 58.93% mAP on RFMiD improve upon RET-CLIP by 3.35% AUC and 7.66% mAP.OPERA also exceeds VisionFM, FLAIR, and RETFound on retinal disease classification.
  • Classification results: 83.05% AUC and 30.58% mAP on Chest X-Ray14 surpass Ark+, global-local integration models, and the ELF ensemble-of-foundation-models baseline.On OrganSMNIST, OPERA achieves 98.8% AUC and 84.3% accuracy without task-specific adaptation.
  • Segmentation results: 82.1% and 77.0% Dice scores on QaTa-COV19 and MosMedData+ outperform nnU-Net and STPNet despite OPERA not being explicitly trained for segmentation.These results support inference-time ensemble adaptation without retraining.
  • Limitations: OPERA’s zero-retraining deployment increases inference cost and system latency because multiple expert agents run in parallel.Zero-retraining specifically excludes backpropagation, gradient computation, and expert-agent parameter updates during deployment.
  • Calibration overhead: Approximately five minutes on standard Chest X-Ray14 validation splits is sufficient for one-time offline calibration using forward passes and analytical AUC computation.The calibration can be performed once and cached for subsequent deployment.
Loading 2607.25108v1…