PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability Testing of Medical Model Performance under Distributed Deployment

Yifei Wang, Xiaohan Zhang, Youtao Ding, Tianlin Li, Xiaoyu Zhang, Yida Yang, Li Pan

arXiv:2609.36525v1eess.IVcs.AI

TL;DR

Medical models are evaluated centrally but often deployed through distributed execution stacks, with an untested assumption that their behavior remains unchanged. The paper introduces paired testing and a distributed-execution-sensitive benchmark; experiments find measurable disagreements across medical model types, including visual test success rates of 0.21–0.43 for single-modality models and 0.32–0.98 for multimodal models.

  • Problem

    Medical-model evaluation and deployment commonly use different execution environments, but whether predictions remain consistent across them is largely untested.

    Source: §1 · p. 1

  • Method

    The paper pairs centralized HuggingFace evaluation with matched distributed executions, holding the checkpoint, input, and decision rule fixed, and applies complementary visual and textual testing procedures.

    Source: §1 · pp. 1–2

  • Results

    Experiments across language, vision, and multimodal models show measurable execution-related disagreements, with visual test success rates of 0.21–0.43 for single-modality models and 0.32–0.98 for multimodal models.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    The benchmark makes evaluation-deployment consistency an explicit criterion for medical-model assessment.

    Source: §1 · p. 2, §6 · p. 8

Abstract

from arXiv · show

Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Although modern frameworks improve serving efficiency through tensor parallelism, mixed precision, kernel fusion, and multi-device communication, they are generally assumed to preserve the behavior observed during centralized HuggingFace evaluation. This assumption creates an evaluation-deployment mismatch: a model may pass offline evaluation but produce a different output after the execution stack changes. To address this mismatch, we propose a testing framework and an improved, distributed-execution-sensitive medical-model benchmark that evaluates the same checkpoint and input under a centralized HuggingFace reference and matched distributed deployments. Extensive experiments across language, vision, and multimodal medical models show that execution changes can produce measurable output disagreements. Across supported visual settings, the test success rate ranges from 0.21 to 0.43 for single-modality models and from 0.32 to 0.98 for multimodal models. The benchmark is aimed at extending medical-model evaluation from capability and security to evaluation-deployment consistency.

1 Introduction

Centralized medical-model evaluation and distributed deployment use different execution environments, yet are often assumed to preserve predictions. The paper makes this mismatch testable with paired evaluations and reports disagreements across medical model types.

1 Introduction

  • Offline evaluation commonly uses centralized single-device HuggingFace, while practical serving uses distributed frameworks for latency and memory constraints (Aminabadi et al. 2022; Kwon et al. 2023; Pope et al. 2023).These frameworks use tensor parallelism, mixed precision, fused kernels, and multi-device communication.Source: §1 · p. 1
  • The untested assumption is that a medical image will receive the same prediction after the execution system changes (Aminabadi et al. 2022; Pope et al. 2023; Kwon et al. 2023).Existing evaluation emphasizes capability and safety, while visual deployment consistency is rarely measured.Source: §1 · p. 1
  • The framework treats deployment disagreement as a reliability dimension beyond aggregate accuracy, and the paper localizes divergence-amplifying layers for targeted precision elevation.The targeted mitigation raises precision only for critical layers, leaving the rest of the deployment pipeline unchanged.Source: §1 · pp. 1–2
  • 0.21–0.43 for single-modality models and 0.32–0.98 for multimodal models are the visual test success-rate ranges across evaluated distributed configurations.The multimodal variation is described as larger and more framework-dependent.Source: §1 · p. 2
  • The benchmark pairs centralized HuggingFace evaluation with matched distributed executions, holding checkpoint, input semantics, and decision rule fixed while changing the execution stack.It uses complementary visual and textual test-generation procedures.Source: §1 · p. 2

2 Related Work

Prior work studies software testing, medical-model capability and safety, and numerical variability in inference systems. This paper connects these areas by testing whether medical visual behavior remains consistent when only the execution stack changes.

2.1 Testing and Deployment Reliability

  • Differential testing and related methods expose failures in learned systems, while prior system-level work finds identical programs can behave differently across platforms (Pham et al. 2019; Guo et al. 2020, 2019; Wang et al. 2024).This work centers the comparison on medical visual behavior, keeping model, image, and prediction rule fixed while changing the execution stack.Source: §2.1 · p. 2

2.2 Medical Foundation Model Evaluation

  • Medical-model benchmarks assess image understanding, question answering, reasoning, hallucination, and safety, but do not fully test whether predictions persist across execution environments (Jin et al. 2021; Ben Abacha and Demner-Fushman 2019; Jin et al. 2019; Pal, Umapathi, and Sankarasubbu 2022; Singhal et al. 2023, 2025; Chen et al. 2023; Zheng et al. 2023; Rückert et al. 2024; Li et al. 2023a,b; Radford et al. 2021).The paper treats visual deployment consistency as a separate, directly measurable reliability dimension.Source: §2.2 · p. 2

2.3 Numerical Variability in Inference Systems

  • Non-associative floating-point arithmetic and execution choices such as fused kernels, mixed precision, and parallel reductions can change numerical results (Goldberg 1991; Higham 2002; Kahan 1996; Demmel, Ahrens, and Nguyen 2016; Ahrens, Demmel, and Nguyen 2020; Micikevicius et al. 2017).Such differences become observable in medical visual outputs when they change the leading class or image-associated answer, rather than when the prediction margin remains large.Source: §2.3 · p. 2

3 Problem Definition

The paper models medical inference as execution of a weighted neural model under a deployment stack, followed by a decision rule. It defines inconsistency by changed predictions across stacks and constructs tests that preserve reference behavior while exposing deployment-specific changes.

3.1 Definition 1 (Medical Model Execution).

  • A medical model is a sequence of weighted neural operators that processes an image or image-text pair into class scores or an answer-token distribution.The execution stack specifies kernels, precision, tensor partitioning, communication, attention implementation, and reduction order.Source: §3.1 · p. 3
  • For image classification, the decision rule takes the class-probability argmax; for visual question answering, it ranks answer options or selects autoregressive tokens conditioned on the image.Medical language-only inference is treated as a complementary instance of the same execution model.Source: §3.1 · p. 3

3.2 Definition 2 (Deployment-Induced Inconsistency).

  • Deployment-induced inconsistency means the same image, checkpoint, prompt, and decision policy produce different predictions when the execution stack changes.It concerns cross-deployment consistency, not whether either prediction is clinically correct in isolation.Source: §3.2 · p. 3

3.3 Definition 3 (Deployment-Sensitive Test Case).

  • A deployment-sensitive visual test adds a bounded perturbation that preserves the reference prediction but changes the distributed prediction.For language tests, an invisible suffix keeps the visible prompt unchanged while inducing different answer choices across executions.Source: §3.3 · p. 3

3.4 Problem Condition.

  • Small execution-level numerical changes alter predictions when they interact with a small margin between leading classes or candidate answers and cross a discrete decision boundary.The primary test uses bounded image perturbations to expose single-card versus multi-card disagreement; the language setting tests the same condition complementarily.Source: §3.4 · p. 3

4 Approach

VDST and TDST test whether matched centralized and distributed executions disagree, using image perturbations or invisible text suffixes to expose deployment-sensitive predictions.

4.1 Visual Deployment-Sensitive Testing

  • VDST searches within an image perturbation budget for inputs that keep the single-card prediction correct but change the multi-card prediction.Its reported success rate measures inputs reaching this asymmetric condition, rather than generic model error.Source: §4.1 · p. 3

4.2 Textual Deployment-Sensitive Testing

  • TDST appends an invisible suffix made from stable single-token characters while leaving the visible medical question unchanged.Retained characters must meet Unicode category or variation-selector criteria and preserve their token mapping under repetition.Source: §4.2 · p. 4
  • TDST reduces the margin between the reference model’s top two answers while keeping them above distractors, then searches discrete suffix tokens for an execution disagreement.Its gradient-guided search tests replacements from the invisible-token pool and stops when the executions select different answer letters.Source: §4.2 · p. 4

5 Experiments

Across language, vision, and multimodal medical models, the experiments test whether changing the execution stack can produce prediction disagreements and whether targeted precision elevation can reduce them. They find disagreements in ordinary questions and test-generated cases, with rates and search costs varying by model and runtime.

5 Experiments

  • The experiments organize testing around natural disagreement, language-model testing, vision and multimodal testing, and localization with precision elevation.RQ1 measures unmodified questions without constructed perturbations or optimization-based test generation.Source: §5 · pp. 4–5, §5.2 · p. 5

5.1 Experimental Setting

  • The benchmark fixes the checkpoint, input, prompt, and decision rule while comparing HuggingFace with matched distributed deployments across language and visual tracks.The language track covers four models, two datasets, and DeepSpeed and vLLM on two GPUs; the visual track includes supported two- and four-GPU model-framework settings. SR measures disagreement exposure, and Avg. Step measures search steps among successful cases.Source: §5.1 · p. 5

5.2 RQ1: Natural Discrepancy Across Deployment Frameworks

  • 45.82% of MedQuAD samples disagreed between standard non-distributed and two-GPU DeepSpeed execution, versus 25.52% across the two MedQA subsets.The MedQA-derived US and Chinese subsets had similar discrepancy rates of 24.85% and 25.80%, respectively.Source: §5.2 · p. 5, Figure 3 · p. 5

5.3 RQ2: Testing Medical Language Models

  • Under DeepSpeed, Qwen2.5-3B-Instruct, HuatuoGPT2-7B, and Qwen3-8B reached 100% SR on both language datasets; vLLM results varied by model and dataset.Qwen2.5-3B-Instruct required 8.57 steps on MedMCQA and 10.70 on PubMedQA under DeepSpeed; Figure 5 compares SR and average steps across frameworks.Source: §5.3 · pp. 5–6, Figure 5 · p. 7

5.4 RQ3: Testing Medical Vision and Multimodal Models

  • Single-modality SR ranged from 0.21 to 0.27 for ResNet-18 and 0.38 to 0.43 for RCLIP, while BLIP2-OPT-2.7B reached 0.98 on two-GPU vLLM.Increasing execution from two to four GPUs did not remove framework-specific sensitivity; the exact values shifted across configurations.Source: §5.4 · pp. 6–7, Table 1 · p. 6

5.5 RQ4: Localization and Precision Elevation

  • Layer-wise localization found divergence-amplifying modules concentrated around numerically sensitive components such as self.rotary_emb and self.norm.The analysis records leaf-module outputs during the forward pass producing the first token and uses relative divergence lift to identify amplification.Source: §5.5 · p. 7
  • Across 16 supported model-framework pairs, targeted precision elevation reduced average SR from 0.56 to 0.21, though residual disagreement remained configuration-dependent.The reduction was especially strong for ResNet-18 and RCLIP, whose SR approached zero; Qwen2.5-VL on vLLM retained substantial disagreement.Source: §5.5 · p. 7, Table 2 · p. 7

6 Conclusion

The paper treats centralized evaluation and distributed deployment as a paired consistency problem, finding observable disagreements when execution changes despite fixed checkpoint, input, and decoding policy. It localizes divergence-amplifying layers and selectively raises their precision, reducing disagreement without changing the whole model.

6 Conclusion

  • Distributed frameworks and GPU configurations can induce prediction disagreements even when the checkpoint, input, and decoding policy are fixed.The benchmark treats evaluation and deployment as a paired system, changing only the execution stack.Source: §6 · p. 7
  • The paper localizes layers that amplify execution differences and selectively improves their numerical precision.This targeted strategy reduces disagreement while avoiding unnecessary changes to the entire model.Source: §6 · pp. 7–8
  • Deployment configurations are not behaviorally neutral, so evaluation-deployment consistency should be an explicit criterion in medical-model assessment.Source: §6 · p. 8

Related papers

Loading 2609.36525v1…