Source-linked AI summary
MedSTP: Preserving Subspace and Transition Evidence for Medical VQA Token Pruning
Mengchen Fan, Qizhen Lan, Xi Xiao, Lin Zhao, Liang He, Tianyang Wang, Xiaoqian Jiang, Pulin Che, Baocheng Geng
TL;DR
Medical VQA pruning must reduce long visual-token sequences without losing localized boundary and texture evidence. MedSTP selects tokens to preserve the full set's dominant evidence subspace and uses a bounded transition prior. It shows strong pruning performance across the evaluated benchmarks, with results reported across both closed- and open-ended RadImageNet-VQA evaluations.
Problem
Medical VQA can depend on localized boundary and texture evidence, yet frozen-VLM pre-decoding selection must choose a fixed token subset without dense supervision, task-specific retraining, or repeated subset decoding.
Source: §1 · p. 2Method
MedSTP progressively selects tokens that add subspace information and applies a bounded token-grid transition prior to promote boundary- or texture-like tokens that already carry subspace support.
Source: §1 · p. 2Results
Takeaways & Limitations
The results support MedSTP as a training-free visual-token pruning framework for medical VQA under fixed budgets.
Source: §7 · p. 9Evaluation is limited to two-dimensional medical VQA with two VLM backbones, and extension to volumetric images and broader clinical tasks remains future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Medical vision-language models (VLMs) for visual question answering (VQA) often pass long visual-token sequences to the language model, making visual-token pruning attractive but risky for medical images, where answers may depend on localized boundary and texture evidence. We study training free pre-decoding pruning and treat pruning as a subspace stable evidence preservation problem. The retained tokens should remain close to the dominant visual evidence subspace induced by the full token set. To this end, we propose \emph{Subspace-Transition Token Pruning} (\method{}), which estimates each token's spectral support from principal component space and constructs a compact representation subspace. To better reflect medical image structure, \method{} further uses a bounded token-grid transition prior to promote boundary and texture sensitive tokens. We evaluate \method{} on the closed-ended SLAKE and OmniMedVQA benchmarks and on RadImageNet-VQA, which provides both closed-ended and open-ended evaluations, using Lingshu-7B and Qwen3-VL-8B backbones. \method{} delivers consistently strong performance on both the closed-ended and open-ended RadImageNet-VQA evaluations. These results show that \method{} is an effective training free pruning framework for medical domain VQA.
1. Introduction
Long visual-token sequences make pruning attractive, but medical VQA can depend on localized boundary and texture evidence that generic selectors may under-prioritize. MedSTP selects a subspace-stable subset and applies a bounded transition prior to favor supported local cues.
1. Introduction
- Medical VLMs pass long image-token sequences to the language model, while medical answers may depend on small boundary or texture changes as well as dominant anatomy.The introduction notes this pressure is central to practical inference for modern backbones such as Qwen3-VL [6], and that localized changes matter.Source: §1 · p. 1
- The fixed-budget selection problem is to preserve leading visual evidence before decoding, when frozen-VLM medical VQA lacks dense supervision, retraining, and repeated subset decoding.Generic saliency or redundancy scores may retain dominant anatomy while discarding lower-priority boundary or texture tokens.Source: §1 · p. 2
- The retained subset should stay close to the full token set's evidence subspace, since high-energy tokens can overrepresent one anatomical pattern while subtle transitions may carry subspace support.The paper frames this as subspace-stable evidence preservation and notes that global participation alone can deprioritize transitions.Source: §1 · p. 2
- MedSTP combines progressive selection of tokens that add previously unrepresented subspace information with a bounded token-grid transition multiplier that promotes supported boundary and texture cues.The bound prevents the transition term from overriding the subspace signal.Source: §1 · p. 2
- The paper evaluates this training-free, pre-decoding selector on SLAKE, OmniMedVQA, and RadImageNet-VQA using Lingshu-7B and Qwen3-VL-8B.Its contributions specify matched insertion points and fixed token budgets.Source: §1 · p. 2
2. Related Work
Prior pruning work spans generic vision-token selection, VLM-specific attention and redundancy cues, and subspace-oriented selection, while medical token reduction has largely targeted other tasks or settings. MedSTP focuses on frozen medical-VQA backbones, selecting original visual tokens before language decoding under a matched budget.
2.1. Visual Token Pruning and Subspace Selection
- Generic vision-token methods prune progressively, reorganize, or merge patches [42] [37] [8], while VLM methods use attention, text relevance, redundancy, or grouping cues [49] [34] [48] [45] [35] [24].These approaches provide practical baselines but generally do not explicitly model localized medical boundary, texture, and transition evidence.Source: §2.1 · p. 2
- Subspace-oriented methods preserve global variance or reconstruct text-conditioned or selected subspaces [5] [36] [33], whereas MedSTP applies token-side PCA and a bounded transition prior to fixed-budget, pre-decoding medical VQA.The compressed object is the visual-token sequence of a frozen VLM, not model weights or a communication budget.Source: §2.1 · pp. 2–3
2.2. Medical VLMs and Medical Token Compression
- Medical VLMs and benchmarks cover image-question answering across modalities, including SLAKE, OmniMedVQA [26], and closed- and open-ended RadImageNet-VQA [9]; the study uses Lingshu-7B [46] and Qwen3-VL-8B [6].Related medical token-reduction work includes segmentation, edge-aware halting, volume scheduling, and 3D medical-image understanding [50] [23] [40], while MedSTP targets training-free frozen-VLM medical VQA.Source: §2.2 · p. 3
2.3. Compact Representations and Knowledge Transfer
- Distributed-learning and knowledge-transfer methods select or weight training information, while MedSTP retains original visual embeddings at inference without synthetic training data, peer selection, or backbone updates.The related work contrasts gradient matching [15], representative knowledge sharing, peer selection [17], and supervision weighting [7] [32].Source: §2.3 · p. 3
3. Method
MedSTP makes a fixed-budget decision before decoding, combining a retained evidence subspace with a bounded local-transition cue. Its design targets preservation of the leading visual evidence structure when pruning tokens.
3. Method
- MedSTP is a training-free, pre-decoding selector that builds a compact subspace from principal-direction participation and incorporates a bounded token-grid transition cue.The selected visual-token subset is passed to the frozen VQA decoder.Source: §3 · p. 3
3.1. Problem Setup
- The ideal selector keeps the pruned answer distribution close to the full-token distribution, but directly optimizing that objective would require repeated decoding over candidate subsets.MedSTP instead uses a representation-level target: preserve the leading subspace induced by the full token set.Source: §3.1 · p. 4
3.2. A Subspace Preservation View
- A retained set is preferable when its restricted second moment stays close to the full-token second moment, while Proposition 1 offers a perturbation view rather than an accuracy guarantee.The bound assumes the residual norm is below half the eigengap; this condition may fail under aggressive pruning.Source: §3.2 · pp. 4–5
3.3. Subspace Construction
- Token-side PCA supplies spectral support by measuring each token’s participation in the top principal directions and using its representation there as an evidence weight.The step-dependent contribution is large when a candidate has substantial representation outside the currently retained span.Source: §3.3 · p. 5
- Tokens are assigned to their strongest projection direction; each nonempty group receives one slot, and remaining slots are allocated according to group size.Cumulative rounding and a deterministic adjustment produce the group budgets.Source: §3.3 · p. 5
3.4. Local Transitions under a Spectral Constraint
- A normalized Sobel-style transition cue from the encoder’s token grid multiplies the current subspace contribution, adding local sensitivity to the spectral criterion.The transition cue is bounded so it modifies rather than replaces the subspace contribution.Source: §3.4 · p. 5
4. Experiments
Experiments compare fixed-budget pruning across three medical VQA benchmarks and two VLM backbones. MedSTP leads the evaluated pruning methods, and its retained-token eigenspace remains closer to the full-token eigenspace across most tested pruning rates.
4.1. Experimental Setup
- Evaluation uses SLAKE, OmniMedVQA, and RadImageNet-VQA with Lingshu-7B and Qwen3-VL-8B, applying pruning after the merger and before language-model insertion under identical prompts, insertion points, and token budgets.Open-ended answer correctness is assessed by two method-blind judges using a shared rubric.Source: §4.1 · pp. 5–6
4.2. Main Results
- Across three benchmarks and both backbones, MedSTP achieves the strongest aggregate performance; OmniMedVQA average accuracy reaches 70.83% with Lingshu-7B and 58.82% with Qwen3-VL-8B.These exceed the strongest competing method by 3.57% and 2.89%, respectively. On SLAKE, average accuracy improves over the strongest competing baseline by 1.55% with Lingshu-7B and 1.67% with Qwen3-VL-8B; on RadImageNet-VQA, gains are 3.50% and 1.60% for closed-ended average accuracy, and 5.20% and 1.25% for method-blind open-ended judge average.Source: §4.2 · p. 6
4.3. Subspace Preservation Diagnostic
- On 200 SLAKE examples, MedSTP aligns the retained-token eigenspace more closely with the full-token eigenspace over most pruning rates, while shrinking budgets generally raise drift and lower accuracy.Figure 4 places MedSTP nearer the low-drift, high-accuracy region as retained-token budgets shrink.Source: §4.3 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6
5. Ablation Studies
MedSTP maintains accuracy more gradually as pruning becomes aggressive, and adding a bounded transition prior improves accuracy at the main 10% keep ratio. Moderate transition weighting works best, while 10% retention also reduces prefilling latency and memory.
5.1. Accuracy Across Pruning Rates
- MedSTP degrades more gradually than other methods on SLAKE and OmniMedVQA as pruning becomes aggressive.The methods remain close under mild pruning, with clearer separation at aggressive budgets; OmniMedVQA separates earlier than SLAKE.Source: §5.1 · pp. 6–7
5.2. Transition Prior Ablation
- At 10% keep ratio, adding the transition prior improves overall accuracy and most group accuracies, with gains concentrated in SLAKE OT and OmniMedVQA RJ and LC.OmniMedVQA KL changes little, so the ablation is better read as answer-level reranking than as a general distribution-matching gain.Source: §5.2 · p. 7, Table 4 · p. 9
5.3. Transition Weight Sensitivity
- OmniMedVQA accuracy peaks at β = 0.1 −0.3 and drops for larger transition weights.The paper uses β = 0.2 by default and treats the transition cue as a bounded correction to spectral support.Source: §5.3 · p. 8, Figure 6 · p. 8
5.4. Efficiency Analysis
- At 10% visual-token retention, LLM prefilling latency falls from 34.58 to 21.43 ms, a 38.03% reduction and 1.61× speedup, while incremental peak allocated GPU memory falls from 84.06 to 42.43 MiB, a 49.52% reduction.On SLAKE with Lingshu-7B, end-to-end latency falls from 80.09 to 74.70 ms, a 6.74% reduction and 1.07× speedup.Source: §5.4 · pp. 8–9
6. Discussion
Across three benchmarks and two backbones, results support that MedSTP preserves diverse representation directions while its bounded transition prior adjusts selection toward localized feature changes. The analysis and evaluation have defined limits.
6. Discussion
- Across three benchmarks and two backbones, results support that MedSTP preserves diverse representation directions while its bounded transition prior adjusts selection toward localized feature changes.Pruning-rate and subspace-drift analyses are consistent with better representation preservation under aggressive compression.Source: §6 · p. 9
- The perturbation analysis does not guarantee answer correctness or prove the discrete selector minimizes the spectral objective.Its small-perturbation condition may also fail under aggressive pruning.Source: §6 · p. 9
- Evaluation is limited to two-dimensional medical VQA with two VLM backbones; volumetric images and broader clinical tasks remain future work.The transition prior is not a lesion detector and cannot recover a local cue with zero contribution to the current selection score.Source: §6 · p. 9
7. Conclusion
The conclusion presents MedSTP as a training-free visual-token pruning rule for medical VQA, combining spectral influence, non-redundant subset construction, and bounded token-grid transition modulation. Experiments report improved fixed-budget pruning on three benchmarks.
7. Conclusion
- MedSTP is a training-free visual-token pruning rule for medical VQA.It combines spectral token influence, non-redundant subset construction, and bounded token-grid transition modulation.Source: §7 · p. 9
- MedSTP combines spectral token influence with non-redundant subset construction.These components are paired with bounded token-grid transition modulation.Source: §7 · p. 9
- Experiments on SLAKE, OmniMedVQA, and RadImageNet-VQA show improved fixed-budget pruning under matched token budgets.Source: §7 · p. 9