Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval

Meng Gao; Yizhen Zhang; Yang Ding; Ziqi Dai; Shuoshuo Zhang; Junjie Wang; Taiqiang Wu; Chufan Shi; Lei Ji; Jian Jiao; Linfeng Zhang; Yeyun Gong; Yujiu Yang

Source-linked AI summary

Skim and Skip: Hierarchical Adaptive Inference for Efficient Multimodal Retrieval

Meng Gao, Yizhen Zhang, Yang Ding, Ziqi Dai, Shuoshuo Zhang, Junjie Wang, Taiqiang Wu, Chufan Shi, Lei Ji, Jian Jiao, Linfeng Zhang, Yeyun Gong, Yujiu Yang

arXiv:2609.01613v1Jun 24, 2026cs.IR

TL;DR

Uniformly dense MLLM retrieval incurs high inference cost while overlooking uneven token contributions and different query depth requirements. SAS combines [EOS]-centric token selection with adaptive depth allocation to preserve relevant evidence and stop inference when representations are sufficiently mature. Across 12 MMEB retrieval tasks, it retains about 99% of dense-baseline average retrieval performance while achieving up to 1.64× end-to-end speedup and 66.3% FLOPs reduction.

Problem
Existing MLLM retrievers process all input tokens through the full model despite uneven token contributions and varying inference-depth requirements across queries.
Method
SAS unifies [EOS]-centric cross-modal information selection with adaptive depth allocation based on representation maturity.
Results
Across 12 multimodal retrieval benchmarks spanning five task types, SAS retains about 99% of the dense baseline’s average retrieval performance while achieving up to 1.64× speedup and 66.3% FLOPs reduction.
Takeaways & Limitations
SAS shows that token- and depth-level computation in dense MLLM retrieval can be removed adaptively while preserving competitive retrieval performance.
Takeaways & Limitations
SAS is evaluated only on retrieval and embedding workloads, not multimodal generation tasks such as VQA or captioning.

Abstract

from arXiv · show

Universal multimodal retrieval (UMR) increasingly adopts multimodal large language models (MLLMs) as unified embedding backbones, but their strong retrieval performance comes at substantial inference cost. Existing methods typically rely on uniformly dense inference, where all input tokens are processed through the entire model and matched using the final-layer [EOS] representation. However, this paradigm overlooks two key forms of heterogeneity in multimodal retrieval: token contributions to the final retrieval embedding are highly uneven, and different queries require markedly different amounts of inference depth. To address this, we propose Skim and Skip (SAS), a hierarchical adaptive inference framework for efficient multimodal retrieval. SAS first performs token-level evidence selection to preserve only the input information most relevant to the final retrieval embedding, and then performs depth-adaptive inference to determine whether the current representation is already sufficient for reliable matching. Experiments on 12 MMEB retrieval tasks show that SAS retains about 99% of the dense baseline's average retrieval performance while achieving up to 1.64 times end-to-end speedup and up to 66.3% FLOPs reduction.

1 Introduction

MLLM retrievers achieve strong multimodal performance but incur high inference cost because dense processing overlooks uneven token contributions and query-dependent depth needs. SAS addresses both forms of heterogeneity through [EOS]-centric token selection and adaptive depth allocation, retaining retrieval quality while reducing computation.

MLLM-based retrievers process every query with all tokens through the full model, creating substantial inference cost.
Existing token-reduction methods can severely degrade retrieval because they overlook the critical role of the [EOS] embedding.
SAS combines [EOS]-centric cross-modal information selection with adaptive depth allocation in a hierarchical inference framework.
Uniform dense inference overlooks uneven token contributions and varying depth requirements across multimodal retrieval queries.
SAS retains about 99% of the dense baseline’s average retrieval performance across 12 multimodal retrieval benchmarks while reducing FLOPs by up to 66.3%.The evaluation spans five task types.

2 Revisiting Uniform Dense Inference in MLLM Retrieval

Evidence from MLLM retrieval shows that token contributions to the [EOS] embedding and the depth needed for reliable retrieval are heterogeneous. These observations motivate selecting retrieval-relevant tokens and allocating inference depth adaptively rather than applying uniform computation.

Are All Tokens Equally Informative?: Retrieval performance remains largely stable under 10–30% random token pruning, with slight gains on some tasks.
Are All Tokens Equally Informative?: Token composition varies substantially across tasks, making fixed modality-dependent selection rules brittle.
Do All Queries Require the Same Inference Depth?: On VisDial, layer 22 reaches 50.9% Hit@1 versus 82.1% at the final layer, showing that intermediate layers already encode retrieval-relevant semantics.
Do All Queries Require the Same Inference Depth?: On MSCOCO text-to-image retrieval, Easy queries reach 55.0% Hit@1 at layer 22, whereas Hard queries reach 7.2%.The disparity indicates that retrieval representations mature at different rates across samples.
Do All Queries Require the Same Inference Depth?: Turning intermediate-layer signals into adaptive depth savings requires intermediate-layer retrieval supervision and a mechanism that assesses representation maturity.

3 Methodology

SAS combines [EOS]-centric token selection with adaptive depth allocation to concentrate computation on evidence relevant to retrieval and stop inference when representations are sufficient.

SAS framework: SAS first applies Skim to retain input information most relevant to the final [EOS] retrieval embedding, then applies Skip to assess representation sufficiency.Skim selects evidence across visual and textual tokens; Skip determines whether deeper processing is needed.
Skim: token influence: Removing a token perturbs [EOS] attention through re-normalization, so token influence depends on both attention mass and value-vector discrepancy.This motivates an influence measure beyond raw attention.
Skim: token influence: The influence score upper-bounds the downstream perturbation caused by removing a token, with larger scores indicating greater importance to the final retrieval embedding.The score aggregates effects across attention heads and uses quantities available during the forward pass.
Skim: cross-modal selection: At skim layer ls, SAS ranks visual and textual tokens under the same [EOS]-based criterion and removes those with the lowest influence scores.Visual tokens are pooled before cross-modal selection, and the selection adds negligible overhead.
Skip: adaptive depth: Skip trains the decision layer ld as a retrieval representation before using a lightweight assessor to predict whether inference can terminate there.Intermediate supervision combines feature-level distillation with contrastive learning, while the assessor uses uncertainty, modality, and semantic features.

4 Experiment

SAS is evaluated on the MMEB retrieval split across 12 benchmarks and five multimodal retrieval task types, using Hit@1 and single-GPU end-to-end latency.

Implementation: The implementation uses Qwen2.5-VL-Instruct 3B and 7B models and also tests generality on a LLaVA-style training architecture.Stage I uses LoRA fine-tuning on 8 NVIDIA H100 GPUs, while Stage II freezes the backbone and trains the assessor on one A100 GPU.
Evaluation: 12 benchmarks across five multimodal retrieval task types form the MMEB evaluation setting for SAS.The evaluation reports Hit@1 and measures end-to-end inference latency on one NVIDIA A100 GPU.

5 Main Results

SAS is evaluated against dense models and representative retrieval baselines, showing competitive retrieval quality alongside substantial efficiency gains. The accelerated 7B model can also outperform and outpace a smaller dense model.

Efficiency–Accuracy Trade-off: SAS retains nearly 99% of the dense baseline’s average retrieval performance while reducing FLOPs by up to 66.3%.The reported examples are 71.7% →70.9% for 7B and 68.9% →68.0% for 3B.
Revisiting the Parameter–Latency Trade-off: The SAS-accelerated 7B model is faster than the dense 3B model while outperforming it by about 2.0% in retrieval accuracy.The paper attributes this to stronger representations enabling more frequent early termination for easy queries.
Performance Comparison: The 7B SAS model achieves an average Hit@1 of 70.9% across diverse retrieval tasks and remains competitive with strong recent MLLM-based baselines.Table 4 reports Hit@1 for the compared retrieval systems.
Performance Comparison: SAS consistently outperforms prior methods on fine-grained semantic matching tasks such as VisDial and WebQA.The paper links this performance to complementary token filtering and adaptive depth allocation.

6 Discussion and Ablation

The ablations examine how influence-aware token scoring, selection timing, and adaptive depth allocation affect retrieval efficiency and quality. Together, the results support complementary benefits from reducing token and layer computation.

6.1 Qualitative Analysis of Influence-Aware Token Scoring: Influence-aware scoring separates kept and pruned tokens more clearly than raw [EOS] attention and better captures marginal contribution to the retrieval embedding.The score aggregates perturbation effects across heads and accounts for downstream propagation.
6.1 Qualitative Analysis of Influence-Aware Token Scoring: Influence-aware spatial maps identify broader coherent high-score regions, whereas raw attention highlights only isolated peaks.This indicates that retrieval-relevant evidence can be distributed across multiple jointly contributing tokens.
6.2 Ablation Study: Setting the selection layer l_s = 10 achieves the best trade-off because earlier pruning removes insufficiently mature cross-modal representations and later pruning leaves little room for depth checking.The compared early layers are 6–8 and the late layer is 13.
6.2 Ablation Study: Under 75% visual and 50% textual reduction, SAS retains 68.0% performance, compared with 39.3–60.6% for vision-only methods and 58.3% for random text pruning.The alternatives are FastV, DART, and random text pruning.
6.2 Ablation Study: l_d = 12 gives the best exit-layer trade-off, reaching 50.6% average Hit@1 at the intermediate layer and 69.4% at the final layer.The study compares l_d ∈ {8, 12, 16, 20}; earlier exits produce weaker intermediate representations.
6.2 Ablation Study: Combining evidence selection and depth adaptation yields a 1.44× speedup at 68.0% accuracy, versus 1.26× and 1.17× for the individual components.The combined result indicates complementary reductions in token- and depth-level computation.

7 Conclusion

The paper concludes that SAS combines influence-aware token filtering with difficulty-aware early exit for efficient universal multimodal retrieval. Across multiple benchmarks, it improves inference efficiency while preserving competitive retrieval performance.

Conclusion: SAS integrates influence-aware token filtering and difficulty-aware early exit to reduce redundant token and layer computation in universal multimodal retrieval.Token filtering mitigates quadratic complexity, while early exit allows easy samples to terminate at intermediate layers.
Conclusion: Comparisons with dense MLLM backbones across multiple benchmarks show substantially improved inference efficiency with competitive retrieval performance.

Limitations

The paper identifies practical limits in SAS’s current validation and implementation: hardware execution does not fully realize theoretical savings, and generative workloads remain unevaluated. Its technical derivation also relies on bounded-dynamics assumptions for stability.

Practical limitations: SAS’s prototype uses dense attention kernels, so variable-length sequences can make wallclock gains fall short of theoretical FLOPs reductions.The paper frames this as an engineering gap and suggests length-bucketed batching, fused indexing, and sparse or variable-length attention kernels.
Evaluation scope: SAS is evaluated only on retrieval and embedding workloads, not multimodal generation tasks.Extending it to VQA or captioning introduces additional KV-cache and pruning or exit-location constraints.
Technical basis: The influence score is derived by bounding the projected perturbation caused by removing a token and aggregating the bound across attention heads.The resulting score is used to rank tokens for filtering and is related to minimal first-order contribution to the layer output.

B.3 Global Stability Analysis via Neural ODEs

SAS’s skimming perturbation remains bounded under stated Neural ODE assumptions, while its termination classifier combines uncertainty, modality, and semantic features to support depth adaptation.

Global Stability Analysis via Neural ODEs: Under the K-Lipschitz assumption, the final trajectory divergence remains bounded when pruning minimizes the initial influence-score perturbation.The analysis models residual connections as Euler discretization and applies Gronwall’s Inequality.
Termination Classifier: The termination classifier uses 27 uncertainty and confidence statistics, including top-1 similarity and the top-1/top-2 margin, to capture ranking stability.A larger top-1/top-2 margin typically indicates a decisive prediction, while a narrow margin suggests ambiguity.
Termination Classifier: The resulting 95-dimensional feature vector is fed to an MLP, while signed Log1p transformation and Layer Normalization handle skewed feature distributions.The MLP uses two hidden layers of dimensions 128 and 64, with Layer Normalization, ReLU, and Dropout.
Termination Classifier: The classifier also incorporates a 4-dimensional modality embedding and a 64-dimensional semantic projection of the backbone representation.These features adapt decisions to input type and semantic context without retaining the full embedding dimension.
Termination Classifier: Focal Loss addresses class imbalance by emphasizing ambiguous sufficiency decisions, using α = 0.8 and γ = 3.0.The loss reduces the dominance of easy-to-classify samples during classifier training.

D.1 Selection of the Selection Layer (ls)

The selection-layer ablation identifies ls = 10 as the best trade-off between representation maturity and computational savings, while SAS preserves retrieval performance under aggressive token reduction.

Selection-layer choice: Pruning after layer 12 preserves redundant tokens for too many layers and weakens efficiency gains, despite being safer for representation quality.This establishes the trade-off between delayed pruning and computational savings.
Selection-layer choice: Pruning too early at layers 6 or 8 reduces final Hit@1 to around 66.3%–66.4% versus 69.2%, indicating unstable early representations.The ablation evaluates selection layers {6, 8, 10, 11, 12, 13} at intermediate L12 and final L36.
Selection-layer choice: Selecting tokens at ls = 10 achieves the best overall result and leaves a buffer before the first sufficiency check at ld = 12.The remaining tokens can redistribute attention and consolidate cross-modal semantics after pruning.
Comparison with token-pruning strategies: SAS sustains 68.0% performance with 75% visual and 50% textual token reduction, whereas VisionZip falls from 68.9% to 39.3%.Random text pruning reaches 58.3%; SAS preserves tokens using marginal contribution to the [EOS] retrieval embedding.
Decision-layer choice: The main experiments use ld = 12 because it provides a more stable decision point and better practical early-exit behavior than ld = 10.The two decision layers achieve comparable final-layer performance.

G Detailed Ablation on Supervision Strategies

The supervision ablation shows that single objectives trade off shallow-layer utility against final-layer performance, whereas the hybrid strategy improves both.

Evaluation coverage: The ablation reports intermediate and final Hit@1 across visual retrieval tasks including CIRR, NIGHTS, OVEN, and VisualNews_i2t.Additional visualizations cover VisDial, MSCOCO_t2i, and Wiki-SS-NQ.
Supervision strategies: Using solely Lcon produces 49.4% Hit@1 because of optimization difficulties at shallow layers.This result reflects weak intermediate utility under the single-constraint objective.
Supervision strategies: The Hybrid Strategy reaches 57.2% Hit@1 at L12 and 69.5% at Llast, improving early-exit utility while raising peak final-layer performance.Table 9 compares intermediate and final retrieval accuracy across supervision strategies.
Supervision strategies: Using solely Ldistill over-regularizes the network, reducing final-layer Hit@1 to 69.2%.The ablation identifies a cost to emphasizing distillation alone.