PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects

Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, Huan Li

arXiv:2604.05546v2cs.CL

TL;DR

LVLM inference is hindered by visual token dominance, which creates compute, quadratic-attention, and memory-bandwidth bottlenecks across the pipeline. The survey develops a stage-wise taxonomy covering encoding, prefilling, and decoding, and concludes that efficient progress requires coordinated full-stack optimization. Pilot findings support future directions including hybrid compression, relaxed verification, progressive state management, and hardware-algorithm co-design.

  • Problem

    Visual token dominance creates distinct encoding, prefilling, and decoding bottlenecks, while prior reviews lack an end-to-end account of their interdependence.

  • Method

    The survey organizes efficient LVLM inference by lifecycle stage and by information density, long-context attention, and memory-bandwidth interventions.

  • Results

    Random relaxation retains 81.4% to 97.6% of original output quality across model settings while providing significant speedups.

  • Takeaways & Limitations

    Efficient LVLM inference requires shifting from isolated algorithmic improvements toward hybrid, modality-aware, progressive, and hardware-algorithm co-designed optimization.

  • Takeaways & Limitations

    The survey may omit undocumented proprietary optimizations and gives less depth to document understanding, heterogeneous multi-image scheduling, energy efficiency, and theoretical compression bounds.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) enable sophisticated reasoning over images and videos, yet their inference is hindered by a systemic efficiency barrier known as visual token dominance. This overhead is driven by a multi-regime interplay between high-resolution feature extraction, quadratic attention scaling, and memory bandwidth constraints. We present a systematic taxonomy of efficiency techniques structured around the inference lifecycle, consisting of encoding, prefilling, and decoding. Unlike prior reviews focused on isolated optimizations, we analyze the end-to-end pipeline to reveal how upstream decisions dictate downstream bottlenecks, covering compute-bound visual encoding, the intensive prefilling of massive contexts, and the ''visual memory wall'' in bandwidth-bound decoding. By decoupling the efficiency landscape into the axes of shaping information density, managing long-context attention, and overcoming memory limits, this work provides a structured analysis of how isolated optimizations compose to navigate the trade-off between visual fidelity and system efficiency. The survey concludes by outlining four future frontiers supported by pilot empirical insights, including hybrid compression based on functional unit sensitivity, modality-aware decoding with relaxed verification, progressive state management for streaming continuity, and stage-disaggregated serving through hardware-algorithm co-design. Our literature repository is at https://github.com/SuDIS-ZJU/Efficient-LVLMs-Inference.

1 Introduction

LVLM inference faces visual token dominance across compute, attention, and memory regimes. This survey addresses fragmented prior work with a stage-wise, end-to-end taxonomy connecting upstream choices to downstream bottlenecks.

  • Visual inputs produce orders of magnitude more tokens than text, creating quadratic-attention and visual-memory-wall constraints.A Qwen2.5-VL-72B example exceeds 40K tokens and 13 GB of cache for 20 images.
  • Inference spans compute-bound encoding, quadratic-attention prefilling, and memory-bound decoding, so isolated optimization can shift rather than reduce the end-to-end bottleneck.
  • Prior reviews focus on isolated techniques and lack a holistic account of how upstream encoder decisions affect downstream efficiency.
  • The survey organizes techniques around information density, long-context attention, and memory bandwidth to analyze composition and fidelity-efficiency trade-offs.

2 Preliminaries and Inference Dynamics

LVLM inference connects visual encoding, projection, and autoregressive generation, with visual tokens typically dominating text tokens. The resulting stages exhibit distinct compute, attention, and memory bottlenecks that determine latency and motivate stage-aware optimization.

  • The canonical pipeline encodes visual input into patch embeddings, projects them into LLM latent space, and autoregressively generates text from a joint context.
  • Visual token dominance means Nv ≈576–4,000+ and Nv ≫ Nt, creating the structural imbalance underlying the analyzed bottlenecks.
  • Encoding is compute-bound, while reducing visual tokens lowers prefilling complexity from O((Nv + Nt)^2) to O((N′v + Nt)^2) and shrinks KV cache size linearly.
  • Prefilling is jointly constrained by quadratic attention and KV-cache materialization, with large Nv increasing HBM write traffic.
  • Decoding is strictly memory-bound because each output step repeatedly loads model weights and visual KV states from HBM.
  • The survey therefore organizes methods by reducing encoding latency and token count, mitigating prefilling attention, and optimizing memory-bound decoding.

3 Efficiency Techniques at Encoding

Encoding efficiency reduces upstream visual processing cost and the token density propagated into later stages. The surveyed methods combine architectural redesign, adaptive input reduction, and encoder-side compression to allocate computation toward informative content.

  • The encoding workflow separates architectural optimization of encoders and adapters from input reduction through keyframe selection, adaptive resolution, and token compression.
  • Efficient Vision Encoders: Vision encoders regulate initial visual token density, motivating dynamic, density-aware architectures instead of static fixed-resolution backbones.
  • Efficient Vision Encoders: Efficient image and video encoders use structural reparameterization, distillation, hierarchical compression, hybrid encoding, temporal adaptation, and long-context extensions.
  • Efficient Modality Adapters: Modality adapters are evolving into active information bottlenecks that pack richer semantics into fewer latent tokens rather than preserving one-to-one feature mappings.
  • Keyframe Selection: Keyframe selection removes redundant frames before encoding, using either portable heuristics or learnable policies conditioned on downstream objectives and queries.
  • Adaptive Resolution: Adaptive resolution allocates higher visual fidelity to semantically difficult samples, extending content-adaptive encoding to query-conditional video processing.
  • Encoding-Side Token Compression: Encoder-side compression is prompt-agnostic, using similarity metrics or encoder attention to retain salient visual tokens without textual-query or LLM interaction.

4 Efficiency Techniques at Prefilling

Prefilling efficiency focuses on reducing massive visual contexts and their quadratic attention cost through token compression and sparse attention. The survey emphasizes hybrid, modality-aware strategies that preserve textual fidelity while compressing redundant visual states.

  • Token Compression: Prefilling-side compression reduces visual-token quantity within the LLM backbone’s latent space, directly mitigating quadratic attention costs.These methods use cross-modal semantic signals available after projection and organize optimization around diversity or attention.
  • Token Compression: Diversity-guided methods retain semantically covering subsets by minimizing embedding similarity among highly correlated visual tokens.Clustering and Determinantal Point Processes identify or merge redundant tokens.
  • Token Compression: Attention-guided methods use self-attention as a proxy for token utility, including early-layer pruning, attention-distribution matching, and learnable gating.StreamingVLM additionally uses text and visual attention sinks to maintain reasoning stability over long video contexts.
  • Hybrid Compression: Hybrid systems combine outer-LLM diversity filtering with inner-LLM attention pruning to separate geometric redundancy from semantic reasoning.The integration targets LVLM modal asymmetry by compressing redundant visual states while preserving textual fidelity.
  • Sparse Attention: Sparse attention restricts prefilling computation to salient regions, with modality-aware kernels, chunk-based video attention, and native sparse training extending beyond generic LLM heuristics.The survey concludes that sparsity objectives should be integrated directly into modality-aware training architectures.

5 Efficiency Techniques at Decoding

Decoding is memory-bound because each output step loads model weights and dynamic KV-cache states, creating a visual memory wall when visual contexts are large. The surveyed responses combine modal KV-cache compression, speculative execution, and adaptive reasoning depth.

  • Decoding Bottleneck: Decoding latency is governed by memory bandwidth required to load model weights and dynamic KV-cache states at every output step.The section frames KV-cache compression, speculative execution, and efficient reasoning as three responses to this bottleneck.
  • KV Cache Compression: LVLM-specific KV-cache methods exploit higher visual-token redundancy than textual-token redundancy across token, layer, head, and bit granularities.Token-level methods prune or retrieve visual states to separate the massive prefill context from the active working set.
  • KV Cache Compression: A unified retrieval, pruning, and quantization framework is proposed because current methods typically operate in isolation.The survey reports a pilot exploration of this hybrid paradigm in Appendix D.1.
  • Speculative Decoding: Speculative decoding faces a visual memory wall because processing massive visual KV contexts can erode the draft model’s efficiency gains.Existing LVLM adaptations use visually specialized draft models, distillation, semi-autoregressive heads, or adaptive visual compression.
  • Speculative Decoding: The survey argues that visual tasks permit relaxed verification because rigid exact-match verification ignores semantic flexibility in visual descriptions.This hypothesis is reported as empirically validated in Appendix D.2.
  • Efficient Reasoning: Efficient reasoning dynamically adjusts Chain-of-Thought length, while future work could prune individual steps rather than processing an entire generated chain.Existing approaches mainly use coarse instance-level routing or expansion mechanisms.

6 Challenges and Future Directions

The survey identifies four future directions spanning representation, generation, continuity, and system deployment. Together, they move toward heterogeneous compression, semantic-aware decoding, progressive streaming state, and hardware-algorithm co-design.

  • Overview: The proposed frontiers target representation, generation, and continuity, with end-to-end system co-design integrating them into hardware-aware deployment.The survey presents co-design as necessary for realizing the algorithmic directions cohesively.
  • Representation: Hybrid Compression: Hybrid compression should assign retrieval, pruning, and quantization operators according to the sensitivity of each LVLM component.Uniform compression and budget allocation alone are described as insufficient for heterogeneous entropy.
  • Generation: Modality-Aware Decoding: Modality-aware decoding requires visual draft alignment and relaxed verification to overcome the visual memory wall.The proposed verification shift replaces rigid exact matching with semantic-aware validation.
  • Continuity: The Streaming Pivot: Streaming continuity requires progressive state management across encoding, prefilling, and decoding rather than holistic offline processing.Suggested directions include streaming visual memory management, progressive token compression, and locality-aware KV-cache compression.
  • End-to-End System Co-Design: Hardware-algorithm co-design should map distinct inference stages to specialized hardware while addressing bandwidth saturation and pipeline bubbles.Disaggregated serving architectures motivate architectural tailoring combined with semantic-aware predictive scheduling.

7 Literature Selection Protocol

The survey curates literature through broad exploration, targeted venue and date filtering, and bidirectional citation tracking. This protocol combines thematic discovery with focused screening and citation-based coverage expansion.

  • Broad Exploration: Broad exploration searched Google Scholar for major themes, representative architectures, and terminology related to LVLMs and efficient inference.This phase established the initial candidate pool.
  • Targeted Filtering: Targeted filtering screened papers from major NLP, machine learning, AI, and computer vision venues, plus relevant arXiv preprints.The survey primarily focused on work published from 2020 to early 2026.
  • Bidirectional Citation Tracking: Bidirectional citation tracking followed references backward from foundational LLaVA and BLIP-2 papers and forward to recent extensions and Qwen-series systems.This step was used to improve literature coverage.

8 Positioning in the Evolving Landscape

The survey positions itself against LLM- and MLLM-centric reviews by analyzing efficient LVLM inference across the complete encoding, prefilling, and decoding pipeline. Its distinctive contribution is a bottleneck-aware, stage-specific synthesis connecting computation, memory traffic, cache locality, and sequence length.

  • Existing LLM-centric surveys organize efficiency research across data, model, and system levels, often linking algorithmic design to hardware bottleneck diagnosis.
  • MLLM-centric surveys commonly emphasize data preparation, architectural taxonomies, edge or resource-constrained deployment, modality-specific models, or token compression.
  • This survey instead provides a stage-aware taxonomy spanning encoding, prefilling, and decoding across the end-to-end LVLM inference pipeline.
  • Its bottleneck-aware analysis tracks the transition from compute-bound encoders to bandwidth-bound decoding through memory traffic, cache locality, and sequence length.
  • The survey synthesizes design principles around dynamic, density-aware mechanisms and stage-disaggregated serving architectures as directions for future research.

9 Conclusion

The conclusion systematizes efficient LVLM inference across encoding, prefilling, and decoding, emphasizing that visual token dominance shifts bottlenecks across the pipeline. It identifies future algorithmic directions centered on adaptive compression, verification, state management, and serving.

  • Efficient LVLM inference is organized around encoding, prefilling, and decoding, with bottlenecks shifting from compute-bound visual encoding to memory-bound autoregression.
  • The analysis identifies three modality-centric shifts: hybrid orchestration instead of uniform compression, semantic approaches instead of rigid verification, and adaptive mechanisms across inference.

11 Limitations

The survey’s coverage is constrained by omitted proprietary optimizations and by emphasis on image and video redundancy, while several efficiency dimensions remain outside its scope.

  • Undocumented optimizations in rapidly released proprietary models may be omitted from the survey.
  • Document understanding and heterogeneous multi-image scheduling receive less depth because their discrete token structures diverge from the work’s continuous temporal focus.
  • The survey focuses on latency and memory throughput, leaving energy efficiency and theoretical compression bounds for future investigation.

12 Ethical Considerations

Efficiency-oriented LVLM inference can reduce energy consumption and broaden access, but lossy compression may degrade safety guardrails and increase hallucination rates. The survey therefore calls for evaluation protocols that monitor these ethical dimensions alongside latency.

  • Efficiency techniques aim to reduce energy consumption and democratize access to multimodal systems.
  • Lossy compression can degrade safety guardrails and increase hallucination rates.
  • Evaluations should monitor safety and hallucination risks alongside latency metrics.

E.1 Hardware Specifications

Using an NVIDIA A100-SXM4-80GB with FP16 tensor-core assumptions, the Roofline analysis characterizes LVLM inference as three distinct workload regimes: compute-bound encoding, mixed prefilling, and memory-bound decoding.

  • Hardware Setup: The Roofline model uses an NVIDIA A100-SXM4-80GB as its reference hardware platform with FP16 tensor-core operations.The hardware specifications are summarized in Table 11.
  • Decoding: Decoding has arithmetic intensity Ia ≈1.0 and is strictly memory-bound because each generated token requires loading the model’s full weight matrix.Its performance is determined by memory-bandwidth utilization.
  • Prefilling: Prefilling has representative arithmetic intensity Ia ≈160.0 near the hardware ridge point at 153.0, creating a mixed compute-and-memory bottleneck.At context lengths around Nv+Nt ≈512, performance is sensitive to both data movement and arithmetic throughput.
  • Encoding: Encoding has arithmetic intensity Ia ≈1200.0, nearly an order of magnitude above the ridge point, classifying visual feature extraction as strictly compute-bound.The workload consists of dense, highly parallel Vision Transformer computations over image patches.
Loading 2604.05546v2…