PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Inference Acceleration for Video Large Language Models: Methods, Trade-offs, and Evaluation

Yuqi Li, Chuanyue Yan, Neil Rodrigues, Xianglong Wang, Xiaoqin Feng, Junming Liu, Xi Xiao, Zhongwei Cheng, Wei Li, Lin Chen, Yingli Tian

arXiv:2610.36327v1cs.CV

TL;DR

Long video sequences impose computational, latency, and memory costs, while useful evidence is unevenly distributed across frames. This survey organizes training-free acceleration by intervention stage and evaluates how stage choice balances evidence retention and efficiency. It concludes that reduced tokens or local latency gains do not necessarily improve end-to-end efficiency.

  • Problem

    Longer video inputs increase processing costs, yet relevant evidence can be brief and unevenly distributed, and the combined effects of intervention position on evidence preservation and realized efficiency remain less explored.

    Source: §1 · p. 1

  • Method

    The survey builds a stage-wise taxonomy, analyzes intervention trade-offs, and combines prior-study evidence with shared-protocol experiments using QLB to assess accuracy–latency performance.

    Source: §1 · p. 2, §3 · p. 2, §4 · p. 6

  • Results

    Across the pipeline, earlier compression offers broader computational savings but risks irreversible evidence loss, and reduced tokens or local latency improvements do not necessarily yield end-to-end efficiency.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Future systems should jointly manage retention, retrieval, and cache updates while maintaining timely responses to incoming queries.

    Source: §5 · p. 8

    The controlled experiments use one backbone, a fixed frame budget, and one GPU platform; PTQ evidence is largely image-based, and system measurements lack complete latency breakdowns.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Video Large Language Models (VLMs) have demonstrated strong capabilities in video understanding and reasoning, but processing long visual sequences incurs substantial computational, latency, and memory overhead. This survey examines training-free inference acceleration, focusing on reducing inference costs while preserving task-relevant spatiotemporal evidence. We organize existing methods by their intervention stages in the inference pipeline, covering input, visual encoding, projector, prefill, and decoding. Post-training quantization (PTQ) is discussed separately as a cross-stage extension with varying calibration requirements. We analyze how intervention position affects evidence preservation and computational savings, and why local efficiency improvements may not translate into end-to-end acceleration. We complement this analysis with quantitative evidence from prior studies and our own experiments under a shared evaluation setting to assess accuracy-latency trade-offs.

1 Introduction

Longer or higher-resolution video inputs increase visual and multimodal processing costs, while task-relevant evidence is unevenly distributed across frames. The survey organizes training-free acceleration by intervention stage and examines how stage choice affects evidence preservation and end-to-end gains.

1 Introduction

  • Training-free acceleration must reduce redundant computation while preserving task-relevant spatiotemporal evidence, because brief actions can be critical and similar frames can differ in relevance.The survey addresses a gap in understanding how intervention position jointly affects evidence preservation and realized efficiency.Source: §1 · pp. 1–2
  • Earlier interventions can avoid more downstream computation, whereas later stages exploit richer representations; local savings do not necessarily yield end-to-end acceleration.The balance between savings and available information makes intervention position central to the survey’s analysis.Source: §1 · pp. 1–2
  • The survey organizes methods across input, visual encoding, projector, prefill, and decoding stages, and treats PTQ separately as a cross-stage extension.It also synthesizes prior studies and conducts shared-protocol experiments using Quality–Latency Balance (QLB) to assess accuracy–latency performance.Source: §1 · p. 2

2 Preliminaries

Video VLMs process visual representations alongside textual queries, then use prefill and decoding to produce output. The survey defines training-free acceleration by the absence of additional training or fine-tuning for the acceleration capability.

2.1 VLMs for Video Understanding

  • A typical video VLM encodes video through a vision encoder and projector, then combines visual representations with embedded query tokens for LLM processing.Prefill builds the KV cache and produces the first token; autoregressive decoding accesses and updates that cache for subsequent tokens.Source: §2.1 · p. 2

2.2 Why Training-Free

  • Training-free methods avoid retraining backbone-specific acceleration modules, which can be costly to maintain as VLM architectures evolve.The survey notes that such methods can be more readily applied to existing models.Source: §2.2 · p. 2
  • Training-free acceleration requires no additional training or fine-tuning for its acceleration capability; frozen auxiliary models are allowed, but their runtime overhead must be counted.Methods requiring acceleration-specific training fall outside the survey’s core scope.Source: §2.2 · p. 2

3 Taxonomy

The taxonomy groups training-free acceleration by where it intervenes in the inference pipeline, distinguishing core stages from PTQ’s cross-stage extension. Across stages, methods trade computational savings against the information retained for downstream reasoning.

3 Taxonomy

  • The taxonomy covers input, encoding, projector, prefill, and decoding interventions, with PTQ treated separately as a cross-stage extension.The stage-wise organization distinguishes computational savings from information available downstream; PTQ techniques may involve calibration or optimization beyond the core training-free scope.Source: §3 · p. 2, §3.6 · p. 5

3.1 Input Stage

  • Keyframe selection reduces processing by retaining fewer frames, using either query-agnostic or query-aware criteria, but limits reasoning to retained temporal evidence.Query-aware scoring or search costs can offset savings; spatial adaptation also reduces visual computation but may obscure fine detail or needed context.Source: §3.1.1 · pp. 2–3, §3.1.2 · pp. 3–4

3.2 Encoding Stage

  • Encoding methods merge redundant features or reuse stable ones to reduce visual computation, but merging may obscure subtle motion and caching risks stale features or matching overhead..Source: §3.2 · p. 4

3.3 Projector Stage

  • Projector-stage selection reduces LLM context using attention or feature-derived importance, while aggregation merges similar tokens; selection can miss query evidence, and merging can blur distinctions or add overhead.These methods act after encoding, so they reduce prefill costs but do not save encoding computation already incurred.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

3.4 Prefill Stage

  • Prefill methods select or aggregate tokens and select or aggregate KV states, reducing later computation or cache costs while risking evidence loss and adding overhead.Token pruning leaves earlier layers to process the original sequence; cache selection may lose historical evidence, and cache aggregation cannot recover prefill computation already spent.Source: §3.4 · p. 4, §3.4.1 · p. 4, §3.4.2 · p. 5

3.5 Decoding Stage

  • Decoding methods use speculative drafting, KV selection, or retrieval to reduce generation costs, but speed depends on drafting and verification overhead, cache access, and retrieval timing.Selection may discard evidence needed by later tokens; retrieval preserves long-range context but adds overhead.Source: §3.5 · p. 5, §3.5.1 · p. 5, §3.5.2 · p. 5

3.6 Post-Training Quantization

  • PTQ covers weight-only, weight-activation, and KV-cache quantization; lower precision can reduce memory demands, but realized speed depends on execution support and overhead.Weight speedups depend on kernel support and dequantization costs; activation quantization benefits require efficient low-precision execution and sufficient accuracy retention, while KV quantization overhead may erase latency gains.Source: §3.6 · p. 5, §3.6.1 · pp. 5–6, §3.6.2 · p. 6

4 Evaluation

The evaluation compares trade-offs across four inference questions, finding that stage savings, evidence retention, and realized system gains do not align automatically.

Evaluation setup

  • The survey combines controlled experiments with prior-study comparisons, using Quality–Latency Balance to compare matched task quality and TTFT or TPOT.Controlled experiments use LLaVA-OneVision-7B with 32 frames on an NVIDIA H100 and average accuracy and TTFT equally across four benchmarks; cross-study comparisons have differing models, budgets, hardware, and protocols.Source: §4 · p. 6

RQ2

  • For VisCache, reducing retention from 40% to 28% lowers TPOT from 56.82 to 49.50 ms/token and raises EgoSchema accuracy from 52.0% to 55.0% (QLB 92.9 to 104.9).DyCoke lowers TPOT from 42 to 31 ms/token as VideoDC score falls from 3.30 to 3.20 (QLB 111.6); retention ratio alone does not determine preserved evidence's utility.Source: Table 2 · p. 7, §4 · p. 7

RQ1

  • EarlyTom reduces mean TTFT by 45.7% while retaining 97.7% of baseline accuracy (QLB 126.6), whereas Curv. merge4 retains 99.6% accuracy with a 19.9% TTFT reduction.VidCom2 reaches QLB 124.8, while FrameFusion increases TTFT by 12.9% despite token reduction; differing retention budgets and implementations prevent isolating intervention position as the sole cause.Source: §4 · p. 7

RQ3

  • 97.3% of FP16 performance is retained by VLM-PTQ on LLaVA-OneVision-7B at W3A16, versus 92.0% for VLMQ and 90.2% for MBQ.In separate five-benchmark comparisons, QIG improves retention from 96.0% to 98.2%, while MASQuant increases it from 91.9% to 92.6% under W4A8; these image-oriented evaluations lack matched latency or memory measurements for video inference.Source: Table 3 · p. 7, §4 · p. 7

RQ4

  • 51.2% prefill and 46.6% generation latency reductions correspond to only 13.6% lower end-to-end latency for V-CAST, with a 1.0-point accuracy loss.FlashVID lowers both stage latencies but raises peak memory by 84.0%; VidCom2 is faster than HoliTom in both reported stages yet has slightly higher end-to-end latency.Source: §4 · p. 7, Table 4 · p. 8

5 Challenges and Future Work

The future-work agenda shifts from isolated compression toward evidence-aware methods that coordinate stages, memory, and system-level efficiency.

Evidence-Aware Compression

  • Evidence-aware compression should coordinate temporal coverage, task relevance, and selective evidence recovery without excessive selection overhead.The survey points to evidence recovery and semantic preservation, adaptive token budgets based on frame importance and query relevance, and question decomposition as complementary directions.Source: §5 · p. 8

Streaming Memory and Decoding

  • Streaming systems should jointly manage retention, retrieval, and cache updates while maintaining timely responses to incoming queries.Compressed visual memory, persistent memory compression, and selective retrieval are cited as explored approaches.Source: §5 · p. 8

End-to-End System Co-Design

  • Future system co-design should coordinate token compression, KV management, and low-precision execution, measuring latency and memory alongside task quality.Token selection, memory access, and kernel inefficiency can offset computational gains.Source: §5 · p. 8

6 Conclusion

The survey concludes that training-free VLM acceleration must balance computational savings with preservation of task-relevant spatiotemporal evidence. It calls for evidence-aware, cross-stage optimization judged by reliable video understanding and realized inference efficiency.

Core conclusion

  • Training-free acceleration must balance computational savings against preserving task-relevant spatiotemporal evidence, rather than merely reducing visual tokens.Intervention position determines which computations can be avoided and what information remains for downstream reasoning.Source: §6 · p. 8

Intervention trade-offs

  • Earlier compression offers broader potential savings but risks irreversible evidence loss, while later intervention uses richer representations but inherits upstream costs.Source: §6 · p. 8

Evaluation goal

  • Reduced token counts and local latency improvements do not necessarily yield end-to-end efficiency; progress should target reliable video understanding and realized efficiency.Source: §6 · p. 8

Limitations

The authors bound their findings to available methods and specific experimental settings, cautioning against universal performance rankings.

Coverage

  • The survey may miss recent or proprietary training-free acceleration methods.Source: Limitations · p. 9

Controlled experiments

  • Controlled experiments use one backbone, a fixed frame budget, and one GPU platform, limiting generalizability.Source: Limitations · p. 9

Evidence scope

  • Published comparisons may use different protocols; PTQ evidence is largely image-based, and system measurements lack complete latency breakdowns.Findings should be interpreted within their evaluation settings rather than as universal performance rankings.Source: Limitations · p. 9

Ethical Considerations

The paper notes that lossy compression may discard critical evidence and increase unreliable predictions, and recommends evaluating these risks alongside accuracy and latency. It defines metrics for task quality, compression, and realized inference cost, including a baseline-normalized quality–latency balance.

Ethical Considerations

  • Lossy compression may discard critical evidence and increase unreliable predictions, so evaluation should assess these risks alongside accuracy and latency.The survey uses publicly available models and benchmarks and documents evaluation settings and comparison protocols for reproducibility.Source: Ethical Considerations · p. 9

A.1 General Evaluation Metrics

  • Task quality, compression, and realized inference cost are assessed using benchmark scores, token or KV-cache retention and precision, and TTFT, TPOT, E2E latency, and peak memory.QLB summarizes the quality–latency trade-off under matched settings; the matched baseline scores 100, and higher values do not imply comparability across settings.Source: §A.1 · p. 16

A.2 Video Understanding Benchmarks

The appendix describes benchmarks spanning several video-understanding tasks and cautions that results should be compared only under matched settings. It also notes that host-model differences affect inference workloads and that literature results are not a unified cross-method ranking.

A.2 Video Understanding Benchmarks

  • Video-MME, EgoSchema, LongVideoBench, and MVBench are used in controlled evaluation, while VideoDC adds a description task to literature-based decoding comparisons.Benchmarks vary in task format, video duration, and evaluation protocol; results should be compared only under matched settings.Source: §A.2 · p. 16

A.3 Host VLMs and Language Backbones

  • Host VLMs differ in visual tokenization, context capacity, and architecture, so similar retention budgets can represent different inference workloads.Reported methods span Video-LLaVA, LLaVA-OneVision, and Qwen-VL models, and their results are not a direct cross-method ranking.Source: §A.3 · p. 16

A.4 Comparison Protocol

  • Comparisons prioritize matched models, benchmarks, input budgets, and measurement protocols; compression ratios alone are not treated as system-performance evidence.TTFT covers model-side visual encoding and prefill but excludes external video preprocessing and subsequent decoding; stage-level and end-to-end measurements are interpreted separately unless a full timing breakdown is available.Source: §A.4 · p. 17

B Scope and Boundaries of Training-Free Acceleration

The paper defines training-free acceleration by the absence of newly trained acceleration-specific parameters, while treating calibration-dependent PTQ separately. It also argues that streaming and offline workloads impose different constraints, and reports workload-dependent performance trade-offs for streaming adaptation.

B Scope and Boundaries of Training-Free Acceleration

  • Training-free acceleration modifies a pretrained VLM’s inference procedure without training additional acceleration-specific parameters; newly trained selectors, predictors, or draft models are outside the core scope.Pretrained auxiliary models may be used without further training, but their inference overhead must be accounted for; calibration-dependent PTQ is treated separately.Source: B Scope and Boundaries of… · pp. 17–18

C Workload Shifts: Streaming and Multi-Turn Inference

  • Streaming and multi-turn workloads change how evidence is accessed, retained, and reused, so conventional accuracy and single-query latency are insufficient to characterize deployment efficiency.Streaming acceleration should also be assessed through causal task performance, memory growth, processing throughput, and response latency.Source: C Workload Shifts: Streaming and… · p. 18, §C.1 · p. 19

C.1 Offline-to-Streaming Workload Shift

  • Offline VideoQA assumes the complete video is available before a query, whereas streaming requires causal processing, bounded historical memory, and frame processing that keeps pace with incoming video.Meeting the sustainable processing-rate condition alone does not guarantee bounded memory or low response latency.Source: §C.1 · p. 18
  • On Qwen2.5-VL-7B, HERMES improves StreamingBench and OVO-Bench by 6.13 and 6.93 points, respectively, while Video-MME decreases by 3.89 points.On LLaVA-OneVision-7B, OVO-Bench rises from 53.35 to 58.27, while offline scores change only modestly; the results indicate a workload-dependent trade-off.Source: §C.1 · p. 19

C.2 Single-Turn to Multi-Turn Interaction

Across the included methods, acceleration must balance computation reuse and reduced decoding costs against evidence retention and output fidelity. Reported results show that accepted progress and decoding speedup need not rise proportionally, while relaxed verification does not preserve the original output distribution.

C.2 Single-Turn to Multi-Turn Interaction

  • For repeated queries on a fixed video, acceleration must balance computation reuse against preserving relevant visual and textual evidence as dialogue history adds context and memory costs.STTM performs query-agnostic spatiotemporal merging, while ReKV reuses previously computed KV states and retrieves query-relevant historical information; aggressive reuse or eviction can discard evidence needed later.Source: §C.2 · p. 19
  • Multi-turn evaluation yields higher METEOR and GPT4-Score values for all listed models, while computation and memory costs of maintaining dialogue context remain unevaluated.For GPT-4o, METEOR rises from 28.81 to 36.84 and GPT4-Score from 45.37 to 60.70.Source: §C.2 · p. 19

D Speculative Decoding for VLM Acceleration

  • Speculative decoding lets a draft model propose multiple tokens for parallel target-model verification, but VLM drafting can be costly because both models process long visual contexts.Effective acceleration depends on draft cost, draft–target alignment, verification overhead, and accepted progress.Source: D Speculative Decoding for VLM… · p. 20

D.1 Draft–Verify Efficiency

  • Sparse-to-Dense combines sparse drafting with dense verification, while SpecVLM prunes draft-side visual tokens and ParallelVLM overlaps drafting and verification.The methods expose levers of reduced draft computation, overlapping execution, and increased accepted progress; their combined effect must be judged by measured decoding efficiency, not acceptance alone.Source: §D.1 · p. 20

D.2 Empirical Trade-offs

  • On Qwen2.5-VL-32B/7B, LVSpec raises accepted length from 3.29 to 7.76 and speedup from 2.00× to 2.70×, with 99.6% task-performance retention.On LLaVA-OneVision-72B/7B, speedup rises from 2.38× to 2.94× with 99.8% retention; relaxed verification does not guarantee preservation of the original output distribution.Source: §D.2 · p. 20
  • For LLaVA-OneVision-72B/7B, ParallelVLM improves mean accepted length by 54.4% but speedup by 22.6%; on Qwen2.5-VL-32B/7B, the respective gains are 2.2% and 14.7%.These matched comparisons show accepted progress and decoding speedup are not proportional; the source attributes the difference to overlapping execution.Source: §D.2 · p. 20

D.3 Evaluation and Net Gain

Reported self-speculative decoding settings sometimes have speedups below 1×, indicating that drafting and verification overhead can exceed computation saved. Evaluating net gain also requires accounting for output length, iteration costs, and protocol boundaries.

D.3 Evaluation and Net Gain

  • Naive self-speculative decoding reached 0.88×, 0.94×, and 0.86× on three Qwen2.5-VL tasks; a LLaVA-OneVision configuration reached 0.65× on MovieChat (Ji et al., 2026; Song et al., 2024).The tasks were Video Detail Caption, Video Detail Description, and MovieChat, respectively; drafting and verification overhead can exceed computation saved.Source: §D.3 · pp. 20–21
  • Cost per committed output token captures drafting, verification, synchronization, and control overhead while accounting for execution overlap.The diagnostic uses the number of tokens committed per iteration and avoids adding draft and verification times when they execute concurrently.Source: §D.3 · p. 21
  • For fixed TTFT and TPOT, longer responses increase the share of request time available for decoding-side optimization, but the simplified fraction excludes system overhead.Video preprocessing and scheduling may remain outside this quantity, so it is not a measured end-to-end latency fraction.Source: §D.3 · p. 21
  • Speculative decoding evaluations should report model configurations, visual budgets, speculative windows, accepted length, throughput or TPOT, output length, and verification fidelity.Where possible, also measure drafting and verification costs, TTFT, and request-level end-to-end latency.Source: §D.3 · p. 21
  • Table 10 comparisons apply only within matched blocks; Table 7 results from different hosts or protocols are not direct rankings, and Tables 8–9 use distinct scopes or metrics.Table 8 offline and streaming scores are comparable only within the same benchmark; Table 9's dialogue-context difference measures neither acceleration nor a causal effect of dialogue history.Source: Table 7 · p. 22, Table 10 · p. 23, Table 8 · p. 23

Related papers

Loading 2610.36327v1…