PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

Killian Steunou, Yannis Tevissen, Mounîm A. El Yacoubi

arXiv:2609.10355v1cs.CVcs.CLcs.MM

TL;DR

VideoLLMs deliver broad video and audiovisual understanding but face inference costs that grow with frame count, modality inputs and context length. This survey organizes efficiency mechanisms by pipeline stage and assembles controlled and heterogeneous accuracy–cost evidence. Across the reviewed evidence, retaining roughly one quarter of the visual-token budget often preserves near-baseline accuracy, while comparisons remain protocol-dependent and cross-stage gains cannot generally be multiplied.

  • Problem

    VideoLLMs incur substantial computation and memory costs as frame count and context length increase, motivating methods that reduce inference cost without sacrificing semantic performance.

  • Method

    The survey categorizes visual and audiovisual efficiency mechanisms across frame selection, encoding, connector reduction, LLM execution and state, comparing shared-protocol results separately from heterogeneous evidence.

  • Results

    Across heterogeneous evidence, retaining roughly one quarter of the visual-token budget often preserves near-baseline accuracy, with achievable reductions depending on host model, task and evaluation protocol.

  • Takeaways & Limitations

    Efficiency is a system-level trade-off among semantic performance, input coverage, compute, latency and memory, and reducing LLM prefilling or cache costs can make vision encoding the limiting stage.

  • Takeaways & Limitations

    Cross-method evidence is constrained by differing backbones, baselines, token budgets and protocols, while separate stage speedups require compatibility checks before being combined.

Abstract

from arXiv · show

Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.

I. INTRODUCTION

VideoLLMs extend video understanding across visual and audiovisual tasks but incur substantial compute and memory costs. This survey organizes efficiency mechanisms across the pipeline and distinguishes controlled accuracy–cost evidence from heterogeneous comparisons.

  • Scope and motivation: VideoLLMs combine video representations with textual prompts to support captioning, question answering, retrieval, grounding and summarization.The surveyed systems may encode frames alone or synchronized audio alongside visual inputs.
  • Scope and motivation: Hundreds of high-resolution frames, multimodal inputs and long temporal contexts make video encoders and language backbones costly in computation and memory.Efficient VideoLLMs therefore target parameter count, FLOPs per input, latency and memory.
  • Survey organization: Efficiency mechanisms operate across frame selection, vision encoding, connector compression, LLM processing and audiovisual token management.The survey connects upstream temporal coverage and encoding cost to downstream token and memory budgets.
  • Positioning and contributions: Unlike broader surveys, this work includes frame selection and efficient video encoders alongside token-compression mechanisms and examines audiovisual efficiency explicitly.It also compares methods under shared hosts and input settings where possible, separating those results from heterogeneous cross-paper evidence.
  • Survey organization: The survey defines efficiency system-wide as preserving or improving semantic performance while reducing parameter count, FLOPs per input, latency or memory for a task and hardware regime.Power and energy are relevant but rarely reported in the surveyed literature.

II. PRELIMINARIES: SURVEY SCOPE, TASKS AND EVALUATION PROTOCOLS

The survey selects efficiency-focused VideoLLM research by concrete inference-cost effects, covering relevant antecedents, tasks and benchmark protocols. Its evaluation framework emphasizes scope boundaries and comparable input and model settings.

  • Survey scope: A method enters the taxonomy when it targets a pipeline mechanism and reports effects on parameters, FLOPs, retained tokens, latency or memory.The scope focuses on VideoLLMs since late 2022 while retaining earlier sampling and encoder methods that remain current components or direct antecedents.
  • Survey scope: The screening process retained 125 papers from several hundred candidates identified through title, abstract and full-text review.Papers may appear in multiple families when they reduce cost at several stages.
  • Tasks and benchmarks: The survey covers classification, grounding, captioning, retrieval, question answering and dialogue over video with optional synchronized audio and derived text.Task families use established datasets and metrics, while VideoLLM benchmarks emphasize temporal reasoning, long-context comprehension and modality ablations.
  • Tasks and benchmarks: The main efficiency comparisons concentrate on MVBench, Video-MME, EgoSchema and LongVideoBench because these benchmarks recur across the surveyed methods.Table I summarizes the benchmark modalities, durations, video counts, question counts and formats used in the comparisons.

III. VIDEOLLM ARCHITECTURES AND COMPUTATIONAL BOTTLENECKS

VideoLLMs use an encoder–connector–LLM pipeline, with audiovisual systems adding a separate audio path. Computational cost grows from frame and modality encoding through connector outputs, LLM context length, prefilling, decoding, and KV-cache storage.

  • Architecture: VideoLLMs combine sampled-frame encoders, connectors, and pretrained LLMs; audiovisual variants add synchronized audio encoders and connectors.The surveyed architecture is restricted to Video Embedder × LLM systems, whose continuous representations are mapped into the LLM input space.
  • Architecture: Prompting lets one backbone support captioning, question answering, retrieval, temporal grounding, and summarization without task-specific heads.Systems differ mainly in encoders, connectors, language backbones, target video length, and audio use.
  • Scaling factors: Frame count and resolution determine encoder cost and modality-token production, while connector compression controls how many tokens enter the LLM.For frame-wise ViTs, spatial patches per frame are determined by resolution and patch size; temporal tubelets replace T with T/τ.
  • Scaling factors: LLM computation and KV-cache memory scale with context length L = Nt + e Nv + e Na, including text, retained visual tokens, and retained audio tokens.A joint Q-Former instead yields L = Nt + Nq.
  • Bottlenecks: Long-video systems may process hundreds of frames or minute-long clips, making the encoder dominant unless frames are subsampled or pooled.Audio cost varies with temporal density, convolutional stride, pooling, and architecture, and can become material in long audiovisual inputs.
  • Bottlenecks: Prefilling has quadratic attention-score scaling in L, while decoding attends over the cached prefix at O(Ld) per layer and is often constrained by memory traffic.KV-cache memory scales with batch size, decoder layers, context length, key/value width, and bytes per stored element.

IV. TAXONOMY OF EFFICIENCY MECHANISMS IN VIDEOLLMS

The survey organizes efficiency mechanisms by the pipeline stage where they reduce VideoLLM inference cost. Audiovisual methods are included when audio compression or audio-guided visual selection directly lowers the joint workload.

  • Taxonomy: The taxonomy covers input construction and selection, encoder computation, encoded representations and connectors, and LLM execution and state.These stages follow the encoder–connector–LLM decomposition used throughout the survey.
  • Taxonomy: Audiovisual mechanisms enter the taxonomy when they compress audio tokens or use audio to guide visual selection.The inclusion criterion is direct reduction of VideoLLM inference cost.

A. Comparison Protocol

The comparison protocol separates analytical indicators from system-level measurements and records the input and accounting conditions needed to interpret each result. Because papers use heterogeneous modalities, budgets, hardware, and protocols, cross-paper comparisons remain indicative.

  • Measurement scope: Analytical indicators include parameter count, FLOPs per input, and retained-token count or ratio, with stage-specific accounting boundaries.Vision-encoder comparisons pair encoder size and GFLOPs with clip configuration, while connector comparisons report host-LLM size.
  • Measurement scope: Measured runtime depends on operator mix, parallelism, memory access, implementation, batch size, sequence length, precision, device, and software stack.System-level measurements include latency, throughput, and peak memory; streaming systems additionally report processing rate or response latency with bounded memory.
  • Comparability: Frame counts, resolutions, modalities, and temporal budgets are not standardized across papers, so equal GFLOPs claims may describe different inputs.Frame-sampling and streaming methods report accuracy against frame or time budgets, but these budgets are not standardized.
  • Comparability: The survey uses only explicitly reported values, records model variants and input settings, and does not infer FLOPs or latency from architecture alone.Token-retention budgets are not converted into FLOPs, and cross-paper comparisons are treated as indicative evidence.
  • Taxonomy: The taxonomy assigns each mechanism to the pipeline stage whose computation it removes, covering input selection, encoder computation, connectors, and LLM execution or state.Methods can appear in multiple families when they reduce cost at several stages.

B. Input Construction and Selection

Input construction reduces cost before encoding by controlling which frames, resolutions, patches, or layouts enter the pipeline. Selection ranges from fixed temporal coverage to content-based, learned, query-conditioned, and generative strategies.

  • Overview: Frame sampling reduces processed frames T, lowering encoder cost and visual tokens while also reducing LLM prefilling through context length.Discarded frames cannot be recovered downstream.
  • Fixed coverage sampling: Uniform or strided sampling is a deterministic, model-free baseline, while TSN samples one snippet from each of K temporal segments for fixed-budget coverage.Controlled evaluation finds that sampling choices alone can change video-QA results, and uniform sampling can be strongest for some small VLMs on Video-MME.
  • Content-based coverage: Content-based methods partition videos or use proxy embeddings to allocate samples where temporal redundancy makes fixed windows inefficient.KTS-based approaches support long-form classification and temporal localization.
  • Learned query-free selection: Learned query-free samplers adapt frame selection without an inference-time user query, using visual policies, scorers, or predicted future utility.AdaFrame adds early stopping, while PEEK distills caption-conditioned teacher rankings into a visual temporal scorer.
  • Query-conditioned selection: Query-aware methods score frame–text relevance while enforcing diversity or coverage under a token budget.LDDR also allocates per-frame resolution and linearizes determinantal-point-process selection from quadratic to linear complexity in frame count.
  • Learned and generative selectors: Generative and reinforcement-learned selectors optimize relevance, coverage, and redundancy for long videos or transfer selection across multiple LLM hosts.GenS uses a separate VideoLLM, HFS uses Gumbel-Softmax with student–teacher learning, and Qin et al. use a 0.4B plug-in selector.

2) Patch, Resolution, and Input-Layout Budgeting:

Input budgeting reduces video cost by trading spatial detail and temporal coverage against a fixed token or compute budget. Selection quality must be assessed with downstream accuracy and scorer overhead, not frame count alone.

  • Patch, Resolution, and Input-Layout Budgeting: Q-Frame and LDDR allocate per-frame resolution by relevance, while F2C trades spatial resolution for longer clips under a fixed token budget.TS-LLaVA combines downsampled frames into a thumbnail grid, then samples additional encoded tokens later in the pipeline.
  • Patch, Resolution, and Input-Layout Budgeting: AutoGaze selects multi-scale patches before the ViT, while MeToM merges low-information regions using codec residual energy.VATT earlier established that sparse input patches and audio tokens can reduce encoder GFLOPs as the drop rate increases, with an accuracy trade-off.
  • Patch, Resolution, and Input-Layout Budgeting: Query-aware sampling can gain up to 5 points on LongVideoBench over uniform sampling when questions identify sparse evidence, but benefits largely disappear on Video-MME.TSPO leads the shared table with a 3.5M-parameter selector, while reduced-budget selectors match the 64-frame uniform reference with 32 frames.
  • Patch, Resolution, and Input-Layout Budgeting: Frame-sampling comparisons require a fixed downstream model, frame budget, benchmark and split because otherwise sampler, representation, connector and LLM capacity are confounded.The survey therefore restricts its main table to methods sharing a LLaVA-Video-7B, approximately 64-frame protocol.
  • Patch, Resolution, and Input-Layout Budgeting: Encoder-efficiency comparisons must keep architectural antecedents evaluated on recognition or retrieval separate from integrated VideoLLM results.The survey distinguishes efficient architectures, intermediate-token reduction, and computation reuse or substitution.

1) Efficient Encoder Architectures and Operators:

Efficient encoder architectures reduce feature-extraction cost through lightweight convolutional, state-space, compact, sparse, token-merging and temporal-reuse mechanisms. Reported gains vary with evaluation views, task protocols and whether savings extend beyond the encoder.

  • Efficient Encoder Architectures and Operators: TSM adds a parameter- and FLOP-free channel shift to a 2D CNN, while X3D expands a small image model along temporal, spatial, width and depth dimensions.These convolutional designs target temporal modeling at low compute.
  • Efficient Encoder Architectures and Operators: VideoMamba reports 6× higher throughput and 40× lower GPU memory than TimeSformer-Ti at 64 frames.The comparison uses an A100-80G and batch size 128; state-space encoding replaces quadratic self-attention with linear-time processing.
  • Efficient Encoder Architectures and Operators: LiteFrame cuts end-to-end latency by 35% relative to InternVL3-8B while processing 8× more frames.MoE-ViE activates 1.1B of 3.5B parameters per token and reaches roughly three quarters of the latency of a dense encoder 1.7× its size.
  • Efficient Encoder Architectures and Operators: ToMe raises ViT-L video throughput by 2.2× with a 0.2%–0.3% accuracy drop through token merging between encoder blocks.EarlyTom merges frame features during encoding and selects spatial tokens after encoding, spanning encoder and connector stages.
  • Efficient Encoder Architectures and Operators: ResidualViT reduces per-frame encoding cost by 53%–56% within 1.7 points of CLIP R@1 on Charades-STA.STC reuses cached features for 75% of tokens, reducing ViT-encoding latency by 24.5% and LLM prefilling latency by 45%.
  • Efficient Encoder Architectures and Operators: CoPE-VideoLM replaces dense RGB encoding for most frames with motion and residual information from a video codec using a delta-encoder under 15M parameters.The mechanism runs the image encoder only on frames stored in full by the codec.

4) Discussion and Synthesis:

Encoder trade-offs depend on evaluation views, downstream token counts and the full pipeline rather than architecture alone. The survey therefore treats reported operating points as indicative and separates encoder-only evidence from integrated VideoLLM measurements.

  • Discussion and Synthesis: VideoMambaPro reaches 84.0 top-1 at 4.7 TFLOPs, but none of the state-space backbones matches Hiera or UniFormerV2 at any compute.Convolutional backbones occupy the low-compute regime, while pooling-attention transformers span the widest accuracy range and reach the highest absolute accuracies.
  • Discussion and Synthesis: MViTv2 and Video Swin report nearly identical Kinetics-400 accuracy at 1.13 versus 3.38 TFLOPs because they use five versus twelve evaluation views.This illustrates why the table supports comparisons between reported operating points rather than attribution of gaps to architecture alone.
  • Discussion and Synthesis: Encoder-only FLOPs do not establish end-to-end VideoLLM efficiency because output token count, connector processing and LLM computation also contribute.The downstream answering context can be reduced only after encoding costs have already been paid unless upstream mechanisms also reduce them.
  • Discussion and Synthesis: VisionZip retains 6.6% of visual tokens for a 7.8× prefilling speed-up, while LLaVA-PruMerge achieves 14× average visual-token compression.These connector-level reductions shrink the LLM input but do not by themselves remove upstream encoding cost.
  • Discussion and Synthesis: TimeChat-Online reports an 82.8% token reduction, approximately 98% retained streaming accuracy and 1.76× faster responses.Its rule drops encoded tokens whose content is unchanged between successive frames and transfers without training to Qwen2.5-VL.
  • Discussion and Synthesis: OmniZip reports a 3.42× speed-up and 1.4× memory reduction at 35% token retention in a Qwen2.5-Omni-7B configuration.Because audio supplies the visual-selection signal, the audio encoder cost should be included in reported savings.

2) Grid Pooling and Downsampling:

Grid pooling, learned resampling and memory compression reduce the representation delivered to the answering model. These mechanisms target spatial, temporal, audiovisual or historical-token budgets at different pipeline locations.

  • Grid Pooling and Downsampling: STORM’s learned Mamba temporal connector cuts computation by up to 8× and decoding latency by 2.4–2.9× at a fixed frame count.Other grid-based connectors use pixel-shuffle, adaptive pooling, two-stream projection or spatial-temporal convolution.
  • Grid Pooling and Downsampling: TS-LLaVA builds a fixed 3,456-token budget from a detail thumbnail plus tokens sampled across 50 frames.The layout combines a compact global representation with sampled temporal detail.
  • Grid Pooling and Downsampling: HyperCLOVA X 8B reduces the audio rate from 25 Hz to 1 Hz after its adapter, while the survey attributes no described visual-token reduction to that report.The survey classifies only the audio downsampling mechanism and does not attribute unrelated visual-token or training-cost savings to it.
  • Grid Pooling and Downsampling: Learned resamplers use cross-attention with latent queries to construct compact representations, including LLaMA-VID’s two tokens per frame and LLaVA-Mini’s single vision token.Audiovisual variants enforce joint budgets or query features at fine and coarse temporal resolutions.
  • Grid Pooling and Downsampling: MovieChat, MA-LMM, ∞-Video, VidCompress and BLIP-3-Video variants maintain compressed visual representations or memory banks while constructing outputs.These methods compress historical context outside the answering LLM rather than only reducing the current input grid.
  • Grid Pooling and Downsampling: AdaCM2 prunes the Q-Former video cache using cross-modal attention and reports a 65% reduction in GPU memory for videos beyond two hours.This cache is part of the connector and is distinct from the answering LLM’s KV cache.

5) Discussion and Synthesis:

Controlled comparisons show that efficiency depends on both the retained budget and where compression acts: post-encoder methods can remain near baseline at 25%, while aggressive budgets expose differences among temporal, spatial, and joint methods.

  • Controlled comparisons: At a 25% budget, post-encoder methods stay within 1.5% of the uncompressed baseline average, while DyCoke loses over 7%.The comparison uses a single frozen LLaVA-OneVision-7B host at matched token budgets.
  • Controlled comparisons: At 10% retention, PruneVid and HoliTom degrade more gracefully than spatial-only VisionZip because they model temporal redundancy.
  • LLM-side reduction: FastV prunes the lowest-attention visual tokens after an early decoder layer, roughly halving prefilling FLOPs.
  • LLM-side reduction: At a 25% token budget, progressive query-conditioned reduction in Mamba–Transformer hybrids yields a 3.8–4.2× prefilling speed-up at near-baseline accuracy.
  • LLM-side reduction: MMInference reaches up to 8.3× prefilling speedup at million-token contexts with at most 0.4-point accuracy differences across reported 7B hosts.
  • KV-cache reduction: KV-cache methods target memory through eviction, quantization, or compression; VidKV reports up to 94% lower peak GPU memory.

5) KV-Cache Offloading and Retrieval:

KV-cache offloading and retrieval extend long-video processing by moving or selectively recovering historical state, but comparisons remain protocol-dependent and expose trade-offs among accuracy, memory, and stage-specific savings.

  • Offloading and retrieval: ReKV offloads KV blocks to CPU RAM or disk and retrieves query-relevant blocks at question time for streaming video answering.StreamKV and StreamingTOM similarly combine cache compression with question-conditioned retrieval or selective dequantization.
  • Controlled decoder comparison: At 24.5% of baseline prefilling FLOPs, HieraVid remains within 0.2–2.1 points across five shared-backbone settings.The comparison does not isolate decoder reduction because HieraVid also reduces tokens before the LLM.
  • Evidence limits: Cross-method comparison is intrinsically limited because methods use different backbones, baselines, token budgets, and evaluation protocols.
  • Trade-offs: ReKV preserves streaming accuracy best but keeps peak GPU memory near the full-cache level, whereas hard-capped eviction lowers memory by about 10 GB while losing roughly six RVS-Ego points.
  • Evidence limits: Reported near-baseline accuracy at 25% visual-token retention does not imply that every VideoLLM can discard 75% of visual tokens without loss.
  • Cross-stage effects: Joint stage reductions can improve end-to-end cost, but their gains cannot be attributed to either stage without an ablation separating the contributions.
  • Open evaluation gaps: Under a fixed LLM budget, token reduction can admit 10–20× more frames, but audiovisual costs remain difficult to isolate because audio encoders are often omitted.
  • Audiovisual efficiency: Audio-guided visual compression requires context-dependent allocation because speech may refer off-screen while visible events may lack informative sound.

VI. CONCLUSION

The survey organizes VideoLLM efficiency mechanisms across the encoder–connector–LLM pipeline and concludes that efficiency must be evaluated as a system-level trade-off. It highlights token-budget reductions, shifting bottlenecks, temporal coverage, and the need for standardized accuracy–compute comparisons.

  • Efficiency mechanisms act at four stages: input construction and selection, encoder computation, encoded representations and connector, and LLM execution and state.
  • Retaining roughly one quarter of the visual-token budget often preserves near-baseline accuracy, although reductions depend on the host model, task and evaluation protocol.
  • Reducing LLM prefilling and cache costs can make vision encoding the limiting stage.
  • In video systems, saved compute can be reinvested in processing more frames, so efficiency gains must be interpreted alongside temporal coverage and frame-encoding cost.
  • Progress requires reproducible accuracy–compute protocols with common backbones, inputs, FLOP-accounting boundaries and shared system measurements.
Loading 2609.10355v1…