PaperlayerView exact revision on arXiv

Source-linked AI summary

DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models

Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang

arXiv:2411.15024v3cs.CVcs.LG

TL;DR

Video LLM inference is costly because video inputs produce many visual tokens, and changing attention across decoding steps makes one-shot pruning risky. DyCoke merges temporally redundant tokens and dynamically prunes the KV cache; it reports faster inference and strong benchmark performance without training.

  • Problem

    Video inputs generate many visual tokens that raise inference costs, while changing frame attention during decoding makes one-shot pruning prone to discarding important tokens.

    Source: §1 · pp. 1–2

  • Method

    DyCoke merges similar tokens across frames and dynamically prunes visual tokens in the KV cache during decoding, without training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    1.54× inference speedup on LLaVA-OV-7B, with the lowest memory consumption and highest accuracy, is reported; benchmarks also show performance surpassing prior counterparts.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    DyCoke’s temporal merging and dynamic pruning maintain high inference accuracy while compressing visual tokens and supporting longer video sequences within the same computational budget.

    Source: §1 · p. 2

Abstract

from arXiv · show

Video large language models (VLLMs) have significantly advanced recently in processing complex video content, yet their inference efficiency remains constrained because of the high computational cost stemming from the thousands of visual tokens generated from the video inputs. We empirically observe that, unlike single image inputs, VLLMs typically attend visual tokens from different frames at different decoding iterations, making a one-shot pruning strategy prone to removing important tokens by mistake. Motivated by this, we present DyCoke, a training-free token compression method to optimize token representation and accelerate VLLMs. DyCoke incorporates a plug-and-play temporal compression module to minimize temporal redundancy by merging redundant tokens across frames, and applies dynamic KV cache reduction to prune spatially redundant tokens selectively. It ensures high-quality inference by dynamically retaining the critical tokens at each decoding step. Extensive experimental results demonstrate that DyCoke can outperform the prior SoTA counterparts, achieving 1.5X inference speedup, 1.4X memory reduction against the baseline VLLM, while still improving the performance, with no training.

1. Introduction

Video inputs generate tens of thousands of tokens, making VLLM inference costly; unlike image inputs, video attention shifts across frames during decoding, so one-shot pruning can discard important tokens. DyCoke combines temporal merging with dynamic KV-cache pruning, and reports faster inference with high accuracy and no training.

1. Introduction

  • Video inputs can produce tens of thousands of tokens, making attention’s quadratic complexity a source of prohibitive training and inference costs.Prior work reducing LLM size can diminish complex reasoning capabilities.Source: §1 · p. 1
  • Video attention shifts among visual tokens across decoding iterations, so single-stage pruning risks omitting important tokens and compromising video comprehension.The attention distribution is described as highly sparse, with different tokens gaining or losing importance over time.Source: §1 · p. 2
  • DyCoke merges similar tokens across frames, then dynamically prunes less important KV-cache tokens while retaining pruned tokens for possible later activation.It selects a distinct token set at each decoding step and requires no fine-tuning or parameter modifications.Source: §1 · p. 2
  • 1.54× inference speedup on LLaVA-OV-7B, with the lowest memory consumption and highest accuracy, is reported for DyCoke.The method is training-free; temporal merging reduces tokens by 50%–60%, followed by an additional 70%–90% reduction per decoding iteration.Source: §1 · p. 2
  • DyCoke combines temporal token merging and dynamic KV-cache reduction to compress visual tokens without training, supporting longer video sequences within the same computational budget.The contribution summary reports evaluations across several video inference benchmarks.Source: §1 · p. 2

2. Related Work

Prior VLLMs use many visual tokens, increasing inference time and memory demands, while fine-tuning can require substantial resources. DyCoke is positioned as a training-free token-compression approach tailored to video’s temporal characteristics.

2.1. Video Large Language Models

  • VLLMs encode long videos into tens of thousands of visual tokens, increasing inference time and memory demands; some video systems require substantial resources for fine-tuning.The section motivates a VLLM-specific compression method that requires no fine-tuning.Source: §2.1 · p. 3

2.2. Efficient Multi-Modal Large Language Models

  • Prior efficiency approaches include quantization, compact backbones, and mixture-of-experts models, while reducing backbone size can compromise reasoning capability.The passage presents visual-token compression as an alternative efficiency direction.Source: §2.2 · p. 3
  • Earlier token methods merge or prune tokens using temporal aggregation or attention, while DyCoke proposes training-free dynamic compression tailored to video-token temporal characteristics.Examples discussed include ToMe [1], TESTA [36], TempMe [40], FastV [2], and LLaVA-PruMerge [39].Source: §2.2 · p. 3

3. Proposed Method

Video LLM inference has prefilling and decoding stages, with visual and text representations combined as input and key/value states stored for token generation. DyCoke merges temporally redundant visual tokens before inference, then dynamically prunes and restores KV-cache tokens as attention changes during decoding.

3.1. Background on Video LLM Inference

  • Video LLM prefilling encodes frames into visual tokens and combines them with prompt-text tokens, while decoding generates tokens using the updated KV cache.The model projects frame embeddings into the text feature space before concatenating visual and text tokens.Source: §3.1 · p. 3

3.2. Our Method: DyCoke

  • DyCoke first merges visually similar tokens across frames to reduce temporal redundancy, then dynamically prunes visual tokens used for attention computation.The two stages operate during prefilling and decoding, respectively.Source: §3.2 · p. 4, Figure 3 · p. 4
  • TTM samples adjacent frames in four-frame windows, compares corresponding tokens between odd and even groups using cosine similarity, and prunes redundant tokens.Within the odd group, it retains the first frame’s full tokens and prunes the remaining similar tokens; its processing time is under 10^-3 seconds for 32 frames.Source: §3.2 · p. 4
  • Because visual-token attention is sparse and shifts across decoding stages, DyCoke dynamically retains tokens with high attention rather than relying on a fixed token set.The method computes attention scores for visual tokens at a selected layer and retains the top-p% set in the KV cache.Source: §3.2 · p. 5
  • DyCoke stores pruned tokens in a DP cache and updates the KV cache when attention similarity is low, allowing tokens with increased attention to be restored.Tokens that fall outside the top-p% set are removed from the KV cache and stored for possible later activation.Source: §3.2 · p. 5

4. Experimental Results

DyCoke is evaluated across video QA, video description, inference efficiency, and ablations against token-compression baselines. The results report strong task performance with reduced inference cost and examine the contributions of temporal merging and dynamic pruning.

4.1. Evaluation Setups and Implementation Details

  • The evaluation uses established video-to-text benchmarks and compares DyCoke with FastV and LLaVA-PruMerge under matched hardware conditions.Implementation covers LLaVA-OneVision 0.5B, 7B, and 72B; pruning ratios are set using calculated FLOPs for fair comparison.Source: §4.1 · pp. 5–6

4.2. Main Results

  • DyCoke significantly outperforms FastV and PruMerge at similar or lower computational cost across video QA benchmarks and model sizes.Table 1 compares methods on video QA and description benchmarks.Source: §4.2 · p. 6, Table 1 · p. 6
  • DyCoke exceeds the original model’s average performance on PercepTest and VideoMME, achieves the best quantitative MVBench results, and shows minimal degradation on VideoDC.The MVBench evaluation covers the more challenging multiple-choice video QA task.Source: §4.2 · p. 6, Table 2 · p. 6
  • Figure 4 shows a case where FastV yields a wrong answer while DyCoke retains the correct answer, and another where compression corrects the full-token model’s mistake.The examples use LLaVA-OV 7B on MVBench.Source: Figure 4 · p. 7

4.3. Efficiency Analysis

  • 1.4× speedup on 7B models and 1.54× for longer visual sequences are reported against full-token inference on MVBench.For VideoDC generation, DyCoke preserves performance comparable to the original model while reducing latency; it also improves short- and medium-video performance under a fixed computational budget.Source: §4.3 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7

4.4. Ablation Study

  • Random token selection in TTM substantially reduces video-understanding performance, while over-pruning causes a sharp drop.The authors interpret these findings as evidence that TTM mitigates temporal redundancy.Source: §4.4 · p. 8, Table 5 · p. 8
  • Replacing dynamic pruning with one-shot pruning reduces performance across tasks, especially on VideoDC; dynamic pruning is stable when the attention evaluation layer L is greater than zero.The authors motivate dynamic pruning by observing that visual-token attention varies across decoding iterations.Source: §4.4 · p. 8, Table 5 · p. 8

5. Conclusion

DyCoke is a training-free, dynamic token-compression method for VLLMs that combines temporal merging with decoding-stage token reduction. Across video QA and reasoning tasks and three model sizes, it surpasses prior SoTA counterparts while improving performance, speed, and memory use.

5. Conclusion

  • DyCoke combines temporal merging of similar frame tokens with further dynamic reduction of visual tokens used during decoding.The method is training-free and is described as the first dynamic token-pruning method specifically tailored for VLLMs.Source: §5 · p. 8
  • Across video QA and reasoning tasks on 0.5B, 7B, and 70B VLLMs, DyCoke consistently surpasses prior SoTA counterparts.The paper reports up to 1.5× speedup and 1.4× memory reduction, with performance still improved.Source: §5 · p. 8

A.1. Brief Overview

MVBench evaluates temporal reasoning across 20 video tasks, from perception to cognitive reasoning. The model-configuration comparison identifies hidden-state size, FFN intermediate size, and transformer-layer count.

A.1. Brief Overview

  • MVBench covers 20 tasks across nine categories, assessing temporal reasoning from basic perception to advanced cognitive tasks.Tasks include action recognition, object localization, scene transformation, episodic reasoning, and counterfactual inference.Source: §A.1 · p. 11
  • The LLaVA-OV model configurations compare 0.5B, 7B, and 72B models using hidden-state size d, FFN intermediate size m, and layer count T.The passage defines d, m, and T as configuration dimensions.Source: Table 8 · p. 11

A.2. Supplementary Experimental Data

The supplementary material reports MVBench performance and inference speedup for two LLaVA-OV sizes across input-frame counts, and lists computational hyperparameters for three VLLM sizes.

A.2. Supplementary Experimental Data

  • Tab. 3 reports compressed-token performance and inference speedup for LLaVA-OV-0.5B and 7B on MVBench across varying input-frame counts.Per-submetric accuracy for the 32-frame case is provided in Tab. 7.Source: §A.2 · p. 11

B. Model Hyperparameters

  • The supplementary hyperparameter table gives n, m, d, and T for the three VLLM sizes used in the FLOPs evaluation.These denote token count, FFN intermediate size, hidden-state size, and total transformer layers, respectively.Source: B. Model Hyperparameters · p. 11

C. Computing Cost Evaluation.

The computing-cost evaluation estimates total FLOPs across prefilling and decoding, with decoding cost accounting for KV-cache contributions over R prediction iterations. FLOPs enable computational comparisons but do not directly measure final inference speed.

C. Computing Cost Evaluation.

  • The FLOPs evaluation estimates prefilling and decoding computation for transformer layers with MHA and FFN modules.The prefilling estimate uses n, d, and m; decoding accounts for KV-cache contributions across R iterations.Source: C. Computing Cost Evaluation · pp. 11–12
  • R is set to 100 prediction iterations for the experiments, and total FLOPs are expressed for a model with T transformer layers.The displayed equation follows this setup.Source: C. Computing Cost Evaluation · p. 12
  • FLOPs quantify token computation for fair comparison, but do not directly indicate final inference speed.Figure 5 is captioned as performance versus K for different input-frame counts; the caption gives no outcome values.Source: C. Computing Cost Evaluation · p. 12, Figure 5 · p. 12

D. Ablation Study about k and Input Frames

The ablation examines how compression performance varies with k and the number of input frames, finding that its effect depends on frame count.

D. Ablation Study about k and Input Frames

E.1. Compatible with Flash Attention

DyCoke adds no significant computational overhead when combined with Flash Attention, though the authors note remaining limits for highly dynamic videos and mobile deployment. Visualizations show reasoning results consistent with the original after compression on both long and short videos.

E.1. Compatible with Flash Attention

  • Flash Attention adds no significant computational overhead with DyCoke because attention scores are computed only at a specific layer during each decoding iteration.The passage says this computation is substantially less complex than prefilling.Source: §E.1 · p. 12

E.2. Future Work

  • Rapid scene changes or critical time shifts may still incur minor information loss, and deploying full LLMs on mobile devices remains challenging.The authors identify finer-grained compression and techniques such as quantization and distillation as future directions.Source: §E.2 · p. 12

F. More Visualizations

  • The visualizations indicate that reasoning results after compression remain consistent with original results for both long and short video inputs.Source: Figure 6 · p. 13, Figure 7 · p. 13

Related papers

Loading 2411.15024v3…