Source-linked AI summary
Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
Jinlong Li, Liyuan Jiang, Haonan Zhang, Nicu Sebe
TL;DR
Video LLMs are inefficient because long videos generate many visual tokens, while existing compression methods can miss temporal dependencies and discard informative context. AOT constructs local-global token anchors and uses optimal transport to aggregate intra- and inter-frame context in a training-free pipeline. Across multiple video benchmarks and models, it prunes 90% of video tokens while retaining 97.6% of original performance and reducing computation to 8.3% of original FLOPs.
Problem
Video LLM inference is computationally expensive because videos produce many visual tokens, while existing methods may require training or fail to exploit temporal dependencies and preserve informative context.
Method
AOT selects local- and global-aware token anchors within frames and applies optimal transport to aggregate informative intra- and inter-frame context into compact anchors without training.
Results
90% token pruning, 8.3% of original FLOPs, and 97.6% of original model performance were reported across MVBench, LongVideoBench, EgoSchema, and VideoMME.
Takeaways & Limitations
AOT provides a training-free token reduction framework with competitive performance across video benchmarks under aggressive compression.
Takeaways & Limitations
The inter-frame OT pruning module remains heuristic, and noisy temporal clip boundaries can group visually dissimilar frames and degrade performance in complex videos.
Abstract
from arXiv · showhide
Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token \textbf{A}nchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global \textbf{O}ptimal \textbf{T}ransport (\textbf{AOT}). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: https://tyroneli.github.io/AOT.
1 University of Trento 2 Tsinghua University 3 University of Electronic Science and Technology of China
Video LLMs face substantial inference costs because video processing produces long visual-token sequences, while existing compression methods often miss temporal dependencies or discard informative context. AOT addresses this by selecting local-global token anchors and using optimal transport to aggregate spatiotemporal information, achieving strong performance under aggressive compression.
- Limitations of Existing Methods: Existing compression methods may require training, operate inside the LLM, ignore video-specific characteristics, or fail to exploit temporal dependencies across sampled frames.These limitations make it difficult to reduce redundancy while preserving critical semantic and contextual information.
- AOT: AOT selects semantically important and spatially diverse token anchors through grid-wise local selection and attention-guided global selection within each frame.The two selections preserve both local priors and global information.
- AOT: Optimal transport models anchors and unselected tokens as discrete distributions, measuring their relationships and globally aggregating necessary context into the remaining anchors.Inverse cosine similarity forms the cost matrix, and each anchor consolidates information under the optimized transport plan.
- AOT: For inter-frame pruning, AOT uses the first frame in each clip as anchors, ensembles similar tokens across consecutive frames, and retains dissimilar tokens for temporal dynamics.The strategy supports uniform sampling or adaptive clustering of frame clips and can be efficiently solved with Sinkhorn-Knopp iteration.
- Motivation: Video LLMs can process tens of thousands to one million visual tokens, substantially increasing inference time, memory consumption, and computation.The challenge is especially pronounced for long and complex videos.
3. Methodology
AOT performs training-free spatiotemporal token compression by selecting local-global anchors and using optimal transport to aggregate informative visual context within and across frames.
- Optimal Transport: AOT models visual tokens as discrete distributions and learns a minimum-cost transport plan under mass constraints.The cost matrix is based on token similarity, and Sinkhorn optimization provides a fast iterative solution.
- Local-Global Token Anchors Establishment: Global anchors are selected using attention received from the [CLS] token, while local anchors preserve fine-grained details through grid-wise window selection.Global selection uses averaged attention across heads; local selection applies TopK within non-overlapping windows.
- Local-Global Token Anchors Establishment: The final anchor set unions global and local tokens while excluding locally selected tokens from global selection to reduce duplication.This balances global semantic coverage with local spatial detail.
- Spatiotemporal Pruning: Within each frame, OT assigns unselected tokens to anchors and updates each anchor through mass-normalized contextual aggregation.The weighting coefficient λ_intra controls the contribution of transported context before producing the compressed token set.
- Spatiotemporal Pruning: Across frame clips, the first frame supplies temporal anchors, and OT processes subsequent frames to assign similar tokens while retaining temporal variants.The inter-frame transport plan is solved with a fast Sinkhorn-based solver.
4. Experiments
AOT is evaluated across four video benchmarks and two video LLMs against training-free compression baselines, with performance, FLOPs, frame scaling, and component contributions examined. Results show substantial token and computation reduction while preserving video understanding performance and spatiotemporal context.
- Inference Cost: Reducing visual tokens before the LLM yields larger savings than pruning only inside early LLM layers because decoding contributes about 2% of total FLOPs.The reported FLOPs comprise prefilling and decoding costs across Transformer layers.
- Main Results: AOT preserves 97.6% of vanilla performance after pruning 90.0% of visual tokens on average across benchmarks.The comparison covers token retention budgets of 25%, 20%, 15%, and 10%; VisionZip can drop by up to 8.4% at 10% retention.
- Frame Scaling: AOT remains effective as frame counts increase, outperforming other compression methods at 16 and 64 frames and maintaining approximate context length at 128 frames.The reported behavior targets settings with sparse temporal information and extensive spatiotemporal context.
- Efficiency Analysis: Intra-frame and inter-frame optimal transport adds about 2.11 milliseconds at 100 Sinkhorn-Knopp iterations, less than 1% of total inference time.Under 32 frames on one A100, intra-frame OT takes about 0.51 milliseconds and inter-frame OT about 1.60 milliseconds.
- Ablation Studies: Ablations show that combining local and global anchors and using optimal transport across both intra- and inter-frame levels improves performance, especially under aggressive compression.Removing OT degrades performance, while global-only anchors are competitive but inferior to the combined strategy.
5. Conclusion
AOT aggregates informative semantics and context from tokens marked for merging or removal, using local-global token anchors and Optimal Transport to preserve video fidelity under compression.
- AOT aggregates necessary semantics and contexts into remaining tokens instead of simply merging or removing tokens.
Abstract
The supplementary material adds theoretical analysis, implementation details, dynamic-frame experiments, and ablations examining token-anchor quality.
- The supplementary material provides detailed theoretical analysis of Optimal Transport.
- It describes more detailed implementation procedures for AOT.
- Additional experiments examine dynamic frame clips and the practical advantage of AOT.
- An ablation study randomly selects intra-frame token anchors to illustrate the importance of high-quality anchor establishment.
6. AOT Approach Details
AOT models token anchors and unselected tokens as discrete distributions, then uses Optimal Transport and Sinkhorn iterations to derive transport plans for intra- and inter-frame aggregation.
- Optimal Transport: Token anchors and unselected tokens are represented as discrete distributions with probability weights over embedding-space support points.The distributions use discrete probability vectors and Dirac delta functions.
- Optimal Transport: AOT defines token-pair cost as one minus similarity, specifically using cosine-based distance between anchor and unselected-token embeddings.The cost matrix records the pairwise costs used by the transport plan.
- Optimal Transport: Optimal Transport learns a transport plan that minimizes total matching cost between token distributions.The transport matrix T satisfies marginal constraints matching the original discrete distributions.
- Sinkhorn Iteration: The Sinkhorn formulation adds entropy regularization to accelerate optimization of the transport plan.The regularized objective retains transport constraints while introducing an entropy term controlled by λ.
- Intra-Frame Inference: Intra-frame inference computes a transport plan and updates each anchor through mass-normalized aggregation from unselected tokens.The resulting compressed token set consists of updated intra-frame anchors.
- Inter-Frame Inference: Inter-frame inference initializes clip anchors from the first frame, aggregates similar tokens across subsequent frames, and retains dissimilar tokens as temporal dynamics.The procedure outputs clip-level anchors and temporal-dynamics tokens.
- Sinkhorn Iteration: The Sinkhorn-Knopp iteration repeatedly updates scaling vectors before constructing an approximate optimal plan.After sufficient updates, the plan is obtained as a diagonal-scaled kernel matrix.
7. More Implementation Details
The implementation uses LLaVA-OneVision-7B and LLaVA-Video-7B with 8 NVIDIA A100 GPUs, varying intra-frame anchor counts across retention budgets.
- AOT is implemented on LLaVA-OneVision-7B and LLaVA-Video-7B using 8 NVIDIA A100 GPUs.
8. Dynamic Clustering Frame Clip
AOT uses dynamic temporal segmentation to form adaptive frame clips based on video complexity. This preserves temporal structure and intra-segment similarity while retaining competitive or superior benchmark performance.
- Dynamic Clustering Frame Clip: Dynamic Temporal Segmentation partitions videos according to complexity using global-level features.It creates fewer partitions for simple scenes and finer partitions for more complex ones.
- Dynamic Clustering Frame Clip: The approach mitigates fixed-length segmentation, which can group visually dissimilar frames despite preserving temporal order.
- Dynamic Clustering Frame Clip: AOT with Dynamic Clustering consistently achieves competitive and superior performance across various video benchmarks.
9. Random Token Anchors Selection Ablation
The ablation shows that high-quality intra-frame token-anchor selection is important for AOT performance.
- Random Token Anchors Selection Ablation: Randomly selecting noisy intra-frame token anchors drastically deteriorates performance.
10. More Visualizations
The paper visualizes initial intra-frame token anchors across consecutive sampled frames to illustrate how optimal transport aggregates information from unselected tokens.
- More Visualizations: Figures 5 and 6 visualize initial intra-frame token anchors across consecutive sampled frames for MVBench and VideoMME.Spatial pooling makes temporal token marking difficult, so the visualizations focus on initial intra-frame anchors.
11. Limitation and Future Works
The paper identifies heuristic temporal-anchor construction and noisy temporal boundaries as limitations of inter-frame pruning. It proposes fine-tuning and auxiliary geometry- or motion-aware supervision as future directions.
- Limitation: Inter-frame OT pruning remains heuristic because constructing high-quality temporal token anchors lacks a principled solution.
- Limitation: Noisy temporal boundaries can place visually dissimilar frames in one clip, degrading performance in complex video scenarios.
- Future Works: Fine-tuning or instruction tuning with OT is proposed as a future direction for more competitive and efficient token reduction.
- Future Works: Auxiliary signals such as depth, camera motion, correspondence, 3D grounding, and trajectories could encourage geometry- and motion-aware anchor preservation.
- Future Works: Combining OT aggregation with auxiliary objectives may support persistent world representations and temporally coherent reasoning.