Source-linked AI summary
Visual Token Coding for Video Multimodal Large Language Models
Chenxin Fang, Tao Chen, JunChao You, Jun Peng, Yiyi Zhou, Rongrong Ji
TL;DR
Video MLLMs face excessive visual-token costs, and conventional codec signals do not directly provide suitable semantic information for MLLMs. VTC transfers structured video coding to semantic token compression, while VTCDy adds dynamic resolution, budgeting, and spatial selection. Across three MLLMs and multiple benchmarks, VTCDy retains 100.1% of average Qwen3-VL performance at a 50% token budget and 97.8% at 25%, without additional model tuning.
Problem
Video MLLMs use excessive visual tokens, while conventional codec signals do not directly provide suitable semantic information for MLLMs.
Method
VTC organizes visual tokens as semantic I/P frames and uses frame-wise token residuals to estimate redundancy, with VTCDy adding dynamic resolution, token allocation, and spatial coverage selection.
Results
VTCDy retains 100.1% and 97.8% of average Qwen3-VL performance at 50% and 25% token budgets, respectively, across three video MLLMs and multiple benchmarks.
Takeaways & Limitations
Video coding structure offers a practical basis for efficient long-video MLLMs under constrained visual-token budgets without additional model tuning.
Takeaways & Limitations
High-frame-rate codec signals cannot directly characterize relations between frames sparsely sampled by video MLLMs.
Abstract
from arXiv · showhide
In this paper, we propose a new token compression paradigm for video Multimodal Large Language Models (MLLMs), termed Visual Token Coding (VTC). Inspired by classical video coding principles, e.g., HEVC, VTC performs structured compression by predicting the I/P frames of a video and measuring their frame-wise residuals to estimate token redundancy. Based on this baseline framework, we also enhance VTC with a set of novel dynamic designs, such as Dynamic Resolution Input (DyRSO), Dynamic Token Allocation (DyTA), and Spatial Coverage Top-K (SC-TopK), and term this new approach $VTC_{Dy}$. To validate VTC, we apply it to three MLLMs and conduct experiments on multiple video understanding benchmarks. The experimental results show that VTC$_{\mathrm{Dy}}$ achieves an average performance retention of 100.1% with a 50% token budget for Qwen3-VL, while still retaining 97.8% of the average performance when the token budget is reduced to 25%. Moreover, as a plug-and-play design, VTC requires no additional tuning of MLLMs for token coding. Our code is available at https://github.com/Msr233/VTC.
Introduction
VTC applies video-coding principles to compress visual tokens in video MLLMs, addressing excessive token costs while preserving semantic information. Its dynamic extension VTCDy improves compression and retention across existing MLLMs and benchmarks.
- Motivation: Up to 460k visual tokens may represent an hour-long video at 1 FPS, creating prohibitive computation and memory costs for video MLLMs.This motivates token compression for efficient video-language understanding.
- Motivation: Traditional video coding compresses frames using reference I-frames, predictive P-frames, and residuals while minimizing information distortion.VTC transfers this structured organization from RGB frames to visual feature tokens.
- Challenges: Direct codec transfer is insufficient because video-MLLMs still consume reconstructed P-frames, while codec signals lack suitable semantic information for direct MLLM use.Existing codec-inspired approaches may also require additional vision-encoder tuning and lack general applicability across MLLMs.
- VTC: VTC partitions visual tokens into semantic I/P frames and uses token-wise residuals to represent P-frame redundancy for pruning.The baseline is designed as a general paradigm for existing MLLMs.
- VTCDy: VTCDy adds Dynamic Resolution Input, Dynamic Token Allocation, and Spatial Coverage Top-K to improve compression ratio and performance retention.It performs token-level compression and is directly applicable to existing MLLMs, with query independence, temporal modeling, and spatial integrity.
Related Work
Prior work compresses visual tokens through pruning, merging, dynamic selection, or codec-guided signals. VTC differs by transferring structured coding to semantic token compression without relying solely on conventional codec primitives.
- Video MLLMs: Video MLLMs extend image-based models with temporal modeling, instruction tuning, and long-context scaling, while long-video systems use hierarchical compression, visual memory, or adaptive frame sampling.Representative models include LLaVA-OneVision, LLaVA-Video, Qwen2.5-VL, Qwen3-VL, and LLaVA-OneVision-2.
- Visual Token Compression: Visual token compression methods reduce sequences through pruning, merging, or dynamic selection, with video-oriented methods incorporating query or cross-frame cues for temporal redundancy.Representative approaches include FastV, VisionZip, DyCoke, PruneVID, KVTP, FrameFusion, STTM, HoliTom, and FlashVID.
- Codec-Guided Methods: Codec-guided methods use motion vectors, residuals, bit-cost flows, or predictive coding to improve video understanding, but existing methods add codec-aware components and require additional adaptation.The paper identifies a need to make codec primitives usable as semantic evidence.
Preliminary
Video coding reduces redundancy by predicting frames from references, representing only residual changes, and allocating bits according to coding complexity. VTC transfers these principles to visual-token compression while addressing sparse video-MLLM sampling.
- Video coding principles: Video coding organizes reference information through predictive coding, residual representation, and resource allocation under a limited coding budget.This structure preserves critical visual information while exploiting spatial and temporal redundancy.
- Prediction and residuals: An inter-frame predictor estimates the current frame from a historical reference, and the difference is represented as a residual.The residual captures newly introduced content relative to the reference frame.
- Prediction and residuals: A larger residual commonly indicates motion, scene changes, or complex texture and therefore requires a larger coding budget.Residual magnitude provides a local measure of frame change.
- Visual Token Coding: Codec signals cannot directly characterize relations between sparsely sampled frames in video MLLMs.This limits direct transfer of high-frame-rate video-coding signals to sparse MLLM inputs.
- Visual Token Coding: VTC transfers video-coding principles to visual tokens, using token-wise residuals and I/P-frame partitioning to represent P-frame redundancy.The proposed framework aims to make structured compression applicable to existing MLLMs.
Method
Visual Token Coding transfers reference prediction and residual coding from video coding into visual feature space, then extends this baseline with dynamic temporal, budget, and spatial designs. The resulting VTCDy distributes a fixed video-level token budget across frames and spatial locations while retaining semantic I/P-frame structure.
- Visual Token Coding: VTC transfers reference prediction and residual coding into visual feature space, using semantic I/P frames to identify token redundancy.I frames preserve scene references, while P frames represent changes relative to references; residual energy measures newly introduced information.
- Basic VTC: The baseline partitions the encoded feature sequence into consecutive chunks and allocates each chunk a budget proportional to its pre-pruning token count.Each chunk is compressed before the next is processed, while only its last full reference feature is retained.
- Basic VTC: For each later frame, VTC classifies semantic I/P frames from residual energy relative to historical energy and caps the number of I frames per chunk.P-frame budgets are evenly divided in Basic VTC, with Top-KP tokens selected by residual energy.
- VTCDy: VTCDy combines Dynamic Resolution Input, Dynamic Token Allocation, and Spatial Coverage Top-K to address limited temporal coverage, non-uniform frame complexity, and spatial concentration.These designs respectively expand temporal coverage, adapt budgets along time, and preserve intra-frame regional coverage.
- Dynamic Resolution Input (DyRSO): DyRSO replaces part of the full-resolution input with low-resolution source frames arranged in multi-resolution GOPs, increasing temporal coverage at fixed visual encoding cost.A 2×2 canvas represents four source timestamps while keeping canvas resolution and pre-pruning ViT token cost equal to a full-resolution frame.
- Dynamic Token Allocation (DyTA): DyTA uses HEVC bitcost as a query-independent complexity prior, assigning larger budgets to more complex temporal positions while preserving minimum allocation and total budget.It computes complexity from mean and 90th-percentile bitcost over causal intervals, then apportions the remaining P-frame budget.
- Spatial Coverage Top-K (SC-TopK): SC-TopK divides the token grid into regions and selects tokens to preserve regional coverage while prioritizing large residual changes.This addresses the tendency of residual-based global Top-K selection to concentrate tokens in a few locally changing regions.
Experiments
Experiments evaluate VTC and VTCDy across three video MLLMs and multiple long-video benchmarks under matched token budgets. VTCDy improves performance retention, adapts token allocation to temporal novelty, and reduces model-side latency at 25% retention.
- Evaluation setup: VTCDy is evaluated on three MLLMs and four long-video understanding benchmarks, with accuracy reported for multiple-choice VideoQA tasks.The benchmarks span diverse genres and durations, including videos up to two hours.
- Performance across MLLMs: 100.1%, 101.6%, and 100.3% average retention are achieved by VTCDy at 50% retention for Qwen3-VL, LLaVA-OneVision, and LLaVA-OneVision-2, respectively.Baseline VTC retains about 98% and 96% average performance at 50% and 25% retention, respectively.
- Comparison with token compression methods: 100.5% on LongVideoBench and 98.2% on MLVU are preserved by VTCDy at 25% retention, exceeding the second-best retention by 1.9 and 2.4 points.VTCDy achieves the best performance on most benchmarks, while direct codec transfer alone underperforms FlashVID.
- Efficiency: 4109 to 1644 ms (2.50×) is the model-side latency reduction for VTCDy at 25% retention on one RTX 3090.The reduction in LLM inference time outweighs pruning overhead and yields the lowest total latency among evaluated methods.
- Ablation study: 69.2 is the MLVU average after adding SC-TopK, rising from 67.3 for VTC through 68.2 with DyRSO and 68.9 with DyTA.The ablation attributes complementary roles to resolution, temporal budgeting, and spatial coverage; LVBench improves from 40.1 to 41.2.
- Ablation study: 0.3 points is the variation in both MLVU average and LVBench across semantic I-frame thresholds from 1.2 to 2.0.The results suggest that the coding structure is not sensitive to the threshold.
- Ablation study: P90 performs best on LVBench, while the combined mean and high-percentile bitcost score achieves the strongest MLVU average.These results support using complementary global and local complexity cues for P-frame allocation.
- Dynamic allocation: 7.7%–29.6% is the token-allocation range across P frames for VTCDy, compared with 16.8% uniformly for baseline VTC.VTCDy assigns more tokens to substantial content changes, fewer to near duplicates, and detects a scene cut as a new semantic I frame.
Conclusion
The paper proposes VTC, a video-coding-inspired framework that organizes visual tokens as semantic I/P frames and allocates them using temporal residuals. Its dynamic variant, VTCDy, retains near-full average performance at reduced token budgets across three MLLMs without additional tuning, supporting structured coding for efficient long-video understanding.
- Conclusion: VTC organizes visual tokens as semantic I/P frames and allocates them according to temporal residuals.VTCDy adds dynamic resolution, temporal budgeting, and spatial coverage.
- Conclusion: 100.1% and 97.8% of average Qwen3-VL performance are retained by VTCDy at 50% and 25% token budgets, respectively.These results are reported across three MLLMs without additional model tuning.
- Conclusion: Structured prediction and residual coding can complement frame-wise importance scores by exposing redundancy at the video level.The paper presents video coding structure as a practical basis for efficient long-video MLLMs under constrained visual-token budgets.