Source-linked AI summary
AdaCodec: A Predictive Visual Code for Video MLLMs
Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang
TL;DR
Video MLLMs redundantly encode temporally similar frames as independent RGB images, limiting coverage and increasing token use and latency. AdaCodec instead sends full tokens for costly reference frames and compact motion-and-residual P-tokens for predictable frames, improving benchmark performance at matched or smaller visual-token budgets while reducing latency.
Problem
Per-frame RGB encoding redundantly retransmits temporally shared content, creating a coverage–detail tradeoff under finite context windows while increasing token cost and latency.
Method
AdaCodec allocates full ViT tokens to costly reference frames and compact motion-and-residual P-tokens to predictable intermediate frames.
Results
Across eleven benchmarks, AdaCodec improves over per-frame RGB baselines at matched or smaller visual-token budgets, including a 1/7-budget setting that surpasses the 224k baseline on all long-video benchmarks.
Takeaways & Limitations
AdaCodec improves the video MLLM performance–cost frontier by converting temporal redundancy into compact visual representations before tokens enter the LLM.
Takeaways & Limitations
Experiments use fixed input resolution, a uniform per-P-frame token budget, and do not evaluate streaming video.
Abstract
from arXiv · showhide
Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at $1/7$ the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.
1 Introduction
AdaCodec replaces redundant per-frame RGB encoding with a predictive visual code that sends full reference frames selectively and compact motion/residual P-tokens otherwise. Across eleven benchmarks, it improves the performance-cost frontier, surpassing the 224k-token baseline at 32k tokens on all long-video benchmarks while reducing visual-token use and latency.
- Motivation: Adjacent frames repeat prior visual content, making independent RGB encoding inefficient and creating a coverage–detail dilemma under finite context windows.Token cost grows roughly linearly with sampled frames, so sparse sampling can miss short events while dense sampling consumes context.
- Motivation: Predictive visual coding transmits prediction errors instead of raw signals, using full reference content and motion/residual information for predictive frames.This principle is motivated by both biological prediction-error coding and modern video codecs.
- AdaCodec: AdaCodec allocates full ViT tokens only when conditional predictive cost is high and otherwise represents intermediate frames with compact motion- and residual-derived P-tokens.A pcost-driven reset starts a new reference frame when prediction becomes costly.
- AdaCodec: AdaCodec combines an MLLM-oriented predictive codec, a compact P-frame tokenizer, and a two-stage alignment pipeline for existing MLLM architectures.The resulting visual interface interleaves reference-frame tokens with predictive codes.
- Results: 32k visual tokens surpass the 224k Qwen3-VL-8B baseline on all long-video benchmarks, while AdaCodec improves the performance-cost frontier across eleven benchmarks.At a matched 224k visual-token budget, it achieves the strongest open-source comparison results on all three long-video benchmarks and two of three temporal benchmarks.
2 Related Work
Prior video MLLM efficiency work reduces visual cost through frame selection, frame-space subsampling, post-encoding token compression, or temporal pooling. AdaCodec is positioned instead as a predictive visual interface, drawing on compressed-domain video representations while differing from encoder-internal sparsity and generative reconstruction tokenizers.
- Video MLLM efficiency: Prior video MLLM efficiency methods reduce cost through frame selection, frame-space subsampling, token compression, and temporal pooling.Encoding hundreds of visual tokens per sampled frame makes visual sequences scale with video length and sampling density, quickly dominating context length and compute.
- Positioning AdaCodec: AdaCodec differs from OneVision-Encoder’s encoder-internal codec patch sparsity and InfoTok’s generative reconstruction tokenizer by serving as a predictive code at the visual interface.OneVision-Encoder targets patch-level sparse computation and encoder efficiency, while InfoTok learns adaptive discrete tokens for video reconstruction.
- Compressed-domain video understanding: Classical and modern codecs represent sparse keyframes in full and encode remaining frames with predictive side information.Compressed-domain video understanding has also used motion vectors as low-cost optical-flow surrogates and jointly modeled I-frame, motion, and residual modalities.
- Codec-aware video MLLMs: Recent codec-aware video MLLMs use I-frames, motion vectors, standardized codec primitives, or refined block-motion representations for long-video understanding.EMA builds GOP-level representations, CoPE-VideoLM aligns codec primitives with MLLM representations, and ReMoRa refines noisy block-motion representations.
3 Method
AdaCodec adapts predictive video coding to video MLLMs by representing reference frames fully and intermediate frames with motion-and-residual P-tokens. Its method combines content-adaptive GOP construction, a dual-branch token pipeline, and two-stage training for P-frame representation learning and multimodal alignment.
- Method overview: AdaCodec encodes intermediate frames as motion-and-residual updates, uses a dual-branch visual-token pipeline, and trains the code in two stages.The stages learn the compact P-tokenizer under frozen visual supervision and then align the visual code with the language model.
- Motion and residual encoding: Each GOP begins with a full I-frame, while every subsequent P-frame is predicted from the preceding sampled frame using motion vectors and residual signals.For each macroblock, local matching provides the motion vector, and the pixel difference between current and reference blocks provides the residual.
- Adaptive GOP construction: AdaCodec starts a new GOP when the frame-level predictive cost ℓ_t exceeds γ, reserving I-frames for frames with substantial novel content.The threshold is chosen to target a median of 8 P-frames per GOP, and GOPs are capped at K_max P-frames for temporal-length constraints.
- Deployable architecture: The deployable model uses a reference-frame encoder and a P-tokenizer that processes five-channel inputs combining residuals with two-channel motion tensors.The P-tokenizer widens a pretrained ViT patch embedding from 3 to 5 channels, copying RGB kernels and zero-initializing motion-vector channels.
- Tokenization: A GOP with K P-frames uses N_I + K N_P tokens instead of (K + 1)N_I, with N_P ≪ N_I, and preserves temporal order in the multimodal prompt.Reference and P-frame tokens are mapped into the LLM visual embedding space before insertion into the prompt.
- Two-stage training: Stage 2 uniformly samples multiple adaptive GOPs under a fixed visual-token budget, preserves their temporal order, and updates only the MLLM language model.Stage 1 optimizes the P-tokenizer and auxiliary feature predictor while visual encoders remain frozen; Stage 2 discards the predictor and freezes visual-side modules.
4 Experiments
AdaCodec is evaluated across eleven long-video, temporal, and general video-understanding benchmarks under matched visual-token budgets. It preserves or improves accuracy with substantially lower token use, while adaptive predictive coding and denser temporal coverage provide further accuracy and efficiency gains.
- Evaluation setup: AdaCodec is evaluated on eleven benchmarks spanning long-video, temporal, and general video understanding.The experiments cover MLVU, LongVideoBench, LVBench, TempCompass, MotionBench, TOMATO, Video-MME, MVBench, NExT-QA, PerceptionTest, and EgoSchema.
- Compactness preserves accuracy: At 1/7 of the baseline’s tokens, AdaCodec maintains accuracy across long-video, temporal, and general benchmarks.Long-video gains are +0.5, +0.8, and +0.2 on MLVU, LongVideoBench, and LVBench; temporal gains are +1.5, +1.9, and +4.1 on TempCompass, MotionBench, and TOMATO.
- Extra coverage converts into accuracy: At a matched token budget, AdaCodec improves over the baseline on every benchmark.Long-video gains are +3.1, +5.4, and +0.4 on MLVU, LongVideoBench, and LVBench, while temporal gains are +1.6, +3.0, and +4.3 on TempCompass, MotionBench, and TOMATO.
- System efficiency: 84.6% reduction in measured token cost accompanies an average GOP length of 10.21 frames on real evaluation videos.Under the longest-GOP regime, AdaCodec incurs an 11.8% token cost relative to per-frame RGB encoding; measured cost is 15.4% of baseline.
- System efficiency: 9.26s to 1.62s is the reduction in time-to-first-token, while AdaCodec’s average score rises from 74.0 to the reported higher value under matched hardware and decoding settings.Across 11,347 unique videos, AdaCodec uses 8,550.4 visual tokens per video against 55,893.2 for the per-frame RGB baseline, an 84.7% reduction.
- Budget scaling: 32k visual tokens already surpass the 224k-token baseline, and AdaCodec dominates across the full 32k–224k budget sweep on all three long-video benchmarks.This indicates the advantage is not tied to a single operating point.
5 Conclusion · Appendix
AdaCodec redesigns the visual-token interface for video MLLMs by allocating full ViT tokens to costly reference frames and compact motion-and-residual P-tokens to predictable frames. The paper also identifies fixed-resolution inputs, uniform P-frame budgets, and missing streaming evaluation as limitations and future directions.
- Appendix: AdaCodec experiments use a fixed input resolution, leaving dynamic-resolution input to future work.The limitation concerns the input-resolution setting used in the experiments.
- Appendix: AdaCodec uses a uniform per-P-frame token budget of NP = 16 rather than adapting allocation to motion or residual complexity.Adapting the budget per frame could further improve the efficiency–accuracy frontier.
- Appendix: AdaCodec is not evaluated on streaming video, although its causal I/P structure with incremental motion search could support streaming in principle.The passage identifies potential support for high frame rates and substantial streaming capability without reporting a streaming evaluation.
- 5 Conclusion: AdaCodec redesigns predictive visual code for the visual-token interface of video MLLMs.It is presented as a series of codec redesigns rather than an adaptation of a fixed playback codec stream.
- 5 Conclusion: AdaCodec allocates full ViT tokens to high-cost reference frames and compact motion-and-residual P-tokens to predictable intermediate frames.This removes repeated visual evidence before it enters the LLM context while preserving needed temporal changes.
- 5 Conclusion: The design targets repeated visual evidence by encoding predictable intermediate frames compactly instead of transmitting full frame representations.The approach preserves temporal changes needed by the model while reducing repetition in the LLM context.
A Codec Redesign Details
AdaCodec differs from a standards-compliant playback codec through four core redesigns and three additional configuration choices tailored to MLLM-token sequences rather than playback bitstreams.
- Component-wise comparison: Table 7 compares AdaCodec component by component with a playback-oriented codec.The table covers every component in which AdaCodec departs from the standards-compliant playback design.
- Core redesigns: The upper block contains the four core redesigns highlighted in Section 3.These rows reproduce the main-text redesigns.
- MLLM-token target: The lower block lists three choices specific to targeting an MLLM-token sequence: color space, frame types, and entropy coding.These choices follow from optimizing for an MLLM-token sequence instead of a playback bitstream.
B Model Implementation Details
AdaCodec constructs P-tokens from five-channel residual-motion tensors using an architecture-matched ViT with appended learnable latent tokens. Its implementation preserves Qwen3-VL’s pretrained visual interface while using Stage 1 reconstruction only for tokenizer training.
- P-token construction: Each P-frame forms a five-channel residual-motion tensor, which an architecture-matched pretrained ViT converts into P-tokens using appended learnable latent tokens.The latent tokens are attached before the visual transformer.
- P-token construction: A block attention mask keeps patch-token computation unchanged while allowing latent tokens to aggregate information from all predictive-representation patches.The resulting P-tokens are learned summary tokens conditioned on the predictive representation.
- Stage 1 reconstruction module: Stage 1 trains the P-frame tokenizer with an auxiliary predictor that reconstructs teacher features from the I-frame embedding and P-token states.The reconstruction head is removed before Stage 2, leaving only the tokenizer for downstream multimodal alignment.
- Qwen3-VL ViT interface: AdaCodec matches Qwen3-VL’s Conv3D stem, 2 × 2 spatial merger, and DeepStack visual-injection path without modifying the pretrained backbone.Single I-frame or P-frame tensors are duplicated temporally before entering the original Conv3D stem.
C Training Details
AdaCodec is trained in two stages using a shared public video-instruction data source, with distinct data formats for teacher-feature alignment and next-token training. Stage 2 uses a 64k-token phase followed by a 224k-token phase, while training runs on 64 NVIDIA H800 GPUs for approximately 12 days.
- Data and training stages: Stage 1 samples reference/intermediate P-frame/target tuples for teacher-feature alignment, while Stage 2 uses paired instruction-response examples for next-token training.The Stage 2 instruction-tuning mixture contains 3,904,313 training examples.
- Training hyperparameters: 64k visual tokens are used for 40,000 Stage 2 training steps, followed by 224k visual tokens for 5,000 steps.These are the two Stage 2 training phases reported in the training hyperparameters.
- Compute resources: 64 NVIDIA H800 GPUs support training runs spanning approximately 12 days of wall-clock time.Latency measurements use a single H800, while codec-build timing uses a 16-core consumer CPU.
D Benchmark Descriptions
The paper evaluates AdaCodec on eleven public video benchmarks spanning long-video understanding, temporal perception, and general video understanding. These benchmarks test multimodal reasoning across diverse durations, domains, tasks, and time-varying attributes.
- Benchmark scope: 11 public video benchmarks are used to evaluate AdaCodec.The benchmark suite covers multiple aspects of video understanding.
- Long-video benchmarks: Long-video benchmarks include heterogeneous content, videos up to one hour, and multi-task or referring-reasoning questions requiring retrieval of relevant temporal context.Examples include movies, surveillance, egocentric clips, cartoons, and gameplay.
- Temporal benchmarks: Temporal benchmarks test attributes such as speed, direction, and fine-grained motion beyond static single-frame cues.TempCompass pairs videos with fixed static content but differing time-varying attributes, while MotionBench evaluates fine-grained motion comprehension.
- General video-understanding benchmarks: General video-understanding benchmarks span short, medium, and long videos, six visual domains, 30 subfields, and 20 multiple-choice temporal-understanding tasks.Video-MME and MVBench assess video-centered questions and temporal understanding beyond a single frame.
E Efficiency and Latency Details
AdaCodec achieves substantial token savings through content-dependent GOPs and compact P-frame codes, while maintaining a strong latency advantage over per-frame RGB encoding. Its tradeoff is additional model memory from a dedicated P-frame tokenizer branch.
- Token efficiency: At the maximum 16-P-frame chain length, AdaCodec reaches a visual-token ratio of 0.118, an 88.2% reduction versus per-frame RGB input.Its default allocation is 256 tokens per I-frame and 16 tokens per P-frame.
- Token efficiency: Aggregated across evaluation videos, the realized average GOP length is 10.21 frames, or 9.21 P-frames per GOP.GOP length is content-dependent; substituting this empirical average into Eq. (10) yields a 15.4% vi…
- Latency: Including the 0.12s codec-build step, AdaCodec’s TTFT is 1.74s, still 5.3× faster than the per-frame RGB baseline’s 9.26s, while E2EL retains a 3.4× gap.Codec construction takes about 7% of AdaCodec’s 1.62s TTFT on a 16-core consumer CPU.
- Memory footprint: AdaCodec increases peak GPU memory by 1.9 GB (5.4%) over the per-frame RGB baseline under FP16.The added P-frame tokenizer uses one ViT-sized visual branch with about 576M parameters, equal to 7% of the 8.14B-parameter backbone.
F Adaptive GOP Behavior Details
AdaCodec adapts GOP length to video complexity: stable scenes receive longer prediction intervals, while dynamic content triggers earlier reference-frame refreshes. This behavior preserves more temporal information within the same context budget and explains a +20.6 MLVU anomaly-recognition gain.
- Case studies: MLVU anomaly videos show long low-pcost intervals followed by only late I-frame refresh bursts.The pattern reflects surveillance-style footage with relatively fixed cameras and slowly evolving scenes.
- Case studies: Dynamic NExT-QA videos exhibit multiple spikes that trigger earlier I-frame refreshes.The case study visualizes how AdaCodec responds to frequent content changes at the video level.
- Adaptive allocation: Longer GOPs serve temporally stable videos, while frequent scene or motion changes produce shorter GOPs under the pcost-guided construction.This matches token allocation to video complexity and preserves more of stable sequences within a fixed context budget.
- Accuracy impact: +20.6 MLVU anomaly-recognition gain is explained by exposing a more complete temporal record within the same context budget.Stable videos receive longer GOPs, allowing AdaCodec to preserve more of the original sequence for the MLLM.
G Ablation Protocol and Per-Axis Analysis
This section describes a compute-efficient ablation protocol and evaluates AdaCodec across major codec-design axes. Variants are compared under controlled training conditions on a fixed one-third data subset before the selected configuration is retrained on the full corpus.
- Ablation Protocol: Ablations use 1,301,438 examples, randomly sampled as one-third of the full training corpus, to reduce wall-clock and GPU costs while revealing stable qualitative trends.The full pipeline was not repeated for every codec variant because that would be prohibitively expensive.
- Ablation Protocol: All variants share the backbone, optimizer, learning rate, input resolution, frame rate, decoding protocol, reduced split, and optimization-step count.This isolates relative rankings among design choices rather than absolute leaderboard performance.
- Ablation Protocol: After selecting the final operating point, the corresponding configuration is retrained on the full training data.Subset-trained ablations are intended for relative design comparisons, not absolute leaderboard performance.
- Per-Axis Analysis: The analysis varies P-token count NP, maximum GOP P-frame count Kmax, pcost threshold target, macroblock size, and GOP construction strategy.Compared GOP strategies include adaptive pcost-guided construction and fixed-length baselines with nP ∈{8, 16} P-frames per GOP.