Source-linked AI summary
CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modeling
Sayan Deb Sarkar, Rémi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai Dusmanu
TL;DR
VideoLMs struggle to represent long videos within limited context windows because sparse keyframe sampling can miss temporal details and full-image processing is costly. CoPE-VideoLM encodes motion vectors and residuals with lightweight transformer encoders aligned to image features, achieving large efficiency gains while maintaining or exceeding performance across 14 benchmarks. Its current scope is limited by support for I- and P-frames, tensorized codec primitives, and a fixed P-frame fusion window.
Problem
VideoLMs must fit long videos into limited context windows, but sparse keyframe sampling can miss macro- and micro-level temporal information while full-image processing increases computational overhead.
Method
CoPE-VideoLM aggregates temporally ordered motion vectors and residuals with lightweight transformer encoders, aligns them with image-encoder representations through pre-training, and integrates them with a VideoLM.
Results
Across 14 video understanding benchmarks, CoPE-VideoLM maintains or exceeds performance while reducing time-to-first-token by up to 86% and token usage by up to 93%.
Takeaways & Limitations
Codec-aware tokenization offers a competitive alternative to traditional keyframe sampling, providing richer temporal representations with lower computational overhead.
Takeaways & Limitations
The approach currently lacks B-frame support, uses tensorized codec primitives, and applies a fixed P-frame fusion window that is suboptimal for tasks with varying motion.
Abstract
from arXiv · showhide
Video Language Models (VideoLMs) enable AI systems to understand temporal dynamics in videos. To fit within the maximum context window constraint, current methods use keyframe sampling which often misses both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their tokens for each frame incurs substantial computational overhead. We address these limitations by leveraging video codec primitives (specifically motion vectors and residuals) which natively encode video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight transformer-based encoders that aggregate codec primitives and align their representations with image encoder embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach, CoPE-VideoLM, reduces the time-to-first-token by up to 86% and token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we maintain or exceed performance on 14 diverse video understanding benchmarks spanning general question answering, temporal and motion reasoning, long-form understanding, and spatial scene understanding.
1 Introduction
VideoLMs face context-window, temporal-coverage, and computational challenges when representing long videos with sparse full-image keyframes. CoPE-VideoLM addresses these gaps by encoding temporally ordered codec primitives alongside keyframes, reducing redundancy while preserving motion and appearance.
- Motivation: Fixed context windows force VideoLMs to sample a limited number of frames, even though video information scales with duration.Open-source models may sample a fixed number such as 64 frames regardless of video length.
- Motivation: Sparse keyframe coverage can miss macro-level events and micro-level details, while redundant consecutive frames make equal token allocation suboptimal.Full-image processing also increases prefill time and delays time-to-first-token.
- Codec primitives: Video codecs encode inter-frame changes as motion vectors and residuals, preserving temporal structure while minimizing redundant full-frame information.In typical streaming, I-frames occur every 5–10 seconds, with subsequent changes encoded separately.
- Research gap: Prior codec-domain methods discard residuals, ignore cross-modal dependencies, lose temporal ordering, or fail to preserve dynamics and appearance together.The paper identifies the absence of a variable-length, temporally ordered representation combining both motion and appearance.
- Approach: CoPE-VideoLM uses transformer encoders to aggregate motion vectors and residuals, align them with image-encoder features through pre-training, and fine-tune them with a VideoLM.Codec-aware tokens are arranged temporally alongside keyframe tokens during inference.
- Efficiency: Up to 86% lower TTFT and up to 93% lower token usage result from skipping redundant RGB information with codec primitives.The approach uses a lightweight dual-branch encoder and can vary primitive density to trade fine-grained representation against token count.
2 Related Work
VideoLM research has pursued temporal understanding and efficiency through multimodal architectures, token compression, and compressed-video representations. Existing approaches reduce tokens or exploit codec signals, but prior codec methods have limitations in modality interaction, residual use, or temporal ordering.
- Video Language Models: VideoLMs extend image-based multimodal language models with temporal reasoning over dynamic visual content.Typical systems combine a vision encoder, modality adapter, and LLM backbone.
- Token Compression: Efficiency methods include frame downsampling, spatial or temporal pooling, similarity-guided merging, compact latent representations, and attention-based token pruning.These approaches seek to remove visual redundancy while preserving semantic fidelity.
- Compressed Video Representation: Compressed-video methods use motion vectors and residuals to bypass costly full-frame processing, especially for action recognition.Early systems trained separate CNNs on I- and P-frame signals but ignored inter-modal dependencies and temporal ordering.
- Compressed Video Representation: Prior codec-based approaches either separate motion and residual processing, discard residuals, or summarize GOP signals without preserving temporal order.These limitations motivate representations that jointly retain dynamics and appearance in temporal sequence.
3 Method
CoPE-VideoLM uses video codec structure to represent I-frames with dense RGB tokens and P-frames with compact, temporally ordered tokens derived from motion vectors and residuals. Lightweight encoders, fusion, and alignment pre-training reduce redundancy while preserving temporal coverage for VideoLMs.
- Codec preliminaries: I-frames provide full visual representations, while P-frames encode changes through motion vectors and residuals relative to reference frames.P-frames therefore contain incremental temporal information and are smaller than I-frames.
- Codec-aware tokenization: The pipeline processes I-frames with a frozen RGB vision encoder and P-frame codec primitives with a lightweight Δ-Encoder, then interleaves their tokens temporally.The resulting sequence is consumed by the LLM without architectural modifications.
- Codec structure: The method retains causal P-frames because they depend only on past references, unlike B-frames whose future dependencies complicate real-time processing.GOP configuration and frame grouping control the trade-off between compression efficiency, quality, random access, and token use.
- Δ-Encoder: Motion vectors and residuals are represented as sparse tensors and compressed by specialized transformer branches into compact Δ-tokens.The motion branch uses an MLP and motion transformer, while the residual branch uses a lightweight ResNet-18 and residual transformer.
- P-frame fusion: P-frame fusion groups consecutive P-frames relative to an earlier reference, trading fine-grained temporal representation for fewer tokens.For a 30 FPS video with GOP size 240, fusion with s=30 reduces per-GOP tokens to M+7N, compared with M+239N for full P-frame modeling.
- Training paradigm: Training first aligns the Δ-Encoder with the image-encoder embedding space, then integrates it into a VideoLM for end-to-end fine-tuning.The alignment uses reference and warped transformers to connect motion and residual information with target RGB tokens.
4 Experiments
CoPE-VideoLM is evaluated across 14 benchmarks and multiple efficiency settings, showing competitive or superior understanding while reducing visual-token and runtime costs. Its codec-aware representation supports temporal, long-form, and diverse video understanding under constrained budgets.
- Evaluation setup: The Δ-encoder represents P-frames by combining motion and residual information into compact tokens alongside keyframe tokens.
- Evaluation setup: 14 benchmarks span general video QA, temporal and motion reasoning, long-form instruction following, and spatial scene understanding.
- Effectiveness of Δ-tokens: Under aggressive compression, CoPE-VideoLM maintains strong video-QA performance with over an order-of-magnitude token reduction, while higher frame densities can surpass the 64-frame baseline.
- Benchmark comparisons: CoPE-VideoLM achieves the highest open-source accuracy on PerceptionTest, ActivityNet-QA, TempCompass, TOMATO, and CVRR-ES.
- Benchmark comparisons: The model performs best among open-source methods on Video-TT, Video-MMMU, and LVBench while remaining competitive on LongVideoBench.
- Runtime and memory: 86.2% lower TTFT and 56.1% faster E2EL are achieved versus the 64-frame LLaVA-Video-7B baseline for generating 64 tokens at 1 FPS.
- Runtime and memory: The most compact configuration processes videos up to 8 hours at 1 FPS within a 1M-token context.
5 Conclusion
CoPE-VideoLM uses codec-aware tokenization to improve VideoLM efficiency while maintaining competitive performance across 14 benchmarks. The approach reduces time-to-first-token and identifies several directions for future codec support and efficiency improvements.
- The current approach supports I- and P-frames but lacks B-frame support because of their complex non-causal dependencies.
- Future work includes operating directly on block-wise motion vectors and quantized DCT coefficients, adapting P-frame fusion windows, and studying codec sensitivity.The paper specifically identifies codec type, bitrate, and encoding quality as open questions.
- Up to 86% lower time-to-first-token demonstrates substantial efficiency gains across 14 video understanding benchmarks.The conclusion characterizes codec-aware tokenization as an alternative to traditional keyframe sampling and reports competitive performance.
- Codec-aware tokenization leverages motion vectors and residuals to provide richer temporal representations with lower computational overhead than sparse keyframe sampling.
A Video Decoding Illustration
The decoding illustration shows how codecs reconstruct a current frame from a previous frame, motion vectors, and residuals. The Δ-Encoder compresses these codec primitives into tokens aligned with vision embeddings without decoding RGB pixels during fine-tuning.
- During decoding, motion vectors move the previous I-frame or reconstructed frame, after which residuals are added to obtain the current RGB frame.
- Figure 5 progresses from the previous frame through motion vectors and residuals, motion-compensated reconstruction, and the final reconstructed frame.
- The Δ-Encoder converts motion vectors and residuals into N = Kτ + Kδ compact tokens aligned with the vision encoder’s embedding space.The module has fewer than 15M parameters and operates in the compressed domain during VideoLM fine-tuning.
- Separate transformer branches compress motion-vector and residual features using learned query tokens.The motion branch patchifies normalized motion fields, while the residual branch uses a truncated ResNet-18 before transformer compression.
- Auxiliary warped and reference branches pre-train the Δ-Encoder to reconstruct target-frame token representations without decoding RGB pixels.
C Additional Training Details
The paper pre-trains the Δ-Encoder with auxiliary branches and then fine-tunes the VideoLM using a fixed mixture of keyframes and P-frames.
- Pre-training runs for 113K iterations on 16×A100 GPUs with global batch size 1024 and per-GPU learning rate 6.25×10^-5.AdamW, cosine scheduling, and 1000 warmup steps are used for the two-day pre-training run.
- VideoLM training uses 4 keyframes and 4 P-frames per GOP for 10.9K steps on 64×A100 GPUs.The global batch size is 128, with hyperparameters matching LLaVA-Video.
D Training Data and Evaluation Benchmarks
Training uses the 1.39M-sample LLaVA-Video-178K corpus, while evaluation spans 14 benchmarks covering general QA, temporal and motion reasoning, long-form understanding, and professional-video comprehension.
- Training Data: The training corpus totals 1.39M samples from LLaVA-Video-178K, covering captioning, open-ended QA, and multiple-choice QA.Its sources include academic benchmarks and YouTube, with videos organized by duration.
- General Video QA: General Video QA benchmarks assess fine-grained perception, causal and temporal reasoning, complex activity understanding, and spatial, temporal, and semantic comprehension.
- Temporal and Motion Reasoning: Temporal and motion benchmarks test speed, direction, ordering, duration, state transitions, scene changes, and object interactions.They also expose systematic failure modes through compositionally varied questions.
- Long-form and Instruction Following: Long-form benchmarks evaluate multi-step reasoning and comprehension over interleaved or several-minute videos, including professional videos across disciplines.
E Scale of Training Data
CoPE-VideoLM’s comparison with LLaVA-Video shows that benchmark gaps on select tasks are tied to training-data scale and composition, not necessarily codec formulation.
- CoPE-VideoLM underperforms LLaVA-Video on select benchmarks, including VideoMME.The authors examine this gap through incremental LLaVA-Video training stages.
- 61.9 →63.4 VideoMME accuracy follows adding image alignment data, which CoPE-VideoLM lacks.
- 63.2 →61.9 VideoMME accuracy follows adding three QA datasets, indicating sensitivity to training-data mixture.
- CoPE-VideoLM outperforms or matches comparable LLaVA-Video configurations while using fewer samples.
F Spatial Video Question Answering
The spatial QA evaluation tests whether codec-aware tokens retain geometry-relevant information despite compressed representations and limited token budgets. CoPE-VideoLM remains competitive with VideoLMs and approaches leading 3D VLMs after fine-tuning.
- Evaluation setup: SQA3D and ScanQA test situated reasoning and spatial understanding by associating multi-view observations with 3D structure.
- Evaluation setup: The spatial QA results use a short-video setup with 6 keyframes and 6 P-frames per GOP to align with a 32-RGB-frame setting.
- Results: CoPE-VideoLM matches state-of-the-art VideoLMs using one-quarter of LLaVA-Video-7B’s tokens.
- Results: Fine-tuned CoPE-VideoLM becomes comparable to leading 3D VLMs without camera poses or 3D point-clouds.
- Efficiency comparison: CoPE-VideoLM outperforms post-hoc pruning approaches across three benchmarks while achieving faster TTFT.P-frames bypass the vision encoder entirely.
H.2 Two-Stage Training
The ablations show that CoPE-VideoLM benefits from both codec-token use and two-stage training. Codec tokens preserve temporal information, while pre-training the Δ-Encoder improves the subsequent end-to-end VideoLM optimization.
- Two-stage training: Two-stage training pre-trains the Δ-Encoder for image-embedding compatibility before end-to-end VideoLM fine-tuning.
- Two-stage training: Direct joint training without Δ-Encoder pre-training performs worse than the two-stage setup.
- Δ-token ablation: Zeroing P-frame Δ-tokens causes substantial performance degradation, showing that the VideoLM actively uses codec information.
- Δ-token ablation: I-frame-only inference still outperforms LLaVA-Video-7B at matched keyframe densities, while adding Δ-tokens improves accuracy further.
- Higher frame rates: 1 to 2 FPS improves MVBench by over 1.8% and TempCompass by nearly 2%, but both decline slightly at 3 FPS.The authors attribute the higher-rate decline likely to train-test mismatch because training used a 1 FPS fusion window.
- Next-frame retrieval: The retrieval model achieves 94.86% recall at five frames, demonstrating that codec primitives preserve useful motion and appearance cues.
I Qualitative Results
Qualitative comparisons across diverse videos show that CoPE-VideoLM provides fine-grained multi-turn answers with consistent speedups and substantially fewer tokens than LLaVA-Video-7B.
- The qualitative examples span fast-motion sports, indoor scenes, and outdoor activities.
- CoPE-VideoLM produces correct, fine-grained answers while maintaining consistent speedups at substantially reduced token counts.KV-cache reuse narrows the absolute latency gap after the second turn, but shorter cached sequences preserve relative speedups.