Source-linked AI summary
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
Feilong Tang, Xiang An, Yunyao Yan, Yin Xie, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Chunyuan Li, Shikun Feng, Changrui Chen, Huajie Tan, Ming Hu, Manyuan Zhang, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, Jiankang Deng
TL;DR
Existing video models often construct dense token sequences from sparsely sampled frames, despite video’s redundant structure and sparse discriminative updates. OV-Encoder aligns representation learning with codec-exposed predictive structure through sparse patch selection, 3D positional encoding, and cluster discrimination. It reports stronger performance across multimodal evaluations, while its spatial selection has a documented bias toward central regions that can weaken peripheral scene coverage.
Problem
Existing video transformers predominantly use frame-centric representations that create dense token sequences and assume equivalent importance across spatial regions and temporal frames.
Method
OV-Encoder aligns spatiotemporal representation learning with video’s predictive structure by selectively encoding codec-informed patches, using 3D-RoPE and cluster discrimination for unified visual representations.
Results
OV-Encoder consistently surpasses strong vision backbones across multimodal benchmarks, with 17.1% and 8.1% Top-1 accuracy improvements over SigLIP2 and DINOv3 on Diving-48 under a 2048 patch budget.
Takeaways & Limitations
Codec-aligned patch-level sparsity supports a unified, structure-preserving encoder that selectively represents informative regions while preserving dense temporal coverage under fixed token budgets.
Takeaways & Limitations
Central motion and residual signals can bias sparse selection toward image centers, reducing peripheral coverage and weakening global scene and fine-grained action representation.
Abstract
from arXiv · showhide
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs. Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics. Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
1 Introduction
OV-Encoder argues that visual modeling should follow video’s predictive structure, where most content is redundant and discriminative motion or residual information is sparse. It implements this principle through codec-guided sparse patch encoding and reports stronger multimodal performance than strong vision backbones across broad evaluation settings.
- Motivation: Video contains substantial predictable redundancy, so uniform dense-grid computation spends capacity on static or easily inferred regions rather than sparse discriminative evidence.Video codecs expose this structure through complete I-frames and motion-compensated P-frame residuals.
- Results: 17.1% and 8.1% Top-1 accuracy improvements over SigLIP2 and DINOv3, respectively, are reported on Diving-48 under attentive probing with an identical patch budget of 2048.OV-Encoder also outperforms multiple strong baselines under dense-patch evaluation.
- Method: Codec Patchification selects informative visual patches from dense video and unifies video, chunk-wise sampling, and single-image inputs with 3D-RoPE.The formulation encodes only 3.1%-25% of regions while supporting irregular spatiotemporal token layouts.
- Method: A self-supervised cluster discrimination objective jointly models object-level and motion-level semantics using structured semantic clusters.The objective provides modality-agnostic visual representation learning rather than relying only on pixel- or feature-level prediction.
- Results: OV-Encoder outperforms Qwen3-ViT and SigLIP2 across multimodal benchmarks under LLM-based probing, including image, video, and document understanding settings.The evaluation uses controlled multimodal fine-tuning protocols and spans 16 benchmarks.
2 Approach
OneVision-Encoder unifies codec-guided video, chunk-wise, and image patchification with a shared Transformer, using sparse visual tokens and relative 3D positional encoding. Its training framework combines object- and motion-level clustering signals for unified representation learning.
- Input and encoding: Up to 64-frame clips are processed with a shared ViT for unified spatiotemporal representation learning.The approach extends beyond the commonly used 16-frame clips and supports higher spatial resolutions.
- Codec Patchification: Codec factorization represents each GOP with one intra-coded frame and predicted frames described by motion vectors and residual signals.Motion vectors encode block-level displacements between a frame and its motion-compensated prediction.
- Codec Patchification: Salient patch masks select a fixed proportion of patches using aggregated codec-derived motion magnitude and residual energy.The selected mask contains |Ωi,n,τ| = ⌊rP0⌋ patches from the patch grid.
- Input formulations: The framework combines dense video-codec, chunk-wise, and single-image spatial patchification before uniformly tokenizing inputs with a shared Transformer backbone.Single-image inputs preserve spatial layout through deterministic row-wise patchification, while chunk-wise sampling selects one frame per temporal chunk.
- Efficiency: 87.5% overall clip-level token reduction is achieved under the default 64-frame, GOP-size-32, 2048-token configuration with P0 = 256.The reported reduction is computed as 1 − 2048/(64P0).
- Representation learning: 3D-RoPE uses relative spatiotemporal offsets to support coherent attention across codec, chunk-wise, and static-image token layouts.The scheme represents temporal and spatial relations jointly and degenerates to spatial offsets for static inputs.
3 Pretraining Dataset
The pretraining data pipeline uses automated processing to create large-scale supervision from image and video datasets, including deduplication, clustering, and multi-label annotation.
- The pipeline generates high-quality supervision signals for massive-scale pretraining data through automated annotation and processing.
- Image data from LAION-400M and COYO-700M is deduplicated, feature-clustered into two million classes, and assigned the nearest Top-10 class centers as multi-label supervision.
- Video data construction draws from HowTo100M, Panda-70M, Kinetics-710, and Something-Something-V2, using uniform sampling of fixed 8-frame clips for feature extraction.
4 Experiments
The experiments evaluate OV-Encoder under controlled multimodal and representation-learning settings, comparing it with established vision encoders and analyzing training-stage contributions.
- OV-Encoder is pretrained with a two-stage pipeline using image, video, and OCR data on 128 A800 GPUs.
- Stage 1 trains an image model on 13B image samples with two million classes, 10 positive labels per sample, and sampled negative class centers.
- Stage 2 adds OCR and video data, trains on 4B samples, and applies codec-guided processing to 64-frame clips with two I-frames and 62 P-frames.
- The multimodal evaluation fixes the language model while varying the visual encoder across image- and video-centric benchmarks under unified instruction-tuning conditions.
- OV-Encoder consistently outperforms SigLIP2 across 16 video, image, and document benchmarks under identical multimodal fine-tuning conditions.
- OV-Encoder outperforms Qwen3-ViT across 16 understanding benchmarks despite using approximately 100B caption tokens versus more than 2.1T for Qwen3-ViT.
- Stage 2 consistently outperforms Stage 1 on multimodal and OCR-related benchmarks while maintaining strong general visual reasoning performance.
4.4 Patch-Efficient Video Understanding Comparison
Codec-guided patch selection redistributes a fixed token budget toward motion-rich regions across dense video, improving video understanding while reducing patch processing and surviving controlled interventions.
- OV-Encoder processes 64-frame source videos without temporal downsampling, extracting codec-guided spatiotemporal patches from motion vectors and residuals.
- 75.0%–96.9% less patch processing enables broader temporal coverage than dense frame-centric processing under the same token budget.
- Codec-selected motion patches are necessary for the observed gains: replacing them with non-motion patches degrades performance across benchmarks, especially motion-sensitive datasets.
- Replacing selected patches with motion patches from unrelated videos causes larger drops on fine-grained temporal benchmarks, indicating semantic specificity of motion cues.
- Preserving semantically correct motion patches produces the strongest performance, whereas semantically mismatched patches and position shuffling are most detrimental.
- Residual magnitudes and motion vectors concentrate selection on temporally salient regions, such as pedestrians tracked across their trajectories.
- Codec patch extraction retains temporally salient patches under a fixed token budget, unlike uniform frame sampling, which can discard fine-grained inter-frame motion.
5 Related Work
Related work spans vision transformers, video sampling, and codec compression, with prior methods increasingly exploiting temporal redundancy for efficient visual representation.
- Vision Transformers and self-supervised methods such as DINOv2 and DINOv3 have become dominant approaches for visual representation learning.
- Conventional video methods sample frames and process all patches within each selected frame, producing long token sequences and high preprocessing cost.
- H.264/AVC and HEVC established motion compensation, transform coding, entropy coding, and more expressive block partitioning for video compression.
- Deep video compression and transformer methods exploit temporal redundancy to reduce tokens, including approaches based on run-length tokenization and compressed video streams.
- The paper uses existing codec signals for visual understanding rather than improving compression efficiency or replacing video codecs.
6 Conclusion
OV-Encoder is presented as a unified self-supervised framework that aligns visual representation learning with video’s predictive structure. Its codec-inspired patch-level sparsity supports strong performance across multimodal and representation-level evaluations.
- OV-Encoder departs from frame-centric modeling by aligning representation learning with the predictive structure of video signals.
- Codec Patchification selectively encodes informative regions while preserving dense temporal coverage under fixed token budgets.
- A shared 3D rotary positional encoding supports sparse spatiotemporal token layouts, chunk-wise temporal modeling, and single-image inputs.
- Cluster discrimination jointly models object-level and motion-level semantics without external supervision.
- OV-Encoder achieves state-of-the-art performance under both LMM probing and attentive probing.
7 Contributors
The contributor list identifies core contributors separately from project leaders. Core contributors are marked in bold in the paper.
- Contributors: The paper lists Feilong Tang, Xiang An, Yunyao Yan, Yin Xie, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Chunyuan Li, Shikun Feng, Changrui Chen, Huajie Tan, Ming Hu, and Manyuan Zhang as contributors.Core contributors are indicated in bold in the paper.
- Project Leaders: The project leaders are Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, and Jiankang Deng.
8 Implementation Details
Implementation uses a unified Vision Transformer pipeline for images and videos, with virtual temporal positioning and codec-based patch sparsity. Multiple input modes share the same backbone and positional encoding.
- Model Architecture and Configuration: OneVision-Encoder Large uses a 24-layer Vision Transformer with hidden dimension 1024, 16 attention heads, and 14 × 14 patches.The model uses GELU activations, Layer Normalization, and Flash Attention 2.
- Unified Patch-based Input Representation: Images are treated as single-frame videos, while videos use a 5D representation mapped to a virtual grid of 64 frames.The mapping provides uniform temporal position encoding across dense, sparse, and Codec inputs.
- Unified Patch-based Input Representation: Visible indices preserve each selected patch’s temporal position when inputs do not explicitly cover all 64 frames.This decouples temporal coverage from token density for sparse inputs.
- Codec-style Patch Selection: Codec-style selection fuses HEVC motion vectors and prediction residuals into patch-level saliency scores.Motion vectors capture displacement, while residuals capture appearance changes not explained by motion compensation.
- Codec-style Patch Selection: Only the top-K globally ranked patches are retained, typically preserving 3.1%–25% of patches and yielding 75%–96.9% compression relative to dense processing.Selected patches are reassembled and passed to the Vision Transformer using sparse visible indices.
- Video Processing Modes and Position Encoding: Pretraining mixes Codec patchification, uniform frame sampling, and Tiling-style spatial concatenation, all using the same ViT backbone and shared 3D Rotary Position Embedding.The temporal RoPE dimension represents positions in the 64-frame virtual grid, while spatial dimensions encode patch row and column.
9 Controlled Evaluation Pipeline
The controlled evaluation decouples the trained vision encoder and compares OV-Encoder with Qwen3-ViT and SigLIP2 under aligned multimodal evaluation conditions.
- For comparison with Qwen3-ViT, OV-Encoder is integrated with Qwen3-1.7B and trained through Stage 1 and Stage 1.5 under the LLaVA-OneVision-1.5 framework.The aligned vision encoder is then decoupled for evaluation under the LLaVA-Next-Videos instruction-tuning setup.
- Figure 7 presents a controlled evaluation pipeline that decouples the encoder for comparison against Qwen3-ViT and SigLIP2.
10 Spatial Bias Analysis
Codec-guided patch selection concentrates tokens near the frame center, reflecting where salient motion cues are often found but weakening peripheral and global scene coverage. Chunk-wise patchification redistributes tokens more uniformly without increasing the token budget.
- Spatial Bias Analysis: Codec-guided selection concentrates most tokens in central frame regions, producing a pronounced spatial center bias.The analysis uses a random subset of 200,000 training videos.
- Spatial Bias Analysis: Central concentration reflects camera framing and subject placement, where salient motion cues and residual signals are typically denser.
- Spatial Bias Analysis: Peripheral undercoverage weakens representation of global scene structure and fine-grained action cues.
- Spatial Bias Analysis: Chunk-wise patchification makes selected-token distributions more uniform by reallocating tokens toward peripheral and boundary regions.
- Spatial Bias Analysis: This spatial rebalancing is achieved through principled token reallocation rather than an increased token budget.
11 Token Allocation Case Study
The case studies examine codec-style token allocation under fixed budgets for videos whose evidence is distributed across continuous motion or concentrated in brief events. Codec-style allocation preserves denser temporal evidence than uniform frame sampling in both regimes.
- Token Allocation Case Study: Fixed-budget comparisons plot time against accumulated visual tokens, with timestamp rugs and I-frame anchors showing each strategy’s temporal allocation.Case 1 uses 2048 tokens against uniform 8-frame sampling; Case 2 uses 4096 tokens against uniform 16-frame sampling.
- Continuous Motion: In continuous-motion videos, missing intermediate timestamps breaks trajectory continuity and harms recognition.
- Continuous Motion: Under 2048 tokens, codec-style allocation samples a 64-frame timeline before saliency-guided P-frame selection, whereas uniform sampling uses eight timestamps.
- Sparse Events: In event-centric videos, uniform sampling can miss short decisive segments entirely, even when the token budget is sufficient.A cooking example includes a pour segment lasting only 1.0 second, from 38.0–39.0s.
- Token Allocation Case Study: Codec-style extraction selectively retains temporally salient patches under a fixed token budget, unlike uniform frame sampling or dense video processing.