PaperlayerView exact revision on arXiv

Source-linked AI summary

CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding

Yulong Liu, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Guibo Zhu, Sirui Han, Dianhai Yu

arXiv:2609.39924v1cs.CVcs.AI

TL;DR

Visual-token counts grow with video duration and resolution, and post-hoc compression can mismatch dense encoder training with compact deployment. CoVisco combines codec-native input, segmented attention, and learnable abstract tokens while preserving patch tokens for optional selection. It provides competitive video-oriented performance with as few as 400 visual tokens in the evaluated 64-frame setting, though image and visual-document performance remains weaker.

  • Problem

    Visual-token counts grow with video duration and resolution, while post-hoc compression can mismatch dense encoding with compact deployment and remove fine-grained evidence.

    Source: §1 · p. 2

  • Method

    CoVisco uses codec-native inputs, segmented abstract-mediated attention, and learnable abstract tokens while retaining patch tokens for optional downstream selection.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    CoVisco is competitive with selected larger models on video-oriented tasks and exposes as few as 400 visual tokens in the evaluated 64-frame setting, while image and visual-document performance remains weaker.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The compact abstract-token interface offers a video-oriented alternative to exposing all patches, while selected patches can restore local evidence for fine-grained reasoning.

    Source: §1 · p. 2, §5 · p. 15

    OCR results are transfer measurements without OCR-specific pretraining, instruction-tuning scale is modest, abstract-token capacity is fixed at Q = 100, and the effective token budget remains task-dependent.

    Source: §5 · p. 15

Abstract

from arXiv · show

Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mismatch between the representation used during training and the compact interface required at deployment. We present CoVisco, a codec-native vision encoder with native token compression for unified image-video understanding. By combining codec-native input support with segmented attention, CoVisco can encode long visual inputs in a single forward pass without forming dense patch-to-patch interactions across all frames. Each temporal segment is equipped with learnable abstract tokens that learn a compact segment-level representation, while fine-grained patch tokens remain available throughout the encoder. Alternating intra-segment and abstract-communication layers preserve video-level context through the abstract-token channel. A lightweight selector further exposes either abstract tokens alone or abstract tokens augmented with a runtime-selected subset of patch tokens, yielding a compact visual interface that reduces the visual context and prefill burden of downstream MLLMs while retaining fine-grained evidence when needed. Pretrained with contrastive objectives on 565M image--text pairs and 6.4M videos, CoVisco shows competitive performance on video-oriented embedding and multimodal understanding benchmarks. In the evaluated four-segment, 64-frame setting, abstract-only inference uses only 400 visual tokens while achieving video-understanding performance close to, and on some benchmarks exceeding, OneVision-Encoder. Selected patch tokens further improve fine-grained video reasoning. Project URL: https://github.com/ernie-research/CoVisco.git

1 Introduction

CoVisco addresses the scaling and train–deploy mismatch of visual-token compression with a codec-native encoder that learns compact segment representations while preserving fine-grained patch tokens. Its unified interface supports long-video encoding and adjustable visual-token exposure, with competitive video-oriented results but weaker image and visual-document performance.

1 Introduction

  • Visual-token counts rise with video duration and resolution, while post-encoding compression can mismatch dense training representations and remove sparse evidence needed for fine-grained tasks.A 256-frame video at 448 × 448 corresponds to roughly 262K patch tokens.Source: §1 · p. 2
  • CoVisco combines codec-native inputs with segmented attention and abstract-mediated communication to encode long videos without dense cross-segment patch attention.Its selector exposes abstract tokens alone or adds selected patch tokens, preserving fine-grained evidence for detail-sensitive tasks while reducing downstream visual context.Source: §1 · pp. 2–3
  • Learnable abstract tokens form compact segment representations, while the encoder preserves patch tokens and supports a selector-based dynamic image-video interface.The design supports codec, uniform, and collage inputs, plus abstract-only or abstract-plus-top-K deployment.Source: §1 · pp. 2–3
  • CoVisco is competitive with selected larger models on video-oriented tasks, while image and visual-document performance remains weaker, particularly on OCR-heavy benchmarks.In the evaluated 64-frame setting, it exposes as few as 400 visual tokens to the downstream LLM.Source: §1 · p. 3

2 Related Work

Prior work reduces video inputs through sampling, codec-derived signals, or streaming memory, and compresses tokens either after encoding or within the encoder. CoVisco combines codec-native and uniform inputs with abstract-mediated segmented attention, while retaining patch tokens for adjustable downstream use.

2.1 Vision encoders and codec-native video processing

  • Long-video approaches trade temporal coverage, computation, and fine-grained evidence through sampling, codec-aware processing, or compact streaming memory.CoVisco supports codec-native and uniform frame inputs, routing cross-segment communication through abstract tokens for either.Source: §2.1 · p. 3

2.2 Visual token compression and compact interfaces

  • Post-hoc compression can mismatch the dense encoder interface, while some learned query interfaces aggregate away patches; CoVisco supervises abstract tokens and preserves patches for optional use.The related work discusses concerns about attention-based redundancy estimates and whether common benchmarks isolate compression sensitivity.Source: §2.2 · p. 4

3 Method

CoVisco is a unified image-video ViT that combines compact abstract-token representations with preserved patch tokens and segmented attention. Its training and selector provide adjustable visual-token interfaces for downstream language models.

3 Method

  • CoVisco combines a codec-native vision encoder with a compact token interface, whose model, attention, pretraining, selector, and curriculum components are described in the method.Figure 1 summarizes the design.Source: §3 · p. 4

3.1 Problem setup and model interface

  • Each video is divided into segments with Q learnable abstract tokens alongside patch tokens, and the encoder supports runtime-configurable segment and frame counts.The shared ViT-L backbone has 24 layers, hidden size 1024, and about 300M parameters; images use the single-segment case.Source: §3.1 · pp. 4–5, Figure 1 · p. 5

3.2 Abstract tokens as a compact interface

  • Abstract tokens form the compact supervised representation, while patch tokens remain available for optional downstream use rather than being pruned during the ViT pass.The paper calls this learned compact output interface inside the encoder “native compression”; contrastive supervision acts directly through abstract tokens.Source: §3.2 · p. 6

3.3 Segmented attention and abstract-mediated communication

  • Alternating intra-segment and abstract-communication layers restrict cross-segment information flow to abstract tokens, reducing the leading patch-to-patch term from O(N^2) to O(N^2/S).Intra-segment layers use full attention within each segment; abstract-communication layers connect local tokens with abstract tokens across segments.Source: §3.3 · p. 6

3.4 Four-dimensional positional encoding

  • Four-dimensional rotary positions encode segment, temporal, height, and width coordinates, with the least positional bandwidth assigned to segments and the most to space.Patch positions use the full unpruned spatiotemporal grid before positions are gathered for visible patches; abstract tokens have a reserved temporal index and 10 × 10 spatial layout.Source: §3.4 · pp. 6–7

3.5 Contrastive pretraining of CoVisco ViT

  • Pooled abstract tokens are trained with image–text, image–image, and video–text SigLIP losses, using image and video streams with preextracted target embeddings.Training data include 480M LAION-2B image-caption pairs, 85M additional image pairs, and 6.4M videos; images use dynamic resolutions, while videos use four segments and codec or uniform sampling.Source: §3.5 · p. 7

3.6 Token selection and token strategies for MLLM

  • CoVisco trains a lightweight, segment-aware selector that ranks encoder patch tokens and supports abstract-only, abstract-plus-top-K, and full modes; abstract-only uses exactly 400 tokens for 64 frames across four segments.Source: §3.6 · pp. 7–8

3.7 Training data and curriculum

  • Training proceeds through ViT contrastive pretraining, selector and language-model training with the ViT frozen, and instruction tuning with both the ViT and selector frozen.Instruction tuning uses 740K LLaVA-Next image instructions and 800K LLaVA-Next-Videos samples.Source: §3.7 · p. 8

4 Experiments

The experiments evaluate CoVisco as a shared embedding space, a selectable visual interface for an LLM, and a long-video encoder under inference-time segmentation changes. Results show useful video transfer and compact-interface performance, alongside weaker image and OCR-heavy performance and stable long-video scores across tested configurations.

4.1 Evaluation as a unified embedding model

  • At 224 resolution, CoVisco scores 71.5/64.0 on ImageNet-1k val/v2, 66.4 on COCO image-to-text retrieval, and 91.1 on Flickr image-to-text retrieval; at 448, these rise to 73.3/66.1, 67.0, and 92.1.The authors note a gap to the strongest L/14 and L/16 image encoders, despite transfer across image and video inputs.Source: §4.1 · pp. 9–10
  • 65.1 on video classification and 46.2 on video retrieval show CoVisco's clearest MMEB-V2 strength is video, despite its 0.3B vision tower.On video classification, it exceeds the listed 2B and 8B open-source baselines except the Qwen3-VL embedding models.Source: §4.1 · p. 10
  • Image classification reaches 55.1, while visual-document scores are 61.8/42.6/63.8 on ViDoRe-v1, ViDoRe-v2, and VisRAG, below stronger larger baselines, particularly on OCR- and document-centric tasks.The authors frame MMEB-V2 as evidence of cross-task transfer, not state-of-the-art performance on every benchmark.Source: §4.1 · p. 11

4.2 LMM Probing Evaluation

  • 400-token abstract-only CoVisco is competitive with references forwarding 10,368 tokens; at 0.2 retention, VideoMME rises 53.5 to 57.0 and LongVideoBench-Val-V 47.3 to 52.7.In the codec setting, 0.4 retention improves VideoMME from 52.9 to 55.7 and LongVideoBench-Val-V from 48.0 to 50.5.Source: §4.2 · p. 11, Table 3 · p. 12
  • DocVQA rises from 20.5 in abstract-only mode to 75.1 at a fine-grained retention ratio of 0.8, while image-centric scores remain below the strongest baselines, especially on OCR-heavy tasks.Codec-guided selection reduces encoder input; the learned selector controls visual context forwarded to the language model.Source: §4.2 · p. 12

4.3 Long video understanding with different segment configuration

  • Across 64–256 uniform frames, MLVU varies from 61.2 to 62.6; LongVideoBench-Val increases from 47.3 at 64 frames and four segments to 49.9–50.3 at 128–256 frames.At 128 frames, repartitioning from 8 to 16 segments changes MLVU from 61.2 to 62.3, while LongVideoBench-Val differs by 0.3 points.Source: §4.3 · p. 14, Table 5 · p. 14
  • Abstract tokens carry global cross-segment context while intra-segment attention models local evidence, a mechanism consistent with useful representations under finer segmentation.The experiments change segment configurations at inference without retraining the language model.Source: §4.3 · p. 14

5 Conclusion

CoVisco combines a compact abstract-token interface with segmented attention and optional patch-token access for unified image-video encoding. Its clearest reported benefits are on video-oriented tasks, while fixed abstract-token capacity and weaker OCR transfer remain limitations.

5 Conclusion

  • Across evaluations, CoVisco offers abstract-only or abstract-plus-top-K outputs, with stable performance across frame counts, segment configurations, and sampling modes.The abstract interface is not tied to one temporal layout or inference budget.Source: §5 · p. 15
  • The compact interface is most clearly beneficial on video-oriented tasks, while image and visual-document performance remains weaker, particularly on OCR-heavy benchmarks.The model uses a 0.3B vision tower, and the authors characterize its results as broad transfer rather than state-of-the-art performance on every benchmark.Source: §5 · p. 15
  • The current design uses no OCR-specific pretraining, modest instruction-tuning scale, and a fixed abstract-token capacity of Q = 100.The authors identify task-dependent token budgets and OCR-oriented transfer as future-work areas.Source: §5 · p. 15

A Detailed model architecture

The encoder uses one shared abstract-token matrix alongside segment-specific visual inputs. Images form one segment, while videos are divided into S temporal segments.

A Detailed model architecture

  • An image is represented as one segment, whereas a video is divided into S temporal segments.The visual input after patch embedding and sampling is represented using S segments, m visible patch tokens per segment, and hidden dimension d = 1024.Source: A Detailed model architecture · p. 20
  • The abstract tokens use a single shared parameter matrix rather than a separate parameter set for each segment.The matrix is expanded along the segment and batch dimensions at runtime, so segments share the parameterization while having independent token positions at input.Source: A Detailed model architecture · p. 20
  • With the default Q = 100, abstract-only inference exposes SQ tokens as the compact representation.The shared abstract-token matrix is expanded over segments before being prepended to the patch tokens.Source: A Detailed model architecture · p. 20

A.1 Attention routing and positional inputs

CoVisco alternates local intra-segment attention with abstract-mediated communication across segments. This routing preserves local patch access while restricting cross-segment exchange to abstract tokens.

A.1 Attention routing and positional inputs

  • The 24-layer ViT alternates intra-segment and abstract-communication layers, applying self-attention followed by a feed-forward block.In intra-segment layers, each segment's tokens attend only to tokens from that same segment.Source: §A.1 · pp. 20–21
  • In abstract-communication layers, each segment's queries can read its local patches and all segments' abstract tokens, but not other segments' patches.This lets segments exchange information through the abstract-token channel while retaining local patch evidence.Source: §A.1 · p. 21
  • After the final layer, the encoder returns abstract tokens and fine-grained patch tokens as separate downstream output paths.Source: §A.1 · p. 21
  • The 4D rotary embedding uses each visible token's segment, temporal, height, and width coordinates.Patch coordinates are gathered from the full virtual grid after sampling; abstract tokens use reserved summary positions.Source: §A.1 · p. 21

A.2 Output paths and downstream interfaces

The encoder retains both abstract-token and patch-token outputs. Contrastive pretraining pools the abstract path, while downstream language-model alignment can expose abstract tokens alone or add selected patches.

A.2 Output paths and downstream interfaces

  • After the final transformer layer, hidden states are split into abstract tokens and fine-grained patch tokens.Only abstract tokens are sent to the attention-pooling head during contrastive pretraining, then projected into image-image, image-text, and video-text target spaces.Source: §A.2 · p. 21
  • A lightweight selector ranks patch tokens and exposes either abstract tokens alone or abstracts concatenated with selected patches to the language model.The selected visual tokens pass through the MLLM's MLP projector; the ViT is frozen during this alignment stage.Source: §A.2 · p. 21
  • Changing K at inference changes only how many fine-grained tokens reach the language model, not encoder computation or pretrained parameters.Source: §A.2 · pp. 21–22

A.3 Token selector details

The selector scores fine-grained patch tokens in the context of their segment’s abstract tokens, then exposes selected patches alongside abstract tokens. Selection and ranking are performed independently within each segment.

A.3 Token selector details

  • Patch selection is performed independently within each segment, so patches from different segments never compete.The batch and segment dimensions are temporarily flattened for selection.Source: §A.3 · p. 22
  • The default two-layer scoring transformer uses patch tokens as queries and abstract tokens as cross-attention keys and values.Its default layers use 8 attention heads, a 4d feed-forward hidden dimension, GELU, and zero dropout.Source: §A.3 · p. 22
  • A final LayerNorm and linear score head produce one logit per patch after the second selector layer.Source: §A.3 · p. 22
  • Independent sigmoid scores rank patches within each segment, and invalid padded positions cannot be selected.The selector returns top-K indices, the full score map, and selected features gathered from the original ViT patch tokens.Source: §A.3 · p. 22
  • During training, continuous scores gate hard top-K features to provide a gradient path; evaluation uses deterministic scores.Optional Gumbel perturbations support stochastic exploration during training, and selected features are concatenated with abstract tokens before the MLP projector.Source: §A.3 · pp. 22–23

B Attention-cost analysis details

CoVisco replaces dense cross-segment patch attention with local segment attention and global communication through abstract tokens. This reduces the leading patch-to-patch attention term, while abstract communication and implementation details still affect cost and runtime.

B Attention-cost analysis details

  • Intra-segment layers attend only within each segment’s n = m + Q tokens, rather than across the full sequence.Here N is the patch-token count, S the segment count, m = N/S, and Q the abstract-token count per segment.Source: B Attention-cost analysis details · p. 23
  • Abstract-communication layers let each token attend to its local segment patches and the abstract tokens from all segments.This global channel adds attention cost beyond the reduced patch-to-patch term.Source: B Attention-cost analysis details · p. 23
  • The leading patch-to-patch attention term falls from N^2 to N^2/S, while global communication is restricted to abstract tokens.The design trades dense cross-segment patch attention for local segment computation plus a compact global channel.Source: B Attention-cost analysis details · p. 23
  • With fixed segment length, increasing S keeps m and Q constant, but abstract-communication cost still depends on S and the abstract-token budget.The analysis covers encoder-side attention operations, not the LLM-side token budget.Source: B Attention-cost analysis details · p. 23
  • The operation-count analysis excludes kernel efficiency, memory layout, and data-loading overhead, so wall-clock latency requires separate evaluation.Source: B Attention-cost analysis details · p. 23

C H.265/HEVC codec-native input construction

CoVisco constructs codec-native inputs from the original H.265/HEVC stream without re-encoding. Codec selection controls tokens entering the ViT, while a separate learned selector controls the visual tokens sent to the language model.

C H.265/HEVC codec-native input construction

  • The codec-native pipeline uses the original H.265/HEVC stream and its existing GOP structure without re-encoding.The codec provides motion and residual signals from inter-coded frames and spatial references from intra-coded frames.Source: C H.265/HEVC codec-native input… · p. 24
  • Codec selection takes the top Kcodec candidates within each segment, yielding SKcodec tokens entering the ViT.The selected count is equal across segments within an input, though the per-segment budget can vary between training and inference configurations.Source: C H.265/HEVC codec-native input… · p. 24
  • Codec selection controls sparse H.265/HEVC input to the encoder, while the learned selector separately controls visual context forwarded to the LLM.For OneVision-Encoder in the codec setting, the same 10,368 tokens serve as both encoder input and LLM visual tokens.Source: C H.265/HEVC codec-native input… · p. 24
  • Codec-matched CoVisco and OneVision-Encoder settings use 10,368 tokens entering the ViT at CoVisco input resolutions of 224 × 224 and 504 × 504.The comparison also includes OneVision-Encoder’s 64-frame codec input and uniform-frame references.Source: C H.265/HEVC codec-native input… · p. 24
  • Dense-input segment-count experiments retain sampled frame tokens and test 64-, 128-, and 256-frame inputs without the codec token budget.Uniform-frame and frame-collage inputs use the downstream abstract-token and learned-selector interfaces but not H.265/HEVC-derived candidate selection.Source: C H.265/HEVC codec-native input… · p. 25

D Controlled evaluation protocol

The controlled protocol holds the language backbone, projector family, and instruction-tuning data fixed while varying the visual encoder and token interface. It compares codec-native and uniformly sampled visual inputs under separately tracked encoder-input and LLM-side token budgets.

D Controlled evaluation protocol

  • All comparison paths use Qwen3-4B-Instruct-2507, the same MLP projector family, and the same instruction-tuning data.The paths differ in visual encoder and visual-token interface to assess retained performance under a fixed decoder and training recipe.Source: D Controlled evaluation protocol · p. 25
  • The protocol compares CoVisco, a codec-native OneVision-Encoder-style baseline, and a uniformly sampled frame-based SigLIP2-style encoder.Codec and frame variants are reported separately because they assume different access to compressed-video signals.Source: D Controlled evaluation protocol · p. 25
  • CoVisco’s codec-matched input budget is 10,368 tokens entering the ViT, while its abstract-only and abstract-plus-top-K budgets count tokens forwarded to the LLM.OneVision-Encoder’s codec setting uses 10,368 tokens for both encoder input and LLM visual context.Source: D Controlled evaluation protocol · p. 25
  • The codec comparison uses the original H.265/HEVC stream without re-encoding and selects equal token counts per segment.The matched CoVisco codec-input budget is 10,368 at 224 × 224 and 504 × 504; uniform references use 8 sampled frames.Source: D Controlled evaluation protocol · pp. 25–26
Loading 2609.39924v1…