PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Encoders Built on Image Representations

Jusheng Zhang, Wenhao Wang, Longqi Cai, Liangzhe Yuan, Yuxiao Wang, Ming-Hsuan Yang

arXiv:2610.06616v1cs.CV

TL;DR

The paper asks whether compact video encoding can preserve frame-specific evidence without early frame mixing or forwarding a redundant image sequence. It uses independent image encoding, coordinated budgeted selection, and optional temporal refinement. Across 13 benchmarks and three backbones, it matches full-image aggregate performance with about 28%–35% of the visual tokens.

  • Problem

    Native video encoding couples neighboring frames before selection, while forwarding all independently encoded image tokens leaves relevance and redundancy unresolved.

    Source: §1 · p. 1

  • Method

    A frozen image encoder forms frame-specific candidates, a question-aware selector allocates a fixed budget, and a lightweight refiner adds neighboring-frame context to retained anchors.

    Source: §1 · pp. 2–3

  • Results

    Across 13 benchmarks and three backbones, the pathway matches full-image aggregate performance using about 28%–35% of its visual tokens.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The results support preserving frame-specific evidence before allocation and adding temporal context after selection without expanding the output budget.

    Source: §1 · p. 2, §6 · p. 11

    Selection can make evidence unavailable, and a larger refiner cannot recover information it cannot observe.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

The design of a video encoder determines when frames begin to interact and which frame-specific visual evidence remains accessible to the language model. Native video pathways couple neighboring frames during visual encoding, whereas image pathways preserve independently computed frame representations but incur a much larger visual-token cost when all image tokens are forwarded. We ask a basic question: whether a compact video encoder can instead be built on image representations. To answer this question, we separate three operations that are often coupled: per-frame representation, cross-frame token allocation, and temporal interaction. A frozen image encoder first produces frame-specific candidates. A question-aware selector then allocates a fixed token budget across frames using relevance, diversity, and cross-frame correspondence, after which a lightweight learned refiner reads neighboring-frame context and writes residual updates only to the retained anchors. This preserves source positions and keeps the visual output at the fixed budget. Across 13 benchmarks and three vision-language backbones, the resulting pathway matches full-image aggregate performance while using only about 28%-35% of its visual tokens. Specifically, on Qwen3-VL-8B, it achieves a 13-benchmark macro-average of 62.75 with 1,535 visual tokens, compared with 62.58 for the full Image pathway at 4,424 tokens and 59.49 for native Conv3D at 2,212 tokens. On Qwen3-VL-32B, it reaches a 13-benchmark macro-average of 66.28, compared with 66.09 for Image, while providing a 2.16x end-to-end speedup. These results show that compact video encoding does not require early temporal mixing: frame-specific evidence can be preserved first, allocated jointly, and temporally contextualized after selection.

1. Introduction

The paper asks how to retain question-relevant frame evidence without early temporal mixing or forwarding a redundant image sequence. It proposes image-first selection and refinement, reporting full-image aggregate performance at a smaller visual-token budget.

1. Introduction

  • Native Conv3D couples neighboring frames before selection, while full-image input preserves frame-specific candidates but leaves relevance and redundancy unresolved in a longer sequence.A brief contact change may be harder to isolate from neighboring states after coupling; the concern is reduced independent selectability, not inevitable motion erasure.Source: §1 · p. 1
  • The paper asks whether compact video encoding can retain frame-specific image evidence without mixing frames before selection or forwarding a redundant image sequence.Repeated backgrounds cannot substitute for brief events such as take-off or landing contact.Source: §1 · pp. 1–2
  • The proposed pathway independently encodes frames, allocates a fixed budget using relevance, diversity, and correspondence, then optionally adds neighboring-frame context to retained anchors.The refiner writes residual updates while preserving source positions and the fixed output budget.Source: §1 · p. 2
  • The analysis separates evidence unavailable after selection from prediction error and characterizes temporal interaction omitted across the selection boundary.It motivates preserving candidates before compression without claiming that later interaction is universally better.Source: §1 · p. 2
  • Across 13 benchmarks, the image-first pathway matches full-image aggregate performance using about 28%–35% of its visual tokens.On Qwen3-VL-8B, it scores 62.75 with 1,535 tokens versus Image’s 62.58 with 4,424 tokens; on Qwen3-VL-32B, it scores 66.28 versus 66.09 and provides a 2.16× end-to-end speedup.Source: §1 · p. 2

2. Problem Formulation

The formulation treats compact video encoding as allocating a fixed output budget over frame-specific source evidence. It distinguishes information lost through selection from interaction omitted when context is inaccessible.

2. Problem Formulation

  • The formulation seeks a compact input of B slots, unlike Conv3D’s coupled observations or Full Image’s unallocated candidate set.The budget is allocated to relevant, complementary source evidence while additional temporal context remains separately accessible.Source: §2 · p. 3

2.1. Source-Aligned Candidates and a Budgeted Interface

  • Independent frame encoding forms a dense candidate pool, while source-aligned selection applies the same retained indices across embeddings, positions, and intermediate features without changing them.The selected mask may depend on the whole video and question; refinement can update embeddings while leaving source positions and intermediate features unchanged.Source: §2.1 · pp. 3–4

2.2. Separating Missing Evidence from Prediction Error

  • The conditional prediction-error decomposition separates unavailable evidence from error in using the selected record, which includes the adaptive selection mask.The unavailable-evidence term is the minimum expected divergence among predictors that observe that record; teacher agreement is not answer correctness.Source: §2.2 · p. 4
  • Adding retrieved context expands the observed record and cannot increase its information floor, though a learned predictor can still perform worse.This additional context need not consume output-token slots.Source: §2.2 · p. 4

2.3. When Can Temporal Interaction Be Postponed?

  • In the fixed-operator diagnostic, selection and temporal interaction agree for every input exactly when the operator has no interaction from unselected to selected indices.The diagnostic does not claim that the native backbone is linear or equivalent to this operator.Source: §2.3 · pp. 4–5
  • Reading context from unselected indices restores some omitted interaction while keeping both operation orders at B output rows.The stated residual is an operator-specific diagnostic, not an accuracy bound for an adaptive neural encoder.Source: §2.3 · p. 5
  • Selection chooses which source evidence occupies the compact sequence, while refinement controls which additional temporal evidence can inform retained observations.Neither task requires mixing frames when image candidates are first formed.Source: §2.3 · p. 5

3. Method

The method preserves separate frame candidates until a coordinated selector fixes the output budget, then optionally adds temporal context to retained anchors. The refiner reads beyond the selected set without expanding the visual output.

3. Method

  • The pathway independently encodes candidates, jointly allocates a fixed budget to relevant and complementary evidence, and optionally refines retained anchors with temporal context.The image encoder, language model, and selector remain frozen; only the refiner is learned.Source: §3 · p. 5

3.1. Preserve Frame-Specific Candidates Before Selection

  • Independent encoding keeps frame-specific candidates separately addressable through selection, preserving the choice of features to retain without claiming lossless pixel encoding.Selection retains the same indices across image streams without pooling embeddings or assigning new coordinates; unselected features can remain accessible until refinement.Source: §3.1 · p. 5, Figure 3 · p. 6

3.2. Select Question-Relevant and Complementary Evidence

  • The selection principle favors question-relevant changes and discounts repeated evidence, while treating it as a surrogate rather than an exact optimization of the information floor.A useful subset also retains observations whose value comes from being seen together.Source: §3.2 · p. 6
  • Frame quotas allocate the budget across time, and within-frame DPP plus adjacent-frame correspondence favors diverse candidates and linked observations.The greedy procedure returns exactly B distinct source anchors but does not claim global subset optimality or alter the retained embeddings.Source: §3.2 · p. 6

3.3. Zero-Expansion Temporal Refinement

  • Each selected anchor retrieves feature-similar neighboring-frame tokens, which bottleneck attention uses to write a bounded residual while keeping one output embedding per anchor.Context may come from preceding or following frames in the offline setting; source positions and intermediate features remain unchanged.Source: §3.3 · pp. 6–7
  • Only refiner parameters are learned, using answer supervision and full-image-teacher KL distillation; disabling refinement recovers the training-free selection pathway.Zero expansion describes the B-to-B visual output interface, not the cost of encoding frames, retrieving context, or refining anchors, and does not itself imply speedup.Source: §3.3 · p. 7

4. Experiments

The experiments evaluate the pathway across 13 benchmarks, multiple task families, and three backbones, comparing it with full-input references and budget-matched compression methods. The results report aggregate performance, efficiency, and component ablations.

4. Experiments

  • The evaluation spans 13 benchmarks, three backbones, and general video understanding, temporal reasoning, long-video understanding, intent recognition, and fine-grained perception.Full-input references uniformly sample 32 frames; compression methods target approximately 1.5K tokens, with rankings restricted to budget-matched methods.Source: §4 · p. 7

4.1. Main Results

  • At comparable budgets, Avg.(6) scores of 51.11, 57.23, and 60.44 exceed the strongest compression baseline by 2.96, 1.92, and 1.82 points, respectively.The method uses 1,536, 1,535, and 1,535 tokens versus Image’s 5,561, 4,424, and 4,424, while its Avg.(6) scores are comparable to and slightly above Image’s 50.81, 57.08, and 60.28.Source: §4.1 · pp. 8–9, Table 1 · p. 8

4.2. Cross-Benchmark Generalization

  • Avg.(13) scores are 59.43, 62.75, and 66.28, near or slightly above Image’s 59.26, 62.58, and 66.09 while using about 28%–35% of its visual tokens.The method exceeds the strongest compression baseline by 2.55, 2.89, and 2.42 percentage points, respectively; Figure 4 normalizes scores to Image for each backbone and benchmark.Source: §4.2 · p. 9, Figure 4 · p. 9
  • Improvements over the strongest compression baseline occur on 4/7, 5/7, and 6/7 benchmarks across backbones, but MVBench scores trail by 4.69, 2.13, and 3.01 points.Improvements on EGO, VideoAds, LVBench, and STAR hold across all three backbones; MVBench is a consistent exception.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3. Efficiency Analysis

  • On Qwen3-VL-32B-Instruct, Ours reduces tokens by 65.3%, TTFT by 56.9%, and end-to-end latency by 53.7% versus Image, with a 2.16× speedup.Peak GPU memory decreases by 1.39 GB, while Avg.(13) is 66.28 for Ours versus 66.09 for Image; most latency savings occur before autoregressive decoding.Source: §4.3 · p. 10, Table 2 · p. 10

4.4. Ablation Study

  • On Qwen3-VL-8B, dynamic allocation raises Avg.(13) from 60.12 to 61.74, coordination to 62.64, and temporal refinement to 62.75.Additional Qwen2-VL-7B ablations examine sensitivity to correspondence quality, candidate-frame coverage, and visual-token budget.Source: §4.4 · p. 10

5. Related Work

Prior video encoders differ in when they introduce temporal interaction, while compression methods reduce visual-token sequences. This study separates representation source, cross-frame allocation, and post-selection interaction in controlled comparisons.

Video encoder design

  • C3D and I3D couple frames inside the visual encoder, while frame-based systems aggregate independently encoded images afterward.Video Transformers also express temporal choices through attention factorization.Source: §5 · p. 10

Token reduction

  • Recent methods prune, merge, or compress existing visual streams, whereas this study also varies which representation is available before selection.Related work includes FastV, SparseVLM, VisionZip, FastVID, VidCom2, and FlashVID (Chen et al., 2024; Zhang et al., 2025a; Fan et al., 2026; Liu et al., 2025; Shen et al., 2025; Yang et al., 2025).Source: §5 · p. 11

Study formulation

  • The controlled formulation holds image representations and output budgets fixed while varying coordinated selection and post-selection refinement.It separates representation source, cross-frame token allocation, and temporal interaction.Source: §5 · p. 11

6. Conclusion

The study finds that coordinated selection alone accounts for the primary gains over training-free compression baselines, with refinement adding a small complement. Its formulation preserves independently selectable image evidence, allocates a fixed budget across frames, and adds temporal context afterward.

6. Conclusion

  • The image-first design preserves frame-specific candidates, jointly allocates a fixed output budget, and introduces temporal context afterward through lightweight refinement.The conclusion reports full-image aggregate performance across 13 benchmarks and three backbones using about 28%–35% of the visual tokens.Source: §6 · p. 11

A. Extended Related Work

  • The formulation distinguishes independently selectable image candidates from token allocation and temporal interaction, complementing prior work on temporal fusion and token compression.It asks which evidence remains selectable under fixed-budget selection before or after temporal interaction.Source: A. Extended Related Work · pp. 14–15

B. Additional Experimental Results

C. Extended Ablations and Training Details

  • The extended experiments isolate sources of gains, document refiner training, and test robustness to correspondence, candidate-frame coverage, and token budget.This passage states the scope of the experiments without reporting their individual outcomes.Source: C. Extended Ablations and Training… · p. 15

C.1. Training-Fair Comparison

  • Selection-only reaches Avg.(13) scores of 59.37, 62.64, and 66.22 on Qwen2-VL-7B, Qwen3-VL-8B, and Qwen3-VL-32B, exceeding the strongest prior training-free baselines by 2.49, 2.78, and 2.36 points.These scores match or slightly exceed Full Image references at approximately 1.5K visual tokens; enabling refinement adds +0.06, +0.11, and +0.06 points, respectively.Source: §C.1 · pp. 15–16

C.2. Component and Refiner-Objective Ablations

Across the ablations, coordinated allocation and cross-frame correspondence provide the main gains, while the refiner adds a smaller complement. A 1.5K-token operating point compares favorably with larger-budget baselines, with effects varying by budget and system choice.

C.2. Component and Refiner-Objective Ablations

  • Dynamic allocation raises Avg.(13) from 60.12 to 61.74, and cross-frame coordination raises it further to 62.64; combined refiner objectives reach 62.75.Answer supervision alone gives 62.67 and KD alone gives 62.71, versus 62.64 without a learned refiner.Source: §C.2 · pp. 16–17, Table 5 · p. 17

C.3. Refiner Training Details

  • The refiner uses approximately 18M trainable parameters, with the vision encoder, language model, and coordinated selector frozen during training.Training runs for two epochs on eight A100 GPUs for approximately 96 GPU-hours.Source: §C.3 · p. 17

C.4. System-Level Ablations

  • Ignoring cross-frame correspondence lowers Avg.(13) from 59.43 to 58.50, while shuffling correspondences lowers it to 57.53.The ablations use Qwen2-VL-7B and compare against the default configuration.Source: §C.4 · p. 18, Table 6 · p. 18
  • Expanding the candidate pool from 16 to 24 frames raises Avg.(13) from 57.97 to 58.85, and 32 frames adds another 0.58 points.The increase is monotonic over the evaluated range.Source: §C.4 · p. 18, Table 6 · p. 18
  • Reducing the budget from 1,536 to 768 tokens lowers Avg.(13) by 2.80 points, while raising it to 3,072 tokens improves the score by 1.68 points.The approximately 1.5K setting matches prior compression baselines' operating regime while retaining a substantial token reduction relative to Full Image.Source: §C.4 · p. 18, Table 6 · p. 18

C.5. Additional Efficiency Breakdown

  • TTFT falls from 849.1 ms for Image to 365.8 ms for Ours, while post-first-token generation changes from 62.1 ms to 56.0 ms.Selection overhead is included in TTFT and is offset by shorter visual prefill.Source: §C.5 · p. 18

C.6. Multi-Budget Accuracy–Latency Trade-off

  • Ours exceeds the strongest prior compression baseline at each tested budget, with gains of 2.83/3.35/3.65 points at 768 tokens and 2.66/2.75/2.75 at 3K.The reported advantage persists across a fourfold range of output budgets.Source: §C.6 · pp. 18–19
  • At 1.5K tokens, Ours scores 59.43/62.75/66.28 versus strongest 3K baselines' 58.45/61.50/65.10, with latencies of 215.5 vs. 405.2 ms, 260.5 vs. 455.2 ms, and 421.8 vs. 685.2 ms.The multi-budget figure plots accuracy–latency operating points for three backbones.Source: §C.6 · p. 19, Figure 6 · p. 19

D. Qualitative Analysis

The qualitative examples illustrate evidence patterns relevant to compact selection, including small motion, relations across observations, late events, and repeated views. A representative failure highlights difficulty maintaining the identity of similar subjects across a long temporal span.

Successful cases

  • Ours predicts the ground-truth option in all four Figure 8 examples, which involve late evidence, repeated outcomes, object counting, and directional temporal reasoning.The cases are representative illustrations, not estimates of how frequently a behavior occurs.Source: D. Qualitative Analysis · p. 20, Figure 8 · p. 21

Visualization protocol

  • Displayed frames are selected for visualization and do not reconstruct a method's internal token mask or retained-token support.The protocol shows four representative frames from each 32-frame input.Source: D. Qualitative Analysis · p. 21

Failure analysis

  • In the parrot case, Full Image is correct while Ours, Video/Conv3D, and shown compact baselines choose the same distractor, indicating a limitation in long-range subject binding.The queried late action must be associated with the farther parrot among visually similar instances.Source: Figure 9 · p. 22, D. Qualitative Analysis · p. 22

Successful cases

  • Examples illustrate small local motion, joint relational evidence, late decisive events, repeated-action comparisons, and avoiding double-counting repeated views.The authors present these as examples of allocating a compact budget to complementary rather than redundant evidence.Source: D. Qualitative Analysis · p. 22

E. Coordinated Selection: Objective and Analysis

The selection analysis characterizes determinantal diversity and adjacent-frame complementarity, then proves that feasible greedy moves fill the fixed anchor budget without guaranteeing global optimality.

Determinantal marginal gain

  • The DPP marginal measures the additional volume a candidate contributes beyond the directions already represented by selected candidates.Candidates nearly contained in the selected span receive a small marginal gain even when their individual quality is high.Source: E. Coordinated Selection: Objective… · p. 23

Adjacent-frame complementarity

  • Nonnegative adjacent-frame pair rewards are supermodular, so adding them to diversity does not generally preserve submodularity or a diminishing-returns guarantee.The analysis makes no corresponding approximation guarantee for the greedy procedure.Source: E. Coordinated Selection: Objective… · p. 24

Quota-constrained feasible moves

  • With singleton moves available and quotas feasible, the procedure reaches exactly B distinct anchors in finitely many moves.Pair moves can retain correspondence endpoints jointly, while singleton moves ensure completion.Source: E. Coordinated Selection: Objective… · pp. 24–25

Termination at exactly B anchors

  • The feasibility result guarantees budget completion, not global optimality; selected moves depend on relevance, diversity, and correspondence scores.The procedure constructs source-aligned anchors whose value can depend on what has already been retained.Source: E. Coordinated Selection: Objective… · p. 25

F. Proof of the Conditional Prediction-Error Decomposition

The proof identifies the lowest achievable expected KL prediction gap given an observed record: it is attained by predicting the conditional mean of the teacher distribution. Additional context lowers this information floor to the extent that it changes that conditional distribution.

Conditional prediction-error decomposition

  • The conditional mean teacher prediction is itself a probability distribution, formed by taking the coordinatewise conditional expectation given O.The proof uses nonnegativity of KL divergence to establish the minimizing predictor.Source: F. Proof of the Conditional… · p. 25
  • The minimum expected KL divergence over predictors measurable from O is the information floor I(O).It is attained by the conditional mean teacher distribution, p̄_O.Source: F. Proof of the Conditional… · p. 25

Additional context

  • Additional context lowers the information floor exactly to the extent that it changes the conditional teacher distribution; equality holds when that distribution is unchanged almost surely.The comparison is between records O and O+ = (O, C).Source: F. Proof of the Conditional… · p. 26

G. Proof of the Selection–Interaction Identity

The proof expresses the difference between selecting after interaction and interacting after selection as a cross-boundary contribution. It also characterizes when the two orders agree and bounds the hidden contribution under a norm constraint.

Selection–interaction identity

  • The difference between interact-then-select and select-then-interact is the interaction contribution crossing from unselected indices into selected outputs.The identity follows by expanding both operator orders and subtracting.Source: G. Proof of the Selection–Interaction… · p. 26
  • The two operator orders agree for every feature matrix exactly when the interaction operator has no selected-to-unselected cross-boundary component.The proof establishes both directions of this condition.Source: G. Proof of the Selection–Interaction… · p. 26

Partial context retrieval

  • The worst-case hidden interaction contribution is conditional on a fixed operator, selection set, and retrieved context, not a bound on answer accuracy or adaptive nonlinear encoders.The bound assumes a norm-bounded unobserved feature matrix.Source: G. Proof of the Selection–Interaction… · p. 27

H. Temporal Refinement and Learning Objective

Temporal refinement lets fixed, source-aligned anchors read additional context while preserving the language-model output-token budget. The implementation specifies how the selector chooses anchors, inherits frame quotas, rewards correspondence, and retrieves neighboring-frame context.

H. Temporal Refinement and Learning Objective

  • Refinement reads temporal context and updates retained source-aligned anchors, while keeping the visual output at B tokens rather than expanding it.The B→B property concerns output-token count, not the computation or memory used to produce the refined anchors.Source: H. Temporal Refinement and Learning… · pp. 27–29

I. Complete Selector Specification and Hyperparameters

  • The algorithm encodes frames independently, selects exactly B source-aligned tokens under inherited quotas, and optionally refines them using the unchanged dense pool.The resulting sequence is sorted by source time and spatial position.Source: §I · p. 30

I.1. Question-Conditioned Relevance

  • Question relevance is the maximum cosine similarity between a visual candidate and the encoded question or option fragments.The implementation encodes the complete question with up to eight question/option fragments.Source: §I.1 · p. 30

I.2. Token Quality and Per-Frame DPP Kernel

  • Within each frame, candidate quality and diversity in normalized visual-feature space jointly drive selection.The diagonal term is a numerical stabilizer.Source: §I.2 · p. 31

I.3. Frame Quotas Inherited from the Frozen Allocator

  • The selector inherits non-uniform frame quotas from the frozen SITE allocator and changes source-token identities within those quotas.It does not compute a separate learned quota predictor.Source: §I.3 · p. 31

I.4. Adjacent-Frame Correspondence Graph

  • Correspondence edges link adjacent-frame mutual top-1 matches, and their rewards favor reliable matches whose endpoints show spatial or feature change.The edge is retained only when the forward and reverse match conditions are met.Source: §I.4 · p. 32

I.5. Implemented Selection Objective and Greedy Solver

  • The implemented objective combines per-frame DPP terms and correspondence rewards, but its greedy solver does not claim to globally maximize that objective.It compares feasible singleton and pair scores, adding a pair only when its score is strictly larger than the best singleton score.Source: §I.5 · pp. 32–33

I.6. Temporal Context Retrieval

  • For each selected anchor, context retrieval takes four most-similar tokens from each neighboring frame; retrieved tokens update anchors without joining the output sequence.At video boundaries, the context frames are the two nearest frames on the available side.Source: §I.6 · p. 33

Related papers

Loading 2610.06616v1…