PaperlayerView exact revision on arXiv

Source-linked AI summary

RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling

Can Zhang, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Dianhai Yu, Ruirui Li

arXiv:2609.39563v1cs.CV

TL;DR

VideoLMs and codec-aware front-ends generally treat predictive frames as independent observations, leaving reference dependence and change order to be reconstructed. RESUME carries codec updates through a latent state, improving results on all three temporal benchmarks over both RGB-frame and codec-based baselines at a matched per-frame readout budget.

  • Problem

    VideoLMs encode sampled RGB frames independently, while deployed codec-aware front-ends tokenize each predictive frame independently rather than conditioning its tokens on a carried reference.

    Source: §1 · pp. 1–2

  • Method

    RESUME initializes a compact latent state from an I-frame, causally updates it with fused motion-residual observations, and reads out VideoLM-compatible tokens from the accumulated state.

    Source: §1 · p. 2

  • Results

    2.8, 5.1, and 3.9 points over LLaVA-Video-7B and 0.5, 1.7, and 0.6 over CoPE-7B on TempCompass, TOMATO, and MVBench, respectively.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    At a matched per-predictive-frame budget, state readouts preserve ordered codec changes before they reach the language model and improve temporal reasoning over both RGB-frame and codec-based baselines.

    Source: §5 · p. 9

    The carried state spans predictive frames between successive I-frames; the next I-frame reinitializes it, and long-range aggregation remains the language model’s responsibility.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware front-ends read the motion vectors and residuals that encoding produced, but in their deployed form each predictive frame is still tokenized on its own: the tokens are a function of the current primitives, not of a carried reference. We argue that a more natural function is of both---the current primitives and a carried reference. A clip and its time reversal share the same frames and differ only in the order of changes---an axis that symmetric pooling discards by construction, and that is non-empty in the frozen vision features VideoLMs use---and the codec recurrence already composes those changes in order against a reference state. We introduce RESUME, a stateful codec representation: an anchor I-frame initializes a compact latent state, each subsequent predictive frame is consumed as an update to that state, and a shared readout exposes VideoLM-compatible tokens from the accumulated state. Codec prediction is thereby kept at the representation level and handed to the language model as a trajectory, not as a set of independent token groups. At the same per-predictive-frame token budget as prior codec-aware methods, a predictive frame enters the language model as a readout of what the front-end already knows, not as an encoding of the current primitives alone. Across ten benchmarks, the gains concentrate on temporal reasoning: on all three temporal benchmarks RESUME improves over both the RGB-frame baseline LLaVA-Video-7B (by 2.8, 5.1, and 3.9 points on TempCompass, TOMATO, and MVBench) and the codec-based baseline CoPE-7B, while staying competitive on general and long-form QA. Frozen-transition tests further show anchor dependence, order sensitivity, and useful rollout behavior beyond the training horizon.

1 INTRODUCTION

VideoLMs face a representational trade-off: dense RGB sampling is costly, while sparse keyframes discard intervening changes. RESUME carries codec updates through a latent state, preserving ordered change before readout to the language model.

1 INTRODUCTION

  • Dense RGB sampling preserves transient changes at full image-encoding cost, whereas sparse keyframes are cheaper but discard changes between sampled frames.The stated limitation is representational rather than a lack of context length.Source: §1 · p. 1
  • A clip and its reversal share frames but differ in change order; static pooling is reversal-invariant, while an order-sensitive probe flips on 75–86% of reversed clips.This identifies temporal order as an information axis in frozen vision features that symmetric aggregation discards.Source: §1 · p. 2
  • Codec-aware encoders use motion vectors and residuals, but deployed systems tokenize predictive frames independently, leaving reference dependence and temporal composition to language-model attention.The codec recurrence already applies ordered updates against a changing reference state.Source: §1 · p. 2
  • RESUME initializes a compact latent state from each anchor I-frame, causally updates it with fused motion-residual observations, and reads out tokens at the same per-frame budget as prior codec-aware methods.The state resets at the next anchor, and frozen vision features provide its initial condition and supervision space.Source: §1 · p. 2
  • Training first learns ordered multi-step transitions with intermediate readouts and their changes, then attaches the frozen transition to a VideoLM through a dedicated projector.The language model is trained to use packed state readouts alongside each I-frame token group.Source: §1 · p. 2
  • 2.8, 5.1, and 3.9 points over LLaVA-Video-7B and 0.5, 1.7, and 0.6 over CoPE-7B on TempCompass, TOMATO, and MVBench, respectively.The gains are concentrated on temporal benchmarks, while RESUME stays competitive on general and long-form QA.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work reduces video-token costs through post-hoc compression or codec-derived inputs, but these approaches generally retain independent frame representations or collapse predictive-frame order. RESUME instead combines causal state updates with per-frame readouts.

2.1 FRAME-BASED VIDEOLMS AND POST-HOC COMPRESSION

  • Post-hoc pooling, merging, pruning, and resampling reduce token counts only after dense RGB encoding, so encoding cost is already paid and retained frames remain independent observations.This subsection contrasts frame-based VideoLM workflows with post-hoc compression.Source: §2.1 · pp. 3–4

2.2 CODEC PRIMITIVES AS VIDEOLM INPUT

  • Video-LaVIT discretizes motion but largely discards residuals, EMA aggregates I-frame and motion into a GOP summary that collapses P-frame order, and CoPE-VideoLM tokens at inference depend on current primitives.CoPE-VideoLM's pre-training emulates a warp using a reference re-measured from decoded RGB, but removes the auxiliary modules before VideoLM integration.Source: §2.2 · p. 4
  • Unlike prior state-propagation approaches, RESUME combines causal GOP-local updates, I-frame initialization, early motion-residual fusion, and per-predictive-frame readouts at CoPE's token budget.The related work contrasts this interface with separate motion and residual states, reconstruction-oriented propagation, and ReMoRa's bidirectional scan with a GOP-level readout.Source: §2.2 · p. 4

3 METHOD

Section 3 frames codec-aware video representation as a causal latent transition rather than independent predictive-frame tokenization. It specifies the state/readout interface, its realization and training, and the GOP-level scope.

3.1 FROM INDEPENDENT TOKENIZATION TO STATE TRANSITION

  • Independent tokenization maps each predictive frame’s motion vectors and residuals to tokens alone, whereas the proposed causal transition carries a persistent state and reads it out at each step.The same per-predictive-frame token budget is retained; only I- and P-frames are used because future-referenced B-frames break causal order.Source: §3.1 · p. 5

3.2 REALIZING THE TRANSITION

  • An I-frame initializes reference state; each motion-residual pair is fused as one observation, written through gated cross-attention, and read as compact tokens from accumulated state.The shared readout keeps timestamps in one coordinate system, while its inference frequency can vary without changing the transition.Source: §3.2 · pp. 5–6

3.3 LEARNING A TRAJECTORY, THEN READING IT

  • Training first matches intermediate-state readouts and their changes to frozen-encoder targets, then attaches the transition to a VideoLM and interleaves ordered readouts with I-frame tokens.The second stage leaves the language architecture and objective unchanged; predictive frames enter as readouts of accumulated state, not full images.Source: §3.3 · p. 6

3.4 PREDICTIONS AND SCOPE

  • The evaluation tests anchor dependence, update-order sensitivity, and longer-rollout extrapolation, but state resets at each anchor, so the tests do not establish whole-video memory.The stated scope is a group of pictures; long-range aggregation remains the language model’s responsibility.Source: §3.4 · p. 6
  • The evaluation table covers general, temporal, and long-form QA; RESUME uses at most 64 I-frames and four 8-token predictive readouts per I-frame.Video-MME is evaluated without subtitles, and ActivityNet-QA uses a different scoring model for RESUME than for the prior rows.Source: Table 1 · p. 7

4 EXPERIMENTS

Across ten benchmarks, RESUME’s clearest gains are on temporal reasoning, while frozen-transition tests and matched-I-frame ablations examine whether its readouts preserve ordered, anchor-dependent changes. The experiments also compare static, net-change, and order-sensitive information in frozen vision features.

4.1 EVALUATION BENCHMARKS

  • The evaluation spans ten general, temporal-and-motion, and long-form video QA benchmarks, with up to 64 I-frames per video.Each of four predictive updates per I-frame produces one 8-token readout.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • RESUME improves over LLaVA-Video-7B by 2.8 points on TempCompass, 5.1 on TOMATO, and 3.9 on MVBench, and over CoPE-7B by 0.5, 1.7, and 0.6 points, respectively.The authors conclude that the gains are strongest when benchmarks require temporal evolution, at the same per-predictive-frame readout budget.Source: §4.2 · pp. 7–8
  • General and long-form QA results are mixed: PerceptionTest and ActivityNet-QA improve over both baselines, Video-TT and Video-MMMU match CoPE-7B, Video-MME trails LLaVA-Video-7B by 1.4 points, and LVBench trails CoPE-7B by 3.2.The authors relate the two weaker results to Stage-2 training only on LLaVA-Video-178K, without the academic QA split or image-alignment data, and note Video-MME’s sensitivity to composition.Source: §4.2 · p. 7

4.3 MOTIVATION STUDY: THE INFORMATION HIERARCHY

  • On Figure 3’s reversal test, static pooling has 0.0000 score displacement, difference-symmetric probes flip on 0.68–0.71 of clips, and order-sensitive probes flip on 0.75–0.86.The order-sensitive result is reported across SigLIP and Qwen2.5-VL.Source: Figure 3 · p. 8, §4.3 · p. 8
  • Order-swapped answer pairs are nearly indistinguishable in pooled text space (cos ≈0.98–0.99), motivating reversal as the measure of order sensitivity.Content-different pairs have cosine values of 0.58–0.72; QA ranking is used for static and net-change buckets.Source: §4.3 · p. 8

4.4 CARRIED STATE BETWEEN SAMPLED FRAMES

  • With identical updates applied to different anchors, readouts move 0.79× as far as teacher features and remain closer to their respective anchors on 97% and 86% of videos.Figure 4(a) tests whether the transition retains anchor-dependent context while incorporating the codec update.Source: §4.4 · p. 8, Figure 4 · p. 9
  • Correct update order wins on 65% of videos; beyond the training horizon, carried state beats the memoryless control on 75% of videos at 8 steps and 68% at 16 steps.Resetting to the anchor before each update removes order dependence and yields median error 0.55; carried state remains between the true-frame re-initialization lower bound and memoryless control.Source: §4.4 · p. 8, Figure 4 · p. 9

4.5 ABLATIONS

  • At matched I-frame budgets, adding state readouts improves Video-MME and LVBench accuracy at both 8 and 16 retained I-frames.Table 2 reports gains of 0.8 points on Video-MME (53.3 →54.1) and 0.6 on LVBench (36.7 →37.3) at 8 I-frames; at 16, gains are 1.3 (57.3 →58.6) and 0.9 (39.1 →40.0), respectively.Source: Table 2 · p. 9, §4.5 · p. 9

5 CONCLUSION

RESUME represents codec prediction as a state transition, with I-frame initialization and fused motion-residual updates. Its state preserves reference dependence and ordered change within each I-frame group, while the language model remains responsible for aggregation across groups.

5 CONCLUSION

  • RESUME replaces independent predictive-frame token groups with a representation-level state transition initialized from each I-frame and updated with fused motion-residual observations.This preserves reference dependence and ordered change before information reaches the language model.Source: §5 · p. 9
  • Across experiments, order is measurable in frozen vision features, the carried state responds to anchors and update order, and readouts improve temporal reasoning over RGB-frame and codec-based baselines at a matched per-predictive-frame budget.Source: §5 · p. 9
  • The state is limited to successive I-frames: each new I-frame reinitializes it, leaving aggregation across groups to the language model.Source: §5 · p. 9

A TRAINING DETAILS

Training separates transition learning from VideoLM integration: the transition is first trained against frozen visual features, then attached to a VideoLM while key visual components remain frozen.

A TRAINING DETAILS

  • The frozen SigLIP encoder supplies visual features; the latent state has 16 slots of width 512, and the readout uses 8 tokens.Initialization and readout each use two pre-norm transformer layers; motion vectors use a shared MLP and residuals use a truncated ResNet-18.Source: A TRAINING DETAILS · p. 13
  • Stage 1 trains only the state transition to regress readouts onto frozen frame features over four ordered updates, supervising a readout at every step.Training uses AdamW with peak learning rate 2 × 10−4, weight decay 0.01, gradient clipping at 1.0, and a cosine schedule over 1,500 steps after 100 warmup steps.Source: A TRAINING DETAILS · p. 13
  • Stage 2 starts from LLaVA-Video-7B and trains the language model and a dedicated state projector while freezing the transition, vision tower, and original multimodal projector.It uses next-token prediction on assistant text, masks visual tokens from the loss, and trains for 10,000 steps with up to 64 I-frames and four 8-token readouts per I-frame.Source: A TRAINING DETAILS · p. 13

B TRAINING DATA CONSTRUCTION

Training examples pair LLaVA-Video-178K videos with their question–answer annotations and represent each video as anchor-initialized groups of predictive updates.

B TRAINING DATA CONSTRUCTION

  • Training data uses videos from LLaVA-Video-178K, paired with their question–answer annotations.Source: B TRAINING DATA CONSTRUCTION · p. 13
  • Each video is adaptively partitioned into at most 64 groups, with longer videos yielding more groups.Source: B TRAINING DATA CONSTRUCTION · p. 13
  • Each group begins with an RGB anchor followed by four predictive updates sampled at about one-second strides, with motion vectors and residuals extracted relative to the anchor.I-frames are resized to 384 for the vision encoder; each group’s visual content is combined with its video’s QA annotation.Source: B TRAINING DATA CONSTRUCTION · p. 13

C BENCHMARKS

The evaluation spans general video QA, temporal and motion reasoning, and long-form video. Its benchmarks test varied content, including questions designed to require temporal information.

General video question answering

  • PerceptionTest, NExT-QA, ActivityNet-QA, Video-MME, Video-TT, and Video-MMMU cover diagnostic reasoning, everyday-video questions, activities, short-video follow-ups, and instructional content.PerceptionTest questions span memory, abstraction, physics, and semantics; NExT-QA includes causal, temporal, and descriptive questions; Video-TT uses adversarial follow-ups.Source: C BENCHMARKS · pp. 13–14

Temporal and motion reasoning

  • TempCompass, TOMATO, and MVBench test temporal and motion reasoning, with questions that a single frame or incorrectly ordered frames cannot answer.They cover temporal aspects such as action, direction, speed, and event order, alongside counting, trends, and object interaction.Source: C BENCHMARKS · pp. 13–14

Long-form video

  • LVBench evaluates long videos, with questions on long-range retrieval, entity and event tracking, and reasoning across segments.Its videos range from tens of minutes to several hours, and question counts grow with video duration.Source: C BENCHMARKS · p. 14

D EVALUATION DATA CONSTRUCTION

Evaluation videos are re-encoded into a consistent anchor-and-update structure, and each group contains an RGB I-frame followed by predictive updates. The token-budget comparison measures video duration covered at one frame per second.

D EVALUATION DATA CONSTRUCTION

  • Evaluation videos are re-encoded at 30 fps with I- and P-frames only, one I-frame every 8 seconds, and at most 64 anchor groups per video.Each group contains one RGB anchor and four predictive updates; groups do not cross I-frame boundaries.Source: D EVALUATION DATA CONSTRUCTION · p. 14

E A GROUP OF PICTURES

  • An I-frame is an independently coded RGB image; each predictive frame supplies motion vectors and residuals as one observation that updates the I-frame-initialized state.Motion vectors describe where existing content moves, while residuals capture what motion compensation cannot explain.Source: Figure 5 · p. 15

F TOKEN BUDGET AT ONE FRAME PER SECOND

G INFERENCE LATENCY

Inference latency is measured for 64-second clips sampled at one frame per second while generating 64 text tokens. All three readout configurations are faster than 64-frame LLaVA-Video-7B, while increasing readout groups raises latency.

G INFERENCE LATENCY

  • Each latency sample spans about 64 seconds at one frame per second and generates 64 text tokens using batch size 1, greedy decoding, and a key–value cache.Measurements follow two warm-up runs and average five runs.Source: G INFERENCE LATENCY · p. 16
  • TTFT falls by 0.225, 0.116, and 0.055 seconds, and E2EL by 0.299, 0.199, and 0.140 seconds, versus 64-frame LLaVA-Video-7B.The three configurations are 32+32, 16+48, and 8+56; TTFT is prefill time and E2EL is time to generate 64 tokens.Source: Table 3 · p. 16, G INFERENCE LATENCY · p. 16
  • As configurations shift from 32+32 to 16+48 to 8+56, TTFT rises from 0.461 to 0.570 to 0.631 seconds as fewer I-frames and more readout groups enter the language model.At 8+56, TTFT is close to the 0.619 seconds for 64 I-frames.Source: G INFERENCE LATENCY · p. 16

H TRAINING-DATA SCALE AND DISTRIBUTION

RESUME improves PerceptionTest and ActivityNet-QA over the base and lifts every temporal benchmark, while Stage 2 uses a smaller, differently composed training mixture. The paper notes that Video-MME is sensitive to training-data scale and composition and does not make a controlled attribution.

H TRAINING-DATA SCALE AND DISTRIBUTION

  • Stage 2 fine-tunes on LLaVA-Video-178K alone, excluding the three academic QA corpora and LLaVA-OneVision image-alignment data used for the released checkpoint.The resulting mixture is smaller and differently distributed than the base model’s training data.Source: H TRAINING-DATA SCALE AND DISTRIBUTION · p. 16, Table 4 · p. 17
  • RESUME improves PerceptionTest by +3.3 and ActivityNet-QA by +4.4 over the base, and lifts every temporal benchmark despite its smaller, differently composed Stage-2 mixture.The paper does not attempt a controlled attribution of Video-MME changes to training-data composition.Source: H TRAINING-DATA SCALE AND DISTRIBUTION · p. 16
  • Video-MME changes with training-data composition: the image-alignment corpus raises its score from 63.2 to 63.4, while the academic QA split shifts it from 63.2 to 61.9.These LLaVA-Video training-stage rows are reproduced from Zhang et al. (2024c), and are included to illustrate sensitivity rather than controlled attribution.Source: H TRAINING-DATA SCALE AND DISTRIBUTION · p. 16, Table 4 · p. 17

I TRAINING-FREE PROBES OF THE INFORMATION HIERARCHY

Training-free probes compare how different temporal pooling rules preserve information in frozen vision features. Order-sensitive probes reveal that clips and their reversals are often distinguishable, while symmetric pooling discards that order axis by construction.

Training-free probes of the information hierarchy

  • The three parameter-free probes differ in temporal symmetry: frame-symmetric pooling uses the mean and endpoints, difference-symmetric pooling summarizes consecutive differences, and order-sensitive pooling time-weights them.The probes use frozen features from SigLIP and Qwen2.5-VL at T=8 and T=16.Source: I TRAINING-FREE PROBES OF THE… · p. 17
  • Across both tested lengths, order-sensitive probes flip on 0.75–0.86 of reversed clips; at T=8, linear and exponential probes flip on 0.80 and 0.86.The frame-symmetric probe flips on none at T=8 or T=16, while the difference-symmetric probe flips on 0.68 and 0.71, respectively.Source: I TRAINING-FREE PROBES OF THE… · p. 17
  • At T=16, the frame-symmetric probe scores 0.685 on static questions, while difference probes score 0.195, 0.265, and 0.290; order-question ranking is near random and is not evidence about order.The text-option criterion is nearly order-blind, so question-answering rank is used only for static and net-change buckets.Source: I TRAINING-FREE PROBES OF THE… · pp. 17–18
  • The same hierarchy appears in SigLIP and Qwen2.5-VL: frame-symmetric pooling cannot distinguish reversal, difference-symmetric pooling separates clips through net displacement, and order-sensitive pooling captures order.The section treats temporal order and static appearance as distinct information axes.Source: I TRAINING-FREE PROBES OF THE… · pp. 17–18
  • The probe study identifies an axis a stateful representation should preserve, but trains no transition and compares no architecture.Its evidence is about frozen visual representations, not an ablation of a stateful codec model.Source: I TRAINING-FREE PROBES OF THE… · p. 18

Related papers

Loading 2609.39563v1…