PaperlayerView exact revision on arXiv

Source-linked AI summary

CAST: Modeling Visual State Transitions for Consistent Video Retrieval

Yanqing Liu, Yingcheng Liu, Fanghong Dong, Budianto Budianto, Cihang Xie, Yan Jiao

arXiv:2603.08648v1cs.CV

TL;DR

Standard video retrieval often ignores preceding visual context, allowing semantically relevant clips that are inconsistent in state or identity. The paper introduces CVR and CAST, a lightweight adapter that predicts context-conditioned residual state updates, and reports gains on YouCook2 and CrossTask, competitiveness on COIN, and improved zero-shot performance across frozen backbones.

  • Problem

    Standard retrieval treats clips independently at inference, limiting its ability to evaluate consistency with preceding visual state and identity cues.

  • Method

    CAST is a lightweight adapter that predicts an instruction-conditioned residual Δ from visual context and updates the current state embedding within frozen foundation-model spaces.

  • Results

    CAST improves retrieval on YouCook2 and CrossTask, remains competitive on COIN, and consistently improves over corresponding zero-shot baselines across diverse frozen embedding backbones.

  • Takeaways & Limitations

    CAST also provides a reranking signal for black-box generated videos, promoting more coherent continuations beyond retrieval.

  • Takeaways & Limitations

    CAST uses a fixed context window of L=5 and remains constrained by the representational quality of its frozen backbone.

Abstract

from arXiv · show

As video content creation shifts toward long-form narratives, composing short clips into coherent storylines becomes increasingly important. However, prevailing retrieval formulations remain context-agnostic at inference time, prioritizing local semantic alignment while neglecting state and identity consistency. To address this structural limitation, we formalize the task of Consistent Video Retrieval (CVR) and introduce a diagnostic benchmark spanning YouCook2, COIN, and CrossTask. We propose CAST (Context-Aware State Transition), a lightweight, plug-and-play adapter compatible with diverse frozen vision-language embedding spaces. By predicting a state-conditioned residual update ($Δ$) from visual history, CAST introduces an explicit inductive bias for latent state evolution. Extensive experiments show that CAST improves performance on YouCook2 and CrossTask, remains competitive on COIN, and consistently outperforms zero-shot baselines across diverse foundation backbones. Furthermore, CAST provides a useful reranking signal for black-box video generation candidates (e.g., from Veo), promoting more temporally coherent continuations.

1. Introduction

The paper reframes video retrieval as selecting temporally and identity-consistent continuations rather than independently matching clips to instructions. It introduces CVR and CAST, which models context-conditioned state transitions for retrieval and generation reranking.

  • Motivation: Long-form video composition requires ordering short segments into temporally coherent procedural narratives.Existing approaches select segments independently, prioritizing local semantic alignment and often producing fragmented narratives.
  • Motivation: Context-agnostic clip encoding can cause identity inconsistencies and violations of evolving procedural state.These failures include abrupt changes in actors, environments, or visual style, as well as temporally invalid continuations.
  • Consistent Video Retrieval: Consistent Video Retrieval conditions target-clip retrieval on preceding visual history and the current text instruction.The protocol uses YouCook2, COIN, and CrossTask as controlled procedural testbeds with strict temporal ordering.
  • CAST: CAST predicts an instruction-conditioned residual Δ that updates the current state embedding while retaining identity-relevant information.The lightweight adapter searches for causally plausible continuations and can rerank black-box video-generation candidates toward more coherent results.
  • Benchmark: The CVR benchmark introduces state and identity hard negatives to diagnose consistency failures beyond semantic matching.State negatives are temporally misaligned clips from the same video, while identity negatives are appearance-misaligned clips from different videos.

2. Method

This section reframes video retrieval as context-aware inference and introduces a diagnostic benchmark and CAST adapter for preserving procedural state and identity consistency. CAST predicts instruction- and history-conditioned residual transitions in a frozen embedding space.

  • Problem Formulation: Standard retrieval independently ranks clips by text-video similarity, ignoring preceding visual context and potentially producing semantically relevant but state-inconsistent results.
  • Problem Formulation: CVR conditions target-clip retrieval on the current instruction and recent narrative history to preserve state and identity consistency.
  • Benchmark Construction: The benchmark uses YouCook2, COIN, and CrossTask procedural data with hard negatives that separately test state and identity consistency.
  • CAST Adapter: CAST is a lightweight query-side adapter over a frozen video encoder that predicts the next visual state through a residual update.
  • CAST Adapter: The residual transition adds instruction-conditioned and temporal-context components, allowing procedural change to be modeled while persistent identity and background cues remain in the anchor state.
  • CAST Adapter: CAST treats visual state as an emergent property of embedding geometry rather than assuming a discrete predefined state space.

3. Experiments

Experiments evaluate CAST on three procedural video datasets and show stronger state and identity consistency than context-free or scalar-fusion baselines, with transfer across frozen backbones.

  • Experimental Setup: CAST is evaluated on YouCook2, COIN, and CrossTask using a fixed 1-vs-9 ranking protocol with state, identity, and easy negatives.The protocol separates training and held-out evaluation splits and reports Accuracy, Mean Rank, State Accuracy, and Identity Accuracy.
  • Quantitative Results: On COIN, learned late fusion reaches 44.66% while CAST reaches 40.47%; on CrossTask, learned fusion falls to 25.52%, 21.9 points below CAST.Scalar reweighting can exploit visual similarity when consecutive steps change little but degrades with more substantial state transitions.
  • Universality Across Backbones: CAST nearly doubles zero-shot accuracy on YouCook2 from 36.75% to 71.68% with InternVideo2 and more than triples CrossTask accuracy from 20.61% to 64.36%.Absolute gains are also reported for VideoPrism, GME-Qwen2-VL-2B, and Qwen3-VL-Embedding.
  • Universality Across Backbones: Across backbones, CAST raises Identity Accuracy from roughly 30% to 69–78%, with gains scaling alongside the quality of the frozen embedding space.Both zero-shot and CAST operate within each backbone’s native frozen vision-language space.
  • Qualitative Analysis: Qualitative examples show CAST correcting state and identity errors that context-agnostic retrieval makes despite semantic relevance.CAST retrieves the correct continuation by modeling state transitions conditioned on visual history.
  • Ablation Studies: Residual modeling improves Early Fusion accuracy from 35.99% to 38.95% and State Accuracy from 38.92% to 43.51% on YouCook2.The residual update anchors prediction around the preceding clip instead of reconstructing the next embedding from scratch.
  • Ablation Studies: The dual-path CAST architecture adds +4.05% Acc. over Early Fusion with Residual, while the Full Ensemble balances state discrimination and identity preservation.Adding visual continuity raises identity preservation from 70.38 to 78.48 with only a modest exact-match drop.
  • Context History: Performance improves sharply from history length L=0 to L=1 and largely saturates thereafter, indicating that the immediate predecessor is the primary context cue.Longer history provides only modest additional improvements or small fluctuations across datasets and metrics.

4. Related Work

Related work spans context-free video-text embedding, temporal cross-modal modeling, procedural understanding, and predictive world models. CAST differs by using instruction-conditioned latent state transitions as retrieval anchors.

  • Video-Text Retrieval: Most video-text retrieval methods map clips and queries into a shared embedding space, while temporal models add cross-modal attention but still optimize global alignment.The paper positions CAST as retrieval under procedural context rather than standard feature matching.
  • Connection to World Models: CAST uses an extrapolated visual state as the retrieval anchor to enforce causal consistency beyond standard semantic matching.Its state-transition adapter is designed for procedural context and human intent encoded by the query.
  • Procedural Video Understanding: Procedural video understanding studies causal dependencies, dense captioning, planning, and visual state modeling, but prior approaches typically represent state change discretely.CAST instead treats procedural progression as a latent transition in an embedding space.
  • Connection to World Models: Recent predictive world models forecast future latent states, whereas CAST maps instruction text to visual state transitions for context-conditioned retrieval.The formulation treats procedural state as an emergent geometric property rather than a predefined semantic label.

5. Limitations

CAST has a fixed context window, depends on the frozen backbone’s ability to resolve subtle differences, and does not explicitly geometrically constrain residual transitions.

  • Scope Boundaries: The fixed context window L=5 may miss longer-range dependencies involving objects or state changes introduced several steps earlier.The authors suggest hierarchical memory for longer-horizon reasoning.
  • Scope Boundaries: CAST remains limited when the frozen encoder cannot resolve subtle state differences such as fine-grained textures or object-configuration changes.As a lightweight adapter, it is constrained by the representational quality of the backbone.
  • Model Assumptions: The residual transition Δ lacks explicit geometric constraints separating temporal progression from persistent identity cues.The authors identify structured regularization as a future direction.

6. Conclusion

The paper introduces CVR to address temporal and identity coherence in retrieval and proposes CAST as a lightweight adapter for state-conditioned residual transitions. Experiments across three procedural datasets show retrieval gains and support reranking generated continuations.

  • CAST models procedural steps as state-conditioned residual transitions within a lightweight adapter.
  • CAST improves retrieval under procedural context on YouCook2 and CrossTask while remaining competitive on COIN.
  • CAST consistently improves over corresponding zero-shot baselines across diverse frozen embedding backbones.
  • CAST can rerank black-box generated candidate videos toward more coherent continuations.

A. Appendix: Backbone Details for Universality Experiments

The universality experiments use frozen pretrained backbones with backbone-specific feature extraction, while evaluating zero-shot retrieval and CAST within each backbone’s native text-video embedding space.

  • All universality backbones remain frozen, with zero-shot retrieval and CAST evaluated in each backbone’s native text-video embedding space.
  • Training schedules vary across datasets and backbones, with alternative backbones trained for 30 / 30 / 50 epochs on YouCook2 / COIN / CrossTask.
  • Frame sampling varies by backbone: most use 3 frames, InternVideo2 uses 8, and VideoPrism uses 8 on YouCook2 but 4 on COIN and CrossTask.
  • Clip embeddings are formed by aggregating frame-level features with mean pooling and then applying L2 normalization.
  • The three inference scores measure native semantic similarity, visual continuity from the last context clip, and compatibility with CAST’s predicted next state.

B. Appendix: CAST Training and Inference Details

The appendix specifies CAST’s disjoint evaluation protocol, hard-negative construction, residual architecture, and shared training and inference settings across datasets and backbone experiments.

  • Evaluation separates CAST training, validation-based ensemble selection, and final benchmark splits.
  • CrossTask keeps training and evaluation source videos disjoint, with reported metrics computed only on crosstask_test.json.
  • The fixed benchmark uses one ground-truth clip and nine negatives, including up to three state and three identity negatives.
  • CAST predicts a transition from the query, last context clip, and context history using complementary instruction-conditioned and attention-based context paths.
  • The final next-state embedding uses a direct residual update followed by L2 normalization without an additional gating module.
  • The main CLIP-based experiments use AdamW, batch size 512, dataset-dependent training schedules, and loss weights λ_s = 5.0 and λ_i = 1.0.

C.1. Human Evaluation Protocol for Generation

The generation study compares standard text matching with CAST reranking on Veo candidate continuations and uses blinded human pairwise judgments on non-overlapping selections.

  • The study samples 300 YouCook2 validation prompts, pairing each instruction with the last context-frame image.
  • Veo generates K = 4 candidate continuation videos per prompt using a simple next-step instruction template.
  • Standard Text Match uses only text-video similarity, whereas CAST Reranking combines semantic matching, visual continuity, and predicted next-state compatibility.
  • Annotators compare different selected videos under overall preference, physical plausibility, and temporal logic criteria.
  • Three annotators judge each prompt, with majority voting and human ties recorded when no majority exists.

C.2. Qualitative Analysis of Generation Reranking

CAST reranking addresses procedural state errors that text-only similarity can introduce by modeling the expected visual state transition. In a blind human-evaluation setup, both methods select from the same Veo-generated candidate pool for comparison.

  • Qualitative example: Text-only ranking can over-rank semantically related but procedurally mismatched candidates, such as ingredients already mixed after the instruction to add ginger garlic paste.This semantic-overlap error produces state-inconsistent continuations.
  • Qualitative example: CAST favors candidates matching the expected next procedural state by explicitly modeling the visual state transition Δ.It extrapolates the current visual state under the query instruction and captures the action's causal constraint.
  • Qualitative example: Figure 7 contrasts text-only ranking with CAST, showing CAST selecting a more coherent continuation when semantic similarity distracts the baseline.The comparison focuses on procedural coherence rather than merely containing related objects.
  • Human evaluation protocol: For each prompt, Veo generates K=4 candidates from one context frame and instruction; each reranker selects one output for blind A/B comparison by annotators.The protocol uses the same candidate pool for standard text matching and CAST reranking, with three annotators and randomized presentation order.
Loading 2603.08648v1…