PaperlayerView exact revision on arXiv

Source-linked AI summary

PRISM: Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning

Youngchae Chee, Hosu Lee, Sungjune Park, Junho Kim, Yong Man Ro

arXiv:2608.30388v1cs.CVcs.AI

TL;DR

Cross-view video methods can entangle viewpoint-invariant actions with correlated viewpoint-specific content, limiting semantic consistency under changed compositions. PRISM decomposes videos into view-invariant and view-variant latents, recomposes them under language supervision, and adds a frame-level self-predictive objective. It achieves state-of-the-art cross-view performance, including zero-shot settings and temporal phase prediction beyond in-domain models.

  • Problem

    Existing unified video representations can entangle view-invariant action semantics with correlated view-variant content, undermining cross-view generalization to novel compositions.

  • Method

    PRISM decomposes videos into view-invariant and view-variant latent streams, cross-recomposes them under language supervision, and adds frame-level self-prediction for temporal dynamics.

  • Results

    PRISM achieves state-of-the-art cross-view benchmark performance, including zero-shot settings, and surpasses in-domain models on temporal phase prediction.

  • Takeaways & Limitations

    Compositional latent decomposition is presented as an effective approach to learning view-invariant video representations robust to counterfactual action–scene compositions.

  • Takeaways & Limitations

    PRISM’s decomposition is upper-bounded by biases in the pretrained LVLM captioner used for language-level supervision.

Abstract

from arXiv · show

Cross-view video representation learning aims to capture viewpoint-invariant action semantics despite substantial appearance changes across egocentric and exocentric videos. However, existing methods encode each video as a unified embedding, where view-invariant and view-variant semantics inevitably entangle under co-occurrences - a failure mode we show persists even in cross-view methods explicitly trained for view-invariance. Our key insight is that a view-invariant feature is truly disentangled when it can be sufficiently recomposed with an arbitrary view-variant feature while preserving their independent semantics. Building on this, we propose PRISM, that decomposes video into view-invariant and view-variant latents and recompose them under language supervision encouraging clean decomposition of the two streams. PRISM achieves state-of-the-art results on EgoExo4D, EgoExoLearn, AE2, even surpassing in-domain models under zero-shot setting. Code is available at https://github.com/litcoderr/prism.

1 Introduction

Cross-view video learning seeks viewpoint-invariant action semantics, but unified representations can entangle actions with correlated viewpoint-specific context. PRISM addresses this by decomposing and recomposing view-invariant and view-variant semantics under language supervision, with a temporal self-predictive objective.

  • Motivation: Unified representations can entangle view-invariant actions with correlated view-variant context, causing failures under novel action–scene compositions.For example, “playing tennis” may become associated with the “tennis court” rather than the action itself.
  • Motivation: PRISM defines clean decomposition as preserving a view-invariant semantic identity when recomposed with an off-distribution view-variant counterpart.Language provides controllable semantic compositions beyond observed visual co-occurrences.
  • Method: PRISM decomposes each video into view-invariant and view-variant representations, then uses language supervision to reconstruct corresponding semantic compositions after cross-recombination.The framework is named Predictive Recomposition vIa Semantic Latent DecoMposition.
  • Results: PRISM outperforms state-of-the-art methods in zero-shot cross-view settings and surpasses in-domain models on temporal phase prediction.Evaluations cover cross-view video understanding benchmarks and report stronger view-invariant representations with fine-grained temporal dynamics.
  • Results: PRISM substantially improves retrieval under background-correlation shifts compared with prior cross-view methods.This result evaluates robustness when viewpoint-invariant action semantics and view-variant backgrounds are less predictably correlated.
  • Method: A frame-level self-predictive objective complements clip-level language supervision by internalizing fine-grained temporal dynamics without compromising semantic disentanglement.This objective addresses the limited temporal specificity of language supervision.

2 Related Work

Related work addresses viewpoint changes through vision-language alignment and view-invariant representations, while prior research also identifies persistent action–scene biases. Existing multi-view methods can still conflate shared action and background information.

  • View-Invariant Representation Learning: First-person platforms and robotics motivate visual recognition methods that maintain semantic consistency across egocentric and exocentric viewpoints.Vision-language encoders provide strong representations but remain tied to their training viewpoint.
  • View-Invariant Representation Learning: Prior multi-view methods align visual features across viewpoints or use language-based pseudo-pairs and embedding alignment to learn view-invariant semantics.These approaches include temporal feature alignment and language supervision.
  • Action-Scene Entanglement: Action–scene entanglement is a persistent bias because models may exploit background cues rather than motion semantics.Prior work reports degradation under unseen action–scene compositions and action–scene hallucinations in Video-LLMs.
  • Action-Scene Entanglement: Existing multi-view alignment methods indiscriminately minimize distances between co-observed features, without distinguishing view-invariant actions from view-variant backgrounds.This limitation leaves the source of extracted commonality ambiguous.

3 Proposed Method

PRISM decomposes videos into view-invariant and view-variant representations, then tests their independence through cross-composition and language-level semantic reconstruction. A complementary self-predictive objective adds fine-grained temporal dynamics to both streams.

  • Language-Supervised Decomposition: The Compositional Latent Predictor synthesizes a semantic embedding from cross-composed representations and aligns it with recombined language descriptions.The target combines decoupled view-invariant and view-variant descriptions generated by a pretrained LVLM, then mapped into a text embedding space.
  • Decompose-and-Recompose Schema: PRISM’s Decompositional Encoder separates each video into view-invariant and view-variant frame-level representations.The encoder produces two streams before external supervision is applied.
  • Decompose-and-Recompose Schema: Cross-composition pairs one video’s view-invariant representation with another video’s view-variant representation, breaking natural co-occurrences.The predictor must operate without access to the original intact video composition.
  • Language-Supervised Decomposition: Cross-composition exposes shortcut dependencies between the two semantic factors because their original co-occurrence structure is intentionally broken.Models relying on correlations such as kitchen → cooking cannot consistently reconstruct the correct recomposed semantics.
  • Internalizing Temporal Dynamics: PRISM adds a self-predictive objective because clip-level language supervision provides coarse summaries but lacks fine-grained frame-level temporal specificity.The predictor forecasts future encoder outputs to internalize temporal structures such as lifting, placing, and slicing actions.
  • Internalizing Temporal Dynamics: An EMA target encoder supplies stable future-representation targets, while cosine-similarity training lets the view-invariant and view-variant streams capture temporal dynamics independently.The objective encourages representations to anticipate upcoming transitions rather than only encode the present state.

4 Experiments

PRISM is evaluated for cross-view semantic alignment, fine-grained temporal modeling, objective contributions, dependency robustness, and representation behavior. It achieves strong cross-view and temporal results, with decomposition and temporal prediction improving complementary axes.

  • Cross-View Semantic Alignment: +10.4 in Retrieval and +11.5 in Association over VIEWPOINTROSETTA, with additional gains of +7.46 in Recognition and +7.32 in Anticipation.Skill Assessment remains comparable to VIEWPOINTROSETTA and SigLIP2, at 55.28 versus 55.82 and 55.57.
  • Cross-View Semantic Alignment: PRISM achieves state-of-the-art cross-view performance on Retrieval, Recognition, Association, and Anticipation across EgoExo4D and EgoExoLearn.These tasks assess semantic equivalence, action classification, same-action retrieval, and future-action prediction across viewpoints.
  • Fine-Grained Temporal Modeling: PRISM is the best out-of-domain model across all four AE2 temporal tasks and surpasses the best in-domain model on Phase Ordering and Phase Progression.It exceeds VIEWPOINTROSETTA by +16.13 in Frame Retrieval, +0.55 in Phase Ordering, +26.64 in Action Phase Classification, and +0.80 in Phase Progression.
  • Ablation Studies: The self-predictive temporal objective improves temporal alignment without harming cross-view alignment, while decomposition primarily improves cross-view alignment.Decomposition changes cross-view alignment from 35.6 to 52.2, Ltemp changes temporal alignment from 65.0 to 70.7, and combining both reaches 53.5 and 72.1.
  • Ablation Studies: Exo-only training moderately reduces cross-view alignment but largely preserves temporal alignment, while replacing the captioner causes only marginal differences.Cross-view and temporal alignment change from 53.5 and 72.1 to 48.2 and 71.2 with exo-only data; captioner replacement changes scores by −0.6 and −0.3.
  • Representation Analysis: PRISM representation similarity follows V-I textual similarity while remaining flat along V-V similarity, and its frame embeddings show both semantic separation and temporal continuity.On UNSCENE, PRISM nearly doubles VIEWPOINTROSETTA on R@10 and RSA, while t-SNE trajectories extend continuously over time.

5 Conclusion

PRISM decomposes video into view-invariant and view-variant latent streams and recomposes them under language-level supervision. A self-predictive temporal objective captures fine-grained dynamics, while experiments show consistent state-of-the-art performance.

  • PRISM decomposes video into view-invariant and view-variant latent streams and recomposes them under language-level supervision.This is intended to enforce semantic disentanglement under counterfactual action–scene compositions.
  • A self-predictive temporal objective operates on an independent axis to internalize fine-grained temporal dynamics without compromising decomposition quality.
  • PRISM demonstrates consistent state-of-the-art performance across major cross-view video understanding benchmarks.The conclusion presents compositional latent decomposition as an effective approach to view-invariant video representation learning.

Limitations

PRISM’s semantic decomposition depends on the quality of its pretrained LVLM captioner and is evaluated mainly on procedural human activities involving physical object manipulation. Non-human activities and non-manipulative human interactions remain unexplored.

  • The semantic decomposition is upper-bounded by the quality of the pretrained LVLM captioner providing language-level supervision.Shared LVLM biases that conflate action and scene descriptions could propagate into supervision targets and limit disentanglement quality.
  • The captioner limitation may require debiased captioning models or additional supervision signals independent of language generation.
  • All evaluated benchmarks center on procedural human activities involving physical object manipulation, such as cooking and sports.
  • Generalization to non-human activities and human interactions without tangible physical manipulation remains unexplored.Examples include animal behavior, natural phenomena, conversational turn-taking, social gestures, and emotional exchanges.
  • For conversational, social, and emotional interactions, relevant semantics may not separate neatly into action and scene.

A Implementation Details

The implementation uses frozen vision and text backbones, a Q-Former-based decompositional encoder, language composition, and specified training and preprocessing configurations. Composed sentences are cached and encoded as decomposition targets.

  • Model Architecture: The decompositional encoder uses frozen SigLIP2 and Qwen3-Embedding-0.6B backbones with a depth-4, 8-head Q-Former.The Q-Former produces two per-frame latent vectors of dimension dz=512 for the view-invariant and view-variant streams.
  • Training Configuration: PRISM is trained for 6 epochs on EgoExo4D using 7 NVIDIA A6000 48GB GPUs, with effective batch size 56 and learning rate 7×10^-5.Optimization uses AdamW, weight decay 0.01, and a constant_with_warmup schedule with 10% warmup.
  • Video Preprocessing: Video clips are sampled at 4.0 FPS for up to 32.0 seconds, yielding at most Tmax=128 frames resized to 384×384 pixels.Text inputs use a maximum sequence length of 128 tokens.
  • Cross-view Text Composition: Qwen3-1.7B generates cross-view composed sentences, which are cached per unique pair and encoded into target embedding eA,B for Ldecomp.

A.1 Benchmarks

The evaluation spans multi-view procedural activity datasets and benchmarks covering semantic correspondence, action recognition, association, and fine-grained temporal understanding. UNSCENE specifically tests robustness to counterfactual action–scene compositions.

  • EgoExo4D: EgoExo4D contains 1,286 hours across 5,035 takes, with synchronized ego video from Aria glasses and exo video from 4–5 stationary GoPros.It covers skilled activities including cooking, sports, music, dance, and bike repair.
  • EgoExoLearn: EgoExoLearn uses asynchronously collected ego and exo videos, requiring semantic understanding to establish cross-view correspondence.It provides fine-grained narrations and supports cross-view association evaluation.
  • AE2: AE2 evaluates frame-level temporal understanding across four action-specific sub-datasets with dense per-frame action phase annotations.The sub-datasets cover Break Eggs, Pour Milk, Pour Liquid, and Tennis Forehand.
  • UNSCENE: UNSCENE diagnoses spurious action-scene correlations using web videos with counterfactual action-scene compositions.Its examples contradict typical background contexts, such as fishing inside a bedroom.

A.2 Captioning via LVLM

PRISM generates separate language descriptions for action semantics and filming context, using orientation-aware prompting and language-based recomposition to supervise compositional latents.

  • Prompt design: Each segment receives disjoint V-I and V-V descriptions: action details are separated from camera viewpoint, scene, background, and lighting.V-I captures verbs, hands, tools, target objects, and spatial relations; V-V captures filming context.
  • Captioner model and frame extraction: The captioning pipeline samples 8–16 frames per segment at 2 fps, resizes them to 448×448, and processes them with Qwen3-VL-30B-A3B-Thinking.Sampling uses padded segment duration, keyframe-based seeking, and decode-time downscaling.
  • Output parsing: Generated outputs are parsed into validated JSON, with chain-of-thought blocks and markdown fences removed; failed records are excluded from training.The captioning prompt requires exactly two keys, T V-I and T V-V, each constrained to ≤40 words.
  • Prompt design: Orientation reasoning maps clips to canonical pose patterns so captions avoid screen-relative left/right errors across egocentric and exocentric viewpoints.The protocol first locates body landmarks and then classifies the agent’s pose.
  • Prompt design: When available, ground-truth narration is supplied as a grounding hint, while the model must preserve it and anchor claims to visual frame evidence.The prompt explicitly forbids paraphrasing the narration hint.
  • Cross-view composition: An LLM composer combines V-I from one video with V-V from another into a single sentence describing the cross-view recomposition.The fused sentence preserves concrete action details while using the second video’s description only as scene framing; embeddings from the sentence supervise the compositional latent.
Loading 2608.30388v1…