PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Learning of Structured Dynamics from Videos

Lukas Knobel, Andrew Zisserman, Yuki M. Asano

arXiv:2607.21576v1cs.CV

TL;DR

Video change mixes camera and object motion, leaving structured dynamics underexplored. This paper introduces SDM to separate primary and residual motion from frozen image features, outperforming naïve global-feature baselines and surpassing VGGT-based probes on 3 of 7 benchmarks.

  • Problem

    Frame-to-frame video change entangles camera and object motion, while structured representations separating these causes remain underexplored despite their importance for interpretable motion learning.

  • Method

    SDM uses future-feature prediction over frozen pretrained image features, combining self-supervised real-video learning with weak scene-level supervision on synthetic videos to organize primary and residual motion.

  • Results

    SDM consistently outperforms global CLS and average-pooled baselines across ProbeMotion settings and surpasses VGGT-based probes on 3 of 7 benchmarks.

  • Takeaways & Limitations

    Frozen pretrained image backbones can be remodeled into structured video-dynamics representations with weak synthetic supervision and unlabeled real videos.

  • Takeaways & Limitations

    The decomposition relies on weak synthetic scene-level supervision, while ProbeMotion uses estimated camera motion and linear probes that can introduce filtering noise and measure only linearly decodable information.

Abstract

from arXiv · show

Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data. We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

1 Introduction

The paper addresses the challenge that video change entangles camera and object motion, proposing SDM to organize dynamics from frozen image features into primary and residual motion representations. Evaluated on ProbeMotion, SDM outperforms global-feature baselines and surpasses VGGT-based probes on 3 of 7 benchmarks.

  • Motivation: Frame-to-frame video change entangles camera motion and object motion, while standard self-supervised objectives can model aggregate change without separating these causes.The paper motivates structured motion representations organized around interpretable physical factors.
  • Method: SDM structures temporal change into a primary motion token for the dominant source of change and a residual motion token for remaining dynamics.The model is built on top of self-supervised image backbones and learns this structure through future feature prediction.
  • Method: SDM extracts structured motion representations from frozen pretrained image features rather than training a video model end-to-end.It is trained through future feature prediction and uses weak synthetic supervision indicating whether the camera or scene is static.
  • Evaluation: ProbeMotion evaluates synthetic and real videos spanning camera motion, object motion, and combined dynamics.The suite is designed to assess structured motion representations across these dynamics.
  • Results: SDM consistently outperforms global CLS and average-pooled-feature baselines and surpasses VGGT-based probes on 3 out of 7 benchmarks.The results support remodeling pretrained image representations into structured video-dynamics representations.

2 Method

SDM learns structured motion representations by predicting future frozen image features while decomposing temporal change into primary and residual dynamics. It uses sequential compensation stages, recurrent non-causal motion tokens, and weak scene-level annotations to align the decomposition with scene semantics.

  • Structured Dynamics Model: SDM predicts ft from ft−1 in frozen feature space, organizing temporal change into primary and residual motion components rather than using a single entangled transition representation.The model operates on frozen visual features from a pretrained image backbone and predicts future feature maps autoregressively in its motion tokens.
  • Temporal modeling: Motion tokens are recurrent and non-causal across time, allowing temporal information to accumulate while each update compares adjacent frame features.The same updates extend naturally to longer clips by carrying pt−1 and rt−1 forward.
  • Sequential compensation: The primary stage extracts pt from consecutive feature maps and compensates ft−1, while the residual stage extracts rt from the remaining mismatch and refines the intermediate prediction.The pair (pt, rt) forms the structured motion representation used throughout the paper.
  • Training objectives: SDM uses weak static-scene and static-camera annotations to selectively supervise the prediction losses and encourage the primary and residual stages to capture their intended dynamics.Static scenes supervise primary compensation directly, whereas static-camera samples constrain the primary stage to preserve the source representation and leave scene dynamics to the residual stage.
  • Implementation: The implementation uses frozen DINOv2-B/14 patch features from all 12 blocks, concatenated and projected to D=768 before SDM prediction.The predictions are projected back to the concatenated multi-layer feature space before applying the loss.

3 Experiments

Experiments evaluate SDM on ProbeMotion across controlled synthetic and real videos, using frozen-token linear probes for camera, object, and action dynamics. SDM generally outperforms frozen-backbone and DeltaTok baselines, approaches supervised representations, and exhibits structured, specialized motion tokens under ablations and qualitative tests.

  • Evaluation setup: ProbeMotion spans synthetic and real videos with camera motion, object motion, and combined dynamics across Kubric, DL3DV, CameraBench, DAVIS2017, YouTubeVOS, and SSv2-110k.Kubric controls camera and object motion with disjoint training/test objects and backgrounds, while real datasets isolate camera or 2D object motion where possible.
  • Evaluation protocol: Linear probes use frozen SDM motion tokens, with regressors for continuous motion targets and classifiers for CameraBench and SSv2-110k.Camera motion is probed on Kubric, DL3DV, and CameraBench; object motion is probed on static DAVIS2017 and YouTubeVOS; action recognition is probed on SSv2-110k.
  • Baseline comparisons: SDM improves over CLS and AVG-pool baselines on all ProbeMotion tasks except static DAVIS2017, where AVG-pool performs best on 2D object-displacement regression.Direct frozen-backbone descriptors remain substantially weaker on 3D motion probes such as Kubric and DL3DV.
  • Baseline comparisons: SDM beats DeltaTok on five of seven ProbeMotion tasks, with 0.16 lower MSE on Kubric object motion and 9.6 p.p. higher accuracy on SSv2-110k.DeltaTok performs better on static DAVIS2017 and CameraBench, while SDM remains competitive with supervised 3D representations and improves over the strongest supervised baseline by 0.13 MSE on Kubric object motion and 0.04 MSE on static YouTubeVOS.
  • Action recognition: On SSv2-110k, SDM exceeds CLS by 8.2 p.p., AVG-pool by 20.8 p.p., DeltaTok by 9.6 p.p., and Pi3X average-feature concatenation by 3.3 p.p.The Pi3X comparison uses a feature concatenation that doubles the linear probe input dimension relative to SDM.
  • Temporal context: Increasing context from T = 2 to T = 4 reduces Kubric camera-motion MSE from 0.26 to 0.16 and object-motion MSE from 0.22 to 0.19, while CameraBench accuracy rises from 84.7% to 85.3%.The gains occur on temporally coherent benchmarks with linear Kubric motion and clip-level CameraBench labels.
  • Structured motion: Token swapping confirms specialization: p is better for camera motion on dynamic-camera datasets, while r performs best for Kubric object motion.Removing either weak scene-level constraint weakens primary/residual structure, and joint extraction increases overlap between the two tokens.
  • Ablations: Intermediate features, unlabeled real videos, diverse pretrained backbones, and λreg = 0.5 improve or balance performance, with DINOv2/3-based backbones performing best overall.MSE is strongest for Kubric object-motion probing, L1 is slightly stronger on some real-video probes, and λreg trades off Kubric object motion against SSv2-110k action recognition.

4 Related work

Prior work provides strong image, video, and 3D representations, but SDM instead freezes an image backbone and learns explicit primary/residual motion tokens from video with weaker supervision. It is closest to feature-space future prediction while structuring temporal change rather than using unstructured spatial tokens or compact latent actions.

  • Self-supervised image encoders: Self-supervised image encoders provide strong frozen visual features and useful 3D cues, but do not directly expose temporal or motion structure.Examples include DINOv2, AIM, and MAE.
  • Self-supervised video encoders: Self-supervised video encoders learn general spatio-temporal representations from unlabeled videos, whereas SDM freezes an image backbone and learns explicit primary/residual motion tokens.Prior methods commonly use masked video modeling or latent predictive objectives.
  • Training image encoders on videos: Video adaptation methods train image-pretrained models with temporal consistency or correspondence learning, while SDM keeps the image backbone frozen and adds structured motion tokens.Other work trains strong image encoders from scratch using long, uncurated videos.
  • 3D reconstruction models: 3D reconstruction models estimate camera pose, depth, point maps, and related geometry under strong 3D supervision, whereas SDM remodels frozen image features into motion tokens with much weaker supervision.Dynamic-scene and 4D reconstruction extensions are also represented in this line of work.
  • Latent action and world models: Latent action and world models predict future frames or pretrained features without dense annotations, often representing temporal change with spatial future tokens or compact latent actions.SDM is closest to feature-space future prediction but explicitly structures motion into primary and residual components.

5 Conclusion

The Structured Dynamics Model extracts structured primary and residual motion latents from frozen image backbones through future-feature prediction, using weak synthetic supervision and unlabeled real videos. Its tokens specialize by scene dynamics, with residual compensation capturing remaining object-centric motion after primary motion explains camera-induced change.

  • 5 Conclusion: SDM learns structured primary and residual motion latents from frozen pretrained image backbones through future-feature prediction, combining weak synthetic supervision with unlabeled real data.The results suggest frozen image backbones can be remodeled into structured video-dynamics representations with interpretable temporal abstractions.
  • 5 Conclusion: The primary token captures camera motion in dynamic-camera videos and object motion in static-camera videos.Token specialization follows the dominant scene dynamics.
  • 5 Conclusion: The residual token captures remaining object-centric dynamics when primary motion explains camera-induced change.This supports residual compensation as a useful inductive bias for structured motion representation.

A Dataset and Evaluation Details … A.6 Metrics and probe selection

The evaluation combines controlled Kubric scenes, filtered real-world motion benchmarks, dataset-specific splits and balancing, targeted linear probes, and held-out classification or regression metrics. These choices test camera, object, and combined dynamics while reducing leakage, class imbalance, and probe-selection sensitivity.

  • A.1 Kubric generation: Weak scene-level supervision separates camera-induced motion, object motion, and their combination through moving-camera/static-scene, static-camera/dynamic-scene, and moving-camera/dynamic-scene sequences.Dynamic objects receive random initial velocities, while dynamic cameras follow linear trajectories while looking toward a fixed point.
  • A.2 Static-camera object-motion benchmarks: Real-world object-motion benchmarks draw from DAVIS2017 and YouTubeVOS, retaining timestamps with RGB frames and annotations and summarizing masks by visible area and image-plane centroid.Near-static-camera clips are selected using VGGT-based camera-pose estimates, while objects must exceed 15 pixels and move at least 0.05 in normalized centroid displacement.
  • A.2 Static-camera object-motion benchmarks: Motion targets use normalized frame-to-frame centroid displacements for continuous horizontal and vertical object-motion components in relative image-plane units.The target is computed from the selected object’s mask sequence.
  • A.3 Dataset splits and sampling: Dataset splits use official or video-level partitions where available, a 70/30 pseudo split for CameraBench, and class-balanced or tail-aware sampling to limit imbalance and leakage.Regression sampling retains all motion magnitudes above the 75th percentile and subsamples lower-motion examples to the same count.
  • A.4 Probe targets: Linear probes jointly predict 6D camera motion or 3D Kubric object translation for regression, while classification probes target SSv2 actions or CameraBench’s camera-motion label groups.Camera motion uses [vx, vy, vz, yaw, pitch, roll], whereas Kubric objects do not rotate and use [vx, vy, vz].
  • A.5 Probe training: Probes are single linear layers trained on frozen motion representations for 100 epochs with SGD, momentum 0.9, cosine learning-rate decay, and multiple learning rates.Depending on the task, probes use the primary motion token p or residual motion token r; features are precomputed without input augmentations.
  • A.6 Metrics and probe selection: Classification reports averaged top-1 accuracy, regression reports normalized-target mean squared error, and results select the best held-out metric across epochs and learning rates.This selection reduces sensitivity to probe optimization hyperparameters.

A.7 DeltaTok evaluation

DeltaTok is evaluated as a strong self-supervised motion baseline using its released Kinetics-700 ViT-B model and inference protocol. Its comparison with SDM is compute- and resolution-mismatched, favoring DeltaTok through larger inputs, more iterations, and larger batch size.

  • Evaluation protocol: DeltaTok is evaluated as a strong self-supervised baseline using the released Kinetics-700 ViT-B model and the same protocol as SDM.Frames are resized to 512×512, and the final delta token from each clip serves as the motion descriptor.
  • Compute and resolution mismatch: DeltaTok uses 512×512 inputs, DINOv3 features, 1.6M iterations, and effective batch size 1024, whereas SDM uses 224×224 inputs, DINOv2-with-registers features, 200k iterations, and batch size 128.The models therefore differ in both input-resolution and compute budgets.
  • Compute and resolution mismatch: 8× more optimization iterations and an 8× larger effective batch size give DeltaTok a substantially higher training budget than SDM.The passage also notes that DeltaTok operates at substantially higher spatial resolution.

A.8 Supervised model evaluation · A.9 Sample visualization details · B Additional results

The paper evaluates supervised geometry-oriented models as frozen feature extractors under the same probing protocol as SDM, and specifies procedures for visualizing projected features and prediction errors. These evaluations use standardized preprocessing, feature extraction, PCA-based RGB projections, and shared error-map normalization.

  • A.8 Supervised model evaluation: Supervised 3D and geometry-oriented models are evaluated as frozen feature extractors using their default preprocessing and the same linear-probing protocol as SDM.Motion descriptors are derived from consecutive frames using the same temporal-concatenation or difference baselines as frozen image features.
  • A.8 Supervised model evaluation: VGGT-1B inputs are RGB-converted, resized to width 518, rounded to height divisible by 14, center-cropped to 518×518, and scaled to [0, 1].Features use either the camera token or mean-pooled patch tokens from the final aggregated token map.
  • A.8 Supervised model evaluation: Depth Anything 3 uses DA3-GIANT-1.1 with its official inference pipeline and camera-token or mean-pooled patch-token features from the final backbone output.The supplied passage also begins describing Pi3X preprocessing but is truncated.
  • A.9 Sample visualization details: Dense feature maps are projected to RGB with sample-specific PCA fitted jointly on normalized feature tokens collected from all compared sequence frames.Tokens from all frames are flattened into one feature matrix before PCA fitting.
  • A.9 Sample visualization details: The first three principal components become RGB channels, each independently scaled to [0, 1] using 1st and 99th percentiles before spatial reshaping and bilinear upsampling.The percentile statistics are computed on the PCA fit set.
  • A.9 Sample visualization details: Pixel-wise error maps use L2 distances between predicted and ground-truth feature maps in projected space, normalized with a shared 99th-percentile scale and overlaid as heatmaps.The heatmaps are displayed over the corresponding RGB video frames.

B.1 SSv2 camera motion analysis

SSv2 clips exhibit limited camera motion, with motion levels between CameraBench’s slow- and regular-speed subsets and rotation closer to the slow-speed subset. This motivates probing action-related object dynamics as a dominant source of temporal change.

  • Camera-motion analysis: The analysis randomly sampled 5k SSv2 clips and estimated average frame-to-frame camera rotation and translation with VGGT.The same procedure was applied to CameraBench, separating slow-, regular-, and fast-speed samples.
  • Camera-motion analysis: SSv2 camera motion is lower than in regular- and fast-speed CameraBench clips and closer to the slow-speed subset in rotation.Rotation is reported in degrees per second, while translation uses VGGT’s normalized scene-scale units per second.
  • Camera-motion analysis: Because camera-induced change is small, action-related object dynamics are expected to dominate temporal change in many SSv2 clips.This reasoning parallels the static-camera DAVIS2017 and YouTubeVOS object-motion benchmarks.

B.2 Baseline results · B.3 Temporal context · B.4 Structured motion

The ablations identify effective motion descriptors for frozen and supervised representations, while temporal context and structured tokens reveal when different motion components are most informative. SDM benefits from longer temporal context and its primary and residual tokens specialize by dominant motion source and benchmark.

  • B.2 Baseline results: Feature differences of average-pooled DINOv2 features perform best overall, especially on regression tasks, with descriptor quality depending on pooling strategy and task.The ablations compare concatenation and feature differences for direct-feature baselines and supervised-model descriptors.
  • B.3 Temporal context: SDM benefits most from longer context on temporally coherent benchmarks, including linear Kubric motion and SSv2-110k, with improvements continuing beyond 5 frames.The same trend holds for the multi-frame average-pooled feature-difference baseline, which remains below SDM with longer context.
  • B.3 Temporal context: The temporal-context analysis reports full results with standard deviations and compares SDM against a multi-frame baseline using average-pooled feature differences.SDM and the baseline are evaluated with longer temporal contexts than the 5-frame training context.
  • B.2 Baseline results: The descriptor ablations compare CLS tokens, average-pooled patch features, camera tokens, and average-pooled feature maps across DINOv2, VGGT, and Depth Anything 3.DINOv2 descriptors include concatenation and feature differences across consecutive frames, plus multi-frame variants averaging frame-pair descriptors over time.
  • B.4 Structured motion: The primary and residual tokens specialize according to the dominant motion source across camera-motion, object-motion, and action-prediction benchmarks.The primary token is strongest for camera motion in dynamic-camera videos, object motion in approximately static-camera videos, and SSv2-110k action prediction.
  • B.4 Structured motion: The residual token is stronger for Kubric object motion, while on static DAVIS2017 it outperforms the primary token.These results are reported with standard deviations in the full structured-motion results.

B.5 Ablation · B.6 Additional visualization

Ablations show that SDM’s structured motion decomposition depends on supervision, sequential extraction, intermediate features, real-video training, MSE loss, and regularization. Additional visualizations illustrate how primary and residual compensation progressively reduce feature-map error.

  • B.5 Ablation: Weak supervision and sequential extraction specialize tokens: p predicts camera motion better (0.16 vs. 0.28), while r predicts object motion better (0.19 vs. 0.21).Removing static-scene supervision or jointly extracting tokens weakens this specialization.
  • B.5 Ablation: Static-camera supervision prevents object motion from being absorbed into p: without it, p becomes more predictive of object motion than r (0.14 vs. 0.38).Removing static-scene supervision also makes r nearly as predictive of camera motion as p (0.21 vs. 0.18).
  • B.5 Ablation: Using all encoder layers improves Kubric camera error from 0.50 to 0.16, DL3DV error from 0.89 to 0.59, and SSv2-110k accuracy from 13.6% to 23.1%.The 12× increase in raw feature dimensionality raises training time by only 1.2×.
  • B.5 Ablation: Adding unlabeled real videos lowers DL3DV error from 0.91 to 0.59 and raises CameraBench accuracy from 76.6% to 85.3% and SSv2-110k accuracy from 10.7% to 23.1%.The gains occur across all evaluated real benchmarks.
  • B.5 Ablation: MSE lowers Kubric object-motion error from 0.30 to 0.19 and camera-motion error from 0.18 to 0.16, despite L1 performing better on the real-video benchmarks.L1 yields DL3DV error 0.57 vs. 0.59, CameraBench accuracy 87.4% vs. 85.3%, and SSv2-110k accuracy 24.4% vs. 23.1%.
  • B.5 Ablation: Setting λreg = 0.5 balances Kubric object-motion performance (0.19) and SSv2-110k accuracy (23.1%).Smaller λreg values favor SSv2-110k, whereas larger values improve Kubric object-motion probing.
  • B.6 Additional visualization: Primary compensation significantly reduces camera-motion-induced feature-map error, especially in the background, while residual compensation further reduces error from scene dynamics.The visualization compares uncompensated, primary-compensated, and residual-compensated predictions on a CameraBench sample.
  • B.6 Additional visualization: Additional visualizations mark matched token–target pairs in green and show that MSE loss improves Kubric object probing.These panels provide qualitative support for token specialization and the loss ablation.

C Limitations · D Information about used assets · E Hardware resources

The paper acknowledges supervision, evaluation, and probe limitations, documents the pretrained models and datasets through Table 14, and reports the computational resources and costs of training and evaluation. Experiments used a four-GPU H100 node, with main results and ablations requiring approximately 132 and 396 GPU-hours, respectively.

  • C Limitations: Medium static-camera regularization balances object motion and action prediction.
  • C Limitations: Weak synthetic scene-level supervision anchors decomposition, while fully self-supervised constraints could improve scalability.
  • C Limitations: ProbeMotion’s estimated camera motion for static DAVIS2017 and YouTubeVOS can introduce filtering noise, while linear probes assess only linearly decodable information.
  • E Hardware resources: Experiments used a local-cluster node with 4 NVIDIA H100 GPUs and 128 CPU cores; each SDM run took approximately 11 hours, with up to 75GB RAM and 25GB VRAM per GPU.ProbeMotion evaluation took approximately 25 minutes.
  • E Hardware resources: 132 GPU-hours covered the three-seed main results, while 9 ablation runs required approximately 396 GPU-hours.The main-results estimate is approximately 11h×4×3 = 132 GPU-hours.
  • D Information about used assets: Table 14 summarizes the existing datasets and pretrained models used in the paper.The paper also notes that Table 14 provides more detailed asset information.
Loading 2607.21576v1…