PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time

Ziqi Ma, Hongqiao Chen, Georgia Gkioxari

arXiv:2609.35704v1cs.CV

TL;DR

Camera-controlled video models can follow camera paths but often fail to generate evolving scenes, and adapting them risks harming camera control. DynaTokens trains scene-specific tokens to add localized dynamics while keeping the base model frozen. It outperforms existing camera-controlled models on VBench2 dynamic categories and WorldScore without compromising camera control.

  • Problem

    Camera-controlled video models struggle to generate dynamic scenes, and adapting them is challenging because camera conditioning is entangled with dynamics generation.

    Source: §1 · p. 1

  • Method

    DynaTokens trains a few scene-specific tokens to steer video latents toward localized dynamics while keeping the base model frozen.

    Source: §1 · pp. 1–2

  • Results

    DynaTokens surpasses current camera-controlled world models on VBench2 dynamic-related categories and WorldScore without compromising camera control.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper identifies a structural mismatch between global LoRA updates and spatially localized object dynamics, and aligns its adaptation design with the learning target.

    Source: §1 · p. 2

Abstract

from arXiv · show

Video generation must account for two sources of motion, one induced by the observer's camera path and the other caused by scene dynamics. An ideal camera-controlled video model should account for both motions: let users move the camera while evolving the scene dynamics. While current models handle camera-induced motion well in static settings, they struggle for dynamic scenes: objects are static, move incorrectly, or degrade in generation quality. We introduce DynaTokens, a lightweight set of learnable scene-specific tokens that teach dynamics to an existing camera-controlled world model. Our method is motivated by a simple asymmetry between the two sources of motion: whereas camera motion affects the generated view globally, object dynamics are spatially localized. Through cross-attention, DynaTokens trains the learnable tokens from a few example trajectories for a scene while keeping the base model frozen, and enables dynamics under new query camera paths. DynaTokens achieves a better simultaneous dynamics-camera tradeoff on VBench2 and WorldScore evaluations than LoRA, block finetuning, and specialized trainable-layer baselines. Analyses of token attention, ablations, and motion temporality suggest that matching the trainable interface to the structure of the learning target is important for effective adaptation. Project website: https://glab-caltech.github.io/dynatokens/

1 Introduction

Camera-controlled video models can follow camera paths but struggle to generate evolving scenes, and adapting them can compromise camera control. DynaTokens uses scene-specific tokens to teach localized dynamics while preserving global camera control, with reported gains on dynamic-video evaluations.

1 Introduction

  • Camera control affects video patches globally, while scene dynamics can be spatially local, motivating tokens that teach dynamics without altering global scene generation.This decoupling is intended to enable dynamic scenes while preserving camera controllability.Source: §1 · p. 1
  • A few scene-specific tokens are trained with the base model frozen from about 15 trajectories, including imperfect examples, to generate prompt-aligned, temporally continuous videos on unseen camera paths.The examples may be short, contain background artifacts, or have motion discontinuities.Source: §1 · p. 2
  • DynaTokens surpasses current camera-controlled world models on VBench2 dynamic-related categories and WorldScore without compromising camera control.LoRA, block finetuning, and Test-Time Training layers struggle; the method also supports physics and stylistic changes.Source: §1 · p. 2
  • The paper analyzes token design and attention, probes motion temporality, and identifies potential for camera-controlled video generation beyond synthetic environments.It frames these analyses as insights for test-time adaptation and future world-model development.Source: §1 · p. 2
  • DynaTokens addresses a structural mismatch: LoRA applies global parameter updates, whereas object dynamics are spatially localized.The design aligns the trainable interface with the localized learning target.Source: §1 · p. 2

2 Related work

Related work covers camera-conditioning approaches, methods for adding dynamics to camera-controlled world models, and test-time adaptation strategies. The paper positions DynaTokens as a test-time-training architecture for localized dynamic learning.

2 Related work

  • Camera-controlled video models condition generation through noise warping, PRoPE, action conditioning, or Plücker ray embeddings.DynaTokens is designed to minimally affect camera conditioning while selectively teaching dynamics.Source: §2 · p. 2
  • Prior dynamics approaches either loosen 3D constraints in general-purpose world models or use 4D-aware generation; BulletTime has short, small-range camera movement, while Atlas requires multiple input views.The section also identifies limited training-data dynamics as a challenge for camera-controlled models.Source: §2 · p. 2
  • Test-time training has used LoRA or specialized TTT layers, and DynaTokens is introduced as an architecture for localized dynamic learning in world models.The cited applications include ARC, scientific discovery, and long-video generation.Source: §2 · p. 2
  • Figure 1 depicts dynamic tokens cross-attended by video patch tokens, trained on about 15 camera trajectories for inference on unseen paths.Its attention visualization is presented as corroboration of localized dynamics.Source: Figure 1 · p. 3

3 Method

DynaTokens treats scene dynamics as localized changes and learns scene-specific tokens while keeping the pretrained camera-controlled model frozen. Trained on a few trajectories, it can generate dynamics under unseen camera paths.

3.1 Motivating example

  • The motivating model decomposes scene dynamics into a localized gate m and a dynamics operator G, applying the change only where the gate is active.The example’s gate covers the object’s positions in both static and dynamic frames.Source: §3.1 · p. 3

3.2 Our approach: dynamic tokens

  • In each double-stream block, video patches cross-attend to learned tokens whose attention weights gate relevant patches; the frozen-base experiments use 16 tokens of dimension 64.The cross-attention output is added back to the model alongside self-attention paths with and without camera control.Source: §3.2 · p. 4

3.3 Test-time training of dynamic tokens

  • Training uses about 15 curated videos from a few trajectories and a diffusion-forcing-type objective to reduce exposure bias and support generalization to unseen camera paths.The training videos may contain artifacts and cover only a few camera trajectories.Source: §3.3 · pp. 4–5
  • DynaTokens learns correct dynamics under unseen camera trajectories, whereas fine-tuning or LoRA on the same curated data yields poor results.The comparison is reported for training on the same data.Source: §3.3 · p. 4

4 Experiments

Experiments test whether DynaTokens can add scene dynamics to camera-controlled video models while preserving camera control. Across benchmark, human, generalization, and qualitative evaluations, it improves dynamics and supports varied changes under camera movement.

4 Experiments

  • Evaluation covers VBench2 dynamics categories and WorldScore motion accuracy and camera control, with camera-motion adjustments to their original static-camera evaluation setups.The VBench2 sample includes 120 videos; the WorldScore sample includes 90 videos.Source: §4 · p. 5

4.1 Comparing with state-of-the-art camera-controlled models

  • 28% improvement in Dynamic Spatial Relations, 35% in Motion Order Understanding on VBench2, and 21% improvement in WorldScore Motion Accuracy, while maintaining camera control.Table 1 compares DynaTokens with state-of-the-art camera-controlled video models across these dynamics metrics and camera control.Source: §4.1 · p. 6, Table 1 · p. 7
  • DynaTokens wins 71%–95.5% of dynamics pairwise comparisons against each baseline; its camera result is a close match with WorldPlay at 48%.The rankings comprise 100 comparisons by five independent annotators.Source: §4.1 · p. 7

4.2 Comparison with other test-time methods

  • DynaTokens has a clear dynamics advantage across both benchmarks and all metrics over LoRA, LoRA-no PRoPE, block finetuning, and TTT layer.LoRA struggles to disentangle dynamics from camera control; last-block finetuning is insufficient, while TTT layer preserves camera but distorts objects and learns dynamics ineffectively.Source: §4.2 · p. 7

4.3 Generalization to Unseen Camera Paths

  • Trained on approximately 15 camera paths, DynaTokens generalizes to diverse unseen paths, which differ from seen paths by 39.4 degrees geodesically on average.Evaluated unseen paths also differ completely in their first 28 frames’ actions.Source: §4.3 · p. 8, Table 2 · p. 8

4.4 Enabling diverse dynamics: physics and stylistic change

  • DynaTokens learns physics changes and global style transformations while maintaining camera navigation; the physics results are qualitative because available IoU and MSE evaluation assumes a static camera.Examples include evolving physics and transforming a cityscape into line art or a room into a spaceship; Lingbot and Genie 3 struggle with physics, and Lingbot fails the style transformations.Source: §4.4 · p. 8, Figure 5 · p. 8

5 Ablations and Analyses

Ablations examine token design, LoRA’s limitations, and how long learned dynamics persist. The current token design performs best overall across dynamics and camera metrics, while dynamics can continue after an initial token-driven kick-start.

Why does LoRA fail to capture dynamics?

  • LoRA’s global patch updates mix patches and interfere with camera control; removing LoRA from the PRoPE path worsens dynamics while only slightly improving camera performance.The authors attribute the limitation to LoRA’s inability to represent localized functions and the shift in the PRoPE product from QKV updates.Source: §5 · pp. 8–9

Ablating the design of DynaTokens

  • The current DynaTokens design performs best overall on VBench2 dynamics and camera metrics; one token compromises dynamics, while removing context noising sharply reduces Dynamic Spatial Relations.No context noising is slightly better on Motion Order Understanding, but harms later spatially correct motion under the mismatch between teacher-forced training and self-prediction at inference.Source: §5 · p. 9, Table 3 · p. 9

Probing the temporal aspect of DynaTokens

  • Applying dynamic tokens only to the first 1/8 of a video yields over 64% successful dynamic trajectories, and applying them to the first half yields 88%.The authors suggest the base model can continue motion once it is kick-started through autoregressive chunk-by-chunk generation.Source: §5 · p. 9

Longer videos

  • The learned dynamics are not limited to the training duration, and the paper demonstrates application to longer videos.The longer-video examples are shown in Appendix Figure 19.Source: §5 · p. 9

6 Conclusions and Discussions

DynaTokens is a lightweight, test-time-trained add-on that teaches localized scene dynamics while preserving camera control. The paper reports strong benchmark results, analyzes how the design separates local dynamics from global camera control, and describes extensions to physical dynamics and style transfer.

6 Conclusions and Discussions

  • DynaTokens greatly surpasses current state-of-the-art camera-controlled video models on VBench2 dynamic categories and WorldScore.The paper also reports that it can extend to physical dynamics and style transfer.Source: §6 · p. 9
  • DynaTokens selectively teaches localized dynamics while enabling good camera control, whereas designs such as LoRA fail.The paper analyzes token attention as evidence of a localized gating effect and discusses how the design disentangles local dynamics from global camera control.Source: §6 · p. 9
  • The authors present DynaTokens as a path toward generating dynamic-rich, camera-controlled videos and dynamic, navigable worlds.They also say their analysis sheds light on the representation of dynamics and camera control in video models.Source: §6 · p. 9

7 Ackowledgements

The acknowledgements name discussion contributors and project funders. The appendix points to additional qualitative comparisons, physical-dynamics examples, and examples involving multiple moving objects.

7 Ackowledgements

  • The authors thank four contributors for discussions and list the Packard Fellowship, Powell Foundation, Amazon Research Award, and Google Faculty Award as project funders.They also thank Google for Gemini credits.Source: §7 · p. 10

A Additional Qualitative Results

  • Additional qualitative results compare DynaTokens with camera-controlled models and test-time training methods, and show physical dynamics and multiple moving objects.The appendix directs readers to the project webpage for interactive video visualization.Source: A Additional Qualitative Results · p. 13

B Evaluation: Standard Errors

The standard-error comparisons report a significant DynaTokens advantage over baseline world models and alternative test-time training methods. In an additional comparison, VPT/APT struggle to learn dynamics, with the paper attributing this difficulty to their interaction with camera conditioning in global self-attention.

B Evaluation: Standard Errors

  • DynaTokens’ advantage is significant over camera-controlled video-model baselines and alternative test-time training methods in the reported VBench2 and WorldScore comparisons.Tables 4 and 5 report standard errors across test samples or evaluation examples; the supplied passages do not give the individual values.Source: Table 4 · p. 13, Table 5 · p. 13, B Evaluation: Standard Errors · p. 13

C Additional Comparison with VPT/APT

  • On a subset of evaluation scenes, VPT/APT struggle to learn dynamics, often remaining near static settings on unseen trajectories and sometimes duplicating objects.The paper reports a significant DynaTokens advantage in Table 6.Source: C Additional Comparison with VPT/APT · p. 13, Table 6 · p. 13
  • VPT/APT add trainable data tokens to global self-attention, where camera conditioning occurs, entangling the camera and data paths.The paper contrasts this with DynaTokens, whose tokens do not participate in that global self-attention.Source: C Additional Comparison with VPT/APT · p. 14

D Additional Analysis of DynaTokens’ Attention Map Localization

The attention-localization analysis compares token attention with moving-object masks and finds greater overlap for DynaTokens than for LoRA or text attention. The authors caution that coarse latent patches make exact object-boundary alignment an inappropriate expectation.

D Additional Analysis of DynaTokens’ Attention Map Localization

  • DynaTokens’ attention has significantly higher overlap with moving-object SAM masks than LoRA or text attention, which perform similarly to random chance.The analysis uses IoU and relaxed IoU-d1; the authors note that the goal is localization, not exact object segmentation, and latent patches span 16x16 pixels across 4 physical frames.Source: D Additional Analysis of DynaTokens’… · p. 14
  • 5.95 is DynaTokens’ Ripley’s L statistic, compared with 4.44 for LoRA and 2.17 for the text keyword (L(3)-3).The paper says the higher score shows a clear lead by DynaTokens.Source: D Additional Analysis of DynaTokens’… · p. 14
  • Additional attention visualizations include a challenging case with four ducks moving in different directions and are presented as further support for localized dynamics.The paper compares DynaTokens attention maps with LoRA and text-based maps.Source: D Additional Analysis of DynaTokens’… · p. 14

E Robustness to Training Trajectories

DynaTokens remains effective when trained with few trajectories or when some training trajectories contain artifacts. On a 30-video subset, it retains an advantage over baselines under both conditions.

E Robustness to Training Trajectories

F Benchmark Evaluation Details

The evaluation adapts dynamic-scene benchmarks and camera-control scoring to camera motion. It also examines robustness to corrupted training trajectories.

F Benchmark Evaluation Details

  • DynaTokens’ performance remains high when 33% of training trajectories are swapped for trajectories with artifacts.Table 9 studies training-data quality by swapping good trajectories with artifact-containing ones; it reports no metric values in the supplied caption.Source: Table 9 · p. 15
  • VBench2 uses Dynamic Spatial Relations and Motion Order Understanding, with object-presence checks to handle objects leaving the camera view.Evaluation uses Gemini 2.5 Flash, GPT-5 verification, and ViPE-based camera estimation; camera score combines rotation and translation errors geometrically.Source: F Benchmark Evaluation Details · p. 15
  • WorldScore’s Motion Accuracy compares subject flow with surrounding flow, while Camera Control scores recovered camera trajectories.The adapted Motion Accuracy uses SEA-RAFT flow and SAM2 masks, setting object flow to zero when localization fails and using background median flow; Camera Control uses DROID-SLAM and scale-aligned translation.Source: F Benchmark Evaluation Details · p. 15
  • For baseline evaluation, WorldPlay action strings are converted to camera matrices; Hydra receives the initial frame repeated as a video.The setting provides no history video for Hydra.Source: F Benchmark Evaluation Details · p. 15

G Test-Time Training Details

The training appendix describes how imperfect trajectory supervision is curated and used to train DynaTokens, then discusses a limitation of the base model and LoRA’s failure mode. It also states the work’s intended research use and societal risks.

G Test-Time Training Details

  • Curated supervision combines dynamics synthesis, camera-aligned views at trajectory keyframes, and interpolated clips stitched into a reviewed video.The dynamics clip is produced with a static camera; HY-WorldPlay projects sampled state frames along camera-trajectory prefixes, and Kling o3 bridges consecutive views.Source: G Test-Time Training Details · pp. 15–16
  • Stitching can create background inconsistencies and motion discontinuities, but localized additive cross-attention lets tokens learn dynamics while retaining the frozen model prior.The curated samples may have inconsistent background appearance because keyframes can project unseen space differently.Source: G Test-Time Training Details · p. 16, Figure 6 · p. 16
  • Training uses 16 dynamic tokens of dimension 64; 15 trajectories totaling 900 frames take 80 minutes to train.The reported training memory is around 22 GB on 4 GPUs.Source: G Test-Time Training Details · p. 16
  • Dynamic tokens should not be expected to fully fix dynamics that the pretrained model handles poorly, such as complex physics.Source: G Test-Time Training Details · p. 17

H Additional Analysis on LoRA’s Failure

  • LoRA’s global entanglement across spatial patches makes it difficult to learn dynamics without affecting camera control; unaffected patches’ token keys and values remain mostly stable with DynaTokens.The analysis suggests LoRA-No PRoPE may hinder learning by creating a discrepancy between PRoPE and non-PRoPE streams.Source: H Additional Analysis on LoRA’s Failure · p. 17

I Societal Impact

  • The paper notes risks including misleading synthetic media, bias amplification, surveillance, and harmful automation, and says it is intended for scientific research.It does not release artifacts designed for deceptive or malicious use and encourages transparent, careful, and safety-conscious deployment.Source: I Societal Impact · p. 17

Related papers

Loading 2609.35704v1…