PaperlayerView exact revision on arXiv

Source-linked AI summary

Reimagine Video Dynamics

Yu Yuan, Yawen Lu, Guoxian Song, Kevin Duarte, Ratheesh Kalarot, Di Chang, Xijun Wang, Stanley H. Chan

arXiv:2609.36496v1cs.CV

TL;DR

Video editing often changes appearance but offers limited control over dynamics, which are entangled with visual context in joint video latents. RVD learns an editable dynamics token through reconstruction and uses a language-guided editor to transform it; experiments demonstrate dynamics editing, retiming, and appearance-controlled re-rendering.

  • Problem

    Video editors often modify appearance but provide limited dynamics control, and joint video latents entangle motion with visual context, making precise dynamics editing difficult.

    Source: §1 · p. 1

  • Method

    RVD learns a compact dynamics token by reconstructing video from the first frame, then uses language guidance and counterfactual-pair training to transform source tokens into target dynamics.

    Source: §1 · pp. 1–2

  • Results

    RVD supports video dynamics editing, training-free retiming, and appearance-controlled re-rendering; qualitative comparisons show it changes temporal evolution where other compared editors often preserve the original motion.

    Source: §1 · p. 2, §4.4 · p. 9

  • Takeaways & Limitations

    The dynamics token provides a common interface for editing how a scene evolves, retiming video, or re-rendering the same motion under new visual context.

    Source: §1 · p. 2, §5 · p. 10

    Rapid motion can be temporally aliased, camera and object motion are not explicitly separated, and complex multi-subject interactions can produce identity or contact errors.

    Source: F LIMITATIONS · p. 24

Abstract

from arXiv · show

Most video editing methods focus on changing the appearance of the source video, while offering limited control over its dynamics. We introduce Reimagine Video Dynamics (RVD), a framework that disentangles a compact, editable dynamics token from visual context. We learn this token through self-supervised reconstruction: given the first frame as visual context, a renderer must recover the original video from the dynamics token, encouraging it to capture how the scene evolves rather than how it looks. This disentanglement allows video dynamics to be edited directly while preserving visual context. We develop a language-guided dynamics-token editor that transforms source dynamics into target dynamics, and train it with a scalable counterfactual video-pair pipeline and a two-stage training strategy. Extensive experiments show that RVD enables effective video dynamics editing, training-free retiming, and appearance-controlled re-rendering.

1 INTRODUCTION

RVD addresses limited control over video dynamics by separating a compact dynamics token from visual context. It learns the token through reconstruction and edits it with language guidance, supporting dynamics editing, retiming, and appearance-controlled re-rendering.

1 INTRODUCTION

  • Joint video latents mix appearance and dynamics, making precise motion edits that preserve the surrounding scene difficult.Examples include changing a run to a walk or making a runner stop before an obstacle.Source: §1 · p. 1
  • RVD separates visual context, which preserves appearance and scene identity, from a spatiotemporal token that describes temporal evolution.Changing context re-renders the same motion under a new appearance; changing the token alters motion within the same scene.Source: §1 · pp. 1–2
  • Self-supervised reconstruction trains a dynamics encoder and renderer to recover the source video from its first frame and dynamics token.The first frame supplies visual context, encouraging the token to carry complementary information about scene motion.Source: §1 · p. 2
  • A language-guided editor transforms source dynamics tokens into target tokens, using an MLLM-produced semantic memory to guide a dynamics editing transformer.The edited token is combined with the first frame and rendered into the final video.Source: §1 · p. 2
  • A scalable counterfactual-pair pipeline and two-stage training strategy teach realistic dynamics distributions first, then precise source-to-target transformations.Fine-tuning uses pairs with shared visual context but different dynamics and accompanying edit instructions.Source: §1 · p. 2
  • RVD supports dynamics editing, training-free retiming by temporal resampling, and appearance-controlled re-rendering with dynamics held fixed.The framework presents the dynamics token as a common interface for these forms of video manipulation.Source: §1 · p. 2

2 RELATED WORK

Prior video-editing work largely emphasizes appearance, while dynamics control remains less explored and often relies on specialized signals or implicit representations. Existing video representations also trade off appearance fidelity, semantic structure, and explicit modeling of temporal evolution.

2.1 VIDEO EDITING MODELS

  • Video-editing models commonly change appearance, while dynamics-editing approaches use explicit motion signals, specialize in deletion, plan trajectories, or leave control implicit.A compact dynamics representation editable separately from visual context remains largely unexplored.Source: §2.1 · p. 2

2.2 REPRESENTATIONS FOR VIDEO DYNAMICS

  • Video VAEs jointly preserve appearance, texture, and motion; semantic representations emphasize scene structure, while predictive representations learn spatiotemporal features.The paper uses a frozen V-JEPA 2 encoder as the starting point for its compact dynamics token.Source: §2.2 · pp. 2–3

2.3 PHYSICS-AWARE VIDEO GENERATION AND EDITING

  • Physics-aware generation places physics in simulation before or after generation, learned physical or motion priors, or inference-time planning and rewards.Precise dynamics editing of real videos remains underexplored.Source: §2.3 · p. 3

3 RENDERER: DISENTANGLING DYNAMICS BY RECONSTRUCTION

The renderer separates visual context from temporal evolution by reconstructing video from a first frame, optional caption, and compact dynamics token. Controlled tests and appearance-editing evaluations support this separation, while ablations identify design choices that improve reconstruction and token injection.

3 RENDERER: DISENTANGLING DYNAMICS BY RECONSTRUCTION

  • The video is modeled as complementary visual-context and dynamics factors: changing context should preserve motion, while changing dynamics should alter temporal evolution.Their composition is generative rather than pixel addition.Source: §3 · p. 3

3.1 RENDERER AND TRAINING

  • The renderer reconstructs video from visual context—first frame and optional caption—and a token extracted by the dynamics encoder.The context path supplies the scene and appearance, so the token carries the missing dynamics.Source: §3.1 · p. 3
  • The dynamics encoder compresses frozen V-JEPA 2 features through a learnable bottleneck, while a separate conditioning branch injects the token into the renderer.The bottleneck reduces temporal, spatial, and channel dimensions; the conditioning branch aligns the token and video-latent spatial grids.Source: §3.1 · p. 3
  • Residual centering limits global appearance leakage, RMS constraints limit the dynamics signal’s magnitude, and a diffusion-time gate emphasizes motion formation early in generation.The gate reduces token influence later, when the model focuses more on visual details.Source: §3.1 · p. 4

3.2 EXPERIMENTS: DOES THE TOKEN DISENTANGLE DYNAMICS FROM APPEARANCE?

  • With the first frame and extracted token, the renderer reconstructs source motion at 19.261 dB PSNR; removing the token lowers PSNR to 16.663 dB and reduces motion consistency.The comparison tests whether the compact token preserves motion information needed by the renderer.Source: §3.2 · p. 4
  • Fixing the dynamics token while changing visual context preserves source motion, while changing only the dynamics input produces different event trajectories under the same context.These controlled interventions support the division of roles between appearance context and temporal evolution.Source: §3.2 · p. 4
  • Across 600 appearance edits, RVD leads in instruction following, edit completion, and dynamics preservation while remaining competitive in imaging quality.The reported metrics are IF, Pass, JEPA, and Imaging; representative comparisons cover three edit types.Source: §3.2 · p. 5

3.3 RENDERER ARCHITECTURE ABLATION

  • Table 2 ablations identify dynamics conditioning as essential for reconstruction; an 8×8×128 bottleneck balances fidelity and efficiency, while 2D RoPE and residual centering provide the clearest injection gains.Injecting into every block or removing the noise gate has a smaller effect.Source: §3.3 · p. 5

4 EDITOR: EDITING THE LEARNED DYNAMICS SPACE

RVD edits motion in a structured dynamics-token space, using a language-guided editor trained on counterfactual pairs. The reported benchmark and qualitative comparisons show effective dynamics editing, while token analyses and retiming tests demonstrate that the learned representation is structured and directly manipulable.

4.1 IS THE DYNAMICS TOKEN STRUCTURED AND EDITABLE?

  • Across 248 real videos, the compressed token retains spatial, motion-state, and channel variation; temporal interpolation slows motion and subsampling speeds it up without training.The analysis reports 0.995 normalized coverage entropy, spatial rank 63/63, and channel covariance rank 128/128; retiming preserves visual content.Source: §4.1 · p. 6

4.2 VIDEO DYNAMICS EDITOR

  • Given a source dynamics token and edit instruction, a semantic encoder guides a transformer that edits the token directly in the learned dynamics space.The edited token is substituted into the renderer to produce the target video.Source: §4 · p. 5, §4.2 · p. 6

4.3 EDITOR TRAINING: COUNTERFACTUAL DATA AND TWO-STAGE STRATEGY

  • The editor is pretrained on 169K pseudo pairs, then fine-tuned on about 20K counterfactual pairs spanning 17 transformations; ablations support both stages and the token-space losses.Pretraining teaches the distribution of realistic dynamics tokens; counterfactual fine-tuning learns precise source-to-target transformations.Source: §4.3 · pp. 7–8

4.4 DYNAMICS EDITING RESULTS

  • Compared with Ditto, OmniVideo2, and VINO, RVD changes requested motion more directly; examples cover physical outcomes, activities, locomotion, and object states.The other methods often preserve the original motion and fail to realize the requested change, while RVD often maintains the visible subject and scene in the examples.Source: §4.4 · p. 9
  • RVD leads all four reported metrics across 300 scenes and 17 edit directions.IF and Pass measure requested-edit success; VBench background consistency and temporal flickering are output-internal diagnostics.Source: Table 4 · p. 9, §4.4 · p. 9

5 CONCLUSION

RVD uses a separate dynamics pathway in its renderer while retaining Wan’s native image and text conditioning. The renderer is trained on unpaired real clips, with caption dropping and token perturbations to support reconstruction from predicted dynamics tokens.

5 CONCLUSION

  • RVD provides a compact editable dynamics token and a reusable interface for dynamics editing, retiming, and appearance-controlled re-rendering.The renderer learns the token by reconstructing video from the first frame, separating temporal evolution from visual context.Source: §5 · p. 10

A.1 RENDERER ARCHITECTURE

  • DynCrossAttention injects dynamics into the first 30 of 40 Wan blocks, while native image, text, and noisy-latent paths remain intact.It attends to same-time 8 × 8 dynamics slices; centering limits global appearance shifts and an RMS cap bounds the residual.Source: §A.1 · p. 14, Figure 10 · p. 14

A.2 RENDERER TRAINING

  • The renderer trains on about 169K real clips without motion annotations or paired edits, dropping captions and perturbing or masking tokens for robustness.Training first learns the bottleneck and adapters, then enables rank-32 Wan LoRA.Source: §A.2 · p. 15

A.3 EXPERIMENTS: DOES THE TOKEN DISENTANGLE DYNAMICS FROM APPEARANCE?

Controlled tests examine whether the dynamics token preserves motion when visual context changes and whether changing the token changes motion under fixed context. The tests report reconstruction, motion-preservation, editing, and appearance re-rendering outcomes.

A.3 EXPERIMENTS: DOES THE TOKEN DISENTANGLE DYNAMICS FROM APPEARANCE?

  • Reconstruction reaches 19.261 dB PSNR and 0.5761 SSIM; removing dynamics lowers them to 16.663 dB and 0.5216, while caption-free rendering scores 18.630 dB and 0.5770.The full and caption-free outputs follow the source event, whereas removing dynamics changes or suppresses its progression.Source: §A.3 · p. 15, Figure 11 · p. 15
  • With the dynamics token fixed across 600 appearance interventions, matched flow cosine is 0.727 and motion-profile correlation 0.501, versus 0.312 and 0.012 for mismatched controls.The result indicates that motion remains tied to the fixed token after visual context changes.Source: §A.3 · p. 15, Table 5 · p. 16
  • With target visual context fixed, the edited source token raises IF from 4.563 to 4.705 and Pass from 90.2% to 95.5% over target context alone, and achieves the highest DINO appearance consistency.The edited token follows the requested outcome while retaining source-specific timing.Source: §A.3 · p. 16
  • Across 600 appearance edits, RVD leads IF (4.438), Pass (86.8%), and JEPA similarity (0.9050), while remaining competitive in Imaging.The benchmark compares seven methods, with one output per case and no reranking.Source: §A.3 · p. 16

A.4 RENDERER ABLATIONS

Renderer ablations test conditioning, bottleneck capacity, and injection design while holding the reconstruction setup and evaluation fixed. The results identify which components contribute to reconstruction quality and where added capacity or complexity does not help.

A.4 RENDERER ABLATIONS

  • Removing dynamics causes the largest conditioning drop: PSNR falls by 2.598 dB, showing the token carries information unavailable from target context alone.The no-caption variant removes text while retaining the first frame and token.Source: §A.4 · p. 16
  • The 8 × 8 × 128 bottleneck is the selected efficiency–fidelity tradeoff; 16 × 16 adds only 0.332 dB PSNR at four times the spatial-token count.The 4 × 4 grid loses spatial detail, while increasing channels beyond 128 does not improve reconstruction.Source: §A.4 · p. 16
  • Removing 2D RoPE or centering lowers PSNR to 18.222 and 18.427 dB, while extending injection to 40 blocks or removing the noise gate gives no PSNR gain.The ablations identify spatial alignment and centered residuals as the clearest supported design benefits.Source: §A.4 · p. 16

B.1 POST-BOTTLENECK TOKEN ANALYSIS

The post-bottleneck token retains distributed spatial and temporal variation, which supports training-free retiming. The section also describes the editor architecture, counterfactual training data, and two-stage editor training.

B.1 POST-BOTTLENECK TOKEN ANALYSIS

  • Across 248 clips, residuals span all 128 channel directions and the full 8 × 8 grid (entropy 0.995; spatial rank 63/63), motivating spatiotemporal self-attention.Residual RMS distributions shift across source-motion groups but overlap because appearance, viewpoint, and motion magnitude also vary.Source: §B.1 · pp. 16–17

B.2 TRAINING-FREE RETIMING

  • Retiming uses temporal resampling for 0.5× and keeps every second token through step 20 for 2×; the frozen renderer produces the corresponding progress changes.At matched normalized times, half-speed outputs progress less, double-speed outputs further, and 1× reconstructions follow source timing.Source: §B.2 · p. 18, Figure 13 · p. 18

C.1 EDITOR ARCHITECTURE

  • Text, target first frame, and learned queries independently form semantic memory that guides a four-layer spatiotemporal editor through cross-attention and AdaLN.The prediction head returns the edited token to its original shape; the target-token branch and token loss are training-only.Source: §C.1 · p. 18, Figure 14 · p. 19

C.2 COUNTERFACTUAL DATA

  • The counterfactual pipeline yields 9,946 usable pairs from 11,702 candidates, with 9,964 passing quality control (85.1%) across 17 edit directions.Accepted pairs are reused in both directions by swapping source and target and reversing the instruction.Source: §C.2 · p. 19, Table 6 · p. 20

C.3 EDITOR TRAINING

  • Training first uses 169K static-to-real pseudo pairs, then fine-tunes on 17 directed counterfactual transformations; the two stages take about five days on eight H200 GPUs.Source-token noise with standard deviation 0.1 improves robustness.Source: §C.3 · p. 20
  • Two-stage training raises AC-cos from 0.2629 to 0.2892 and lowers AC MSE from 0.6977 to 0.6352; loss ablations show distinct effects on alignment and temporal errors.Removing cosine alignment lowers AC-cos by 0.0134; removing variance and spectrum matching raises their errors by 0.2079 and 0.3276, respectively.Source: §C.3 · p. 20

C.4 DYNAMICS EDITING EXPERIMENTS

On the dynamics-edit benchmark, RVD leads the reported metrics, and additional examples compare event changes against instruction-based baselines. The renderer benchmark evaluates appearance edits using measures of instruction following, motion preservation, and imaging quality.

C.4 DYNAMICS EDITING EXPERIMENTS

  • RVD leads all four reported metrics—4.837 IF, 98.3% Pass, 0.9711 background consistency, and 0.9919 temporal smoothness—while VINO reaches 3.957 IF and 75.0% Pass.The paper uses IF and Pass for edit-success conclusions and treats VBench metrics as complementary checks.Source: §C.4 · pp. 20–21
  • In additional cases, RVD changes the event while instruction-based baselines often retain the source dynamics.Figure 15 compares five dynamics edits at aligned relative time, with each row reporting its instruction.Source: §C.4 · p. 21, Figure 15 · p. 21

D.1 RENDERER BENCHMARK

  • The 600-case renderer benchmark compares RVD with six baselines on identical source videos and instructions without reranking, using IF/Pass, JEPA motion preservation, and imaging quality.Cases cover foreground replacement, background replacement, and style transfer across varied scenes and motion patterns.Source: §D.1 · pp. 21–22

D.2 EDITOR BENCHMARK

The editor benchmark measures 17 directed transformations across 300 public-web cases, comparing complete editing systems under a fixed protocol. Its edit-success result includes target-context planning, optional first-frame editing, dynamics-token editing, and rendering.

D.2 EDITOR BENCHMARK

  • The benchmark covers 17 directed transformations across 300 public-web scenes and 1,200 outputs from RVD, Ditto, OmniVideo2, and VINO.The source videos and file hashes are disjoint from editor training data.Source: §D.2 · p. 22
  • RVD’s 98.3% Pass is measured over all 300 cases as a complete-system result, not the isolated contribution of the dynamics editor.The protocol includes target-context planning, optional first-frame editing, token editing, and rendering.Source: §D.2 · p. 22
  • IF and Pass assess requested temporal changes, while background consistency and flickering are output-internal diagnostics that cannot alone establish source-scene or motion preservation.Low flickering can reflect motion suppression; the judge evaluates 12 sampled frames and does not see RVD’s internal target context or features.Source: §D.2 · p. 22

E WHEN DOES THE DYNAMICS EDITOR HELP?

The editor adds source-aware motion control beyond the target context alone, with its largest gains on trajectory and rhythm edits where the desired path or timing is ambiguous.

E WHEN DOES THE DYNAMICS EDITOR HELP?

  • The controlled comparison fixes the target frame, caption, instruction, renderer, sampling schedule, and seed while varying only the dynamics input.It tests no token, the unedited source token, a matched donor token, and the edited source token.Source: E WHEN DOES THE DYNAMICS EDITOR HELP? · p. 22
  • 95.5% Pass versus 90.2% for target context alone on 112 matched cases; the edited source token improves instruction following, edit success, and appearance consistency.The comparison fixes the target context and sampling conditions, varying only the dynamics input.Source: Table 7 · p. 23, E WHEN DOES THE DYNAMICS EDITOR HELP? · p. 23
  • The largest gains occur in trajectory and rhythm tasks, including starting, stopping, speed change, distance control, and avoidance.These instructions can leave the motion path or timing ambiguous even when the target frame specifies the desired state.Source: E WHEN DOES THE DYNAMICS EDITOR HELP? · p. 23
  • A pairwise judge prefers the edited source token in 65.2% of cases, versus 33.0% for a matched donor token and 1.8% ties.The comparison provides evidence that edited-source dynamics better preserve source-specific trajectory, timing, and rhythm while completing the requested edit.Source: E WHEN DOES THE DYNAMICS EDITOR HELP? · p. 23

F LIMITATIONS

The discussion identifies limits in the editor’s motion coverage and determinism, bottleneck disentanglement, and complex interaction scope, alongside constraints on evaluation and inference.

F LIMITATIONS

  • The editor is trained on finite counterfactual transformations and predicts one deterministic target trajectory, limiting open-ended edits with multiple valid motions.The authors identify stochastic generation and broader counterfactual supervision as natural next steps.Source: F LIMITATIONS · pp. 23–24

Q1. Why use V-JEPA 2 instead of learning from scratch or using optical flow?

  • V-JEPA 2 supplies a frozen motion-aware video representation without dynamics labels; optical flow captures apparent 2D displacement but not action or interaction semantics.The authors also note that learning the full visual representation from scratch would discard this video prior and increase training burden.Source: Q1. Why use V-JEPA 2 instead of… · p. 24

Q2. What does the learned bottleneck add beyond directly compressing V-JEPA features?

  • The learned 128-channel token reduces feature volume by approximately 64× and improves reconstruction over retaining the full 1024-channel width after the same temporal and spatial preprocessing (19.261 dB PSNR, 0.5761 SSIM versus 18.467 dB, 0.5659).Compression does not guarantee strict statistical independence, and residual donor cues can still remain.Source: Q2. What does the learned bottleneck… · pp. 24–25

Q3. What do the controlled interventions establish?

  • Controlled interventions support functional disentanglement: changing visual context preserves motion, while changing dynamics alters event evolution under fixed context.The edited source token raises Pass from 90.2% to 95.5% on 112 matched cases; these tests do not establish strict statistical independence.Source: Q3. What do the controlled… · p. 25

Q4. Why is the dynamics editor deterministic?

  • Each training pair specifies one target token, so deterministic regression tests reliable transformation but cannot represent multiple equally valid motions for an instruction.The two-stage strategy raises AC-cos from 0.2629 to 0.2892 and reduces AC MSE from 0.6977 to 0.6352.Source: Q4. Why is the dynamics editor… · p. 25

Q5. Can RVD edit complex multi-object interactions?

  • The benchmark covers 300 scenes and 17 directed edits, centered on one dominant subject or interaction; complex multi-agent physical reasoning remains unestablished.Several moving subjects, persistent occlusion, or rapid contact changes can produce identity merging, incorrect participant selection, or incorrect contact order.Source: Q5. Can RVD edit complex multi-object… · p. 25

Q6. What is the main inference cost?

  • The 14B Wan renderer and diffusion sampling dominate inference cost; dynamics encoding takes one frozen V-JEPA 2 pass and the editor adds about 24.9M trainable parameters.The renderer trains 437.3M parameters through the bottleneck, adapters, and LoRA while the 14B backbone remains frozen.Source: Q6. What is the main inference cost? · p. 25

Q7. Why are reconstruction PSNR and SSIM lower than video-autoencoder results?

  • RVD is a conditional generator, not a pixel-complete codec, so omitted texture must be regenerated rather than transmitted.Removing dynamics lowers PSNR from 19.261 to 16.663 dB and SSIM from 0.5761 to 0.5216; removing the caption gives 18.630 dB and 0.5770 SSIM.Source: Q7. Why are reconstruction PSNR and… · p. 25

Related papers

Loading 2609.36496v1…