PaperlayerView exact revision on arXiv

Source-linked AI summary

ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow

Jin Cao, Zian Meng, Kaipeng Zhang

arXiv:2607.28362v1cs.CVcs.AIcs.LG

TL;DR

Interactive video world models lack a unified way to specify diverse dynamics precisely, while demonstrations transfer poorly across appearances. ShadowDancer learns dynamics from paired videos with resampled appearance, enabling reusable action assets and improving action transfer and long-horizon rollout across diverse dynamics families.

  • Problem

    Interactive world models lack a unified interface that specifies diverse dynamics both precisely and generally for frame-level control.

  • Method

    ShadowDancer constructs shadow pairs with shared dynamics and resampled appearance, then uses cross-shadow prediction to learn unified latent dynamics for reusable action assets.

  • Results

    ShadowDancer improves action transfer and long action rollout over latent-action and interactive world model baselines across human, camera, gameplay, and robot dynamics.

  • Takeaways & Limitations

    Demonstrated dynamics can be replayed as reusable action assets in new environments without action labels, motion estimators, or fine-tuning.

  • Takeaways & Limitations

    Real-world video lacks exact shadow pairs, so it contributes visual diversity and realism priors while synthetic pairs provide the invariance signal.

Abstract

from arXiv · show

We present ShadowDancer, a novel approach to any-action, frame-level control of interactive video world models. The obstacle is representational: existing interfaces either encode an action loosely, leaving how it unfolds for the model to improvise, or encode it exactly through structured signals that serve one family and are hard to acquire, so precise control across diverse dynamics remains impractical. Demonstration videos are the natural remedy, specifying any dynamics frame by frame; yet a video shows its dynamics only through one particular appearance, a single shadow of the underlying dynamics, so actions learned from demonstrations transfer poorly to new scenes. ShadowDancer addresses this with two key innovations: (1) shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by our Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it; and (2) cross-shadow prediction, which learns actions by predicting one shadow from the other, so that whatever the pairing resamples is discarded by construction and whatever it preserves becomes the action, yielding a unified dynamics representation that drives a block-causal world model. Any demonstrated clip thus becomes a reusable action asset, replayed in new environments without action labels, motion estimators, or fine-tuning. Experiments demonstrate improved action transfer and long action rollout over strong latent-action and interactive world model baselines across diverse dynamics families, with an average blinded win rate of 86% in rollout comparisons. We show video results at https://ShadowDancer-1.github.io

1. Introduction

ShadowDancer treats any-action, frame-level control as an action-representation problem and learns transferable dynamics by observing synchronized videos of the same behavior under resampled appearance. Its shadow pairs and cross-shadow prediction produce dense, reusable action assets that can be replayed in new environments.

  • Motivation: Existing interfaces trade precision for generality: loose controls leave trajectories to the generator, while exact structured signals are difficult to acquire and limited across dynamics.The bottleneck is action representation rather than video-generation capacity.
  • Method: ShadowDancer constructs frame-synchronized shadow pairs that replay the same dynamics under independently resampled appearance.The Shadow Library implements this pairing protocol across animation suites, open-world games, and robotic simulators.
  • Method: Cross-shadow prediction extracts latent actions from one shadow and predicts the other using its appearance context, forcing the representation to transfer dynamics rather than reconstruct appearance.Whatever the pairing preserves defines the controllable action, while resampled factors become invariant by construction.
  • Use: A single demonstrated clip becomes a dense action asset extracted in one frozen-encoder pass, composable with other assets and replayable in a new environment.The demonstration transfers timing, amplitude, and style from the observed trajectory instead of leaving execution details to the generator.

2. Related Work

Interactive video world models typically use symbolic or textual controls, while latent action models infer controls from unlabeled video. ShadowDancer instead defines reusable dynamics representations through shadow-paired data, transferring the invariance-by-construction principle to frame-level dynamics.

  • Interactive Video World Models: Interactive video world models support planning and simulation but commonly control generation through symbolic keyboard/mouse states or free-form text prompts.These interfaces specify what happens without fully specifying how the action unfolds.
  • Latent Actions as Control Interfaces: Latent action models infer controls directly from unlabeled video for interactive generation, cross-embodiment policy learning, and world-model pretraining.ShadowDancer focuses on the latent representation as a reusable dynamics signal transferable across appearances.
  • Invariance by Construction: Self-supervised representations inherit invariances from their training pairs: varied factors are discarded, while shared factors are retained.Examples include image augmentations preserving visual identity and image-caption pairs preserving language-expressible semantics.
  • Invariance by Construction: Shadow pairs move augmentation into the renderer by replaying trajectories while independently resampling appearance, enabling cross-shadow prediction to learn dynamics invariance.Pixel-space augmentation struggles to resample appearance while preserving motion frame by frame.

3. Method

ShadowDancer makes dynamics controllable by pairing synchronized videos that preserve dynamics while independently resampling appearance, then learning a unified latent representation through cross-shadow prediction. A diffusion world model uses this latent for transferable control and the source stream for motion detail, enabling demonstrated actions to be reused as variable-length assets.

  • Unified representation: The unified dynamics representation is z alone, using one encoder, latent space, and interface across dynamics families, while a = (z, s) combines z with source-detail stream s for generation.The frozen 3D-VAE encodes s, which supplies high-frequency motion detail that the invariance-constrained action bottleneck is too small to carry.
  • Shadow pairs: Shadow pairs preserve the same frame-level dynamics while independently resampling appearance, making the preserved factor the controllable action.Pair construction defines which factor belongs to dynamics: preserving body motion while changing camera assigns body motion to dynamics, whereas preserving camera motion while changing scene assigns camera motion to dynamics.
  • Cross-shadow prediction: Cross-shadow prediction encodes source transitions into latent actions that predict the synchronized target shadow, forcing z to retain shared dynamics rather than appearance.The KL-constrained bottleneck removes source-context information that cannot improve prediction of the independently rendered target, while self-pairs recover the standard single-video latent-action objective.
  • Interactive world model: A block-causal video diffusion generator conditions on z for transferable control and on source assets for motion detail, then rolls out the demonstrated dynamics from a first frame.At inference, demonstrations are encoded into per-frame latent actions, and the action is stored as a variable-length dynamics segment that can be reused in new environments.
  • Shadow Library: Shadow Library implementations replay dynamics across independently varied characters, scenes, cameras, environments, and physical setups, while real video enters as degenerate self-pairs.Synthetic shadow pairs provide the identifying invariance signal; self-pairs broaden visual diversity and realism priors without exact shadows.

4. Experiments

Experiments evaluate ShadowDancer’s unified action interface across diverse dynamics, testing transfer to new environments, long-horizon composability, component contributions, and unseen modded actions. Results show broad qualitative and quantitative advantages, including faithful transfer and an 86% average blinded win rate in rollout comparisons.

  • Experimental coverage: The Shadow Library covers human motion, robotic manipulation, first-person and third-person games, and camera control.Experiments use paired appearances that replay identical dynamics across varied characters, environments, and rendering conditions.
  • Action transfer: A single model transfers held-out actions across human motion, combat, third-person action, robot manipulation, and camera control.Actions are extracted from one video and generated from the first frame of its synchronized shadow in a new environment.
  • Action transfer: ShadowDancer leads every tested dynamics family, while Olaf-World produces warped subjects and spurious motion from entangled appearance and dynamics.The shadow-trained latent re-enacts demonstrated dynamics faithfully in new environments.
  • Long action rollout: Long rollouts chain canonical action assets across navigation, aiming and firing, skill casts, human motion, and robot manipulation.Evaluation emphasizes action control, action fidelity, and long-horizon consistency rather than visual fidelity.
  • Ablations: Pairing is the foundation of transfer, while source assets independently provide strong action and spatial detail.The ablation compares an unpaired latent with the full model and finds pairing decisive; assets match or surpass the bottlenecked latent.
  • Unseen-action transfer: Modded actions from an unseen character and a two-handed sword test transfer beyond the training dynamics and character assets.Walking, turning, and sword-attack clips are recorded on one map and reused as extracted action assets.

5. Conclusion

ShadowDancer enables any-action, frame-level control of interactive video world models from demonstrations by learning unified dynamics representations through shadow pairs and cross-shadow prediction. It turns demonstrated clips into reusable action assets for replay in new environments and supports interaction by showing rather than telling.

  • Core contribution: ShadowDancer enables any-action, frame-level control of interactive video world models by demonstration.Its approach uses shadow pairs and cross-shadow prediction to learn a unified dynamics representation.
  • Core contribution: Shadow pairs and cross-shadow prediction discard resampled attributes and preserve the dynamics, allowing clip dynamics to become reusable action assets replayed in new environments.The representation is constructed so that what pairing resamples is discarded and what it preserves becomes the action.
  • Implications: The resulting interaction paradigm drives world models by showing rather than telling, with applications ranging from entertainment to simulation-based training of embodied agents.The paper discusses limitations and future directions in Supplementary Sec. A.

Supplementary Material … B.1. Factor-Selective Readout of the Unified Representation

ShadowDancer’s factor-selective readout separates camera and dynamics information by supervising distinct heads with pairing protocols that preserve the relevant factor. The supplementary material also notes deployment and data-collection limitations, especially the need for preprepared demonstrations and the difficulty of constructing real-world shadow pairs.

  • A. Limitations and Future Directions: ShadowDancer requires action assets prepared before deployment, so unseen dynamics must first be captured or authored.At deployment, the model consumes reference clips or stored assets.
  • A. Limitations and Future Directions: Shadow pairs are easy to construct in games and simulators but much harder to collect in the real world.Games and simulators can replay dynamics under resampled appearance on demand, whereas real-world reruns are rarely possible.
  • B.1. Factor-Selective Readout of the Unified Representation: The inverse-dynamics encoder uses a spatiotemporal transformer with causal temporal attention to infer z_t at frame t+1.Its posterior conditions on the full causal prefix, qϕ(z_t | x_1:t+1), rather than only the transition.
  • B.1. Factor-Selective Readout of the Unified Representation: Pairs preserving only camera trajectory supervise the camera head, while pairs preserving only scene motion supervise the dynamics head.Camera and subject are resampled or frozen to define which factor is preserved.
  • B.1. Factor-Selective Readout of the Unified Representation: Pairs preserving both camera and scene dynamics supervise the full head.The pairing protocol routes supervision according to which factors each pair preserves.
  • B.1. Factor-Selective Readout of the Unified Representation: A per-sample mask, (m_cam, m_dyn) ∈ {0, 1}^2, selects one head so each head is supervised only by its preserved factor.The mask determines whether camera, dynamics, or full-factor supervision is applied.
  • B.1. Factor-Selective Readout of the Unified Representation: The dynamics factor is defined by each family’s pairing protocol rather than fixed in advance.For robot manipulation, the pairs preserve the arm trajectory.

B.2. Conditioning Architecture · C. Formal Guarantee for Shadow Learning

The conditioning architecture combines globally modulated action summaries, token-level cross-attention, and source-video assets to preserve both low- and high-frequency motion information. The formal guarantee shows that cross-shadow prediction identifies shared dynamics under observability, separation, overlap, and capacity conditions, with scope and limitations made explicit.

  • B.2. Conditioning Architecture: Action tokens are grouped by latent video frame and their average modulates each timestep embedding for global, low-frequency control.The grouping follows temporal compression r=4, with Z_f covering the corresponding per-frame actions.
  • B.2. Conditioning Architecture: Cross-attention preserves all per-frame action tokens, while zero-initialized attention starts as a no-op and positively initialized modulation avoids vanishing gradients.The attention context initially contains projected action tokens, and the positive modulation gate was chosen because zero initialization stalled training in practice.
  • B.2. Conditioning Architecture: Because the dz=32 bottleneck cannot carry precise motion detail, the source video is supplied as generator assets through concatenated latents, availability masks, context tokens, and semantic embeddings.These inputs preserve high-frequency information such as limb articulation and contact timing while the bottleneck remains constrained against appearance leakage.
  • C. Formal Guarantee for Shadow Learning: The formal result characterizes the coarsest statistic sufficient for predicting one shadow from another as the shared dynamics, up to an invertible reparameterization.It does not assume independent latent components or recovery of a privileged coordinate system.
  • C. Formal Guarantee for Shadow Learning: The guarantee requires independent context resampling, source observability, and target overlap with separated transition kernels, so distinct dynamics differ on target contexts with nonzero probability.These conditions formalize replaying the dynamics while resampling other factors and prevent decoder-side context from perfectly revealing the dynamics.
  • C. Formal Guarantee for Shadow Learning: Under compactness, continuity, and a continuous embedding of D into the latent dimension, differentiable neural encoders and predictors approximate the dynamics representation and target mapping arbitrarily closely.The encoder discards source context up to an invertible coordinate change, while the decoder attains arbitrarily small cross-shadow reconstruction error.
  • C. Formal Guarantee for Shadow Learning: Exact reconstruction identifies the dynamics almost surely, and a minimal exact-reconstruction code is a one-to-one reparameterization of D.For stochastic renderers, the construction requires rendering randomness in decoder-available context or a neural density decoder; statistical learning additionally needs stated optimization and capacity conditions.
  • C. Formal Guarantee for Shadow Learning: For heterogeneous action families, identification holds within each pairing family; unified encoders and latent spaces enable cross-family sharing architecturally, not as a theorem consequence.The guarantee applies only to exact shadows satisfying the independence condition, while failed separation identifies only dynamics equivalence classes and degenerate self-pairs provide no identifiability guarantee.

C.2. Identification Theorem · D. Additional Experimental Details and Ablations · D.1. Composition of the Shadow Library

The identification theorem shows that cross-shadow prediction isolates the underlying dynamics representation: it is sufficient, minimal, invariant to resampled source-context factors, and equivalent to the dynamics variable up to bijective reparameterization. The accompanying corollary connects this result to Bayes-optimal likelihood learning, while the supplied implementation detail states that Table 4 organizes Shadow Library sources by pairing, supervised control channel, scale, and sampling share.

  • C.2. Identification Theorem: Under Assumptions A1–A3, Γ(X) is cross-shadow sufficient and minimal, and is invariant to every source-context factor resampled by shadow construction.The theorem also states that every cross-shadow sufficient representation determines D.
  • C.2. Identification Theorem: Every cross-shadow sufficient representation Z = s(X) determines D, while D and any minimal sufficient Z determine one another up to one-to-one reparameterization.This establishes equivalence between the learned representation and the underlying dynamics variable.
  • C.2. Identification Theorem: Cross-shadow likelihood reaches the full-source population negative log-likelihood if and only if Z is cross-shadow sufficient; therefore every Bayes-optimal code contains D.Every minimal Bayes-optimal code is a one-to-one reparameterization of D.
  • C.2. Identification Theorem: The theorem identifies a representation rather than a preferred coordinate system, so any bijective transform φ(D) is equally valid.A nonminimal Bayes-optimal encoder may retain source appearance, but that information is unnecessary for cross-shadow prediction.
  • C.2. Identification Theorem: The deterministic theorems do not guarantee exact sufficiency or minimality for the finite-rate sampled Gaussian channel used in implementation.The β-weighted KL term instead provides a rate–distortion approximation that favors a compact code.
  • D.1. Composition of the Shadow Library: Table 4 organizes Shadow Library sources by pairing strategy, supervised control channel, approximate scale, and sampling-mixture share.The supervised channels are specified through the per-sample masks (mcam, mdyn) of Sec. B.1.

D.2. Training and Inference Details

Training uses a latent action model with a fixed Gaussian prior and self-paired inputs, then converts the world model into a block-causal generator. Action-transfer and long-rollout evaluations use distinct target resolutions and training procedures.

  • Training: The LAM uses β=0.01, a fixed prior N(0, I), and half the world model’s spatial resolution.Real videos enter as degenerate self-pairs.
  • Training: Roughly one third of training samples are self-pairs, with self-pair probability 0.5 for human body dynamics and 0 for other paired sources.Unpaired real-video and static-scene camera sources are always self-paired.
  • Training and inference: The world model is fine-tuned bidirectionally with flow matching before conversion to a block-causal generator with 3 latent frames per block.Inference rollout uses a key–value cache.
  • Evaluation settings: Action-transfer comparisons use 480×720 outputs with a 240×360 LAM, whereas the long-rollout model generates 544×960 outputs with a 272×480 LAM.The long-rollout model is warm-started from a lower-resolution checkpoint and fine-tuned at target resolution for 10k steps on 8×H200 GPUs.

D.3. Evaluation Details … D.6. What the Action Latent Carries

The evaluation combines controlled rollout comparisons, latent-level probes, distribution-quality checks, and intervention tests to assess action control, realism, representation content, and transfer. Results indicate that ShadowDancer’s latent carries commanded dynamics while source assets primarily preserve appearance and detail.

  • D.3. Evaluation Details: D.3. Evaluation Details: The comparison protocol keeps both systems under the same first frame, command sequence, and rollout boundaries before judging their behavior.The judge is instructed to evaluate actions rather than image quality and must choose A or B independently for each axis.
  • D.3. Evaluation Details: D.3. Evaluation Details: Rollouts are compared in corresponding 240-frame segments, yielding roughly 64 anonymized comparison pairs per baseline.A VLM makes forced choices on action control, action fidelity, and long-horizon consistency, and a human audit reviews a random 20% of judgments.
  • D.4. Representation-Level Probes: D.4. Representation-Level Probes: The latent is tested through cross-pair reconstruction transfer and linear probes for character and scene information.The cross-pair ratio is MSEcross/MSEself, comparing reconstruction with the target’s own action against the shadow source’s action.
  • D.5. Distribution-Level Quality: D.5. Distribution-Level Quality: FVD is computed over 16-frame windows with stride 8 on the three large reconstruction families, while camera quality uses trajectory error.The robot split is excluded from stable FVD estimation, and ShadowDancer roughly halves Olaf-World’s FVD.
  • D.6. What the Action Latent Carries: D.6. What the Action Latent Carries: Intervention tests replace or shuffle the latent, replace the source assets, or reduce asset resolution to one quarter before measuring PSNR against target A.The interventions use two shadow pairs from different sources and start from the base input (zA, sA).
  • D.6. What the Action Latent Carries: D.6. What the Action Latent Carries: On four of five families, replacing or shuffling z sharply lowers PSNR, whereas degrading s barely changes it.This supports that commanded action resides in the latent, while assets provide appearance and high-frequency detail.

D.7. Architecture Ablations

The architecture ablations motivate a small latent bottleneck and identify a limitation of the probing method on exact-replay motion families. Real-video self-pairs support inheriting dz=32, while some assets already fully encode the motion and prevent channel separation.

  • Latent size and self-pair ratio: The dz=32 latent size carries over from prior latent-action models because real video enters training as self-pairs under the same reconstruction objective.The choice is also consistent with keeping the bottleneck too small to smuggle appearance.
  • Latent size and self-pair ratio: The invariance argument favors a bottleneck that is too small to encode appearance, supporting the selected latent dimensionality.
  • Probe limitation: On body-motion and character-render families, source assets already depict the motion fully, so the probe cannot separate the two channels.This limitation is consistent with the assets acting as a strong carrier for exact replay.
Loading 2607.28362v1…