PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionStream: Real-Time Video Generation with Interactive Motion Controls

Joonghyuk Shin, Zhengqi Li, Richard Zhang, Jun-Yan Zhu, Jaesik Park, Eli Shechtman, Xun Huang

arXiv:2511.01266v5cs.CVcs.LG

TL;DR

Motion-conditioned video generation has been too slow and non-causal for real-time interaction. MotionStream adds motion control to a video model, distills it into a causal streaming student, and uses fixed-window attention strategies for long horizons. It achieves state-of-the-art motion-control results with sub-second, up to 29 FPS generation, while remaining limited by scene-change and complexity handling.

  • Problem

    Existing motion-conditioned video generation is limited by prohibitive latency and non-causal processing that prevents real-time interaction.

  • Method

    MotionStream distills a motion-controlled bidirectional teacher into a causal student using self-rollout, joint text-motion guidance, attention sinks, and rolling KV caches.

  • Results

    MotionStream achieves state-of-the-art motion-control results with sub-second latency, reaching 29 FPS on a single GPU.

  • Takeaways & Limitations

    MotionStream enables interactive, potentially infinite-length video generation with users continuously guiding trajectories, cameras, or transferred motion.

  • Takeaways & Limitations

    Fixed attention sinks can preserve the initial scene when environments change continuously, limiting adaptation to complete scene changes.

Abstract

from arXiv · show

Current motion-conditioned video generation methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present MotionStream, enabling sub-second latency with up to 29 FPS streaming generation on a single GPU. Our approach begins by augmenting a text-to-video model with motion control, which generates high-quality videos that adhere to the global text prompt and local motion guidance, but does not perform inference on the fly. As such, we distill this bidirectional teacher into a causal student through Self Forcing with Distribution Matching Distillation, enabling real-time streaming inference. Several key challenges arise when generating videos of long, potentially infinite time-horizons -- (1) bridging the domain gap from training on finite length and extrapolating to infinite horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without incurring growth in computational cost due to increasing context windows. A key to our approach is introducing carefully designed sliding-window causal attention, combined with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we properly simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Our models achieve state-of-the-art results in motion following and video quality while being two orders of magnitude faster, uniquely enabling infinite-length streaming. With MotionStream, users can paint trajectories, control cameras, or transfer motion, and see results unfold in real-time, delivering a truly interactive experience.

1 INTRODUCTION

MotionStream targets interactive motion-controlled video by replacing slow, non-causal generation with streaming synthesis. It combines motion-conditioned teaching, causal distillation, efficient components, and long-horizon attention strategies to achieve fast, high-quality control.

  • Interactive motion-controlled video remains limited by slow generation, non-causal processing, and inability to produce more than a few seconds.A 5-second clip can take 12 minutes, while users cannot see partial results before the motion specification is complete.
  • MotionStream autoregressively synthesizes video in a streaming manner while reacting to user-drawn motion trajectories on-the-fly.
  • The motion-controlled teacher combines lightweight trajectory conditioning with joint text-motion guidance to balance trajectory adherence and natural secondary motion.The approach avoids the computational overhead of ControlNet-style architectures.
  • 17 FPS at 480P and 10 FPS at 720P are achieved with sub-second latency on one H100 GPU, reaching 29 FPS with optimized VAE decoders.
  • The system combines efficient architectural components, distillation, joint text-motion guidance, and a Tiny VAE to accelerate streaming generation.
  • Attention sinks, local attention, and extrapolation-aware distillation are introduced to prevent drift during long-term streaming.
  • MotionStream reports state-of-the-art motion transfer and camera-control results at orders-of-magnitude faster speeds.

2 RELATED WORK

Related work spans controllable video generation, motion-conditioned diffusion, autoregressive video models, and interactive video world models. MotionStream is positioned among approaches seeking causal, high-quality, real-time generation from motion and other controls.

  • Controllable Video Generation: Controllable video-generation research explores structure, camera, and other control signals for precise user control.
  • Controllable Video Generation: Motion is used as a conditioning signal through optical flow, 2D and 3D trajectories, bounding boxes, and semantic segmentation.
  • Autoregressive Video Models: Autoregressive video research has progressed from GAN-based synthesis toward diffusion models trained with denoising objectives.
  • Autoregressive Video Models: Recent hybrid autoregressive-diffusion methods distill slow teacher models into fast autoregressive students for real-time performance.
  • Interactive Video World Model: Interactive video world models aim to simulate environments for real-time interaction, although many existing approaches require substantial inference compute.

3 MO T I O NST R E A M: STREAMING GENERATION MEETS MOTION CONTROLS

MotionStream builds a motion-controlled bidirectional teacher and distills it into a causal student for streaming generation. Its track conditioning, joint text-motion guidance, autoregressive self-rollout, attention sinks, and rolling KV cache target efficient, stable long-video synthesis.

  • 3.1 Adding Motion Controls to Bidirectional Teacher Models: MotionStream first equips a pretrained video diffusion model with lightweight track conditioning to create a bidirectional motion-controlled teacher.Track embeddings are combined with image, noisy video latents, and text embeddings; the teacher is trained with flow matching.
  • 3.1 Adding Motion Controls to Bidirectional Teacher Models: Sinusoidal track embeddings and direct concatenation preserve trajectory control while avoiding the duplicated network blocks of ControlNet-style architectures.The track head applies temporal compression and a 1 × 1 × 1 convolution, while the core DiT remains unchanged.
  • 3.1 Adding Motion Controls to Bidirectional Teacher Models: Joint text-motion guidance balances natural dynamics from text with trajectory adherence and shape fidelity from track conditioning.The reported teacher weights are w_t = 3.0 and w_m = 1.5; distillation embeds the guidance into a single function evaluation.
  • 3.2 Causal Distillation: The causal student is trained through Self Forcing-style autoregressive rollout with DMD, conditioning on previously self-generated video chunks rather than ground truth.Causal adaptation uses teacher ODE solution pairs, while rollout maintains chunk-level conditioning through the KV cache.
  • 3.2 Causal Distillation: Attention sinks and rolling local windows bridge training and inference, keeping the KV cache fixed while preventing drift during long-video extrapolation.Static sink chunks remain in context as recent chunks roll through a fixed local window; RoPE positions follow cache locations.
  • 3.2 Causal Distillation: The student transfers the teacher’s joint-guidance control through a real-score target defined using the frozen, jointly guided teacher.This design transfers high-fidelity control without adding guidance inference overhead to the student.

4 EXPERIMENTS

Experiments evaluate MotionStream across motion transfer, camera control, guidance, attention, efficiency, and interactive streaming. Results support trade-offs between trajectory fidelity, visual quality, latency, and long-horizon stability.

  • Evaluation Setup: MotionStream is evaluated on DAVIS and Sora videos for motion transfer, and on LLFF for zero-shot single-image novel view synthesis.DAVIS includes challenging occlusions, while Sora provides consistently visible tracks; camera trajectories are derived using monocular depth and camera parameters.
  • Track Representation: Sinusoidal positional encoding with a learnable track head outperforms RGB-VAE encoding in quality and efficiency, achieving two-orders-of-magnitude faster encoding.The table caption reports 40× faster encoding for real-time streaming.
  • Guidance Strategies: Pure motion guidance maximizes trajectory accuracy, while joint guidance balances motion fidelity with natural dynamics and handles imperfect trajectories.The selected weights are w_t = 3.0 and w_m = 1.5.
  • Attention and Extrapolation: A single attention-sink chunk is crucial for long-video extrapolation, whereas additional sinks provide marginal benefit and larger windows can accumulate errors.The ablation uses clips up to 241 frames; removing sink tokens slightly improves speed but degrades long-term stability.
  • Chunk and Window Design: Chunk size 3 is selected as the streaming configuration because chunk size 1 degrades quality and chunks larger than 3 introduce prohibitive latency.The selected configuration is c3s1w1, which maintains high visual quality with small latency and throughput fluctuations.
  • Streaming Applications: The Tiny VAE and causal pipeline support interactive applications including motion transfer, drag-based control, precise camera control, and real-time streaming demonstrations.The Tiny VAE reduces decoding time by over 10× and improves Wan 2.1 from 16.7 FPS with 0.69s latency to 29.5 FPS with 0.39s latency.

5 CONCLUSION

MotionStream is presented as an infinite-length video-generation framework with interactive motion control and stable real-time operation. It combines efficient motion conditioning, causal distillation, attention sinks, rolling KV caches, and state-of-the-art task performance.

  • MotionStream maintains a stable 29 FPS on a single GPU for infinite-length video generation with interactive motion control.

A TRAINING EFFICIENT TINY VAE

This section evaluates Tiny VAE as a faster alternative to the full causal VAE for streaming generation. It reports substantially faster decoding with minimal practical quality differences when paired with the distilled student.

  • Reconstruction and Efficiency: Tiny VAE achieves an order-of-magnitude faster decoding than Full VAEs while outperforming existing community implementations in reconstruction quality.
  • VAE Bottleneck: Full CausalVAE decoding consumes 47% of Wan 2.1’s and 35% of Wan 2.2’s chunk-generation wall time in the base setup.Wan 2.1 provides 8× spatial and 4× temporal compression, while Wan 2.2 provides 16× spatial and 4× temporal compression.
  • Training: Tiny VAE training extends prior compact-decoder designs with LPIPS loss, proper data scaling, and selected hyperparameters.
  • Streaming Quality: Tiny VAE causes minimal quality differences in streaming practice because most quality degradation and drift originate from the diffusion process.
  • Streaming Quality: Tiny VAE increases throughput by 1.75× and 2.3× relative to Full VAE decoding while preserving distilled-model quality advantages over other baselines.

B VBENCH RESULTS AND USER STUDY

VBench-I2V and a user study assess video quality and related dimensions across MotionStream models and baselines. The results show competitive benchmark performance and strong quality relative to other baselines, with backbone capacity influencing visual quality.

  • VBench-I2V: VBench-I2V excludes camera motion and dynamic degree because input trajectories already constrain those dimensions, reducing their relevance as text-prompt metrics.
  • VBench-I2V: MotionStream’s teacher and distilled models consistently achieve competitive performance across all evaluated VBench-I2V dimensions.VBench scores correlate strongly with the underlying backbone, and image-plus-trajectory conditions produce uniformly high scores.
  • User Study: ATI produces more visually favorable videos than MotionStream in pure quality comparisons, but uses a 10× larger Wan 2.1-14B backbone and often lacks precise trajectory adherence.

C ADDITIONAL ABLATION EXPERIMENTS AND QUALITATIVE RESULTS

The ablations identify chunk size, sampling steps, and attention sinks as central speed–quality controls, while qualitative results demonstrate motion adherence, streaming interaction, and residual failures.

  • Ablations: Chunk size 3 provides the selected balance between interactivity and video quality, whereas smaller chunks degrade quality and larger chunks add prohibitive latency.Chunk size 1 causes significant quality degradation; chunks beyond 3 induce high latency for real-time interaction.
  • Ablations: 3 sampling steps are the primary configuration because more steps yield marginal quality gains while 2 steps noticeably reduce quality.The student is trained for 3-step generation, and the DMD framework permits flexible test-time sampling configurations.
  • Qualitative results: Attention sinks prevent cumulative drift during extended generation, maintaining stable quality where models without sinks increasingly degrade.Figure A3 compares long-video extrapolation with and without attention sinks.
  • Qualitative results: Motion control does not significantly degrade generative quality, although removing motion conditions causes a slight quality drop while preserving effective text following.The comparison uses the larger Wan 2.1 14B I2V model as a high-quality reference.
  • Qualitative results: The streaming demo supports online dragging, static points, pre-drawn multi-point trajectories, camera movement, and pausing or resuming generation.The demo begins with an input image and text prompts before users select track-grid spacing and control scene elements.

D TRAINING DETAILS

MotionStream is trained through staged motion conditioning and causal distillation, combining trajectory extraction, teacher training, ODE adaptation, and Self Forcing-style DMD.

  • Data preprocessing: Training uses 0.6M filtered OpenVid-1M videos plus synthetic Wan samples, with trajectories extracted on a 50×50 uniform grid.OpenVid-1M videos require at least 81 frames and a 16:9 aspect ratio; synthetic data includes Wan 2.1 and public samples.
  • Data preprocessing: The two-stage data strategy first establishes general motion conditioning on OpenVid-1M, then fine-tunes on cleaner synthetic data to improve adherence and reduce artifacts.Distillation uses only synthetic samples because the process converges quickly.
  • Distillation: Self Forcing-style DMD distillation requires only the first frame, text prompt, and motion tracks rather than complete video sequences.The causal adaptation and distillation process converges rapidly, with Self Forcing distillation taking around 400 steps.
  • Teacher model training: The teacher model combines text and motion conditions, using lightweight sinusoidal trajectory embeddings and joint guidance to balance trajectory adherence with natural secondary motion.This conditioning avoids the computational overhead of ControlNet-style architectures.
  • Causal architecture adaptation: Causal adaptation generates 4,000 teacher ODE trajectories and trains a unified student with diverse sparse causal attention masks.The masks support flexible self-rollout configurations across causal window patterns.
  • Implementation: Training uses AdamW, bfloat16 mixed precision, PyTorch FSDP, and exponential moving averages across teacher training and DMD distillation.Wan 2.1 variants train at 832 × 480 resolution, while Wan 2.2 trains at 1280 × 704.

E EVALUATION PROTOCOLS

Evaluation matches methods to their spatial and temporal capabilities, standardizes outputs and hardware measurements, and tests motion transfer and camera control under defined protocols.

  • Evaluation setup: DAVIS evaluation subsamples longer sequences by retaining the first and last frames and uniformly sampling intermediate frames with matching ground truth.This accommodates methods with different default temporal lengths.
  • Evaluation setup: Standard Sora experiments use 81 frames at 16 FPS, while extrapolation experiments extend to 241 frames with an average of 194 frames.Models with shorter temporal contexts receive analogous subsampling.
  • Baseline comparison: Each method is evaluated with its best reported configuration, using method-specific inputs such as RAFT flow, SpatialTracker tracks, or tested track counts.Image Conductor performs best with 100 tracks among the tested settings.
  • Metrics and efficiency: Results are resized to 832 × 480 for metric consistency, and latency and throughput are measured on one H100 GPU using bfloat16 and Flash Attention 3.The resizing particularly standardizes EPE, which computes L2 distance between track coordinates.
  • Camera control: Camera control is evaluated zero-shot on single-image 3D novel-view synthesis using LLFF prompts that specify static scenes with camera motion only.The protocol is designed to minimize unintended object motion.

F LIMITATION AND FUTURE WORK

The paper identifies limitations in scene-change handling, physically implausible trajectories, and complex-content fidelity, and proposes adaptive attention sinks, track augmentation, and larger backbones.

  • Limitations: Fixed attention sinks can preserve the initial scene instead of adapting when environments change continuously, limiting world-modeling scenarios.The limitation is attributed partly to current 2D tracking systems’ inability to encode complete scene transitions.
  • Limitations: Extremely rapid or physically implausible trajectories can produce temporal inconsistencies and distortions in object appearance.The paper suggests track augmentation to simulate imperfect user inputs during training.
  • Limitations: Highly complex scenes, prompts, or intended motions can reduce source-detail preservation, primarily because of backbone capacity limitations.Different image-conditioning mechanisms across backbones also affect robustness to imperfect motion cues.
  • Future work: Wan 2.1 1.3B generally preserves source structures better than Wan 2.2 5B for imperfect flat-grid trajectories.The paper attributes this to Wan 2.1’s input-image cross-attention design and expects larger base models to improve challenging-case stability.

G ETHICS STATEMENT

MotionStream acknowledges misuse risks inherited from realistic video generation and calls for safeguards and responsible deployment. The work also reports publicly available models and datasets, detailed training information, and limited LLM use.

  • MotionStream may enable deceptive media, reflecting misuse risks inherited from the underlying generative technology.
  • The authors call for watermarking, content authentication, controlled access, and responsible deployment alongside capability improvements.
  • The models use publicly available Wan variants and OpenVid-1M and synthetic Wan videos, with training details and implementation specifics provided for reproducibility.
  • LLMs were used to polish writing and presentation, not for research ideation, experimental design, or scientific discovery.
Loading 2511.01266v5…