PaperlayerView exact revision on arXiv

Source-linked AI summary

4DStreamCtrl: Interactive Video Generation with Online 4D Control

Shiqian Li, Chenguo Lin, Zhiguang Liu, Yu Tang, Jiarong Ou, Rui Chen, Yixin Zhu

arXiv:2608.25479v1cs.CVcs.AI

TL;DR

Existing video-control methods do not jointly provide 3D-consistent camera and object control with real-time streaming generation. 4DStreamCtrl unifies these controls through 3D point tracks, trains on mined motion supervision, and distills the model into a causal student that achieves precise control with long, coherent streaming video.

  • Problem

    Existing methods separately address camera control, 2D object trajectories, or offline 3D control, leaving no unified real-time interface for 3D-consistent camera and object motion.

  • Method

    4DStreamCtrl combines a 3D point-track interface, OpenVidHD-Motion3D supervision, a Geometric Motion Head, and causal streaming distillation.

  • Results

    The model achieves state-of-the-art motion-control precision and 3D consistency while sustaining coherent 350-frame clips at memory independent of length.

  • Takeaways & Limitations

    Explicit 3D geometry plus causal inference supports real-time 4D-controllable streaming video generation on a single GPU.

  • Takeaways & Limitations

    PSNR is lower than the reference baseline, so the authors treat EPE and LPIPS as the primary control-quality indicators.

Abstract

from arXiv · show

Generative video models now synthesize footage nearly indistinguishable from reality. Their promise as interactive tools hinges on fine-grained control of how objects and the camera move over time, yet each existing approach captures only part of this: camera-parameter methods steer the viewpoint but cannot move objects, 2D-trajectory methods act in the image plane and ignore depth and occlusion, and recent 3D methods add geometry but run only offline at a fixed length. In particular, none combines 3D-consistent control of both camera and objects with real-time, streaming generation. Here we show that camera motion, object trajectories, and depth can be unified into a single 3D point-track representation, from which one model performs joint camera and object control, depth editing, and motion transfer in a single forward pass. To learn this interface at scale, we mine in-the-wild video for 3D motion supervision, yielding OpenVidHD-Motion3D, and encode it with a lightweight Geometric Motion Head that plugs into a pretrained video diffusion model. Because this encoder is temporally separable, we distill the model into a causal streaming student that generates arbitrarily long video in four denoising steps at memory independent of length. This unified design surpasses prior camera-only, 2D, and offline-3D methods in motion-control precision while covering modalities they address only in isolation. 4DStreamCtrl runs at 20 FPS on a single high-end GPU for 480p video and stays temporally coherent over hundreds of frames, enabling, to our knowledge, interactive 4D-controllable streaming generation for the first time. More broadly, grounding generation in explicit 3D geometry with efficient causal inference points toward interactive world models with closed-loop spatiotemporal control, from controllable simulators to real-time visual imagination for embodied agents.

1 INTRODUCTION

4DStreamCtrl unifies camera motion, object trajectories, and depth in a 3D point-track interface for precise, interactive video control. It combines large-scale motion supervision with causal streaming generation to support long, coherent videos.

  • Existing camera-parameter methods control viewpoints but cannot move objects, while 2D trajectories ignore depth and occlusion and fail under parallax.
  • 4DStreamCtrl represents camera geometry, object trajectories, and depth together as a single 3D-track conditioning signal.This interface supports joint camera and object manipulation, motion transfer, and depth-aware editing in one forward pass.
  • OpenVidHD-Motion3D provides roughly 0.4M in-the-wild clips with 3D point tracks and per-frame camera parameters for training.A lightweight Geometric Motion Head encodes these signals for a pretrained video diffusion model.
  • Causal distillation converts the bidirectional teacher into a streaming student that responds to signals arriving during playback.The student uses causal generation to support online control rather than waiting for a complete clip.
  • 350-frame clips remain coherent at memory independent of length, while the model achieves state-of-the-art motion-control precision and 3D consistency.The reported duration is 14.6 s at 24 FPS.

2 RELATED WORK

Prior work improves motion control through object trajectories, 3D supervision, few-step distillation, or streaming generation, but these directions address different parts of the problem.

  • Motion-conditioned video methods inject object, drag, point-based, or trajectory cues to reduce the underspecification of language-only conditioning.
  • 3D supervision methods recover geometry, cameras, and point dynamics from video, enabling scalable motion signals beyond image-plane control.
  • Few-step distillation reduces the cost of iterative diffusion, while autoregressive and chunk-based generators produce frame blocks sequentially for streaming.

3 METHOD

The method trains a pretrained video diffusion transformer on mined 3D motion supervision, encoding tracks and depth into temporally separable latent-grid features. A causal student then enables few-step, long-horizon streaming generation with constant memory.

  • 3.1 3D TRACK DATASET CONSTRUCTION: OpenVidHD-Motion3D is built by extracting 3D point trajectories and camera parameters from OpenVid-1M, filtering unreliable sequences to retain approximately 0.4M clips.Each example contains RGB frames, a text prompt, 3D tracks, and camera parameters.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: 3D tracks are projected to normalized image coordinates, rasterized on the latent grid, and combined with a separate depth branch before fusion with noisy latents.The architecture uses separate track and depth features to condition the DiT backbone.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: The motion encoder compresses raw frames temporally by preserving the first frame and grouping every four subsequent frames into one latent frame.This yields F = (T−1)/4 + 1 latent frames.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: Track identity uses shared sinusoidal embeddings, while rasterization scatter-adds features and handles multiple tracks occupying the same cell by summation.Empty raster cells and the depth branch implicitly handle occlusion and missing samples without an explicit visibility mask.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: The training objective fine-tunes a pretrained TI2V diffusion transformer using flow-matching supervision conditioned on text, an initial image, and 3D motion features.The network predicts a velocity field for noisy interpolated latents.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: A two-stage curriculum first trains short 256p, 49-frame clips, then scales to 480p, 81-frame clips for finer alignment and longer-horizon consistency.The ablations confirm the effectiveness of this schedule.
  • 3.2 3D MOTION-CONDITIONED VIDEO MODEL: The causal student distills the approximately 50-step bidirectional teacher into four steps while retaining the motion encoder and replacing bidirectional attention with block-wise causal attention.

4 EXPERIMENTS

Experiments evaluate 4DStreamCtrl across joint object-camera control, motion transfer, interactive streaming, and ablations. Results show strong motion and perceptual control, online responsiveness, constant-memory generation, and benefits from explicit 3D conditioning and the proposed motion-head and training designs.

  • Joint object and camera control: The teacher achieves the best reported motion alignment on DAVIS, with EPE 5.29, LPIPS 0.404, and SSIM 0.479.On a matched Wan 2.2-5B backbone, replacing 2D with 3D tracks cuts EPE by roughly 33% and LPIPS by 5%.
  • Joint object and camera control: The causal student reaches 20.6 FPS while maintaining EPE 5.48, within 4% of the teacher and below every causal baseline.It outperforms MotionStream Causal at both Wan 2.1-1.3B and Wan 2.2-5B settings on throughput.
  • Motion transfer: Motion transfer reproduces source 3D-consistent motion, including body dynamics, camera motion, and background parallax, in new visual styles and scene content.The source motion is extracted with SpatialTrackerV2 and applied to a restyled first frame without retraining or per-instance optimization.
  • Interactive streaming control: The streaming student sustains over 20 FPS at 480p with memory independent of generated length, enabling coherent generation over extended sequences.Chunk-wise four-step denoising lowers latency and raises throughput; bounded rolling KV-cache memory stays flat as video lengthens.
  • Ablation study: Explicit 3D tracks outperform 2D conditioning on depth-sensitive behavior, while track identity encoding and low-resolution warm-up materially improve motion metrics.2D conditioning incurs 0.5 EPE and 0.025 LPIPS; removing track-ID embeddings raises EPE from 0.985 to 2.455, and single-stage full-resolution training raises EPE to 3.217.

5 CONCLUSION

4DStreamCtrl unifies 3D control signals with large-scale motion supervision and causal streaming distillation. The resulting framework targets precise, real-time 4D-controllable video generation and interactive world models.

  • 5 CONCLUSION: 4DStreamCtrl unifies camera motion, object trajectories, and per-point depth in one 3D-track interface for 4D control.The interface supports joint control, cross-subject motion transfer, and depth-aware editing.
  • 5 CONCLUSION: OpenVidHD-Motion3D provides roughly 0.4M in-the-wild clips with 3D tracks and camera parameters for learning the control interface.A lightweight, temporally separable Geometric Motion Head injects these signals into a pretrained video diffusion backbone.
  • 5 CONCLUSION: Causal streaming distillation reduces inference from 50 steps to 4 steps while keeping memory independent of video length.The framework enables real-time 4D-controllable streaming synthesis on a single GPU.
  • 5 CONCLUSION: The results suggest explicit 3D geometry combined with efficient causal inference as a path toward interactive world models with closed-loop spatiotemporal control.The paper identifies remaining failure modes and discusses limitations and future directions.

B Training Details A4

The training-details material covers teacher training, streaming distillation, and ablations examining distillation, random seeds, control-point count, and attention sinks.

  • B Training Details A4: Teacher training is covered in Section B.1.
  • B Training Details A4: Streaming distillation is covered in Section B.2.
  • B Training Details A4: The ablations examine distillation, random seeds, control-point count, and attention sinks.

A.1 LONG VIDEO GENERATION

The method generates videos extending to 350 frames while preserving plausible appearance, temporal coherence, and motion fidelity across diverse activities.

  • A.1 LONG VIDEO GENERATION: 350 frames span three generated sequences of gesturing, cooking, and delicate craft work beyond the training clip length.
  • A.1 LONG VIDEO GENERATION: Per-frame SSIM starts above 0.90 and remains above 0.75 throughout the sequences.The gradual decline is attributed to increasing temporal distance from the conditioning frame.
  • A.1 LONG VIDEO GENERATION: Track accuracy remains low for most frames, with transient EPE spikes during rapid motion that recover without persistent drift.A pronounced peak occurs around frame 125 during fast hand movement near a sewing machine.

A.2 WITH OR WITHOUT TRACK CONDITIONING

Point-track conditioning improves spatial consistency and physical plausibility in object interactions, while long generated sequences still exhibit blur and causal object-state failures.

  • A.2 WITH OR WITHOUT TRACK CONDITIONING: Track conditioning keeps a rolling ball’s spatial extent and scale constant, producing natural rolling motion.Without conditioning, the ball progressively enlarges over time.
  • A.2 WITH OR WITHOUT TRACK CONDITIONING: Point tracks constrain a pushed can to a physically plausible rigid-body trajectory while preserving shape and orientation.Without track guidance, the can rotates and deforms implausibly as the sequence progresses.
  • A.2 WITH OR WITHOUT TRACK CONDITIONING: Point tracks act as a spatial prior that complements text guidance without requiring explicit physics simulation.This combination addresses object drift, scale inconsistency, and implausible deformation.
  • A.3 FAILURE CASES: Small faces and background objects progressively blur as videos extend, reducing fine detail such as facial features and object texture.The paper attributes this likely to limited spatial resolution allocated to small regions.
  • A.3 FAILURE CASES: Some sequences violate causality and object permanence, including a cake losing a section and a piping bag appearing later.The paper identifies structured physical or causal reasoning as a promising future direction.

B.1 TEACHER TRAINING

The teacher and student are trained to encode 3D motion signals into a pretrained video diffusion model, then generate video causally in chunks for online control.

  • Teacher model and conditioning: The teacher is a 5B-parameter Wan2.2 text-and-image-to-video DiT operating on 81-frame, 480×832 latent clips.LoRA adapters are inserted into the pretrained transformer while the motion encoder, depth branch, and expanded patch embedding accept additional control features.
  • Teacher model and conditioning: The conditioning tensor concatenates motion and depth features with noisy latents before patch embedding, while the flow-matching objective predicts the noise-to-clean-latent velocity.The first-frame latent is supplied as ground truth and excluded from the loss.
  • Teacher model and conditioning: 3D tracks store camera-space coordinates, camera intrinsics, and visibility, then become normalized image positions and inverse-depth features for training.Each sample uses a fixed N=256 tracks selected from the tracked points in the video.
  • Causal student distillation: The teacher is distilled into a causal 4-step student through ODE pair generation, ODE initialization, and adversarial DMD training.The student uses block-wise causal attention, chunked denoising, and cached preceding context to match streaming deployment.
  • Causal student distillation: Online inference recomputes conditioning chunk by chunk, allowing users to revise tracks or depth during generation without changing the architecture.Temporal separability makes per-chunk encoding bit-identical to full-sequence encoding, while the rolling cache supports constant-memory streaming.

C.1 ABLATIONS ON DISTILLATION

The ablations identify cfg = 1 and lr = 4e-6 as the preferred distillation settings, while seed tests show that trajectory-controlled regions remain stable and ambiguity drives variation.

  • Distillation hyperparameters: At 4K steps, cfg = 1 achieves the best PSNR, SSIM, LPIPS, and EPE among the tested guidance scales.It reaches about 18.5 PSNR, 0.65 SSIM, 0.22 LPIPS, and 0.95 EPE.
  • Distillation hyperparameters: lr = 4e-6 consistently outperforms lr = 3e-6 across every guidance scale and metric.The advantage is widest for PSNR and LPIPS, with no reported instability.
  • Distillation hyperparameters: Higher guidance values increase EPE slightly over training, whereas cfg = 1 keeps motion error low and stable.The study therefore adopts cfg = 1 with lr = 4e-6 as the default.
  • Seed variation: With the first frame and trajectories fixed, regions directly governed by trajectories move almost identically across random seeds.This indicates that the input trajectories provide a strong spatial control signal.
  • Seed variation: Seed-dependent variation concentrates in underspecified regions, including entering hands and occluded pan interiors.Their appearance, timing, and fine-grained states are not fully constrained by the first frame or trajectories.

C.3 EFFECT OF THE NUMBER OF CONTROL POINTS

Control quality improves as the student receives more tracks up to the training density of 256, then degrades when inference uses substantially denser conditioning; the attention sink prevents long-video drift.

  • Control-point density: From 16 to 256 tracks, PSNR rises from 16.25 to 18.27, SSIM from 0.55 to 0.64, and LPIPS falls from 0.29 to 0.23.EPE remains stable between 2.60 and 2.75 over this range.
  • Control-point density: Beyond 256 tracks, quality declines moderately at 512 and sharply at 1024, where PSNR is 16.21, LPIPS is 0.31, and EPE is 3.41.The model handles sparser subsets of the learned conditioning pattern more gracefully than counts above its training density.
  • Control-point density: The student is trained and used with 256 tracks by default because quality peaks at the training count.The control-point study varies inference-time counts from 16 through 1024 with other settings fixed.
  • Attention sink: Retaining the initial token as a persistent attention sink stabilizes color, identity, and visual quality across a 350-frame generated sequence.Without the sink, saturation intensifies, identity drifts, and fine detail becomes overly smooth as frames accumulate.

D LIMITATIONS AND FUTURE WORK

The paper identifies limitations in monocular 3D supervision, fixed-density tracks, and the streaming student’s teacher gap, then proposes extensions toward closed-loop, physics-informed, embodied, and multi-view systems.

  • Limitations: Monocular 3D labels introduce scale ambiguity and become less reliable under rapid motion or severe occlusion.At inference, this can appear as subtle depth-scale inconsistency under extreme camera baselines.
  • Limitations: The fixed 32^32 track grid can underserve localized fine-grained interactions such as dexterous hand manipulation and dense multi-body contact.Variable-density or hierarchical layouts are proposed to improve controllability without increasing the average budget.
  • Limitations: The causal student is 12.5^ faster than the bidirectional teacher but loses about 0.5 dB PSNR and has a marginally higher EPE on DAVIS.Dependencies extending beyond the fixed local attention window may require adaptive memory.
  • Future work: Closed-loop perception could feed corrective 3D tracks from real-time visual trackers back into conditioning to compensate for drift without user intervention.The paper frames this extension as analogous to model-predictive control.
  • Future work: Physics-informed trajectory synthesis could couple the interface with differentiable physics engines or learned dynamics models.
  • Future work: The streaming architecture is proposed for robotic planning, real-time visual imagination, cross-view-consistent generation, and VR or AR content creation.These directions include embodied policy learning, multi-view diffusion, and 4D scene representations.
Loading 2608.25479v1…