PaperlayerView exact revision on arXiv

Source-linked AI summary

Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Control

Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Chi Zhang

arXiv:2509.15130v3cs.GRcs.AIcs.CV

TL;DR

Video diffusion models have useful spatial priors but limited camera control and consistency for 3D/4D tasks. WorldForge adds training-free inference-time guidance, and the paper reports state-of-the-art results for trajectory control and visual quality. Its performance remains limited by the quality of depth estimation.

  • Problem

    Video diffusion models have limited controllability, spatial-temporal consistency, and geometric fidelity for 3D/4D tasks, including precise camera-trajectory following.

    Source: §1 · p. 2

  • Method

    WorldForge guides pretrained video diffusion models at inference time using recursive denoising refinement, flow-gated latent fusion, and dual-path self-correction.

    Source: §1 · p. 2

  • Results

    The authors report state-of-the-art trajectory adherence, geometric consistency, and perceptual quality over leading baselines, including training-intensive pipelines.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    WorldForge offers training-free trajectory-controllable generation for static 3D and dynamic 4D scenes while preserving the base model’s world priors.

    Source: §5 · p. 8

    Performance is inherently bottlenecked by the quality of the underlying depth estimation; complex dynamics or severe depth errors can cause structural distortions.

    Source: §4.8 · p. 22

Abstract

from arXiv · show

Video diffusion models have rich world priors, but their use in spatial tasks is limited by poor control, spatial-temporal inconsistent results, and entangled scene-camera dynamics. Current approaches, such as per-task fine-tuning or post-process warping, often introduce visual artifacts, fail to generalize, or incur high computational costs. We introduce WorldForge, a novel, training-free framework that operates purely at inference time to resolve these issues. Our method comprises three synergistic components. First, an intra-step refinement loop injects fine-grained motion guidance during the denoising process, iteratively correcting the output to ensure strict adherence to the target camera path. Second, an optical flow-based analysis identifies and isolates motion-related channels within the latent space. This allows our framework to selectively apply guidance, thereby decoupling motion from appearance and preserving visual fidelity. Third, a dual-path guidance strategy adaptively corrects for drift by comparing the guided generation against an unguided, reference denoising path, effectively neutralizing artifacts caused by misaligned structural inputs. Together, these components inject precise, trajectory-aligned control without model retraining, achieving accurate motion guidance and photorealistic synthesis. As a plug-and-play, model-agnostic solution, WorldForge demonstrates highly versatile generalizability. Beyond robust zero-shot 3D/4D generation, it readily empowers over a dozen diverse downstream applications, seamlessly enabling tasks like video editing, stabilization, and virtual try-on. Extensive experiments confirm state-of-the-art performance in trajectory adherence and perceptual quality, outperforming both training-dependent and inference-only baselines.

1. Introduction

Video diffusion models offer rich spatiotemporal priors, but limited controllability, consistency, and geometric fidelity constrain their use in 3D/4D tasks. WorldForge combines inference-time guidance mechanisms to improve trajectory control while preserving those priors.

1. Introduction

  • VDMs struggle with precise camera trajectories, spatial-temporal consistency, and geometric fidelity, while entangling scene and camera motion can deform objects or destabilize scenes.These limitations constrain their use in tasks requiring structured spatial reasoning or controllable generation.Source: §1 · p. 2
  • Fine-tuning can be costly and generalize poorly, while warping-and-repainting can produce artifacts and fragmented geometry on out-of-distribution warped inputs.These trade-offs leave fine-grained controllability, generation quality, and generalization challenging to balance.Source: §1 · p. 2
  • IRR inserts a predict-correct loop within each denoising step, replacing predicted content in observed regions with corresponding ground-truth observations.This injects trajectory control signals at every step to align generated motion with the target trajectory.Source: §1 · p. 2
  • FLF uses optical-flow similarity to selectively inject trajectory information into motion-correlated latent channels while leaving appearance-relevant channels unchanged.This is intended to decouple appearance from motion and preserve content fidelity during viewpoint manipulation.Source: §1 · p. 2
  • DSG compares guided and unguided denoising paths to correct the guided path toward the unguided path’s perceptual quality while maintaining camera control.The correction addresses artifacts associated with imperfect depth, occlusions, or misalignments in warped inputs.Source: §1 · p. 2
  • WorldForge combines three inference-time mechanisms to improve trajectory control while preserving video diffusion model priors, without training or fine-tuning.The framework integrates Intra-Step Recursive Refinement (IRR), Flow-Gated Latent Fusion (FLF), and Dual-Path Self-Corrective Guidance (DSG).Source: §1 · pp. 2–3
  • Experiments report state-of-the-art controllability and visual quality across diverse datasets and tasks, including against training-intensive pipelines.The framework is described as model-agnostic and applicable to backbones such as Wan 2.1 and SVD.Source: §1 · pp. 2–3

2. Related Works

The related-work discussion covers static 3D scene generation, trajectory-controlled video generation, and guidance for generative models. WorldForge is positioned as an inference-time guidance approach to precise motion control and warp-artifact correction.

3D Static Scene Generation

  • Static 3D scene generation approaches use video diffusion model priors, while training-free methods preserve pretrained priors but must maintain geometric coherence.The discussion contrasts these methods with costly fine-tuning that can corrupt priors.Source: §2 · p. 3

Trajectory-Controlled Dynamic Video Generation

  • Trajectory-controlled video generation commonly uses fine-tuned adapters or warp-and-repaint, with the latter prone to artifacts from noisy warps.WorldForge instead directly steers the diffusion process through inference-time guidance.Source: §2 · p. 3

Guidance and Control for Generative Models

  • Generative-model guidance ranges from CFG to auxiliary models and iterative refinement, while warp-based 3D/4D methods can suffer noise-induced artifacts.DSG derives a per-step correction signal from guided and unguided predictions without retraining.Source: §2 · p. 3

3. Proposed Methods

WorldForge guides pretrained video diffusion models at inference time, combining trajectory-aware warping with three mechanisms for balancing camera control and visual fidelity. The method intervenes during denoising, selectively guides motion-related latent channels, and corrects divergence between guided and unguided paths.

3. Proposed Methods

  • WorldForge combines depth-based trajectory guidance with IRR, FLF, and DSG to steer pretrained video diffusion models without retraining.The framework is presented as a way to balance controllability with visual fidelity.Source: §3 · p. 3, Figure 2 · p. 4

3.1. Preliminaries

  • The method is framed around diffusion models, guidance strategies, and trajectory-controlled video synthesis.Source: §3.1 · p. 3

3.1.1. Denoising Diffusion Models and Guidance

  • WorldForge modifies the one-step denoised estimate during sampling, and its intervention can extend to solvers that compute or recover that estimate.Its experiments primarily use the flow-based Wan2.1 model.Source: §3.1.1 · p. 3

3.1.2. Trajectory Control via Depth-Based Warping

  • A depth network estimates camera poses and depth maps, which are used to warp source frames into partial target views with visibility masks for guiding generation.This guidance applies only to regions visible in the source views.Source: §3.1.2 · p. 4

3.2. Intra-Step Recursive Refinement

  • IRR fuses warped trajectory content into the one-step denoised estimate, re-noises the fused latent, and feeds it into the next denoising step for continual trajectory correction.The fusion copies observable warped content according to a binary mask.Source: §3.2 · pp. 4–5

3.3. Flow-Gated Latent Fusion

  • FLF scores latent channels by comparing predicted and target optical flow in visible regions, then selectively fuses trajectory information into motion-relevant channels.Flow gating is disabled in early steps, then the number of guided channels is progressively reduced to preserve fine details.Source: §3.3 · p. 5

3.4. Dual-Path Self-Corrective Guidance

  • DSG compares guided and unguided velocity fields, using the guided component orthogonal to the unguided direction and adaptive cosine-based scaling to correct artifacts while preserving natural predictions.The correction strengthens when the paths diverge and lessens when they align.Source: §3.4 · pp. 5–6

4. Experiments

Across 3D static and 4D dynamic generation, WorldForge outperforms baselines under matched evaluation; component and backbone/depth tests probe its mechanisms and transferability.

4.1. Implementation Details

  • Experiments use Wan2.1 I2V-14B primarily, test 70+ single views and 50+ videos, and evaluate generation quality and trajectory accuracy with task-specific metrics.The setup uses a single GPU with at least 69 GB VRAM for Wan2.1; SVD runs 25-frame inference on a 24 GB GPU, and IRR is applied during roughly the first 35–45% of steps.Source: §4.1 · pp. 6–7

4.2. 3D and 4D Trajectory-Controlled Generation

  • WorldForge outperforms training-based and training-free 3D baselines and matches or surpasses costly training-based methods on 4D clips, with stronger fidelity and trajectory alignment.The comparisons use identical evaluation settings; results also report more coherent scene completion and plausible reconstruction of unseen regions.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • Across 40+ static and 30+ dynamic scenes, channel 13 is most often filtered out, channel 8 never is, and 4D scenes show more diverse flow scores.Filtering frequency indicates low optical-flow score and poor motion correlation; the statistics support selective guidance.Source: Figure 4 · p. 7

4.3. Ablation Experiments

  • Ablations show IRR enables trajectory guidance, FLF limits warped-frame noise, and DSG improves detail and plausibility; standard CFG in DSG causes severe artifacts.The complete model gives the best fidelity and control, and the quantitative ablation covers static and dynamic generation metrics.Source: §4.3 · p. 8, Table 3 · p. 8, Figure 5 · p. 8
  • WorldForge transfers to SVD and LongCat-Video and remains robust across tested depth estimators, with performance scaling with base-model capabilities.SVD’s limited implicit world knowledge restricts performance; the method’s depth-model integration is plug-and-play.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

WorldForge is a training-free framework for trajectory-controllable generation in static 3D and dynamic 4D scenes. Experiments report state-of-the-art performance, though iterative guidance does not meet real-time requirements.

5. Conclusion

  • Extensive experiments show state-of-the-art performance on both 3D and 4D generation tasks.The framework aims to balance visual quality, generalization, and precise control in video synthesis.Source: §5 · p. 8
  • WorldForge combines IRR, FLF, and DSG at inference time to decouple motion from appearance and correct trajectory drift while preserving base-model priors.The framework is training-free and targets trajectory-controllable generation in static 3D and dynamic 4D scenes.Source: §5 · p. 8
  • Iterative guidance currently prevents the framework from meeting real-time requirements.Future work targets stronger generative models and distillation to reduce sampling steps.Source: §5 · p. 8

1. Proof of The Equivalence between Diffusion and Flow Models

The section shows that Flow Matching is a special case of diffusion modeling under a linear noise schedule. Its probability-flow vector field matches the derivative of the Flow Matching path.

1. Proof of The Equivalence between Diffusion and Flow Models

  • Diffusion models reverse a forward process that transforms data toward a simple prior, commonly a Gaussian; the PF-ODE offers a more efficient generation route.The PF-ODE shares the SDE’s marginal distributions at each time.Source: §1 · p. 14
  • Flow Matching linearly interpolates data x0 and Gaussian noise ϵ, with schedule parameters αt = 1 − t and σt = t.The section identifies these parameters by comparing its path with the general diffusion forward process.Source: §1 · p. 14
  • Flow Matching’s vector field equals the derivative of its linear path, making it a diffusion-model instance with αt = 1 − t and σt = t.The section therefore places Flow Matching within the broader diffusion modeling framework.Source: §1 · pp. 14–15

2. Evaluation Metrics

The evaluation uses seven metrics spanning static image quality, dynamic video quality, and camera-trajectory consistency. It also specifies preprocessing, reference sets, and a shared protocol for baseline comparisons.

2. Evaluation Metrics

  • Seven metrics assess static-scene quality, dynamic-scene quality, and camera trajectory consistency across realism, semantics, temporal coherence, and motion fidelity.The metrics are FID and CLIPsim, FVD and CLIP-Vsim, and ATE, RPE-T, and RPE-R, respectively.Source: §2 · p. 15

2.1. Static Scene Evaluation

  • FID compares real and generated image-feature distributions, while CLIP similarity measures semantic similarity using pretrained vision-language representations.FID uses Inception-V3 pool3 features; CLIP similarity uses normalized CLIP ViT-B/32 image features.Source: §2.1 · p. 15

2.2. Dynamic Scene Evaluation

  • FVD compares real and generated video distributions using spatio-temporal features, while CLIP-Vsim measures frame-level semantic consistency across video pairs.FVD uses I3D features; CLIP-Vsim extends CLIP similarity to the temporal domain.Source: §2.2 · p. 15

2.3. Camera Trajectory Evaluation

  • ATE measures global camera-position consistency after Sim3 alignment, while RPE-T and RPE-R assess translation and rotation accuracy between consecutive frames.The relative-pose metrics use a one-frame index gap and report root-mean-square errors.Source: §2.3 · pp. 15–16

2.4. Evaluation Details

  • Evaluation specifies image and video resizing and sampling, aggregates reference sets for distributional metrics, and applies the same protocol to all baselines.Trajectory estimates are recovered by SfM and aligned to the reference with Sim3 before metrics are computed.Source: §2.4 · p. 16

3. Implementation Details

The implementation details explain how Flow-Gated Latent Fusion scores motion relevance in latent channels and how its guidance settings vary over denoising steps. The defaults are based on Wan 2.1 and can be adjusted for scene requirements.

3. Implementation Details

  • Flow-Gated Latent Fusion scores each latent channel’s motion similarity at each denoising step to selectively update motion-relevant channels while preserving appearance details.The section presents this as the module’s purpose.Source: §3 · p. 16

3.1. Details for FLF Estimation and Motion Scoring

  • FLF computes dense optical flow between consecutive latent frames and evaluates predicted against ground-truth flow over valid non-occluded pixels using masked endpoint error, masked angular error, and outlier percentage.Source: §3.1 · pp. 16–17
  • FLF normalizes its three flow metrics before combining their inverted errors into the channel motion score.Normalization constants and metric weights are specified for the experiments.Source: §3.1 · p. 17

3.2. Hyperparameter Settings

  • On Wan 2.1’s 50-step sampling schedule, IRR runs during the first 20 steps; FLF progressively relaxes channel filtering to balance structural guidance and detail preservation.Filtering is disabled in steps 0–5, permits at most two channels to retain original predictions in steps 6–10, and at most six from step 11 onward.Source: §3.2 · p. 17, Table 1 · p. 17

4. More Experimental Results

Additional experiments report inference efficiency, depth-model robustness, and qualitative ablations of Dual-Path Self-Corrective Guidance. They also illustrate video-editing operations supported by the method.

4.2. Ablation of DSG and Naive CFG

  • The 50°–70° guided-versus-unguided angular gap, compared with <5° in typical CFG, motivates DSG; naive CFG distorts structure, while full DSG preserves integrity and camera-path adherence.The qualitative ablation contrasts standard CFG, DSG without adaptive βt, and the full DSG framework.Source: Figure 1 · p. 18, §4.2 · pp. 18–19

4. More Experimental Results

  • Depth-model ablations show that the method compensates for errors or missing regions and remains compatible with different depth estimators despite depth-warping variation or noise.The figure attributes this self-correction to the video diffusion models’ learned priors.Source: Figure 2 · p. 18
  • The method’s depth-based warping also supports freezing the camera, video stabilization, and video-content editing.These are presented as additional video effects beyond video re-cam.Source: Figure 3 · p. 19

4.1. Efficiency and Runtime Analysis

  • The framework adds no training cost; IRR contributes the main runtime overhead, while DSG and FLF add substantially less computation than backbone inference.Its inference speeds are reported as comparable to, and often faster than, existing methods; the efficiency comparison is on a single NVIDIA A100.Source: §4.1 · pp. 17–18, Table 2 · p. 19, Table 3 · p. 19

4.3. Ablation on Video Diffusion Models

The framework transfers across video diffusion backbones and depth estimators, supports video editing, and handles challenging camera trajectories and scenes. Its performance remains dependent on depth-estimation quality.

4.3. Ablation on Video Diffusion Models

  • Table 4 compares the framework on SVD, Wan 2.1, and LongCat; the evaluation uses identical inputs after minor architecture-specific hyperparameter tuning.The authors note that SVD’s smaller parameter count and architecture limit its ability to exploit the guidance method’s potential.Source: Table 4 · p. 20, §4.3 · p. 20

4.4. Ablation on Depth Estimation Models

  • Across VGGT, UniDepth, Mega-SaM, and DepthCrafter, the framework maintains high performance and compensates for noisy, erroneous, or disoccluded depth-warped inputs.The authors attribute this resilience to the video diffusion model’s world priors, which support artifact correction and inpainting.Source: §4.4 · p. 20

4.5. Applications in Video Editing

  • Video editing applications include stabilization, camera freezing, viewpoint switching, object removal or addition, subject replacement, and virtual try-on.For content edits, the described process segments the target region, edits the first frame, then processes it with masks; a bounding box can guide new-object placement.Source: §4.5 · p. 20

4.6. Generation on Challenging Scenes

  • For human-centric scenes, the method maintains stationarity and produces natural renderings; for large camera movements, it uses iterative multi-clip generation to create 180° turns and coherent 360° object-centric views.The next clip uses the previous clip’s last frame as its prior, and the 360° sequence connects its final frame to its first.Source: §4.6 · p. 20, Figure 4 · p. 21, Figure 5 · p. 21
  • Under fast motion, occlusions, and non-rigid dynamics, FLF improves results over disabling it, while adaptive thresholding adjusts channel control when flow is unstable.The authors report that this adaptation helps prevent channel misclassification and maintain structural stability.Source: §4.6 · p. 22, Figure 7 · p. 22

4.7. Robustness across Optical Flow Estimators

FLF improves generation quality with different optical-flow estimators, while the paper identifies depth quality as a constraint. Additional qualitative results cover both 3D generation and dynamic video re-filming across backbones.

4.7. Robustness across Optical Flow Estimators

  • FLF consistently enhances generation quality with both Farnebäck and RAFT; a multi-metric score evaluates flow magnitude, direction, and reliability.The authors report that this scoring mitigates single-metric noise and makes lightweight Farnebäck sufficient for the pipeline.Source: §4.7 · p. 22

4.8. Limitations and Failure Cases

  • Severe depth errors or extremely complex scene dynamics can cause structural distortions, and performance is inherently bottlenecked by the underlying depth estimate.The authors state that dynamic gating typically keeps outputs no worse than the baseline without FLF guidance in chaotic cases.Source: §4.8 · p. 22, Figure 9 · p. 23

4.9. More Cases

  • Additional qualitative results report visual fidelity and structural plausibility for single-view 3D generation and dynamic video re-filming on Wan 2.1 and LongCat.The supplied figure captions cover two 3D-generation cases and three dynamic re-filming cases.Source: §4.9 · p. 22, Figure 10 · p. 23, Figure 11 · p. 24
Loading 2509.15130v3…