PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldForge V2: Eliciting Camera-Trajectory Alignment in Video Diffusion Models

Chenxi Song, Yanming Yang, Tong Zhao, Ruibo Li, Joey Tianyi Zhou, Chi Zhang

arXiv:2610.31640v1cs.CV

TL;DR

Video diffusion models have rich spatial priors but do not reliably follow prescribed camera paths. WorldForge V2 combines warped-reference guidance, efficient correction, and selective LoRA adaptation. The paper reports state-of-the-art visual quality and trajectory accuracy, with effective trajectory following possible from a single training scene.

  • Problem

    Pretrained video diffusion models do not reliably follow prescribed camera paths, and fusing warped references can create a denoising-state gap that produces visual artifacts.

    Source: §I · p. 2

  • Method

    WorldForge V2 guides denoising with warped references using IRR, FLF, and DSG, adds v-cache correction, and adapts trajectory-sensitive blocks with LoRA.

    Source: §I · p. 2

  • Results

    WorldForge V2 achieves state-of-the-art visual quality and trajectory accuracy in camera-controlled video generation with limited training data and iterations.

    Source: §I · p. 2

  • Takeaways & Limitations

    Training data from a single scene is sufficient to enable effective trajectory following, including on unseen scenes.

    Source: §I · p. 2

Abstract

from arXiv · show

Video diffusion models have learned rich 3D/4D priors, making them promising for scene generation and video re-rendering. However, their use in spatial tasks is limited by imprecise camera control, spatial and temporal inconsistencies, and entangled camera motion with scene dynamics. Existing approaches, such as task-specific fine-tuning and post-process warping, often introduce visual artifacts, generalize poorly, or incur high computational costs. We propose WorldForge V2 to achieve accurate camera control while preserving the generative capabilities of pretrained video models. We first develop a training-free guidance framework using warped references to iteratively refine generation along a target camera path. Optical-flow analysis identifies motion-related latent channels for selective guidance, helping separate motion control from appearance. A dual-path strategy further compares guided and unguided predictions to correct artifacts from imperfect references. Together, these components guide viewpoint changes while preserving scene appearance and object motion. To reduce refinement costs, v-cache approximates the correction using the preceding denoising prediction, avoiding an additional model evaluation. We further extend motion localization from latent channels to network blocks and introduce Trajectory LoRA, which concentrates adaptation on blocks most responsive to trajectory guidance. A two-stage training strategy teaches these adapters to denoise the states encountered during guided generation, using limited supervision effectively. This enables reliable camera control with very little training data, with a single training scene already supporting generalization to unseen scenes. Quantitative and qualitative evaluations across multiple datasets show that our method achieves state-of-the-art generation quality and trajectory accuracy in camera-controlled video generation.

I. INTRODUCTION

Video diffusion models offer strong spatial and temporal priors but do not reliably follow prescribed camera paths. WorldForge V2 combines warped-reference guidance, efficient correction, and selective adaptation to control trajectories while retaining pretrained generation quality.

I. INTRODUCTION

  • With limited training data and iterations, WorldForge V2 achieves state-of-the-art visual quality and trajectory accuracy in camera-controlled video generation.Source: Figure 1 · p. 1, §I · p. 2
  • IRR, FLF, and DSG address denoising-state mismatch, motion–appearance entanglement, and unreliable warped references in training-free guidance.IRR fuses warped references in the predicted clean estimate before re-noising; FLF selects motion-related channels, while DSG compares guided and unguided predictions.Source: §I · p. 2
  • V-cache approximates clean-estimate correction in noisy latent space using an available denoising prediction, avoiding an additional model evaluation.Source: §I · p. 2
  • Trajectory-sensitive LoRA adapts selected blocks with a two-stage strategy, and training data from a single scene can enable effective trajectory following.Source: §I · p. 2

A. Video Diffusion Priors for Spatial Generation

B. Camera Control in Video Diffusion Models

C. Guidance and Correction Strategies in Generative Models

D. Parameter-Efficient Adaptation for Camera Control

WorldForge V2 uses geometric warping to provide trajectory references, then guides denoising or adapts selected model blocks. Its setup distinguishes the current noisy latent from its predicted clean content and motivates selective, data-efficient control.

D. Parameter-Efficient Adaptation for Camera Control

  • WorldForge V2 extends motion localization from latent channels to trajectory-sensitive network blocks and applies lightweight LoRA only to those blocks.The approach builds on findings that motion- and camera-related representations concentrate in particular channels or layers.Source: D. Parameter-Efficient Adaptation for… · p. 4

III. METHOD

  • Given a source image or video and a target camera path, geometric warping produces a trajectory reference and visibility mask used to guide denoising.Source: §III · p. 4, A. Preliminaries · p. 4

A. Preliminaries

  • The current noisy latent and its one-step clean estimate represent different states, either of which can be modified to inject trajectory evidence.Source: A. Preliminaries · p. 4
  • LoRA represents trainable updates to fixed pretrained weights with low-rank matrices, using fewer trainable parameters than full-model fine-tuning.Source: A. Preliminaries · p. 4
  • Direct noisy-latent insertion combines an externally re-noised visible reference with model-generated content, creating a denoising-state gap.The reference may also contain holes, disocclusions, and depth-boundary errors; the paper describes Warp-as-History and Go-with-the-Flow as alternative trajectory-signal interfaces.Source: A. Preliminaries · pp. 4–5

B. Inference-Time Trajectory Guidance

Inference-time guidance injects warped trajectory references while attempting to preserve appearance and reduce artifacts. IRR fuses in clean-estimate space, FLF selects motion-related channels, and DSG calibrates guided predictions against the base denoising path.

B. Inference-Time Trajectory Guidance

  • IRR fuses the warped reference into the model-predicted clean estimate under a visibility mask, then re-noises the fused result with one Gaussian noise sample.This avoids directly stitching regions associated with incompatible noisy states and reduces their denoising-state gap.Source: B. Inference-Time Trajectory Guidance · p. 5
  • FLF uses optical flow to select channels whose motion agrees with the target trajectory, restricting reference fusion to those channels and reducing appearance distortion.Its schedule keeps more channels during early noisy steps and tightens selection as appearance details emerge.Source: B. Inference-Time Trajectory Guidance · p. 5
  • DSG compares trajectory-guided and base predictions, matching their magnitudes and using the guided component orthogonal to the base direction to reduce artifacts from unreliable references.The paper reports cosine similarity of 0.3–0.6 between the two velocities, compared with nearly parallel branches in standard CFG.Source: B. Inference-Time Trajectory Guidance · pp. 5–6

C. Efficient Noisy-Latent Correction

V-cache reuses the preceding denoising step’s velocity to approximate IRR’s noisy-latent correction with one model evaluation per guided step. Trajectory LoRA restricts adaptation to trajectory-sensitive blocks and trains on v-cache-aligned states, with reported generalization from one scene to unseen scenes.

C. Efficient Noisy-Latent Correction

  • V-cache approximates IRR’s correction in noisy latent space using the preceding step’s cached velocity, requiring one model evaluation per guided step.The first guided step has no cached velocity and uses direct noisy-latent insertion.Source: C. Efficient Noisy-Latent Correction · p. 6

D. Trajectory-Sensitive LoRA Adaptation

  • Trajectory LoRA extends motion localization from latent channels to network blocks, retaining adapters only in blocks identified as sensitive to trajectory signals.The method probes candidate self-attention blocks and ranks their contributions using Shapley analysis.Source: D. Trajectory-Sensitive LoRA Adaptation · pp. 6–7
  • Two-stage training warms adapters on real v-cache states, then constructs additional v-cache-consistent inputs at sampled noise levels.Both stages preserve warped-reference insertion and v-cache correction to reduce the training–inference gap.Source: D. Trajectory-Sensitive LoRA Adaptation · p. 7
  • Training on one scene supports generalization to unseen scenes, and the trained LongCat adapter transfers to other trajectory-injection methods without additional fine-tuning.At inference, only the selected trajectory-sensitive LoRA adapters are loaded.Source: D. Trajectory-Sensitive LoRA Adaptation · p. 7

IV. EXPERIMENTS

Across static 3D and dynamic 4D evaluations, training-free guidance combines camera-path accuracy with generation quality. V-cache reduces noise-state gaps without IRR’s extra correction evaluation, while adapted models improve reconstruction metrics against camera-control baselines.

IV. EXPERIMENTS

  • The experiments assess static and dynamic guidance, V-cache correction, data-efficient adaptation, transfer, and ablations of guidance, block selection, and training.Static evaluation uses FID and CLIP similarity; dynamic evaluation uses FVD and CLIP-V similarity, with additional reconstruction and trajectory metrics for Trajectory LoRA.Source: §IV · p. 7, A. Experimental Setup · pp. 7–8

A. Experimental Setup

  • The LoRA comparison uses 200 test cases across 40 scenes with training and test scenes disjoint.Training pairs are rendered from DL3DV scenes represented with optimized 3D Gaussian Splatting.Source: A. Experimental Setup · p. 7

B. Inference-Time Camera-Trajectory Control

  • The training-free method leads on all static metrics and on dynamic FVD, CLIP-V similarity, RPE-T, and RPE-R, while preserving scene structure under large viewpoint changes.The static comparison covers identical source inputs; the dynamic comparison uses matched source videos and target trajectories.Source: B. Inference-Time Camera-Trajectory… · p. 8, Table I · p. 8, Table II · p. 8

C. V-Cache Correction and Data-Efficient LoRA

  • V-cache reduces inferred-noise gaps similarly to IRR while avoiding IRR’s extra model evaluation for correction.Direct insertion leaves gaps, particularly in log-variance, accompanied by noisy patches and structural errors.Source: C. V-Cache Correction and… · p. 9, Figure 4 · p. 9
  • Both adapted backbones outperform camera-control baselines on all five reconstruction metrics, and H3 achieves the lowest local camera errors.The reported improvements include preservation of observed content and completion of newly exposed regions.Source: C. V-Cache Correction and… · p. 9, Table III · p. 9

D. Additional Results

Additional results show that trajectory control transfers across camera-control methods, while ablations examine guidance components, block selection, training data, updates, and training strategy. Across these tests, selective adaptation and training on guided denoising states support camera control while preserving generation quality.

D. Additional Results

  • The same LongCat adapter improves surface continuity and object detail in WAH [18] and GWF without further fine-tuning.The inference-time guidance also transfers to SVD [1] and LongCat-Video and remains effective with different depth and optical-flow estimators.Source: D. Additional Results · p. 9, Figure 6 · p. 10

E. Ablation Studies

  • LongCat and H3 recover coherent shapes and material detail from incomplete warped references, with fewer rendering remnants and more continuous geometry than baselines.The comparison includes thin structures and partially revealed objects as the camera exposes new content.Source: Figure 5 · p. 10
  • The complete inference-time framework performs best on all four quality metrics, while removing FLF or DSG reduces static and dynamic generation quality.Replacing DSG with its CFG form causes further degradation; Fig. 5 compares qualitative scene generation, including partially revealed objects and thin structures.Source: Table IV · p. 10, E. Ablation Studies · p. 10, Figure 5 · p. 10
  • Fig. 7 shows trajectory-sensitive blocks dispersed through LongCat’s middle and later layers, but concentrated more regularly in H3’s later layers.Sensitivity is estimated after brief probe LoRA training, using approximately 400 steps and 25 trajectory pairs per backbone.Source: E. Ablation Studies · pp. 10–11, Figure 7 · p. 11
  • Selecting roughly the most sensitive three quarters of blocks matches or improves full-block performance, and sensitive-half selection beats random selection on both backbones.LongCat’s 36 selected blocks improve reconstruction and local camera accuracy; H3’s 38 retain comparable reconstruction and reduce camera errors. Low-response selection performs worst.Source: E. Ablation Studies · pp. 11–12, Table V · p. 11
  • Both backbones achieve good reconstruction and camera control after single-scene training, while most gains from added scene diversity arrive with two to four scenes.LongCat gains in reconstruction and rotation accuracy; H3 gains mainly in translation accuracy, and expanding coverage to eight scenes brings limited further benefit.Source: Table VI · p. 11, E. Ablation Studies · p. 12
  • LongCat trains more stably with eight scenes and reaches its best perceptual quality near 1,200 updates, while H3 learns reliably from one scene.Removing LoRA from H3’s less sensitive blocks accelerates convergence and improves perceptual quality over full-block adaptation; later updates refine camera accuracy with diminishing gains.Source: Figure 8 · p. 11, E. Ablation Studies · p. 12
  • The full two-stage training strategy improves reconstruction over Standard FM on both backbones and achieves the best camera accuracy on H3.Anchor warm-up adapts the model to intermediate guided-sampling states; the second stage extends training to guided inputs at additional noise levels. Fig. 9 shows more continuous surfaces and finer detail than No LoRA and Standard FM.Source: E. Ablation Studies · p. 12, Table VII · p. 12, Figure 9 · p. 12

V. CONCLUSION

WorldForge V2 combines training-free guidance, v-cache, and selective two-stage Trajectory LoRA adaptation to control camera paths while preserving video generation capabilities. Evaluations across multiple datasets report state-of-the-art generation quality and trajectory accuracy; long-horizon real-time streaming remains future work.

V. CONCLUSION

  • WorldForge V2 targets accurate camera control while preserving pretrained video models’ generative capabilities.Its training-free guidance uses warped references with motion-aware channel selection and self-correction; v-cache reuses denoising predictions, and Trajectory LoRA applies selective adaptation with two-stage training.Source: §V · p. 13
  • Across multiple datasets, quantitative and qualitative evaluations show state-of-the-art generation quality and trajectory accuracy in camera-controlled video generation.The conclusion identifies long-horizon, real-time streaming generation with camera control as future work for interactive world models.Source: §V · p. 13

1. Equivalence between Diffusion and Flow Models

The appendix relates diffusion noise predictions and Flow Matching velocity predictions through a shared clean-latent estimate. For the linear data-to-noise path considered, the corresponding diffusion schedule is α_t = 1 − t and σ_t = t.

1. Equivalence between Diffusion and Flow Models

  • The camera guidance uses the model’s clean-video-latent estimate, obtainable from either a diffusion noise prediction or a Flow Matching velocity prediction.The paper derives the relationship by expressing the Flow Matching path through a diffusion model’s noise schedule.Source: §1 · p. 16
  • For the considered linear interpolation between data and Gaussian noise, the corresponding diffusion schedule is α_t = 1 − t and σ_t = t.The conditional path’s derivative is ϵ − x_0; its conditional expectation gives the marginal Flow Matching velocity.Source: §1 · p. 16
  • For corresponding noise and velocity predictions, the clean latent is obtained from the noisy latent and Flow Matching velocity using the stated schedule.The passage gives the relation as x_t − t v_θ(x_t, t).Source: §1 · p. 16

2. Evaluation Protocol

The protocol evaluates visual quality, reconstruction and completion, and camera-trajectory accuracy across static scenes, dynamic videos, and learned-control cases. It uses shared comparison inputs and target paths, with metric procedures tailored to each evaluation setting.

2. Evaluation Protocol

  • Evaluation measures visual quality and trajectory accuracy on training-free benchmarks, and reconstruction of observed and newly exposed content against rendered DL3DV targets.The static benchmark includes more than 70 source views from over 40 scenes; the dynamic benchmark includes more than 50 outputs from over 30 source videos.Source: §2 · p. 16, §2.1 · p. 16

2.1. Data and Comparison Protocol

  • The main learned-control comparison uses 200 DL3DV test cases from 40 scenes, with five random camera paths per scene and training scenes disjoint from test scenes.Sequences contain 25–93 frames, with lengths adapted to each method’s supported input format; methods within each comparison share inputs, target trajectories, and evaluation cases.Source: §2.1 · pp. 16–17

2.2. Static Scene Evaluation

  • Static-scene evaluation uses FID for visual quality and CLIP similarity for semantic agreement between corresponding real and generated images.FID compares Gaussian distributions of Inception-V3 image features; CLIP similarity averages cosine similarity between paired-image features.Source: §2.2 · p. 17

2.3. Dynamic Scene Evaluation

  • Dynamic-scene evaluation uses FVD for spatiotemporal feature distributions and CLIP-Vsim for frame-level semantic agreement across corresponding videos.CLIP-Vsim averages corresponding-frame CLIP similarity within each video pair and then across pairs; videos are evaluated using 16 uniformly sampled frames, including the first and last.Source: §2.3 · p. 17

2.4. Reconstruction and Completion

  • Reconstruction and completion are evaluated with PSNR, SSIM, and LPIPS on a common 640 × 360 area, excluding the input frame.Vis-LPIPS scores observed regions and Disocc-LPIPS scores newly exposed regions; empty regions are omitted, and scores are aggregated with equal weight over scenes.Source: §2.4 · p. 17

2.5. Camera-Trajectory Accuracy

  • Camera accuracy uses aligned trajectory RMSE metrics for training-free benchmarks and normalized translation errors plus angular error for DL3DV paths.ATE measures global position agreement; RPE-T and RPE-R measure adjacent-frame translation and rotation errors. DL3DV alignment uses a positive-scale similarity transform, and the input frame is excluded from alignment and scoring.Source: §2.5 · pp. 17–18

3. Implementation Details

Implementation details specify how FLF selects latent channels, how V-Cache approximates IRR’s correction, and how Trajectory LoRA identifies and trains responsive blocks. The settings also cover the guided sampling schedule and tests of training strategy and transfer.

3.1. FLF Motion Estimation and Channel Scoring

  • FLF scores per-channel motion agreement with the rendered reference and fuses it only into selected channels, using displacement, angular, and outlier errors to guide selection.Channels not selected retain the model’s prediction to preserve appearance details.Source: §3.1 · p. 18

3.2. Hyperparameter Settings

  • Because early motion estimates are unreliable, all channels receive reference fusion at steps 0–5; later, up to two or six channels may retain their predictions.For Wan2.1, IRR runs during the first 20 steps of a 50-step UniPC sampling schedule by default.Source: §3.2 · pp. 18–19

3.3. V-Cache

  • V-Cache uses the preceding denoising step’s cached velocity to approximate IRR’s correction in noisy latent space, avoiding an extra correction evaluation.On three held-out cases, V-Cache and IRR reduce regional inferred-noise gaps over much of denoising compared with Direct Insertion.Source: §3.3 · p. 19

3.4. Trajectory LoRA Adaptation

  • A probe LoRA measures block contributions to trajectory-guided velocity changes, and the highest-scoring self-attention blocks are selected for adaptation with pretrained weights fixed.The contribution scores account for interactions between blocks by averaging marginal changes across sampled permutations and validation states.Source: §3.4 · p. 19
  • Trajectory LoRA training first adapts on recorded guided anchors, then expands to constructed guided states and directly noised targets; experiments also vary scene diversity, training strategy, and transfer across guidance methods.The state-construction step does not track gradients; the subsequent denoising prediction updates LoRA parameters.Source: §3.4 · p. 20

4. Additional Experiments

Additional experiments examine component behavior, backbone and estimator compatibility, training efficiency, and control limits. They also demonstrate wide-range camera paths and editing using rendered references.

4.1. Component Behavior and Motion Selection

  • IRR introduces camera motion, FLF reduces appearance distortion, and DSG further corrects the guided prediction; stable channel-filtering differences across more than 40 static and 30 dynamic scenes reflect motion relevance.The complete method preserves coherent scene content along the requested path.Source: §4.1 · p. 20, Figure S1 · p. 20, Figure S2 · p. 20

4.3. Backbone and Geometry Compatibility

  • LongCat-Video leads on CLIP and RPE-T, while Wan2.1 leads on ATE and RPE-R; both improve all reported metrics over SVD under shared rendered-reference guidance.The comparison covers more than 40 static scenes.Source: Table S3 · p. 21, §4.3 · p. 21

4.2. Self-Corrective Guidance

  • Cosine-based projection with adaptive weighting preserves scene structure and fine detail, unlike standard CFG, while removing DSG’s adaptive angular weight weakens consistency.The comparison concerns combining original and trajectory-guided predictions.Source: §4.2 · p. 21, Figure S3 · p. 21

4.3. Backbone and Geometry Compatibility

  • FLF reduces rendering artifacts with both Farnebäck and RAFT, while warped references from four geometry estimators yield recognizable layouts and plausible surfaces despite differing estimates.The geometry estimators are VGGT, UniDepth, Mega-SaM, and DepthCrafter.Source: §4.3 · p. 21, Figure S5 · p. 21

4.5. Challenging Scenes and Control Limits

  • Under fast motion, non-rigid motion, and heavy occlusion, FLF reduces marked distortions while retaining viewpoint changes, but severe depth errors and complex dynamics can still produce visible inconsistencies.Large camera rotations are handled by chaining clips from their final frames, and rendered references also support camera stabilization and content editing.Source: §4.5 · pp. 21–22, Figure S8 · p. 22, Figure S7 · p. 22

4.4. Training Resources and Inference Efficiency

  • 4.09× on LongCat and 5.09× on H3 are the speedups from combining trajectory adapters with released acceleration LoRAs; eight-step H3 samples retain scene layout and object geometry in the examples.The adapters are trained from 50 videos using 1,200 updates for LongCat or 1,500 for H3.Source: §4.4 · p. 21, Table S4 · p. 22, Figure S6 · p. 22

5. Additional Visual Comparisons

Additional held-out DL3DV comparisons show LongCat and H3 preserving scene layout and local appearance where several baseline outputs exhibit visual artifacts. The supplementary visuals also illustrate camera and content editing with rendered references.

Additional visual comparisons

  • Across six held-out DL3DV cases, LongCat and H3 preserve scene layout and local appearance, while several baselines show distorted boundaries, texture loss, or rendering remnants.The examples include indoor layouts, outdoor structures, and finely textured objects.Source: §5 · p. 22, Figure S11 · p. 23

Camera and content editing

  • Rendered references support camera freezing and stabilization, while an edited first frame and region masks propagate appearance changes through the video.The editing setup can also use a bounding box to specify where to add an object absent from the source video.Source: Figure S10 · p. 23

Related papers

Loading 2610.31640v1…