PaperlayerView exact revision on arXiv

Source-linked AI summary

CausalAlign: Causal Alignment of Video Generation Models for Consistent Ego-Motion

Kai Zhu, Yintianrun Zhang, Shuwei Jiang, Ruolin Zuo, Yixin Chen, Duomin Wang, Miao Liu, Yin Li, Daquan Zhou, Yiwu Zhong

arXiv:2610.29915v1cs.CV

TL;DR

Video generation models can produce plausible videos but may drift in camera ego-motion, which matters for consistent geometry and downstream action prediction. CausalAlign diagnoses ego-motion state across denoising and aligns identified layers to a 3D foundation model through delta tokens. It improves ego-motion consistency while preserving visual quality, and the fine-tuned model raises RoboCasa365 mobile-manipulation success from 43.5% to 54.2%.

  • Problem

    Video generation models struggle to capture 3D camera trajectories and maintain consistent viewpoint changes across frames, despite their visually plausible outputs.

    Source: §1 · p. 1

  • Method

    CausalAlign uses linear probing and causal intervention to identify where ego-motion fades, then aligns those VGM states to a 3D foundation model through delta-token space.

    Source: §1 · p. 2

  • Results

    CausalAlign consistently improves ego-motion consistency and temporal dynamics while preserving visual quality; RoboCasa365 success rises from 43.5% to 54.2%.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The improved VGM can facilitate downstream world-action-model applications, including mobile-manipulation tasks.

    Source: §1 · p. 3

    The reference trajectory is used only for evaluation and is not provided as a generation condition.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Diffusion-based video generation models (VGMs) can produce visually plausible videos, yet their camera ego-motion often drifts over long horizons. Camera ego-motion determines how the viewpoint changes between frames and is essential for maintaining consistent geometry and trajectories. To address this problem, we propose CausalAlign, a causal alignment framework with two components. First, we diagnose how precise ego motion is represented, used, and retained within VGMs using linear probes and causal interventions, revealing a rise-and-fall pattern during denoising. Second, we strengthen ego-motion state in VGM by aligning the representations at the identified layers to a 3D foundation model through delta-token space. This space encodes the changes between consecutive frames, allowing the alignment to focus on ego-motion dynamics rather than static scene content. With Wan2.2 fine-tuned on DL3DV and RealEstate10K, CausalAlign can largely improve motion accuracy, geometric consistency, and temporal smoothness, while preserving visual quality. For example, it improves geometric consistency by more than 20% relative to the base model. Moreover, with the fine-tuned VGM, CausalAlign improves the success rate of world action models (WAMs) by 10 points on mobile manipulation tasks of RoboCasa365 benchmark. Overall, CausalAlign improves ego-motion consistency in diffusion-based VGMs and yields noticeable gains in embodied models.

1 Introduction

Video generation models can create plausible videos but struggle to maintain consistent camera trajectories, a concern for downstream action prediction. CausalAlign diagnoses where ego-motion fades and aligns those states to 3D delta-token representations.

1 Introduction

  • Inconsistent camera trajectories remain an open problem despite prior 3D-prior and preference-alignment approaches, and can induce systematic downstream action-prediction errors.The paper frames precise ego-motion representation, use, and retention within VGMs as a central question.Source: §1 · p. 1
  • Ego-motion decodability and causal effect rise and then fall across denoising timesteps and Transformer layers, a pattern termed geometric state forgetting.The state is increasingly used at intermediate stages but not well retained at later stages.Source: §1 · p. 2
  • CausalAlign uses probing and intervention to locate where ego-motion is represented, used, and forgotten, then targets those locations for alignment.The alignment is intended to strengthen the internal state without compromising visual-generation quality.Source: §1 · p. 2
  • Delta tokens encode changes between consecutive frames, reducing redundant static scene content while carrying motion information for alignment to a 3D foundation model.A projector maps hidden-state transitions into the teacher space, and the resulting tokens supervise identified layers.Source: §1 · pp. 2–3
  • CausalAlign preserves visual quality while reducing reprojection error by 22.6% and improving trajectory smoothness by 23.9% relative to the base model.The reported ego-motion-state pattern also changes from rise-and-fall to rise-and-maintain.Source: §1 · p. 3
  • RoboCasa365 success rate rises from 43.5% to 54.2% with the fine-tuned VGM.The paper reports this result for mobile-manipulation tasks.Source: §1 · p. 3

2 Related Work

Prior work uses video-generation models for action-conditioned futures and 3D models for geometric supervision, while geometry-aware adaptation methods leave specific alignment gaps. CausalAlign targets diagnosed locations in the generator and transfers geometric knowledge through delta-token alignment.

2 Related Work

  • Video-generation models increasingly serve as action-conditioned simulators or policies, making ego-motion consistency relevant to world action models.The paper notes that erroneous camera motion can corrupt predicted actions.Source: §2 · p. 3
  • Feed-forward 3D models provide camera and scene representations for supervision; CausalAlign adopts LingBot-Map as its teacher.The paper describes LingBot-Map as a streaming model with temporal state.Source: §2 · p. 3
  • Geometry-aware adaptation includes external 3D caches, preference alignment, and direct feature alignment, which respectively risk cache errors, lack direct internal-state supervision, or miss effective timestep and layer locations.CausalAlign instead aligns internal ego-motion states at locations identified by causal diagnosis.Source: §2 · pp. 3–4
  • CausalAlign training constructs teacher delta tokens, learns and freezes a projector, then optimizes LoRA adapters at identified VGM layers with the full training split.The projector is trained using a small subset; the final stage keeps it frozen.Source: Figure 2 · p. 4

3 Method

CausalAlign diagnoses where ego-motion information is represented and used, then aligns VGM transitions with a 3D foundation model’s delta-token space at selected layers and denoising steps. It uses the resulting model for video generation and initializes a world action model’s video expert.

3.1 Method Overview

  • CausalAlign identifies layers and denoising steps where ego-motion fades, then uses delta-token alignment and LoRA updates to strengthen those states.The 3D foundation model and projectors are frozen after training; the original backbone is retained.Source: §3.1 · p. 4

3.2 Causal Diagnosis of Ego-Motion States

  • Linear probes measure relative-camera-motion decodability, while fixed-prompt, fixed-noise temporal interventions estimate whether hidden states causally affect generated trajectories.The method combines normalized layer and timestep profiles and selects locations with the largest positive decline rates.Source: §3.2 · p. 5

3.3 Delta-Token Alignment

  • Teacher delta tokens encode transitions, and auxiliary prediction tasks encourage them to retain transition information and camera motion.The 3D teacher supplies geometric states; temporal and pose objectives preserve transition structure and anchor tokens to camera motion.Source: §3.3 · pp. 5–6
  • VGM-side projectors map selected-layer transitions into teacher space, and LoRA adapters are trained to align those transitions with teacher delta tokens.Projectors are frozen after training, and gradients pass through frozen projectors and the camera decoder to the LoRA adapters.Source: §3.3 · p. 6

3.4 Inference

  • At inference, the adapted Wan2.2 uses its standard generation procedure without auxiliary modules, extra inputs, or geometric computation.Generation takes an initial frame and text prompt.Source: §3.4 · p. 6

3.5 Application to World Action Models

  • Fast-WAM is initialized with adapted Wan2.2 as its video expert and trained with the original flow-matching objectives to predict executable actions.The policy uses synchronized images from three camera views; future video is a training target, not an inference output.Source: §3.5 · p. 6

4 Experiment

Causal diagnosis identifies where Wan2.2 forms and later forgets ego-motion, and CausalAlign targets those locations with delta-token supervision. Across benchmark comparisons, it improves camera consistency while preserving visual quality, and the fine-tuned model also improves RoboCasa365 success.

4.1 Causal Diagnosis and Target Selection

  • Wan2.2’s decodability and causal effect rise and then fall; CausalAlign’s selected layers and timesteps change this to a rise-and-maintain pattern.Layers 14, 10, 12, and 20 are selected in descending decline-rate order, with layer 14 primary; timesteps 398 and 799 are also selected.Source: §4.1 · p. 7

4.2 Experiments on the Comprehensive Evaluation

  • CausalAlign improves all seven ego-motion metrics over Wan2.2 and leads on six of seven compared methods.It reduces reprojection error from 8.4 to 6.5 pixels (22.6%), raises trajectory smoothness from 0.71 to 0.88 (23.9%), and lowers jerk from 0.182 to 0.151.Source: Table 1 · p. 8, §4.2 · p. 8
  • CausalAlign preserves visual quality comparable to Wan2.2 while improving ego-motion consistency.VBench quality changes from 0.805 to 0.806, I2V score remains 0.958, and VBench smoothness rises from 0.983 to 0.985.Source: §4.2 · p. 9, Table 2 · p. 9
  • CausalAlign + Fast-WAM raises RoboCasa365 average success from 43.5% to 54.2% (+10.7 percentage points).Atomic Seen success rises from 59.1% to 78.1% (+19.0 points); Composite Seen and Composite Unseen success also increase.Source: §4.2 · pp. 9–10, Table 3 · p. 10

4.3 Ablation Studies

  • Structured delta tokens outperform unpartitioned encoders across all six metrics, with cross-probes showing each region performs best on its corresponding task.The structured representation uses 64D ego-motion, 40D geometry, and 24D visual regions.Source: §4.3 · p. 10, Table 4 · p. 10
  • Learned 128D delta tokens outperform 8192D concatenated and 4096D difference features across all six metrics.Compared with the original feature pair, rotation error decreases from 3.243 to 2.530 degrees.Source: §4.3 · p. 10, Table 4 · p. 10

5 Conclusion

CausalAlign diagnoses where VGMs use and then lose ego-motion state, and aligns the identified representations to a 3D foundation model’s delta-token space. Its appendix details the diagnostic and alignment procedures and their evaluation support.

5 Conclusion

  • CausalAlign mitigates geometric state forgetting by targeting identified locations and aligning ego-motion states to a 3D foundation model’s delta-token space.The reported pattern changes from rise-and-fall to rise-and-maintain while preserving visual quality and improving ego-motion consistency and temporal dynamics; the improved VGM also raises WAM success on RoboCasa365 mobile manipulation tasks.Source: §5 · p. 11

Appendix

  • The appendix documents causal diagnosis, target selection, three-stage delta-token alignment, evaluation protocols, and further experiments and ablations.It also extends experiments to text-to-video and camera-trajectory-conditioned generation.Source: Appendix · p. 16

A.1 Causal Diagnosis and Target Selection

  • Target selection uses a fixed diagnostic subset of about 10% of the data, chosen before adaptation from frozen Wan2.2 measurements and excluded from held-out and WAM evaluations.The selected locations remain fixed in subsequent experiments; re-evaluation of the subset after adaptation is only for visualizing geometric-state dynamics.Source: §A.1 · p. 16
  • Diagnosis probes relative camera motion and intervenes on hidden states across layer–denoising locations to measure decodability and effects on generated camera trajectories.The four interventions are Mask, Shuffle, Replace, and Inject; diagnosis uses noised real videos for probe features and free-running generation for intervention analysis.Source: §A.1 · pp. 16–17
  • The diagnostic progress coordinate runs from earlier, higher-noise states to later, lower-noise states, and targets include layer 14 plus auxiliary layers 10, 12, and 20.Layer 14 is primary; the timestep targets use the reindexed coordinate, where s = 398 is earlier than s = 799.Source: §A.1 · pp. 16–17

A.2 Additional Details of Delta-Token Alignment

  • A frozen streaming 3D model supplies geometric states whose consecutive-frame differences form teacher delta tokens, trained with transition reconstruction, temporal prediction, and relative-pose supervision.Layer-specific projectors map VGM transition features into this teacher space; final LoRA updates use pose and delta-token matching losses only at target layers and timesteps, with no teacher or projector computation at inference.Source: §A.2 · p. 18

A.3 More Evidence for geometric state Forgetting

Additional analyses test whether late ego-motion signal decline reflects limited probe access, motion-specific causal effects, and retention rather than geometry formation. They find a rise-and-fall pattern across tested probes and late attenuation despite persistent geometry commitment.

Linear and nonlinear decodability

  • Both linear and MLP probes show ego-motion decodability rising and then falling during denoising, so the late decrease is not specific to linear readout.The result supports reduced accessibility under both tested probe families, without claiming that every possible nonlinear representation is unrecoverable.Source: §A.3 · p. 18, Figure 5 · p. 19

Motion-specific causal intervention

  • Motion-only interventions have substantially larger effects than norm-matched content-only controls, and the motion-specific effect decreases during late denoising.The comparison indicates that the measured intervention effect is primarily attributable to the motion-specific component rather than generic hidden-state perturbation.Source: Figure 5 · p. 19, §A.3 · p. 19

Geometry commitment and late attenuation

  • Geometry commitment rises early and remains high while probe decodability and motion-specific causal effects decline later, supporting a retention-failure interpretation rather than failure to form camera geometry.The three curves compare geometry commitment with the decodability and motion-only causal-effect curves reported in the complementary analyses.Source: §A.3 · p. 19, Figure 5 · p. 19

B Extended Evaluation of CausalAlign

The extended evaluation clarifies the main-paper I2V metrics and training settings, explains why direct ground-truth trajectory error is not used for unconstrained I2V, and reports complementary generation experiments.

B Extended Evaluation of CausalAlign

B.1 Metric Definitions and Normalization

This section defines motion-magnitude proximity scores and a composite gain used in update-placement experiments. The proximity scores compare clip-level means with real-data means, not full motion distributions.

B.1 Metric Definitions and Normalization

  • Translational and angular speeds are computed from estimated camera motion and depth, then summarized as clip-level means.Source: §B.1 · pp. 19–20
  • Mean Trans. Prox. and Mean Angular Prox. measure how close generated clip-level mean motion is to the corresponding real-data mean; they do not measure agreement with the full motion distribution.A value closer to one indicates greater proximity to the real-data mean.Source: §B.1 · p. 20
  • The update-placement composite gain combines rotation and translation-direction RPE, Sim(3) ATE, endpoint error, speed-profile error, and depth AbsRel.The composite compares an adapted model with the base model using direction-corrected normalized scores and fixed bounds from the comparison set.Source: §B.1 · p. 20

B.2 Training and Inference Details

The appendix details training and evaluation protocols, then tests CausalAlign in text-to-video and explicitly trajectory-conditioned generation. The reported results cover ego-motion metrics, visual quality, and two successful RoboCasa365 rollouts.

B.2 Training and Inference Details

  • Training data use 90/10 train/evaluation splits and fixed 30%, 10%, and 100% training-split subsets for the three stages; inference retains only the adapted VGM.No teacher or external geometric cache is used at inference.Source: §B.2 · pp. 20–21
  • The main unconstrained I2V protocol gives generators only the first frame and prompt; multiple future trajectories can be valid, so evaluation uses shared intrinsic metrics rather than ground-truth pose matching.Explicitly trajectory-conditioned results are interpreted separately, with direct trajectory comparison.Source: §B.2 · p. 21

B.3 Text-to-Video Generation

  • CausalAlign improves text-to-video quality, semantic score, FVD, and LPIPS, matches smoothness, and has lower CLIP and SSIM scores.Source: §B.3 · p. 22, Table 5 · p. 22
  • CausalAlign improves all seven text-to-video ego-motion metrics across motion magnitude, geometric consistency, and temporal dynamics.The evaluation uses shared text prompts and geometric estimators.Source: §B.3 · p. 22, Table 6 · p. 22

B.4 Camera-Trajectory-Conditioned Generation

  • CausalAlign achieves the best values on six of seven trajectory metrics under explicit camera conditioning, while FantasyWorld has the lowest RTE.All methods receive the same first frame, ground-truth camera trajectory, and text prompt.Source: §B.4 · p. 22, Table 7 · p. 23

B.5 More Visualization For Mobile Manipulation Rollouts

  • Figure 6 shows two successful RoboCasa365 rollouts: placing a bowl in a refrigerator and moving a bread basket to a counter.Source: §B.5 · p. 22, Figure 6 · p. 23

C.1 Structured Delta-Space Representation

The structured delta-token representation separates motion, geometric, and visual changes between frames. Its regions specialize by task, and the full representation outperforms several readout controls while a high-dimensional spatial-difference control improves three listed metrics.

C.1 Structured Delta-Space Representation

  • The token separates relative camera motion, camera-local geometric changes, and RGB changes; pose, transition, and temporal-prediction losses supervise these structured targets jointly.The three regions are target channels, not three additional losses.Source: §C.1 · p. 23
  • The complete delta token scores SSIM 0.30283 and LPIPS 0.08676, versus 0.29397 and 0.08924 when copying the source image.Source: §C.1 · p. 24
  • Delta regions lead all 12 paired metric comparisons against matched-width unpartitioned encoders, and each region performs best on its designated task at a common width of 24.The specialization evaluation uses probes on 256 scenes across three seeds.Source: §C.1 · p. 24, Table 8 · p. 24
  • Delta leads all six readout metrics over frozen teacher features across three extraction controls, but a 133,120D full-spatial-grid difference improves on delta for AbsRel 0.2109, SSIM 0.3533, and LPIPS 0.08575.Source: §C.1 · p. 24

C.2 Update Placement and Temporal Modulation

Matched adapter-placement experiments identify the diagnosed layer set as the strongest placement, while temporal-modulation comparisons do not establish a clear separation because their confidence intervals include zero.

C.2 Update Placement and Temporal Modulation

  • Placement comparisons use independently trained, norm-matched LoRA updates and a fixed-source protocol with 32 scenes, one seed, and ten inference steps.The composite combines trajectory, speed-profile, and depth metrics; confidence intervals use 50,000 scene-bootstrap resamples.Source: §C.2 · p. 24
  • The adapter at layers 10, 12, 14, and 20 gives the largest gain over the native model, 3.109%, and gains 2.224% over a norm-matched random update.The matched comparison isolates target-layer placement.Source: §C.2 · p. 24, Table 9 · p. 25
  • The ensemble has the largest mean gain over the static temporal update, but each temporal-modulation interval includes zero.Source: §C.2 · p. 24, Table 9 · p. 25

C.3 Auxiliary Objectives and Update Strength

The auxiliary-objective experiments show a trade-off between trajectory gains and geometry or temporal preservation, while additional 5B configuration tables identify the highest reported mean gains among their comparisons.

C.3 Auxiliary Objectives and Update Strength

  • Relative-pose supervision gives the largest trajectory gain, 3.525%, while combining it with motion-dynamics supervision reduces that gain to 1.299% but improves geometric and temporal preservation.The three objective runs share a parent checkpoint and endpoint.Source: §C.3 · p. 25, Table 10 · p. 25
  • Reducing the later-layer update scale from 0.50 to 0.25 raises trajectory gain from 1.750% to 2.209% and geometry gain from −0.506% to 0.761%, while temporal and visual gains fall.The comparisons show that stronger or more numerous constraints need not improve all metric groups together.Source: §C.3 · p. 25

C.4 Additional 5B Configurations

  • Tables 11 and 12 cover 27 configurations grouped by update design and training objective, reporting trajectory, geometry, temporal, and visual improvements with means and 95% confidence intervals.Source: §C.4 · p. 26
  • Scaling the layer-22 update to 0.25 yields the highest mean gain among Table 11 configurations, 0.769%.Source: Table 11 · p. 26
  • Relative-pose alignment at step 72 yields the highest mean gain among Table 12 configurations, 0.982%.Source: Table 12 · p. 26

Related papers

Loading 2610.29915v1…