PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling

Haoyu Wu, Diankun Wu, Tianyu He, Junliang Guo, Yang Ye, Yueqi Duan, Jiang Bian

arXiv:2507.07982v2cs.CVcs.AI

TL;DR

Video diffusion models trained on raw videos may fail to learn meaningful geometry, even though videos depict a dynamic 3D world. Geometry Forcing aligns their intermediate representations with geometric foundation-model features through angular and scale objectives. Across camera-conditioned and action-conditioned generation, the method improves visual quality and 3D consistency, while its performance at larger scales and in highly dynamic environments remains unresolved.

  • Problem

    Video diffusion models often model 2D pixel distributions without capturing meaningful geometric structure underlying dynamic 3D scenes.

  • Method

    Geometry Forcing aligns video-diffusion intermediate features with geometric foundation-model representations using Angular Alignment and Scale Alignment.

  • Results

    GF significantly improves visual quality and 3D consistency across camera-conditioned and action-conditioned video-generation benchmarks, yielding lower FVD scores and more stable scene dynamics.

  • Takeaways & Limitations

    Explicit geometry-aware alignment can enhance the geometric consistency and visual quality of autoregressive video diffusion models across the evaluated settings.

  • Takeaways & Limitations

    GF has not yet been evaluated with larger models or more extensive video datasets, and reliance on VGGT trained mainly on static scenes constrains performance in dynamic environments.

Abstract

from arXiv · show

Videos inherently represent 2D projections of a dynamic 3D world. However, our analysis suggests that video diffusion models trained solely on raw video data often fail to capture meaningful geometric-aware structure in their learned representations. To bridge the gap between video diffusion models and the underlying 3D nature of the physical world, we propose Geometry Forcing, a simple yet effective method that encourages video diffusion models to internalize 3D representations. Our key insight is to guide the model's intermediate representations toward geometry-aware structure by aligning them with features from a geometric foundation model. To this end, we introduce two complementary alignment objectives: Angular Alignment, which enforces directional consistency via cosine similarity, and Scale Alignment, which preserves scale-related information by regressing geometric features from normalized diffusion representations. We evaluate Geometry Forcing on both camera-view conditioned and action-conditioned video generation tasks. Experimental results demonstrate that our method substantially improves visual quality and 3D consistency over the baseline methods. Project page: https://GeometryForcing.github.io.

1 INTRODUCTION

Video diffusion models often learn pixel distributions without meaningful geometric structure, despite videos depicting a dynamic 3D world. Geometry Forcing addresses this gap by aligning intermediate diffusion representations with geometry-aware features, improving visual and 3D consistency.

  • Motivation: Video generation models typically focus on pixel distributions, which can undermine geometric coherence and long-term consistency in videos.These difficulties are especially pronounced in autoregressive settings, where small errors may accumulate over time.
  • Motivation: Explicitly modeling dynamic 3D structure can help, but tractable representations and additional annotation requirements limit scalability.Prior work includes jointly modeling RGB and point-map information.
  • Motivation: Raw-video training alone fails to produce meaningful geometric representations when intermediate video-diffusion features are mapped to depth.The authors investigate this question using a DPT head applied to a pretrained video diffusion model.
  • Method: Geometry Forcing aligns diffusion features with pretrained geometric representations using Angular Alignment and Scale Alignment.Angular Alignment uses cosine similarity for directional consistency, while Scale Alignment regresses geometric features from normalized diffusion features to preserve scale information.
  • Results: GF improves geometric consistency and visual quality on camera-view-conditioned RealEstate10K and action-conditioned Minecraft video generation.On RealEstate10K, GF reduces FVD from 364 to 243; explicit geometry reconstruction may also support structured memory for long-term world modeling.

2 RELATED WORK

World-modeling research spans interactive video generation, explicit dynamic 4D modeling, and methods for maintaining consistency across long sequences. Recent 3D foundation models provide geometric representations that can be incorporated into video diffusion systems.

  • Interactive Video Generation: Interactive video generation extends text-to-video synthesis by conditioning generated environments on interactive control signals.This line of work positions video generation as an approach to world modeling.
  • Interactive 4D Generation: Interactive 4D simulators explicitly model dynamic 3D structures, progressing from dynamic objects toward complex dynamic scenes.Some recent methods also integrate video priors to improve realism and temporal coherence.
  • Consistent World Modeling: Long-sequence consistency methods use frame-level context mechanisms or 3D information such as memory banks indexed by field-of-view overlap.These approaches target the challenge of preserving consistency over long video sequences.
  • 3D Foundation Models: 3D foundation models can directly predict camera poses, depth maps, and dense point clouds from visual inputs.They use end-to-end frameworks with fast and robust inference.
  • 3D Foundation Models: Their accuracy, efficiency, and robustness motivate incorporating 3D representations into video diffusion models to enhance temporal and spatial consistency.The paper connects these capabilities to downstream world-modeling goals.

3 PRELIMINARIES

The method builds on autoregressive video diffusion with flow matching and a Transformer backbone, alongside VGGT features as geometric representations. Training independently corrupts video frames and learns a target velocity field, while inference uses probability-flow ODE sampling and autoregressive conditioning.

  • Overview: The approach combines autoregressive video diffusion models with a 3D foundation model to guide geometric learning during training.The section introduces both components as the basis for Geometry Forcing.
  • Autoregressive Video Diffusion Models: Flow Matching trains a Transformer-based video model by assigning each frame an independent timestep and corrupting frames through interpolation with Gaussian noise.The video is represented as a sequence of frames, and each frame receives its own corruption time.
  • Autoregressive Video Diffusion Models: The training target is a velocity field defined by the difference between noise and clean input, optimized with a Flow Matching loss.A neural network vθ is trained to minimize this loss.
  • Autoregressive Video Diffusion Models: Inference follows a probability-flow ODE and uses an Euler solver to sample data from noise.Autoregressive generation initializes clean context and predicts subsequent frames sequentially conditioned on previously generated frames.
  • Visual Geometry Grounded Transformer: VGGT is a feed-forward model that predicts camera parameters, point maps, and depth maps, using Transformer features as geometric representations.Alternating frame-wise and global self-attention integrates local and global information before task-specific prediction heads.

4 GEOMETRY FORCING

Geometry Forcing addresses the limited geometric structure learned by video diffusion models from raw video by aligning intermediate features with geometry-aware representations. Its angular and scale objectives improve geometric consistency while preserving the video-generation framework and enabling explicit geometry reconstruction during inference.

  • Motivation: Videos are 2D projections of dynamic 3D worlds, but video diffusion models trained on raw video may encode limited dynamic 3D structure.A frozen-model depth probe found that raw-video features did not produce meaningful geometric representations.
  • Challenge: Heavy reliance on annotated 3D data constrains scalability and generalization on large, diverse real-world video datasets.Jointly modeling RGB and geometry is a straightforward alternative, but limited annotated 3D data makes it challenging.
  • Method Overview: Geometry Forcing aligns video-diffusion features with geometric representations from a pretrained foundation model to encourage internalized geometric information.The approach is inspired by representation alignment methods and adds regularization objectives during training.
  • Geometric Representation Alignment: Angular Alignment uses cosine similarity to enforce directional correspondence between diffusion hidden states and target geometric features.A lightweight projector maps diffusion latents to the target shape, and alignment is applied independently at frame and patch levels.
  • Geometric Representation Alignment: Scale Alignment preserves geometric feature scale by predicting target features from normalized diffusion representations, avoiding instability associated with direct magnitude regression.The combined decomposition captures directional and scale attributes while stabilizing training.
  • 3D-Aware Autoregressive Video Diffusion Models: The overall training objective combines the base diffusion loss with weighted Angular Alignment and Scale Alignment losses.This objective is integrated into an autoregressive video diffusion framework.
  • Results and Extensions: Geometry Forcing improves RealEstate10K generation across short-term and long-term settings while enabling explicit 3D geometry reconstruction during inference.The table compares 16-frame and 256-frame generation; reconstructed geometry may support structured memory for long-term world modeling.
  • Discussion: Geometry Forcing is orthogonal to Teacher Forcing, Diffusion Forcing, and Self Forcing because it supervises representation structure rather than autoregressive exposure bias.Its training signal aligns intermediate video-diffusion representations with geometry-aware signals from a 3D foundation model.

5 EXPERIMENTS

Geometry Forcing is evaluated across camera-view-conditioned and action-conditioned video generation, with results showing improved visual quality, temporal stability, and geometric consistency. Ablations attribute these gains to geometric targets, complementary alignment losses, internal feature alignment, and mid-level alignment depth.

  • 5.1 MAIN RESULTS: GF consistently outperforms baselines across FVD, LPIPS, SSIM, and PSNR for both short-term and long-term RealEstate10K generation.The evaluation covers 16-frame and 256-frame settings.
  • 5.1 MAIN RESULTS: GF achieves lower FVD than the baseline for action-conditioned Minecraft generation despite the real-world-to-Minecraft distribution gap.This supports generalization to an out-of-domain environment.
  • 5.2 QUALITATIVE RESULTS: Under full 360° camera rotation, GF reconstructs the initial frame and produces realistic intermediate views, unlike baselines that lose temporal and scene consistency.The comparison uses a single input frame with per-frame camera poses.
  • 5.3 ABLATION STUDIES: VGGT alignment outperforms DINOv2 alignment on short-term and long-term generation, while combining geometric and semantic features provides additional gains.VGGT primarily improves 3D consistency, whereas DINOv2 improves visual quality.
  • 5.3 ABLATION STUDIES: Angular Alignment combined with Scale Alignment performs best, indicating that aligning directional and scale-related information is more effective than either alone or naive MSE.The ablation compares Angular Alignment, the combined objective, and MSE between VGGT and diffusion features.
  • 5.3 ABLATION STUDIES: Internal alignment through GF outperforms rendered-image conditioning, and alignment at layer 3 yields the best FVD-256 while preserving FVD-16 performance.These results favor internal geometric supervision and mid-level feature alignment.

6 CONCLUSION

Geometry Forcing aligns video diffusion representations with geometry-aware features to improve geometric consistency, visual quality, and scene dynamics. The paper identifies scaling to larger models and datasets as an important unresolved direction.

  • Geometry Forcing aligns internal video diffusion representations with geometry-aware features to improve geometric consistency.The framework uses Angular Alignment and Scale Alignment to guide latent features toward 3D-aware representations.
  • Empirical results on camera-conditioned and action-conditioned video generation show improved visual quality, 3D consistency, lower FVD scores, and more stable scene dynamics.
  • GF’s full potential remains unexplored under large-scale training with larger models and more extensive video datasets.The authors identify scale as the work’s primary limitation and suggest that broader training may amplify its benefits.
  • Future work includes scaling GF to larger datasets for 3D-consistent world simulators and using 3D representations as memory for long video generation.

B LIMITATIONS

This section describes the evaluation settings and implementation choices for Geometry Forcing, including datasets, alignment design, training configuration, and inference efficiency.

  • RealEstate10K provides camera poses for 10 million video frames and supports evaluation of 3D consistency and camera navigation at 256 × 256 resolution.
  • Minecraft includes action annotations for evaluating video generation in dynamic environments with camera motion.
  • Geometry Forcing aggregates VGGT features from all transformer blocks and bilinearly interpolates them to 512×512 spatial dimensions for alignment targets.
  • A Conv3D-based projector aligns latent dimensions using independent projectors for each feature layer and target representation.
  • The U-ViT video-generation backbone integrates geometric feature alignment at the third transformer block.
  • Training uses 10,000 RealEstate10K clips of 16 consecutive frames, two epochs, learning rate 8 × 10−6, global batch size 40, and combined diffusion and alignment losses.
  • Inference adds no computational overhead during sampling and is demonstrated with a 50-step DDIM sampler while remaining compatible with standard diffusion samplers.

C.4 METRICS

The paper evaluates multi-view geometric consistency through reprojection error and long-range temporal consistency through revisit error, alongside feature-based reconstruction analyses.

  • Reprojection Error: Reprojection Error measures the average Euclidean distance between projected and observed locations of co-visible 3D points across frames.DROID-SLAM reconstructs the scene and refines camera poses and per-pixel depths through differentiable Dense Bundle Adjustment.
  • Reprojection Error: RPE uses valid feature correspondences, observed pixel locations, reconstructed 3D points, and a camera projection function; lower values indicate better geometric coherence.
  • Revisit Error: Revisit Error evaluates long-range temporal consistency under a 256-frame camera trajectory produced by rotating the initial pose around the Y-axis.The evaluation compares the first and final frames using reconstruction FID.
  • 3D Reconstruction: Geometry Forcing features are passed through VGGT’s depth prediction head to obtain predicted depth maps for 3D reconstruction.
  • 3D Reconstruction: Diffusion-feature reconstruction freezes the pretrained DFoT backbone and trains a DPT head to regress depth maps supervised by VGGT targets.

D.1 ABLATION ON TEACHER MODEL

The ablations examine teacher-model compatibility, geometry supervision choices, alignment context and layers, broader generation tasks, and computational costs.

  • Teacher Model: Geometry Forcing is compatible with different 3D teacher models, including VGGT and Pi3, and achieves significant improvement with Pi3.
  • Explicit and Implicit Geometry: Geometry Forcing achieves the best performance across all metrics on RealEstate10K for long-term 256-frame video generation in the explicit-versus-implicit geometry comparison.
  • Alignment Context Length: Longer alignment context lengths provide more complete 3D information and lead to better results.The ablation varies the number of frames used to extract VGGT features during training.
  • Alignment Layers: Aligning more diffusion-model layers does not necessarily improve performance; the reported comparison evaluates the middle layer against the last three layers.
  • Text-Conditioned Generation: Geometry Forcing improves aesthetic quality, imaging quality, and motion smoothness when extended to text-conditioned video generation.
  • Computational Cost: VGGT feature alignment adds 52.5% execution time and 60.4% total FLOPs per training step but accelerates convergence and reduces overall training duration.
  • Computational Cost: VGGT feature-extraction time rises from 0.1s to 0.8s as input views increase from 1 to 12.

E.2 ANALYSIS OF GEOMETRIC AND SEMANTIC REPRESENTATIONS

Geometric and semantic representations overlap but contribute differently: VGGT alignment primarily improves 3D consistency, while DINOv2 alignment improves visual quality. Geometry Forcing also slows FVD growth in long-term generation, yet reflective materials remain difficult.

  • VGGT alignment primarily enhances 3D consistency, whereas DINOv2 alignment improves visual quality.Combining both representations achieves superior performance compared with either individual alignment.
  • Semantic alignment enhances object realism and visual detail, while geometric alignment preserves structural consistency and shape coherence.
  • Geometry Forcing slows FVD growth in long-term generation by internalizing stable 3D scene geometry.The stated mechanism is reduced error accumulation as more frames are regenerated.
  • The method still struggles with reflective materials, including a glass table that intermittently disappears and reappears across frames.

F SUPPLEMENTARY VISUALIZATIONS

Supplementary visualizations compare Geometry Forcing with strong baselines and isolate the role of scale alignment. GF maintains long-horizon visual and geometric consistency, while scale alignment improves camera-following stability.

  • On RealEstate10K, GF is compared with DFoT, REPA, and VideoREPA using the same first frame and per-frame camera trajectory.
  • GF preserves object shapes and scene layouts over long horizons, while baselines often show drift, shape distortion, or abrupt transitions.The comparison covers both scenes inside and outside the input context.
  • Angular alignment alone maintains basic geometric coherence, but without scale supervision camera motion becomes unstable and object scaling unnatural.
  • Adding scale alignment produces smoother camera-following behavior and more stable, realistic videos.
Loading 2507.07982v2…