PaperlayerView exact revision on arXiv

Source-linked AI summary

VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward

Zhaochong An, Orest Kupyn, Théo Uscidda, Andrea Colaco, Karan Ahuja, Serge Belongie, Mar Gonzalez-Franco, Marta Tintore Gazulla

arXiv:2603.26599v2cs.CV

TL;DR

Video diffusion models often lack geometric consistency, while prior alignment methods can depend on costly RGB decoding and static-scene assumptions. VGGRPO connects diffusion latents to geometry foundation models and applies latent-space GRPO with camera and geometry rewards, reporting consistent improvements across static and dynamic benchmarks.

  • Problem

    Video diffusion models achieve strong visual fidelity but often show geometric drift, unstable camera motion, and inconsistent structure, while existing rewards can be costly and limited to static scenes.

  • Method

    VGGRPO uses a Latent Geometry Model to predict 4D geometry from video latents and optimizes latent-space GRPO with camera-motion smoothness and geometry-reprojection rewards.

  • Results

    VGGRPO yields consistent gains on static- and dynamic-scene benchmarks across camera motion smoothness, geometric consistency, and overall video quality.

  • Takeaways & Limitations

    Latent-space geometry-guided reinforcement provides an efficient and flexible approach to world-consistent video post-training while supporting dynamic scenes.

  • Takeaways & Limitations

    Existing geometric reward formulations and correspondence pipelines remain limited in their ability to extend to complex dynamic videos.

Abstract

from arXiv · show

Large-scale video diffusion models achieve impressive visual quality, yet often fail to preserve geometric consistency. Prior approaches improve consistency either by augmenting the generator with additional modules or applying geometry-aware alignment. However, architectural modifications can compromise the generalization of internet-scale pretrained models, while existing alignment methods are limited to static scenes and rely on RGB-space rewards that require repeated VAE decoding, incurring substantial compute overhead and failing to generalize to highly dynamic real-world scenes. To preserve the pretrained capacity while improving geometric consistency, we propose VGGRPO (Visual Geometry GRPO), a latent geometry-guided framework for geometry-aware video post-training. VGGRPO introduces a Latent Geometry Model (LGM) that stitches video diffusion latents to geometry foundation models, enabling direct decoding of scene geometry from the latent space. By constructing LGM from a geometry model with 4D reconstruction capability, VGGRPO naturally extends to dynamic scenes, overcoming the static-scene limitations of prior methods. Building on this, we perform latent-space Group Relative Policy Optimization with two complementary rewards: a camera motion smoothness reward that penalizes jittery trajectories, and a geometry reprojection consistency reward that enforces cross-view geometric coherence. Experiments on both static and dynamic benchmarks show that VGGRPO improves camera stability, geometry consistency, and overall quality while eliminating costly VAE decoding, making latent-space geometry-guided reinforcement an efficient and flexible approach to world-consistent video generation.

1. Introduction

VGGRPO addresses geometric inconsistency in video diffusion by connecting diffusion latents to geometry priors and optimizing latent-space rewards for stable camera motion and coherent scene structure. It supports both static and dynamic videos while avoiding repeated RGB decoding.

  • Motivation: Video diffusion models often exhibit geometric drift, unstable camera trajectories, and inconsistent scene structure despite strong visual fidelity and generalization.These weaknesses matter for applications such as embodied AI and physics-aware simulation.
  • Limitations of prior work: Existing methods either modify the generator, potentially increasing complexity and constraining pretrained generalization, or use pixel-space alignment with costly VAE decoding.RGB-based rewards are also sensitive to decoding noise and low-level pixel variation.
  • Method: VGGRPO connects video diffusion latents to geometry foundation models through a lightweight Latent Geometry Model that predicts scene geometry directly from latent representations.Using a foundation model with dynamic 4D reconstruction capability extends the approach beyond static-scene assumptions.
  • Method: VGGRPO applies latent-space Group Relative Policy Optimization with camera-motion smoothness and geometry-reprojection consistency rewards.The rewards target stable camera trajectories and coherent 3D structure across viewpoints without repeated VAE decoding.
  • Results: Experiments report consistent gains across static- and dynamic-scene benchmarks in camera smoothness, geometric consistency, and overall video quality.The method is presented as a practical alternative to RGB-based alignment that incorporates geometry priors and supports dynamic scenes.

2. Related Work

Related work improves video geometric consistency through geometry-aware generation or diffusion alignment, but the paper positions VGGRPO as latent-space geometry alignment for dynamic scenes without RGB decoding.

  • Geometrically consistent video generation: Geometrically consistent video generation is motivated by geometric drift in large-scale diffusion models and the need for stable camera motion and coherent scene geometry.These properties are relevant to downstream applications requiring world-consistent video.
  • Diffusion model alignment: Diffusion alignment methods address task-specific objectives that broad data-distribution training may not directly capture.Prior alignment work includes classifier-filtered fine-tuning and policy-optimization approaches.
  • Diffusion model alignment: VGGRPO performs GRPO-based geometry alignment directly in latent space through a Latent Geometry Model, removing the RGB decoding bottleneck for dynamic scenes.Its two rewards target camera-motion smoothness and geometry-reprojection consistency.

3. Methodology

VGGRPO combines a latent geometry model with latent-space GRPO to improve camera stability and cross-view geometric coherence without repeated RGB decoding. Its two complementary rewards target smooth camera motion and consistent scene geometry, while dynamic-scene handling uses geometry predictions that include scene flow.

  • Framework: VGGRPO couples a Latent Geometry Model with latent-space Group Relative Policy Optimization for geometry-aware video post-training.The framework stitches diffusion latents to a geometry foundation model and performs group-based reinforcement optimization directly in latent space.
  • Latent Geometry Model: VGGRPO computes geometry predictions and rewards directly from latent representations, avoiding repeated VAE decoding during group-based updates.This design uses a differentiable latent geometry model and reduces the compute and memory overhead associated with RGB-space reward evaluation.
  • Latent Geometry Model: The Latent Geometry Model replaces the geometry model’s initial layers with a learned 3D convolutional connector that maps VAE latents into intermediate feature space.The connector and downstream layers are trained to align latent-model predictions with the original geometry model’s outputs.
  • VGGRPO Training: The camera reward penalizes translational and rotational acceleration, assigning values near 1 to smooth trajectories and decreasing values as jitter increases.Translation uses discrete camera-center velocities and accelerations; rotation uses angular velocities.
  • VGGRPO Training: The geometry reward reprojects a predicted point cloud into each view and compares rendered and predicted depths over valid pixels, averaging error over the three worst views.For dynamic scenes, predicted scene flow filters dynamic regions before constructing the stable scene representation.
  • VGGRPO Training: The training objective averages separately normalized camera-motion and geometry rewards across K sampled latent videos to form group-relative advantages.GRPO then uses these advantages with importance ratios and a clipped surrogate objective for policy updates.

4. Experiments

VGGRPO improves geometric consistency, camera stability, and video quality across static and dynamic benchmarks, while latent-space rewards reduce computation and support additional guidance studies.

  • Main Results: VGGRPO consistently outperforms baselines on geometry-related metrics across static and dynamic benchmarks, remaining robust under complex non-rigid motion.The evaluation includes 190 static-scene and 200 dynamic-scene captions, with dynamic cases selected for significant non-rigid motion.
  • Main Results: Qualitatively, VGGRPO produces more coherent scene structure and smoother camera motion than baselines in both static and dynamic settings.Baselines show geometric drift, temporal flicker, unstable motion, and weaker robustness on dynamic examples.
  • Additional Studies: The combined motion and reprojection rewards yield the best results: motion optimization stabilizes trajectories, while geometry optimization reduces reprojection artifacts.The ablation identifies the two rewards as complementary rather than interchangeable.
  • Additional Studies: Test-time latent reward guidance improves geometric consistency with modest runtime overhead and no additional training.The guidance is applied once every 20 denoising steps out of 50 total.
  • Additional Studies: VGGRPO preserves general video-generation quality on standard VBench captions, although Dynamic Degree is slightly lower than the baseline.The reported gains are achieved without specifically training for general video quality.
  • Additional Studies: Latent-space reward computation reduces peak memory from 76.80 GB to 68.57 GB and runtime from 54.73 s to 41.33 s.This corresponds to a 13.40 s reduction and a 24.5% runtime reduction for the reward computation step at batch size 4.

5. Conclusion

VGGRPO aligns pretrained video diffusion models toward 4D world-consistent generation through latent-space geometry rewards. Across static and dynamic benchmarks, it improves geometric consistency and camera smoothness while preserving visual fidelity and reducing reward-computation cost.

  • Conclusion: VGGRPO connects video diffusion latents to geometry foundation models through a latent geometry model for 4D scene-geometry reward computation.The framework operates directly in latent space and avoids repeated VAE decoding during group-based policy updates.
  • Conclusion: Complementary camera-motion and geometry-reprojection rewards jointly promote stable trajectories and cross-view coherent scene structure.The formulation applies to both static- and dynamic-scene videos.
  • Conclusion: Across static and dynamic benchmarks, VGGRPO substantially improves geometric consistency and camera smoothness while preserving strong visual fidelity.The conclusion characterizes latent-space 4D geometry rewards as an efficient and scalable alignment approach.

A. Flow-Based Group Relative Policy Optimization

Flow-based GRPO casts diffusion denoising as a sequential decision process and uses group-relative rewards for policy optimization. An ODE-to-SDE conversion supplies stochastic exploration and tractable per-step probabilities while preserving the model’s marginals.

  • MDP formulation: DDPO models iterative denoising as a multi-step MDP whose final clean sample receives an arbitrary downstream reward.The state includes the prompt, timestep, and noisy sample; actions are denoised predictions, and transitions are deterministic.
  • Policy gradients: Exact Gaussian log-probabilities for individual denoising steps enable standard policy-gradient estimators throughout the trajectory.The formulation avoids the intractable marginal likelihood of the full denoising chain.
  • Group-relative optimization: GRPO eliminates PPO’s value-function critic by normalizing rewards within each prompt-specific sample group.Group normalization centers advantages around zero and scales them to unit variance, while each sample’s advantage remains constant across denoising steps.
  • Stable updates: The clipped surrogate objective limits policy-ratio changes, while KL regularization keeps the updated policy near the pretrained reference policy.Positive-advantage trajectories are clipped from above and negative-advantage trajectories from below.
  • ODE-to-SDE conversion: Converting rectified-flow ODEs to SDEs injects controlled noise, preserving timestep marginals while enabling exploration and tractable Gaussian transition probabilities.Setting σ_t=0 recovers deterministic dynamics; larger values increase exploration but produce noisier trajectories.
  • KL regularization: Because policy and reference transitions share isotropic Gaussian variance, their per-step KL divergence reduces to a scaled squared velocity difference.This closed form avoids sampling-based KL estimation and supplies a smooth regularization signal at every denoising step.

A.6. Denoising Reduction

Denoising reduction accelerates online reinforcement-learning data collection for flow models by training with fewer denoising steps, then restoring the full schedule at inference.

  • Motivation: Online reinforcement learning is bottlenecked by sequential denoising, because each collected flow-model sample requires T steps.This cost is especially substantial for large video models.
  • Reduced-step training: Training uses T_train ≪ T_infer, such as 10 versus 40 denoising steps, even though reduced-step samples have lower visual quality.The shorter schedule still produces rewards that distinguish better from worse samples.
  • Inference: The full denoising schedule is restored at inference, and improvements learned from reduced-step training transfer to high-quality generation.The reported speedup is proportional to T_infer/T_train without degrading final test-time sample quality.

B.1. More Training Details

The implementation trains the latent geometry model and VGGRPO with AdamW, LoRA adaptation, gradient clipping, and scheduled or fixed optimization settings.

  • Latent geometry model: The latent geometry model is trained for 20 epochs with AdamW at learning rate 2 × 10^-4 and cosine decay with 100-step linear warmup.It uses no weight decay, maximum gradient norm 1.0, and a 3D convolutional stitching connector.
  • VGGRPO training: VGGRPO uses AdamW with learning rate 1×10^-4 and weight decay 1 × 10^-4, requiring approximately 1536 GPU hours.Unless otherwise stated, LoRA uses rank r=32 and scaling factor α=64, with gradient clipping at norm 1.0.
  • Policy settings: Policy optimization sets the clipping range ε to 1 × 10^-3 and the KL regularization weight β to 0.004.These values are specified alongside the default VGGRPO training configuration.

B.2. More Details of Test-Time Reward Guidance

Test-time reward guidance evaluates geometry directly from noisy diffusion latents and back-propagates a weighted reward gradient to modify sampling. Sparse application reduces added inference cost while retaining geometric improvements.

  • Reward computation: The video diffusion velocity and latent geometry model jointly provide the geometric quantities used to compute the latent-space reward.A noisy latent z_t is processed under text condition p, then passed through the latent geometry model.
  • Reward design: The reward combines camera-motion smoothness and geometry-reprojection consistency with weights λ_motion and λ_geo.The two terms target stable camera trajectories and cross-view geometric coherence.
  • Gradient guidance: Reward gradients are back-propagated through the latent geometry model with respect to the current latent and used to modify the sampling trajectory without RGB decoding.The guidance strength is controlled by s_reward.
  • Sparse guidance: The implementation applies classifier-free guidance and computes reward guidance only at selected denoising steps.The default uses 50 total steps and applies reward guidance once every 20 steps, keeping runtime overhead modest while improving geometric consistency.
  • Algorithm: The pseudo-code decodes geometry directly from latent space, computes both geometry-aware rewards, and differentiates the combined reward with respect to the noisy latents.This procedure is applied conditionally at the configured guidance interval.

B.3. More Quantitative Comparisons

VGGRPO's latent geometry model remains robust under latent perturbations where RGB-based geometry evaluation degrades, while generalization experiments show improved world consistency without sacrificing perceptual quality.

  • Latent Geometry Model: RGB-based geometry evaluation degrades rapidly even when decoded frames remain visually almost unchanged, revealing a real-to-generated content distribution gap.VGGRPO instead evaluates perturbed latents directly, avoiding RGB decoding.
  • Latent Geometry Model: The latent geometry model remains stable across latent noise levels, whereas the RGB-based geometry model degrades substantially as perturbations increase.The comparison uses 50 RealEstate10K test sequences and evaluates Racc@5, AUC@5, and Tacc@5.
  • Latent Geometry Model: VGGRPO's latent rewards remain stable under noise and distribution shifts, providing more reliable geometry estimates than RGB-based rewards.This robustness supports latent-space reward evaluation during post-training.
  • Generalization Performance: Prior post-training baselines provide limited general video-quality gains and can degrade Imaging Quality relative to the base model.VGGRPO avoids this degradation while transferring effectively to the standard VBench caption set.
  • Generalization Performance: VGGRPO achieves the strongest overall performance across most standard VBench metrics while improving world consistency over the base model and prior post-training methods.The results indicate that geometry-aware post-training preserves strong perceptual quality and robust generalization.

B.4. More Qualitative Results

Qualitative comparisons across indoor, outdoor, static, and dynamic scenarios show that VGGRPO produces smoother camera motion and more stable scene geometry than baseline methods.

  • Qualitative Results: Across diverse scenarios, baselines exhibit camera shake, motion blur, geometric distortion, and broken temporal continuity, whereas VGGRPO produces smoother trajectories and stable geometry.The examples include static scenes and dynamic scenes with complex object motion.
  • Qualitative Results: VGGRPO preserves coherent scene structure and temporal continuity in challenging dynamic examples where baselines suffer severe blur and geometric deformation.These results extend the reported benefits beyond static settings.
Loading 2603.26599v2…