VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

Hongyang Du; Junjie Ye; Xiaoyan Cong; Runhao Li; Jingcheng Ni; Aman Agarwal; Zeqi Zhou; Zekun Li; Randall Balestriero; Yue Wang

Source-linked AI summary

VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang

arXiv:2601.23286v4Jan 30, 2026cs.CVcs.AIcs.LG

TL;DR

Video diffusion models struggle to preserve 3D structural and temporal consistency despite strong visual quality. VideoGPA uses geometry-foundation-model-derived preference signals with DPO to align pretrained models, improving geometric coherence and human-aligned quality across video-generation settings. The approach relies on lightweight post-training rather than architectural changes or human annotations.

Problem
Current video diffusion models remain deficient in 3D consistency, failing to maintain structural consistency and temporal stability despite billion-scale pretraining.
Method
VideoGPA derives self-supervised geometric preference pairs from geometry foundation models and uses them to guide DPO-based post-training without human annotations.
Results
VideoGPA consistently improves geometric consistency and perceptual quality over baselines across image-to-video and text-to-video settings, while the conclusion reports substantially improved 3D coherence and temporal stability.
Takeaways & Limitations
The results suggest that geometric failures can be mitigated through lightweight post-training alignment, without degrading general-purpose generation quality.
Takeaways & Limitations
Sequential supervised fine-tuning can constrain the policy and hinder subsequent DPO from discovering optimal 3D-consistent representations.

Abstract

from arXiv · show

While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift. We hypothesize that these failures arise because standard denoising objectives lack explicit incentives for geometric coherence. To address this, we introduce VideoGPA (Video Geometric Preference Alignment), a data-efficient self-supervised framework that leverages a geometry foundation model to automatically derive dense preference signals that guide VDMs via Direct Preference Optimization (DPO). This approach effectively steers the generative distribution toward inherent 3D consistency without requiring human annotations. VideoGPA significantly enhances temporal stability, geometric plausibility, and motion coherence using minimal preference pairs, consistently outperforming state-of-the-art baselines in extensive experiments.

1. Introduction

Video diffusion models show strong visual fidelity but still fail to preserve 3D structure, motivating VideoGPA, which distills geometric priors into lightweight preference alignment.

Video diffusion models support downstream 3D applications, but faithful 3D understanding remains fundamental to those uses.
Pretrained VDMs exhibit structural inconsistency and temporal instability, including object deformation, spatial drifting, and geometry collapse.
Standard denoising objectives favor pixel-level statistical matching without geometric regularization, allowing plausible textures without effective 3D consistency.
VideoGPA distills reconstruction-based 3D knowledge from geometry foundation models into VDMs without retraining from scratch or human annotations.
VideoGPA uses GFM-rendered reconstructions to score geometric consistency, form preference pairs, and guide pretrained VDMs with DPO.The framework uses approximately 2,500 preference pairs and LoRA fine-tuning on approximately 1% of model parameters.

2. Related Works

The paper builds on diffusion-model alignment and geometric foundation models by using reprojection-based consistency to create self-supervised preference pairs for video post-training.

Video diffusion models: Diffusion-transformer video models achieve high-fidelity generation but remain optimized primarily for pixel-level denoising rather than geometric coherence.
Alignment methods: Post-training methods include supervised fine-tuning and reinforcement-learning approaches, with supervised methods often limited in generalization.
Geometric foundation models: Geometric foundation models recover dense structure, camera poses, and pointmaps from sparse or multi-view observations, providing geometric priors for video alignment.
DPO for video diffusion: VideoGPA adapts Diffusion-DPO to v-prediction video diffusion and updates a LoRA-parameterized model using shared noise and timestep samples for each preference pair.
Preference modeling: A geometric foundation model predicts depth and camera poses, reconstructs frames through reprojection, and uses reconstruction error as a self-supervised consistency signal.Lower reconstruction error indicates stronger cross-view geometric consistency, while higher error indicates violations of 3D coherence.
Preference modeling: Multiple generations sharing semantic content but differing in geometry are sampled to expose geometric differences and construct preference pairs.
Evaluation: Models trained with structured motion prompts retain natural-language handling during evaluation while showing improved 3D geometric consistency.

4. Experiments

Experiments evaluate VideoGPA across I2V and T2V generation using geometric, perceptual, and human-aligned metrics, alongside qualitative comparisons and preference studies. VideoGPA consistently improves geometric consistency and human-aligned quality over the evaluated baselines.

Evaluation Setup: The evaluation measures 3D reconstruction error, 3D consistency, and human-aligned video quality using reconstruction, geometry, VideoReward, and preference metrics.Static videos are filtered during training and evaluation so reported metrics reflect motion content.
Image-to-Video (I2V): VideoGPA improves I2V geometry, raising MVCS from 0.945 to 0.950, reducing 3DCS from 0.485 to 0.483, and lowering Epipolar error from 0.585 to 0.539.These gains accompany a 76.0% Overall VideoReward win rate, versus 66.0% for Epipolar-DPO and 35.0% for SFT.
Text-to-Video (T2V): VideoGPA improves T2V reconstruction and consistency, achieving SSIM 0.803, LPIPS 0.411, MVCS 0.953, 3DCS 0.422, and epipolar error 0.548.Its Overall VideoReward win rate is 60.33%, compared with 48.67% for Epipolar-DPO and 15.33% for SFT.
Comparison with GeoVideo: Against GeoVideo, VideoGPA uses 1,500 training steps rather than approximately 10,000 and achieves stronger overall geometric consistency and human-aligned performance.VideoGPA records Epipolar 0.567 versus 0.875, MVCS 0.982 versus 0.819, and OVL 57.64% versus 18.06%.
Human Preference Study: In a blind I2V preference study, VideoGPA receives 53.5% of total wins, versus 22.4% for Epipolar-DPO.Twenty-five participants evaluated randomized groups of four videos using the same prompt and seed.
Qualitative Comparison: Qualitative comparisons show improved structural and geometric consistency, texture stability, challenging-lighting robustness, and object attribute consistency.The comparisons include the base model, SFT, and Epipolar-DPO.

5. Discussion

The discussion argues that scene-level geometric preference modeling is more reliable than local pairwise constraints and that geometric regularization supports motion coherence. VideoGPA’s global reprojection requirement rejects locally plausible but globally inconsistent outputs.

Scene-Level Geometry v.s. Local Constraints: Scene-level geometry requires all frames to admit one coherent 3D explanation, preventing accumulated spatial drift and penalizing collapsed or physically implausible generations.This provides a dense alignment signal for preference optimization.
Scene-Level Geometry v.s. Local Constraints: Local epipolar metrics can produce false positives under severe artifacts, whereas the scene-level metric correctly rejects corrupted sequences.Figure 5 contrasts local pairwise constraints with the global reprojection-based evaluation.
Geometry as a Regularizer for Motion Generation: VideoGPA improves motion coherence in both T2V and I2V despite explicitly targeting geometric consistency in predominantly static scenes.The discussion links this result to higher Motion Quality win rates and Figure 6.
Geometry as a Regularizer for Motion Generation: Geometric constraints act as a regularizer that stabilizes backgrounds and camera trajectories, helping the model disentangle camera movement from object motion.This interpretation connects consistent projective geometry with dynamic generation.

6. Conclusion

VideoGPA aligns pretrained video diffusion models using a scene-level 3D consistency score derived from geometry foundation models. The approach improves 3D coherence and temporal stability through lightweight post-training, while geometric reconstruction remains costly for long videos.

Contribution: VideoGPA uses scene-level 3D consistency from geometry foundation models within preference optimization to improve pretrained video generators.The method does not introduce explicit structural priors or degrade general-purpose generation quality.
Conclusion: The conclusion attributes geometric failures largely to objective misalignment rather than architectural limitations and reports mitigation through lightweight post-training.This claim is presented as the paper’s supported interpretation of the results.
Limitation: Geometric reconstruction scalability remains limited because runtime and memory costs grow with video length.The paper expects lightweight geometry foundation models to address this limitation.

A. Additional Details on Preference Data Construction

The preference-data pipeline generates multiple candidate videos, ranks them by 3D consistency, and filters pairs to preserve meaningful geometric distinctions. For I2V, scripted multi-stage camera motions and static-scene constraints encourage geometric consistency while diversifying trajectories.

Candidate generation: Approximately 3,000 conditioning inputs produce roughly 9,000 candidate videos by sampling three seeds per input.The inputs come from subsets 8k, 9k, 10k, and 11k of DL3DV-10K.
Pair construction: Candidate videos are ranked by 3D consistency, with the best and worst samples forming preference pairs within each generation group.The same multi-stage filtering procedure is applied to I2V and T2V data.
Pair filtering: Pairs are retained only when their 3D consistency scores differ by more than 0.05, ensuring a meaningful geometric preference signal.Pairs whose preferred sample still has a consistency score greater than 0.8 are removed.
I2V prompt construction: I2V prompts combine two or three translation, rotation, or complex-path primitives with temporal connectors to elicit diverse camera trajectories.A fixed static-scene prefix is added so only the camera moves.
I2V prompt construction: The static-scene constraint makes geometric consistency the primary factor distinguishing candidate samples during alignment.The resulting alignment also improves 3D consistency in dynamic scenes, beyond the static training setup.

B. Training Details and GPU Usage

Training uses a unified LoRA-based post-training protocol on 8×A100 GPUs, with compact preference datasets for VideoGPA and Epipolar-DPO and a larger supervised dataset for the SFT baseline.

Optimization protocol: Training runs on 8× NVIDIA A100 GPUs with AdamW, a 5 × 10^-6 peak learning rate, cosine decay, 500 warm-up steps, and batch size 16.LoRA uses rank r = 64 and α = 128, affecting approximately 1% of model parameters.
Dataset sizes: Each preference-alignment model variant is trained on approximately 2,500 curated preference pairs after motion-salience and geometric-margin filtering.VideoGPA and Epipolar-DPO use the same data-construction pipeline for fair comparison.
Dataset sizes: The SFT baseline uses 20,356 DL3DV-10K video-caption training pairs generated with CogVLM2-Video captions.This setup uses standard supervised fine-tuning without additional geometric signals.
Dataset sizes: Table 3 reports curated-dataset statistics and training computational costs on 8×A100 GPUs.The table summarizes dataset sizes, training steps, and wall-clock time.
Optimization protocol: CogVideoX1.5-5B uses a shortened schedule of 500 warm-up steps followed by 1,000 optimization steps.

C. Scalability and VRAM Consumption

The dense 3D consistency score becomes more expensive as the reconstructed sequence grows, motivating a 10-frame alignment choice. On 10-frame clips, its throughput is comparable to GPU-based epipolar metrics.

Frame-count scalability: Runtime and memory consumption increase monotonically with the number of reconstructed frames, while longer sequences substantially increase VRAM usage.Throughput remains competitive for short clips but declines in computational practicality for longer sequences.
Frame-count scalability: T = 10 frames is selected for alignment because it provides dense temporal coverage without significant geometric-consistency gains from longer sampling.Increasing T beyond this range incurs substantially higher computational and memory costs.
Limitations: Long-video post-training alignment remains challenging because larger temporal windows require proportionally higher memory and compute.The paper leaves more efficient geometric foundation models and scalable reconstruction strategies for future work.
Runtime comparison: 11.50 FPS versus 12.10 FPS: the dense consistency metric achieves comparable throughput to GPU-based epipolar methods on 10-frame sequences.The comparison uses a single NVIDIA RTX 6000 Ada GPU.

D. Analysis of Scene and Motion Fidelity

VideoGPA uses KL regularization to limit policy drift and evaluates whether geometric alignment preserves visual richness and motion. The reported metrics remain preserved or improve, with competitive performance reached early during training.

Alignment safeguards: KL regularization prevents VideoGPA from deviating excessively from the original policy distribution during DPO.This addresses the concern that preference fine-tuning could collapse toward simpler generations.
Evaluation metrics: Four metrics assess texture sharpness, fine-grained detail, structural complexity, and motion intensity across I2V and T2V settings.The metrics are Laplacian Variance, FFT High-Frequency Ratio, Edge Density, and Optical Flow Magnitude.
Scene and motion fidelity: 299.56 to 839.26: I2V Laplacian Variance increases substantially after VideoGPA fine-tuning.All four reported scene-complexity and motion metrics are either preserved or significantly improved.
Training-step ablation: Competitive geometric and perceptual performance appears by 1,000 training steps, with only marginal improvements at later checkpoints.Evaluation uses natural descriptive prompts that differ from the scripted camera-motion prompts used for I2V training.

E.1. Limitations of Supervised Fine-Tuning for Geometric Grounding

VideoGPA’s direct preference optimization outperforms supervised fine-tuning for geometric grounding, while alternative geometric backbones provide highly consistent preference signals and the pipeline tolerates label noise.

Sequential interference: SFT10K followed by DPO1K fails to match direct DPO training, suggesting that real-world SFT can prematurely constrain subsequent geometric alignment.The paper attributes this interference to redundant signals and an optimization barrier affecting the later DPO phase.
Optimization strategy: DPO significantly outperforms SFT on identical winner samples and direct DPO surpasses the SFT-to-DPO pipeline.Table 8 summarizes both optimization comparisons as evidence that direct preference optimization is more effective for structural constraints.
Backbone robustness: Agreement Rate measures binary winner agreement, Top-1 Consistency measures best-sample agreement, and Full Ranking Agreement measures identical ordinal rankings across backbones.These metrics compare DUSt3R and DA3 preference rankings with VGGT across generated groups.
Backbone robustness: DA3 achieves 100% consensus with VGGT across more than 3,000 groups, supporting consistent preference signals across geometric foundation models.The reported alignment covers all three hierarchical agreement measures described in the evaluation.
Supervision robustness: VideoGPA maintains strong performance under 20% flipped preference labels, with only a marginal decline in overall preference.The robustness ablation reports consistent gains across metrics despite injected supervision noise.

F. Generalization on Out-of-Distribution Datasets

VideoGPA generalizes beyond its training distribution to WebVid and Panda-70M, improving geometric and preference-based evaluations on both web-scale and dynamically moving videos.

WebVid: VideoGPA achieves the best LPIPS, 3DCS, and Epipolar error on 100 WebVid videos compared with baseline, SFT, and Epipolar-DPO.WebVid is disjoint from the training set and has a different visual distribution and motion profile.
Panda-70M: VideoGPA significantly outperforms the baseline across all VideoReward metrics on 100 dynamic Panda-70M videos.These videos contain complex object movements and camera transitions absent from the relatively static-scene training data.
Panda-70M: 64.0% Overall Preference is achieved on Panda-70M dynamic sequences.The gain supports generalization of motion coherence beyond static-scene training conditions.

G. Additional Evaluation on Wan Family

Evaluation on Wan2.2-TI2V-5B shows that VideoGPA improves geometric consistency most strongly under scripted camera-motion prompts while delivering modest gains under natural captions.

Static-scene camera motion: PSNR improves from 19.40 to 24.41, LPIPS drops from 0.480 to 0.394, and Epipolar error falls from 0.594 to 0.499 under scripted camera-motion prompts.VideoGPA also gains across 3D consistency metrics in this regime.
Static-scene camera motion: VideoGPA achieves VideoReward win rates of 52.00% VQ, 65.00% MQ, and 57.00% OVL under scripted camera-motion prompts.These human-aligned gains accompany the geometric improvements reported for Wan2.2.
Natural captions: Under natural captions, PSNR improves from 24.06 to 24.41, SSIM from 0.815 to 0.823, LPIPS from 0.412 to 0.401, MVCS from 0.941 to 0.945, and 3DCS from 0.424 to 0.411.Epipolar error increases from 0.588 to 0.611 in this regime.
Natural captions: VideoGPA achieves 47.00% VQ, 67.00% MQ, and 56.00% OVL win rates under natural captions.The reported improvements are modest because the base model is already strong in this setting.

H. Emergent Consistency in Dynamic Scenes

VideoGPA’s geometric alignment extends to dynamic scenes, improving motion coherence, physical plausibility, and temporal stability beyond static-scene consistency.

Dynamic object generation: VideoGPA preserves the pirate ride’s rigid-body integrity and maintains a vehicle’s physical plausibility and red-car color consistency after occlusion.Baseline models exhibit color drift or structural distortion in these unconstrained dynamic-object examples.
Dynamic object generation: VideoGPA improves coherence for naturally moving objects such as spinning rides and vehicles despite training on relatively static scenes.The reported gains include superior temporal stability and physical plausibility in complex dynamic scenarios.
Motion-coherent generation: Experiments in I2V and T2V show suppression of geometry collapse and semantic metamorphosis during articulated and non-rigid motion.The paper reports improvements for animal limb movements, ear morphing, object splitting, and object merging.
Physical plausibility: Distilling geometric knowledge from VGGT and DINO is hypothesized to improve physical plausibility by ranking generations toward realistic object structures.The paper contrasts this geometric signal with frame-level Epipolar and vision-language metrics.
Qualitative evaluation: VideoGPA produces more stable geometry under camera motion in image-to-video results, reducing spatial drift and texture flickering.Additional text-to-video results show better preservation of object structure and appearance over time, with reduced deformation and semantic drift.