PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoFix: Geometry Fixing for 4D Consistent Generation

Yeobin Hong, Hyojun Go, Dohun Lee, Jeongjae Lee, Sunghwan Hong, Federico Tombari, Zhaochong An, Jong Chul Ye, Konrad Schindler, Dominik Narnhofer

arXiv:2610.14293v1cs.CVcs.AIcs.LG

TL;DR

Video diffusion models can produce geometric inconsistencies that limit their use as priors for 4D generation, while existing reward approaches can trade visual detail for consistency. GeoFix aligns models with an overlap-calibrated geometric reward, and reports improvements in geometric consistency, video quality, and camera accuracy.

  • Problem

    Video diffusion outputs can contain geometric inconsistencies that limit their use as priors for 4D generation, and current rewards fall short of improving consistency without harming visual richness.

    Source: §1 · pp. 1–2

  • Method

    GeoFix uses FOCAL, which measures cross-view depth consistency in static regions and calibrates agreement against the expected score for camera overlap.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Experiments on WorldScore and VBench show that GeoFix outperforms baselines in geometric consistency and video quality, while also improving camera accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    In 4D Gaussian scene generation, GeoFix produces more stable background geometry and coherent object motion.

    Source: §1 · p. 2

    FOCAL inherits errors from its pretrained 4D reconstructor and dynamic segmentation, and requires extra reconstruction and segmentation computation, plus backpropagation for gradient guidance.

    Source: E LIMITATIONS · p. 22

Abstract

from arXiv · show

Video diffusion models increasingly serve as priors for 4D generation, but the generated videos often lack geometric consistency. Recent approaches address this through post-training by designing geometric rewards, yet this can degrade texture and fine detail, requiring auxiliary appearance rewards and careful reward balancing. To address this challenge, we propose GeoFix, a unified framework that improves geometric consistency and visual quality with a single geometric reward, applicable to both training and inference-time reward alignment. Our reward measures cross-view depth consistency in static scene regions, targeting structural inconsistencies without directly constraining appearance or penalizing valid object motion. Crucially, depth agreement increases with camera field-of-view overlap even in real videos and 4D datasets, allowing a smaller camera movement to be mistaken for better geometry. We account for this effect by calibrating the reward against the overlap-dependent trend. GeoFix achieves a 39% improvement in WorldScore 3D consistency and a 4% increase in VBench Quality Score.

1 INTRODUCTION

GeoFix addresses whether a single geometric reward can improve both geometric consistency and video quality without auxiliary rewards. It uses overlap-calibrated cross-view depth consistency and applies reward alignment during training and inference.

1 INTRODUCTION

  • Prior geometric rewards can improve consistency at the expense of visual detail, motivating the question of whether one geometric reward can improve both without auxiliary rewards.Existing approaches add rewards or preference-data filtering, which introduces reward-weight or sample-selection choices.Source: §1 · p. 2
  • GeoFix combines FOCAL, a stitched 4D reconstructor, and alignment methods to improve geometric consistency and video quality with a single geometric reward.FOCAL measures cross-view depth consistency in static regions, avoiding direct color or texture matching and penalization of valid object motion.Source: §1 · p. 2
  • FOCAL calibrates depth agreement against the expected score for camera overlap, reducing the reward for overlap alone rather than better geometry.Depth agreement is higher at larger field-of-view overlap even in clean real videos.Source: §1 · p. 2
  • A stitched pretrained 4D reconstructor computes FOCAL directly from generated video latents, bypassing RGB decoding and early reconstructor layers.The resulting model supplies depth and camera estimates and produces an explicit 4D Gaussian representation.Source: §1 · p. 2
  • Across WorldScore and VBench, GeoFix outperforms baselines in geometric consistency and video quality, while also improving camera accuracy.The framework is also demonstrated for 4D Gaussian scene generation, producing more stable background geometry and coherent object motion.Source: §1 · p. 2
  • FOCAL supports DiffusionNFT post-training and best-of-N or DPS-style inference-time guidance.Experiments report improvements in geometric consistency, video quality, camera accuracy, and 4D Gaussian scene generation.Source: §1 · p. 2

2 RELATED WORK

Prior work aligns diffusion models through post-training or inference-time steering, while geometric rewards use indirect quality assessments or direct consistency measures. GeoFix proposes a single geometric reward to improve consistency and video quality without auxiliary rewards.

2 RELATED WORK

  • Diffusion-model alignment includes post-training reward or preference optimization and inference-time selection, resampling, or guidance; GeoFix evaluates geometric rewards with both types of alignment.Its evaluations use DiffusionNFT, best-of-N selection, and DPS-style gradient guidance.Source: §2 · p. 2
  • Indirect geometric feedback uses preference models or VLMs to assess rendered-view quality, but depends on appearance and rendering fidelity and can miss structural errors.Direct feedback instead measures reconstruction error, epipolar constraints, geometric correspondence, or depth agreement.Source: §2 · p. 3
  • Directly optimizing consistency does not ensure visual quality; prior methods report blur or oversaturation and use auxiliary objectives or preference filtering.These additions require choices about reward weights or sample-selection thresholds.Source: §2 · p. 3
  • GeoFix addresses the trade-off with a single geometric reward intended to improve consistency and video quality without auxiliary rewards.Figure 2 outlines latent-based FOCAL computation, static-region depth reprojection, and overlap-bias calibration.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD

GeoFix aligns video diffusion models with a geometric reward computed on video latents. Its FOCAL reward calibrates static-region cross-view depth agreement for camera overlap and supports both training and inference-time alignment.

3 METHOD

  • GeoFix computes its geometric reward directly on video latents using FOCAL and a stitched 4D reconstructor.The framework is presented as applicable to both training and inference-time reward alignment.Source: §3 · p. 3

3.1 LATENT-TO-4D MODEL STITCHING

  • A stitched reconstructor maps video latents to depth, camera, intrinsics, and Gaussian outputs; jointly distilled from a frozen reconstructor, it achieves higher reconstruction fidelity.The distillation data include multi-view image sets and monocular videos of dynamic scenes.Source: §3.1 · p. 4

3.2 CROSS-VIEW DEPTH CONSISTENCY REWARD

  • FOCAL compares reprojected cross-view depth only in static regions, avoiding appearance matching and valid-motion penalties; raw agreement rises with camera overlap, biasing rewards toward smaller viewpoint changes.The overlap dependence is observed even in real videos and multi-view 4D datasets.Source: §3.2 · pp. 4–5

3.3 FOCAL: FOV-CALIBRATED GEOMETRIC REWARD

  • FOCAL subtracts the reference-fitted overlap term from raw depth consistency; on held-out real and generated videos, the corrected score–overlap trend has near-zero regression slope.The calibration coefficient is estimated from reference videos and held fixed during alignment.Source: §3.3 · p. 5

3.4 GEOFIX: REWARD-BASED VIDEO ALIGNMENT

  • GeoFix uses FOCAL to fine-tune video diffusion models or select and guide samples at inference time, without auxiliary rewards.Demonstrated alignment methods include DiffusionNFT, Best-of-N, and DPS-style gradient guidance.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

Across training-time and inference-time alignment, GeoFix improves geometric consistency and video quality, while its 4D scene generation preserves structure and meaningful motion. The ablations examine reward sensitivity to geometric corruption, video-quality rewards, and FOV calibration.

4.1 EXPERIMENTAL SETUP

  • Training-time alignment uses Wan2.2-TI2V-5B with supervised and DiffusionNFT losses, while inference-time alignment uses Wan2.2-Fun-5B-Control-Camera and a fixed stitched reconstructor.The reconstructor is fine-tuned from NeoVerse.Source: §4.1 · p. 6

4.2 VDM TRAINING-TIME ALIGNMENT

  • GeoFix achieves the strongest geometric consistency among training-time baselines, improves general video quality without an auxiliary reward, and exceeds VGGRPO’s relative gains across VBench dimensions.It also attains the highest WorldScore 3D and photometric consistency, best motion accuracy and smoothness, and superior or competitive visual quality and text alignment.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7

4.3 INFERENCE-TIME ALIGNMENT

  • Both first- and zeroth-order inference-time steering improve camera accuracy, 4D scene quality, consistency, visual quality, and text alignment, with only marginally reduced motion accuracy.Every camera-accuracy measure and every other WorldScore aspect improves.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 4D SCENE GENERATION

  • Overlaid timestep renderings show GeoFix keeping the background aligned while dynamic content moves through it.MoGe4D generates almost no motion, while 4DNeX shows background drift and discontinuous object motion.Source: §4.4 · p. 8, Figure 5 · p. 8
  • GeoFix performs best overall in 4D scene generation and achieves the highest consistency scores under novel camera trajectories.4DNeX’s higher dynamic degree largely reflects structural inconsistency, while MoGe4D has insufficient motion; GeoFix better preserves structure while maintaining meaningful dynamics.Source: Figure 6 · p. 8, §4.4 · p. 8

4.5 ABLATION STUDIES

  • FOCAL decreases as each of four camera-inexplicable geometric corruptions increases, supporting its use as a signal of geometric inconsistency rather than image fidelity.The test perturbs reconstructed real and generated videos; the static-region mask is transformed with the depth so mask quality does not affect measurement.Source: §4.5 · pp. 8–9, Figure 7 · p. 9
  • Directly optimizing VBench rewards produces saturated videos, and longer training adds blur and grid artifacts.Several VBench metrics reward high interframe DINO or CLIP feature similarity, collapsing variation; none of the video-quality metrics is used as a training reward for GeoFix.Source: Figure 8 · p. 9, §4.5 · p. 9
  • FOV calibration makes the reward invariant to camera motion and prevents reward optimization from suppressing camera movement during training.Without calibration, the model reduces camera motion to collect an easier reward; the calibrated setting also achieves the highest VBench video quality.Source: §4.5 · p. 9

5 CONCLUSION

The conclusion presents FOCAL as a geometric consistency reward and GeoFix as its application within reward alignment. It reports improvements across training-time and inference-time alignment and coherent 4D scenes compared with baselines.

5 CONCLUSION

  • FOCAL evaluates cross-view depth consistency in static regions and calibrates depth agreement against the expected value at a given camera overlap.This targets structural inconsistencies while preserving valid object motion and reduces rewarding greater overlap alone as improved geometry.Source: §5 · p. 9
  • GeoFix applies FOCAL to reward alignment methods to improve geometric consistency and visual quality.FOCAL is computed directly from video latents through a stitched 4D reconstructor, without VAE decoding.Source: §5 · p. 9
  • Experiments across training-time and inference-time reward alignment show improvements in geometric consistency, video quality, and camera accuracy.The conclusion also reports coherent 4D scenes compared with baselines.Source: §5 · p. 9

A.1.1 TASKS AND DATASETS

The appendix specifies the datasets, baselines, training and steering procedures, stitched reconstructor, and evaluation protocols used across GeoFix’s tasks.

A.1.1 TASKS AND DATASETS

  • Three tasks use a NeoVerse reconstructor stitched to the video-diffusion latent space to decode generated latents into 4D Gaussian Splatting scenes.Text-to-4DGS uses 200 prompts, image-to-4D uses 100 image-caption pairs, and camera-controlled image-to-4D uses 355 pairs assigned to ten trajectories.Source: §A.1.1 · p. 14, Table 5 · p. 14

A.1.2 BASELINES

  • Training comparisons use the same SFT data, GPU hours, and frozen reconstructor, while steering compares unguided sampling, Best-of-N, and DPS on the same camera-control model.Image-to-4D baselines use common target-camera rendering; MoGe4D and 4DNeX point clouds are rendered according to their respective representations.Source: §A.1.2 · pp. 14–15

A.1.3 TRAINING

  • Wan2.2-TI2V-5B is fine-tuned with LoRA using a supervised loss and an online DiffusionNFT reinforcement-learning loss, then sampled under shared inference settings.Training uses MultiCamVideo and UltraVideo for supervision and VideoUFO prompts for reinforcement learning; inference generates 41 frames at 480×832 unless otherwise stated.Source: §A.1.3 · p. 15, §A.1.4 · p. 15

A.1.5 TEST-TIME STEERING

  • Test-time steering shares stochastic sampling and FOCAL reward maximization; Best-of-N selects the top of 20 trajectories, while DPS backpropagates through the reconstructor and reward.DPS guides 15 denoising steps and uses a soft FOV-overlap relaxation for gradients while preserving exact overlap in the forward pass.Source: §A.1.5 · pp. 15–16

A.1.6 STITCHED MODEL TRAINING

  • The stitched model distills NeoVerse outputs from frozen Wan 2.2 video-VAE latents, then trains on static DL3DV scenes followed by dynamic UltraVideo videos.Distillation matches point maps, camera poses and intrinsics, depth, Gaussian parameters, and dynamic-scene velocities.Source: §A.1.6 · p. 16

A.2.1 EVALUATED VIDEO

  • Evaluation scores rendered videos to expose cross-frame geometric inconsistency; the VGGRPO comparison instead uses VAE-decoded videos and relative gains over each method’s own base model.The rendered-video protocol uses reconstructor input-view cameras for text-to-4DGS and steering, and a shared novel-view trajectory for image-to-4D.Source: §A.2.1 · p. 16, §A.2.2 · p. 16

A.2.3 METRICS PER TASK

  • Metrics vary by task: text-to-4DGS uses FOCAL and WorldScore, image-to-4D uses VBench, CLIP, and VideoPhy-2, and steering uses camera accuracy and WorldScore.Image-to-4D metrics include physical plausibility, text alignment, quality, consistency, flickering, and dynamic degree.Source: §A.2.3 · p. 16

A.2.4 METRIC DEFINITIONS

The appendix defines the video-quality, consistency, physical-plausibility, text-alignment, and camera-accuracy measures used in evaluation.

VBench

  • VBench Quality Score is the normalized weighted average of seven quality dimensions; in text-to-4DGS, GeoFix raises it from 0.7272 (Base) to 0.7588 (+4.3%).VBench reports eight dimensions overall, with the unweighted mean reported separately as VBench Avg.Source: §A.2.4 · p. 17

WorldScore

  • WorldScore measures eight aspects, including 3D consistency through DROID-SLAM reprojection error and photometric consistency through SEA-RAFT optical-flow error.Camera and object control are omitted because the benchmarks lack the required per-instance annotations.Source: §A.2.4 · p. 17

VideoPhy-2

  • VideoPhy-2-AutoEval scores physical commonsense on a 1–5 scale using 32 sampled frames and its official query template.The evaluator is an mPLUG-Owl-Video model with a LLaMA-7B backbone, fine-tuned on VideoPhy-2 human annotations.Source: §A.2.4 · p. 17

CLIP score

  • CLIP score is the frame-averaged cosine similarity between CLIP ViT-L/14 image and text embeddings, scaled by CLIP’s logit scale.Source: §A.2.4 · p. 17

Camera accuracy

  • Camera accuracy reports relative-trajectory rotation error, median translation-direction error, and normalized ATE after scale fitting.The median is used for translation direction because the first frames have little translation and ill-conditioned directions.Source: §A.2.4 · p. 17

B STITCHED MODEL RECONSTRUCTION RESULTS

The stitched reconstructor is evaluated on novel-time and novel-view synthesis, with reported gains over baselines in most metrics.

Setup

  • Evaluation uses MultiCamVideo’s ten synchronized cameras for novel-time synthesis from 33 frames and novel-view synthesis from 21 context-camera frames.The comparisons include 4DGT, MoVieS, and NeoVerse; predicted poses are aligned at evaluation for NeoVerse and the proposed model.Source: B STITCHED MODEL RECONSTRUCTION RESULTS · pp. 17–18

Results

C ADDITIONAL RESULTS

The appendix adds qualitative examples for text-to-4DGS and comparisons of rendered text-to-video results.

C.1 ADDITIONAL 4DGS QUALITATIVE RESULTS

  • Figure 10 presents additional qualitative text-to-4DGS results.Source: §C.1 · p. 18

C.2 ADDITIONAL TEXT-TO-VIDEO QUALITATIVE RESULTS

  • Figure 11 presents additional 4DGS-rendered text-to-video comparisons against baseline methods.Source: §C.2 · p. 18

D.1 OVERLAP BIAS IN DEPTH ERRORS

Cross-view depth error varies with camera FOV overlap even when reconstruction quality is comparable, so raw depth agreement can mistake smaller camera motion for better geometry. The overlap computation estimates how much the first and last cameras see in common.

D.1 OVERLAP BIAS IN DEPTH ERRORS

  • Low-overlap clips show higher depth reprojection errors than comparable high-overlap clips, so raw depth scores can favor less camera motion over geometric consistency.Small overlap makes reprojection harder near image boundaries, at grazing angles, and on newly revealed or occluded surfaces; this motivates calibration.Source: §D.1 · pp. 18–19, Figure 12 · p. 20

D.2 FOV OVERLAP COMPUTATION

  • The overlap computation uses the first and last predicted cameras, expressed in the first camera’s frame, to compare their fields of view on a reference sphere.The sphere is centered at the first camera, and its radius is set to the 75th percentile of predicted clip depths.Source: §D.2 · p. 20
  • Using the 75th-percentile depth avoids letting unreliable, heavy-tailed sky or distant-background depths place the reference sphere too far from scene content.A sphere placed too far away would make camera translation negligible relative to its radius, causing overlap to reflect almost only rotation.Source: §D.2 · pp. 20–21
  • FOV overlap Ω approximates the area-weighted intersection-over-union of the two camera views on the sphere, ranging from 1 for identical views to 0 for no overlap.Sphere samples are weighted by cos e to compensate for polar oversampling on the azimuth–elevation grid.Source: Figure 13 · p. 20, §D.2 · p. 21

D.3 ANALYSIS OF VBENCH REWARDS

The VBench reward comparison shows artifacts from optimizing individual perceptual proxies, while GeoFix’s geometric reward produces more natural colors and cleaner scene structure. The examples illustrate how proxy rewards can favor artifacts without directly constraining scene geometry.

D.3 ANALYSIS OF VBENCH REWARDS

  • VBench-reward training exhibits excessive saturation and unnatural color casts, while the geometric reward produces more natural colors and cleaner scene structure.Figure 14(a) provides additional qualitative comparisons of these training outcomes.Source: §D.3 · pp. 21–22, Figure 14 · p. 21
  • MUSIQ optimization produces color banding and repetitive high-contrast textures, exploiting contrast and texture responses without constraining geometry or temporal coherence.The imaging-quality predictor assesses distortions such as blur, noise, and exposure errors.Source: §D.3 · p. 21
  • Global CLIP feature similarity optimization yields blurred content and a regular grid, since suppressing changing details can reduce frame-to-frame variation without preserving a sharp background.The metric compares whole-frame features rather than explicitly isolating the background.Source: §D.3 · p. 22
  • Motion-smoothness optimization produces nearly black, featureless output because low-contrast content makes frame interpolation easier without requiring realistic dynamics.The reward minimizes pixel-wise difference between interpolated and actual frames.Source: §D.3 · p. 22
  • DINO subject-consistency optimization replaces recognizable objects with repetitive blue patterns, stabilizing feature similarity without preserving subject identity.The metric does not explicitly verify the intended subject’s presence or geometry.Source: §D.3 · p. 22
  • Aesthetic-reward optimization produces a collage-like image with repeated faces and decorative fragments, as frame-wise scoring leaves spatial arrangement and temporal evolution unconstrained.The reward averages a LAION aesthetic predictor applied to CLIP embeddings of individual frames.Source: §D.3 · p. 22

E LIMITATIONS

GeoFix inherits biases from its pretrained video diffusion model and 4D reconstruction model, and its reward depends on depth, camera, and motion-segmentation predictions.

E LIMITATIONS

  • GeoFix inherits pretrained-model biases; teacher failures can corrupt reward signals, segmentation errors can penalize valid motion or leave inconsistent regions unconstrained, and reward computation adds training and inference cost.Gradient-based guidance additionally requires backpropagation through the reconstructor.Source: E LIMITATIONS · p. 22

Related papers

Loading 2610.14293v1…