PaperlayerView exact revision on arXiv

Source-linked AI summary

Geo-Align: Video Generation Alignment via Metric Geometry Reward

Zizun Li, Haoyu Guo, Runzhe Teng, Chunhua Shen, Tong He

arXiv:2605.23903v1cs.CV

TL;DR

Camera-controlled video re-rendering lacks synchronized real-world multi-view data and metric-aware camera alignment. Geo-Align uses reinforcement learning with geometry-based rewards and hybrid real–synthetic trajectory sampling, outperforming supervised baselines in camera controllability and visual quality across trajectory categories.

  • Problem

    Existing methods rely on scarce synchronized real-world multi-view data and often lack metric-level camera alignment, limiting evidence for real-world trajectory control.

  • Method

    Geo-Align applies reinforcement learning with metric 3D geometry and aesthetic rewards, combining real conditioning videos with scaled synthetic target trajectories without paired multi-view data.

  • Results

    Geo-Align consistently improves camera trajectory fidelity, geometric consistency, camera accuracy, and visual quality over existing baselines across ten target trajectory types.

  • Takeaways & Limitations

    Geo-Align shows that reinforcement learning can enhance precise camera control and visual quality while reducing reliance on synchronized paired multi-view training data.

  • Takeaways & Limitations

    The model can fail under extreme camera motions or large nearby foreground objects, while dynamic-object-heavy inputs may produce flickering or disappearing elements.

Abstract

from arXiv · show

Camera-controlled video generation has achieved remarkable progress in recent years. However, existing video-to-video re-rendering methods primarily rely on Supervised Fine-Tuning using synthetic datasets. At present, there is an extreme scarcity of synchronized, multi-view real-world video data. Consequently, the prevailing paradigm often exhibits limited generalization when processing out-of-distribution real-world videos, with models struggling to accurately adhere to physical scales and camera trajectories. To bridge this gap, we propose Geo-Align, the first Reinforcement Learning framework specifically designed for camera-controlled video re-rendering. Built upon a pretrained model, we optimize the model through a scale-aware perceptual reward mechanism. Specifically, we introduce a metric 3D estimator to extract precise camera trajectories from generated videos, explicitly penalizing deviations in rotation and translation. Furthermore, we meticulously designed a data pipeline strategy based on real-world conditioning videos and target camera trajectories derived from synthetic data, eliminating the reliance on paired data. Extensive experiments demonstrate that Geo-Align consistently outperforms existing supervised learning baselines in both precise camera controllability and visual fidelity, indicating the effectiveness of our method.

1 Introduction

Geo-Align addresses scarce synchronized real-world multi-view data and scale-ambiguous camera supervision by using reinforcement learning to optimize metric camera alignment and visual quality without paired target-view videos. Evaluations on DAVIS across ReCamMaster’s ten trajectory types show improved trajectory fidelity and visual aesthetics over the original model and supervised baselines.

  • Challenges: Real-world multi-view training data are scarce, so existing methods rely mainly on synthetic datasets or difficult-to-construct point-cloud renderings.Video retake requires multi-view video data for supervised training, unlike single-frame camera-controlled generation.
  • Challenges: Scale-less real-world camera annotations and reconstruction-focused SFT losses frequently cause scale drift in generated camera trajectories.Existing real-world videos often lack metric pose information, while standard losses do not explicitly optimize physically meaningful camera alignment.
  • Geo-Align: Geo-Align introduces reinforcement learning that directly optimizes physical camera alignment and visual quality without requiring target-trajectory video pairs.The method uses readily obtainable real-world conditioning videos and avoids the time-synchronized multi-camera ground truth required by prior SFT paradigms.
  • Geo-Align: A verifiable geometry reward compares estimated and target camera trajectories, penalizing rotation and translation errors while mitigating metric-related reward hacking.MapAnything estimates trajectories, and a metric evaluator suppresses degenerate solutions such as slow-moving outputs for rapid targets.
  • Results: On DAVIS, the RL-trained model improves target-trajectory accuracy and aesthetic metrics across ReCamMaster’s 10 trajectory types, achieving SOTA performance.Qualitative comparisons also show improved generated-video quality.
  • Geo-Align: A fusion data strategy combines Citywalk camera-pose priors with rescaled gaming trajectories, improving diversity, bridging scale gaps, and avoiding paired multi-view data.Truncated Gaussian Sampling rescales target trajectories, while aesthetic rewards enhance generated-video quality.

2 Related Work

Related work spans camera-controlled video retaking, feed-forward scene-geometry estimation, and reinforcement-learning alignment for generative models. These directions include geometric warping, stateful online perception, foundational architectures, and GRPO-based optimization.

  • Camera-Controlled Video Retake: Camera-controlled video retake synthesizes novel views by redirecting camera trajectories through generative models.Early approaches use external depth estimators and point trackers to warp input frames before video-diffusion refinement, including TrajectoryCrafter and CogNVS.
  • Feed-Forward Geometry Estimation: Feed-forward models predict scene geometry without traditional SfM optimization, beginning with DUSt3R’s dense point-map regression from unconstrained images.CUT3R and WinT3R add stateful memory and sliding-window mechanisms for efficient online perception, while VGGT, π3, and Depth Anything 3 scale toward unified foundational architectures.
  • Reinforcement-Learning Alignment: GRPO has emerged as an online reinforcement-learning framework for aligning generative models across flow-matching and video-generation domains.Flow-GRPO enables online RL through ODE-to-SDE conversion, MixGRPO improves efficiency with mixed ODE-SDE sliding-window sampling, and GrndCtrl applies GRPO to physically grounded world modeling.

3 Methodology

Geo-Align formulates camera-controlled video re-rendering as reinforcement learning from an input video, text prompt, and target camera trajectory. It combines metric 3D geometry rewards with perceptual-quality rewards, optimizes the pretrained model using GRPO, and trains without paired ground-truth videos through physically rescaled synthetic trajectories.

  • Problem Formulation: The model re-renders an input conditioning video along a user-specified unseen trajectory defined by target camera intrinsics and extrinsics.The conditioning inputs include video frames x1:N and text prompt c, while the target trajectory provides Ktgt1:N and Etgt1:N.
  • Reward Design: The geometry reward uses MapAnything to estimate generated-video translations and rotations, then penalizes translation and angular deviations from the target trajectory.The evaluator extracts predicted camera translations and rotations, enabling explicit spatial alignment across both dimensions.
  • Reward Design: Temporal weighting emphasizes later frames because error accumulation and spatial drift make them the primary bottleneck in trajectory control.The weighting scheme reflects stronger initial-frame adherence and larger later-frame deviations in pretrained video generators.
  • Reward Design: Perceptual and aesthetic rewards complement geometric alignment by measuring sequence dynamics, visual quality, and motion quality to reduce reward hacking and artifacts.VideoAlign supplies sequence-level visual and motion quality scores alongside the geometry reward.
  • Optimization: GRPO removes the value model and, in Geo-Align, the KL penalty, while timestep-aware weighting improves exploration of novel out-of-distribution camera trajectories.The method uses relative scores within output groups for advantage estimation and balances gradients across diffusion stages.
  • Data Pipeline: The training pipeline combines in-the-wild CityWalk conditioning videos with synthetic target trajectories sampled under truncated-Gaussian speed bounds and uniformly rescaled to physical metric ranges.This eliminates paired ground-truth video dependence, prevents unnatural camera jumps, and stabilizes RL optimization.

4 Experiments

Geo-Align is evaluated against warping-based and implicit-extrinsic baselines using DAVIS test cases, camera-trajectory metrics, consistency metrics, and aesthetic evaluation. Results show improved geometric consistency, camera accuracy, visual quality, and robustness under large camera motions, while ablations assess the geometry reward.

  • Baselines: The evaluation compares Geo-Align with TrajectoryCrafter, CogNVS, ReCamMaster, and ReDirector across explicit warping and implicit camera-extrinsic categories.TrajectoryCrafter and CogNVS generate fewer than 49 frames, whereas ReCamMaster and ReDirector generate 81 or more.
  • Evaluation setup: The protocol evaluates 500 DAVIS test cases formed from 50 videos and 10 ReCamMaster camera trajectories per video, with methods assessed at dataset-default lengths or a 49-frame cap.TrajectoryCrafter and CogNVS are restricted to a maximum of 49 frames to prevent performance degradation.
  • Metrics: Metrics include TransErr and RotErr from ViPE, input consistency from MEt3R, geometric consistency from Dyn-MEt3R, and comprehensive aesthetics from VBench.Camera-speed comparisons use a reliable subset of 40 DAVIS videos because large movements can produce featureless frames that cause ViPE estimation failures.
  • Results: Geo-Align improves Geometric Consistency, Camera Accuracy, and overall video quality over the baseline, with especially substantial gains in trajectory-following precision.Under large camera motions, it mitigates subject disappearance and background blurring that affect ReCamMaster and ReDirector.
  • Ablation: The ablation retrains models for 140 steps on 16 A800 GPUs, comparing the full aesthetic-plus-geometry reward with an aesthetic-only variant under the Table 1 protocol.The study evaluates how the proposed reward components contribute to geometric consistency, camera accuracy, and visual quality.

5 Conclusion

Geo-Align introduces a reinforcement learning framework for camera-controlled video retake. It combines metric-3D trajectory-aware rewards with hybrid sampling of real videos and scaled synthetic camera trajectories.

  • Geo-Align is a reinforcement learning framework designed for camera-controlled video retake.
  • Its reward mechanism uses a metric 3D evaluator to optimize how accurately generated videos follow target camera trajectories.
  • Its hybrid data sampling strategy combines real videos with scaled camera trajectories from synthetic data.The strategy is intended to mitigate reliance on scarce time-synchronized multi-view data.

A Appendix · A.1 Limitations

Geo-Align improves camera-trajectory adherence and overall video quality, but remains vulnerable to challenging camera motions, nearby large foreground objects, and dynamic-object-heavy inputs.

  • A Appendix · A.1 Limitations: The reinforcement learning approach improves camera trajectory adherence while enhancing overall video quality.
  • A Appendix · A.1 Limitations: Excessively fast rotations can cause the model to fail.
  • A Appendix · A.1 Limitations: Large translations remain a failure condition for the model.
  • A Appendix · A.1 Limitations: Large foreground objects close to the camera can trigger model failures.
  • A Appendix · A.1 Limitations: Inputs dominated by dynamic objects frequently produce artifacts.
  • A Appendix · A.1 Limitations: These artifacts include flickering or vanishing dynamic elements.

A.2 Assets and Licenses

The paper summarizes its research assets—including datasets, models, and code—and their licenses and accessibility in Table 4. It states that all assets are used according to their respective terms.

  • Assets and Licenses: The paper reports that all research assets are used in accordance with their respective terms.The asset summary includes licenses and accessibility information.
  • Assets and Licenses: Table 4 summarizes the datasets, models, and code used in the research.The table is identified as a summary of used assets.
Loading 2605.23903v1…