Source-linked AI summary
UniGeo: Unifying Geometric Guidance for Camera-Controllable Image Editing via Video Models
Hong Jiang, Wensong Song, Zongxin Yang, Ruijie Quan, Yi Yang
TL;DR
Camera-controllable editing requires consistent scene geometry under changing viewpoints, but prior methods use discrete view mappings and fragmented guidance. UniGeo unifies guidance across representation, architecture, and loss, and experiments report stronger visual quality and geometric consistency across extensive and limited camera motion, while complex scenes and extreme viewpoint changes remain challenging.
Problem
Existing methods rely on discrete viewpoint mappings and fragmented geometric guidance, limiting stable geometric understanding and cross-view structural consistency under continuous camera motion.
Method
UniGeo uses frame-decoupled point-cloud injection, geometric anchor attention, and trajectory-endpoint supervision to unify geometric guidance across representation, architecture, and loss.
Results
UniGeo consistently outperforms existing methods in geometric reliability and visual quality across diverse camera motions; LPIPS decreases from 0.3008 to 0.2377.
Takeaways & Limitations
Unified geometric guidance provides a principled framework for high-fidelity camera-controllable image editing across extensive and limited camera-motion settings.
Takeaways & Limitations
Geometric references may become unreliable in complex scenes or under excessively large viewpoint changes, and inference remains slightly slower than single-frame image diffusion.
Abstract
from arXiv · showhide
Camera-controllable image editing aims to synthesize novel views of a given scene under varying camera poses while strictly preserving cross-view geometric consistency. However, existing methods typically rely on fragmented geometric guidance, such as only injecting point clouds at the representation level despite models containing multiple levels, and are mainly based on image diffusion models that operate on discrete view mappings. These two limitations jointly lead to geometric drift and structural degradation under continuous camera motion. We observe that while leveraging video models provides continuous viewpoint priors for camera-controllable image editing, they still struggle to form stable geometric understanding if geometric guidance remains fragmented. To systematically address this, we inject unified geometric guidance across three levels that jointly determine the generative output: representation, architecture, and loss function. To this end, we propose UniGeo, a novel camera-controllable editing framework. Specifically, at the representation level, UniGeo incorporates a frame-decoupled geometric reference injection mechanism to provide robust cross-view geometry context. At the architecture level, it introduces geometric anchor attention to align multi-view features. At the loss function level, it proposes a trajectory-endpoint geometric supervision strategy to explicitly reinforce the structural fidelity of target views. Comprehensive experiments across multiple public benchmarks, encompassing both extensive and limited camera motion settings, demonstrate that UniGeo significantly outperforms existing methods in both visual quality and geometric consistency.
1 Introduction
Camera-controllable editing must preserve scene geometry across continuously changing viewpoints, but prior methods use discrete view mappings and fragmented geometric guidance. UniGeo addresses this by coordinating geometric guidance across representation, architecture, and loss, achieving stronger visual and geometric results across camera-motion settings.
- Motivation: Camera-controllable editing requires viewpoint changes that preserve strict cross-view geometric consistency for rendering and perception applications.The challenge concerns maintaining scene structure across views, not merely modifying appearance.
- Challenges: Image-diffusion methods target discrete viewpoint mappings, limiting their ability to model continuous camera trajectories and causing unstable generation.Camera motion evolves continuously in 3D space, whereas these methods mainly model mappings between discrete views.
- Challenges: Fragmented guidance, such as injecting geometry only at the representation level, disrupts guidance propagation and can lead to 3D structure collapse.The isolated treatment leaves other model levels disjointed and unable to form unified correspondences.
- UniGeo: UniGeo systematically injects geometric guidance into representation, architecture, and loss function to form stable cross-view geometric understanding with video models.Video models provide continuous-viewpoint modeling capabilities, but fragmented guidance still prevents stable geometric understanding.
- UniGeo: UniGeo uses frame-decoupled point-cloud injection, geometric anchor attention, and trajectory-endpoint supervision to strengthen target-view structural fidelity.These modules respectively provide geometric context, align cross-view features, and emphasize geometry at trajectory endpoints.
- Results: 0.2377 LPIPS and 17.8171 PSNR are reported after changes from 0.3008 LPIPS and 16.9580 PSNR, respectively, on the cited evaluation settings.The reported improvements include Tanks videos with limited camera motion for PSNR.
2 Related Work
Related image-editing methods provide controllable visual modification but generally lack unified geometric guidance across the generation pipeline. This limitation makes structural fidelity difficult to maintain under continuous camera motion.
- Image Editing: Image-editing research has progressed from inversion and fine-tuning methods to large-scale text-to-image and unified autoregressive architectures for semantic control.These approaches support increasingly fine-grained control over edited visual content.
- Geometric Guidance: Geometric guidance in existing methods is typically fragmented, limiting structural fidelity during continuous camera motion.The related-work discussion identifies the absence of unified guidance across the generation pipeline.
3 Background: Rectified Flow for Video Diffusion Models
Video diffusion models operate in a compact latent space produced by a 3D-VAE and can use rectified flow to connect data latents with Gaussian noise over continuous time. A neural network learns the velocity field conditioned on timestep and text or image signals.
- Latent Video Modeling: A 3D-VAE encodes an input video into a compact latent representation before generative modeling, and a decoder reconstructs generated latents into pixels.The latent operations are z0 = E(x) and x̂ = D(ẑ0).
- Rectified Flow: Rectified flow constructs an intermediate latent by linearly blending the data latent z_0 with Gaussian noise ε over continuous time t.The passage gives z_t = (1 - t) z_0 + t ε for t in [0, 1].
- Rectified Flow: A neural network predicts the target velocity v = ε - z_0 from the latent state, timestep, and text or image conditioning signals.This prediction drives the rectified-flow process.
4 UniGeo Model
UniGeo introduces unified geometric guidance across representation, architecture, and loss levels for camera-controllable image editing. Its modules construct trajectory-aligned point-cloud guidance, align cross-view features, and emphasize geometric fidelity at trajectory endpoints.
- Frame-Decoupled Point Cloud Injection: UniGeo lifts the input image into a point-cloud sequence rendered along the target camera trajectory for geometric guidance.The point cloud is reconstructed from the first frame, rendered at estimated camera poses, and aligned to the reference view by replacing the first rendering with the original image.
- Frame-Decoupled Point Cloud Injection: Frame-decoupled injection concatenates target-video and rendered-point-cloud tokens along the frame dimension before feeding them into the DiT backbone.This design allows geometric context to interact flexibly with target video features and mitigates adverse effects from imperfect point-cloud priors.
- Geometric Anchor Attention: Geometric Anchor Attention uses first-frame features as keys and values to align features from subsequent timesteps across views.The proposed attention is added to the original self-attention output, with a scalar α controlling geometric-guidance influence and a zero-initialized new output projection.
- Trajectory-Endpoint Geometric Supervision: Trajectory-Endpoint Geometric Supervision assigns higher loss weights to endpoint frames and lower weights to intermediate frames.The weighting follows a quadratic function controlled by γ, while sparse temporal sampling reduces computation on intermediate frames.
- Trajectory-Endpoint Geometric Supervision: Duplicating the target-view frame across consecutive final timesteps maintains geometric guidance during the last generation stage.The temporal extension is designed to enforce a stable geometric structure at the target viewpoint.
5 Experiments
UniGeo is evaluated across multiple datasets, camera-motion regimes, qualitative comparisons, trajectory visualizations, and ablations. The results show strong visual and geometric performance, while identifying limitations for complex scenes, extreme viewpoint changes, and inference efficiency.
- Quantitative comparisons: UniGeo achieves the best performance across key metrics on DL3DV, RE10K, and Tanks under extensive and limited camera motion, and on MannequinChallenge.It is compared with CameraCtrl, MotionCtrl, ViewCrafter, FlexWorld, and PE-Field.
- Qualitative comparisons: Qualitative comparisons show that UniGeo better preserves scene geometry and produces more coherent novel views, especially under extensive camera motion.It alleviates duplicated structures, distorted geometric relationships, and locally incoherent content.
- Qualitative comparisons: Under limited camera motion, UniGeo maintains stable spatial layouts, scene structure, and fine-grained details across views.
- Intermediate trajectory visualization: Trajectory visualizations show continuous geometric transformations from the input through intermediate frames to the final novel view.The intermediate process maintains structural coherence and follows camera-motion characteristics.
- Ablation study: Ablations show that removing FDPCI increases LPIPS by 0.02 on average and decreases SSIM by 0.06 under extensive motion.Removing GAA degrades all metrics, while α = 1.0 gives the best anchor-attention performance; omitting intermediate supervision produces blurrier results.
- Limitations: UniGeo has reduced geometric accuracy for highly complex scenes or excessively large viewpoint changes, and inference still requires generating multiple frames.Sparse temporal sampling improves efficiency relative to standard video generation models but does not eliminate the frame-generation cost.
6 Conclusion
UniGeo uses video diffusion models and unified geometric guidance to preserve cross-view structure during camera-controllable editing. Experiments show stronger geometric reliability and visual quality, while ablations and failure cases identify conditions that degrade consistency.
- UniGeo integrates geometric guidance across representation, architecture, and loss function while leveraging video diffusion models’ continuity prior.
- Ablations show that removing point-cloud or intermediate supervision causes object duplication, incorrect placement, and increased blur.
- Complex objects challenge geometry and texture preservation, while extreme camera changes impede geometric consistency.
- UniGeo consistently outperforms existing methods in geometric reliability and visual quality across diverse camera motions.