PaperlayerView exact revision on arXiv

Source-linked AI summary

CameraEditor: Camera-Controlled Image Editing via Video-Prior Sequential Modeling

Xin Shen, Chengyou Jia, Keshuo Xing, Zifeng Zhu, Changliang Xia, Bowen Ping, Zhuohang Dang, Hangwei Qian, Minnan Luo

arXiv:2609.01479v1cs.CV

TL;DR

Precise camera-parameter editing remains difficult because instruction-driven models can misinterpret geometry, miss exact adjustments, or damage identity during large perspective shifts. CameraEditor recasts the task as temporally coherent video sequence prediction with geometric references, dynamic routing, and intermediate frames. It reports state-of-the-art spatial alignment and content preservation, while remaining limited by texture smoothing and cascaded inference latency.

  • Problem

    Existing image-editing methods struggle to apply accurate camera-parameter changes to pre-existing images, especially under large perspective shifts that threaten structural identity.

  • Method

    CameraEditor uses video diffusion sequence prediction with explicit geometric reference pairs, dynamic routing, and intermediate Chain of Frames transitions.

  • Results

    CameraEditor achieves state-of-the-art performance in geometric alignment and content preservation on CamEditor-Bench.

  • Takeaways & Limitations

    CameraEditor and CamEditor-Bench provide a temporal editing framework and independent evaluation suite for precise camera control with source identity preservation.

  • Takeaways & Limitations

    Video 3D-VAEs can over-smooth high-frequency details, while the cascaded pipeline lacks end-to-end optimization and increases inference latency.

Abstract

from arXiv · show

Beyond semantic content, camera parameters play a pivotal role in dictating the geometric perspective and appearance of any given image. While recent image editing models excel at semantic and stylistic manipulation, they struggle with explicit camera parameter control. When handling large perspective shifts, instruction-driven models face a dilemma: they either suffer from structural tearing or generate conservative outputs that ignore geometric instructions. To address this, we introduce CameraEditor, a framework that reformulates camera-controlled editing from a spatial problem into a temporal sequence prediction task. By leveraging the temporal coherence of video diffusion models, our approach integrates an explicit geometric perception module with a dynamic reference routing mechanism. This allows us to construct geometrically rigorous visual reference pairs via dynamic panorama cropping, overcoming the ambiguity of text-based instructions. Furthermore, CameraEditor strategically inserts intermediate transition frames to decompose large perspective shifts, providing a robust temporal buffer that preserves content identity and spatial coherence. We construct a training dataset of 5,760 instances. As an independent contribution, we introduce CamEditor-Bench, a model-agnostic evaluation suite of 462 test cases. Extensive experiments demonstrate that CameraEditor achieves state-of-the-art camera control precision and source identity preservation, outperforming existing methods.

1 Introduction

CameraEditor targets precise camera-parameter editing, where instruction-driven models often confuse geometric changes, miss exact adjustments, or lose content identity. It reformulates editing as temporal sequence prediction with visual references and intermediate transitions, supported by dedicated data and evaluation.

  • Camera parameters alter an image’s visual impression even when content remains unchanged, motivating precise camera-controlled editing.
  • Instruction-driven models exhibit terminology misinterpretation, parametric imprecision, and identity-preservation failures under complex perspective shifts.
  • CameraEditor reformulates camera-controlled editing as sequence prediction, using video priors, visual reference pairs, and intermediate transition frames.The transitions decompose large geometric transformations into a traceable trajectory.
  • 5,760 training instances and the 462-instance CamEditor-Bench support the framework’s evaluation and development.The dataset includes detailed camera annotations, while the benchmark is model-agnostic.
  • CameraEditor reports state-of-the-art geometric alignment and content preservation, with gains attributed to explicit geometric conditioning and sequential visual references rather than backbone capacity alone.

2 Related Work

Prior work provides strong image editing, geometric conditioning, camera estimation, and viewpoint synthesis, but precise camera control for existing images remains unresolved. CameraEditor addresses this post-capture editing gap.

  • Instruction-driven editing methods support semantic and localized manipulation but remain limited for precise structural control.
  • Classical and learned camera estimation methods seek geometric calibration, yet direct image-to-camera regression is highly nonlinear and can generalize poorly.
  • Novel-view synthesis methods manipulate viewpoints through intermediate representations but are predominantly object-centric and limited in complex open-domain scenes.
  • Camera-controlled diffusion methods regulate forward generation, but existing-image editing still lacks reliable inversion of latent geometry.This leaves exact camera-parameter manipulation without severe structural degradation as an unresolved bottleneck.

3 Methodology

CameraEditor casts camera-controlled editing as temporally coherent video generation conditioned on explicit geometric references. Dynamic routing aligns references at inference, while Chain of Frames decomposes large viewpoint changes into gradual transitions.

  • Framework overview: The framework predicts a target Chain of Frames sequence using a video backbone, explicit reference visual sequences, and inference-time dynamic routing.
  • Task definition: A camera configuration comprises yaw, roll, pitch, vertical field of view, and radial distortion; CameraEditor controls all except yaw while preserving scene identity.
  • Video generative models: Video editing uses shared source-target latents and flow matching, with noisy states interpolated between pure noise and the clean target latent.
  • Reference visual sequence: Reference image pairs demonstrate the exact camera shift, allowing temporal attention to transfer the geometric transformation to the target image.
  • CoF-based intermediate frames: Large camera shifts create distortions and content inconsistency, so Chain of Frames uses shared interpolated camera poses to generate temporally coherent intermediate steps.Temporal attention models correlations across frames, supporting smooth transformations and content preservation.
  • Dynamic routing: GeoCalib estimates input camera parameters so dynamic routing can select a reference prior with aligned viewpoint conditions during inference.

4 Dataset and Evaluation Benchmark

The dataset pipeline synthesizes camera-annotated temporal training instances from real and UE5 panoramas, while CamEditor-Bench evaluates zero-shot editing across isolated scenes and geometric conditions. Evaluation combines image fidelity, content consistency, and camera accuracy.

  • Dataset construction: The automated pipeline generates 5,760 training instances because continuous camera-transition data is absent from current editing domains.
  • Dataset construction: Real-world and UE5 panoramas provide diverse indoor and outdoor geometric structures for the panorama pool.
  • Dataset construction: Perspective Fields parameterize yaw, pitch, roll, vertical field of view, and radial distortion for aligned reference-target crops.
  • Dataset construction: Super-resolution, five-criterion VLM filtering, and linearly interpolated intermediate frames refine the pairs and form temporally consistent target sequences.
  • Benchmark construction: CamEditor-Bench is a 462-instance, model-agnostic suite designed for diverse semantic content, parameter variations, strict isolation, and zero-shot evaluation.
  • Benchmark scope: Yaw is excluded because large yaw changes require extensive out-of-view generation, shifting the task from controlled editing toward outpainting.
  • Evaluation protocol: Evaluation measures SSIM, LPIPS, CLIP-I, DINO-v2, and Perspective-Field alignment against ground-truth images and target camera conditions.

5 Experiments

Experiments evaluate CameraEditor against open- and closed-source baselines, same-backbone controls, routing strategies, human preferences, and intermediate-frame choices. Results show strong camera alignment and content preservation, with GeoCalib routing and eight transition frames providing key benefits.

  • Experimental Setup: CameraEditor is compared with six open-source models and two closed-source APIs using a deterministic prompt-conversion pipeline.The evaluation covers image quality, content preservation, and camera geometric alignment on CamEditor-Bench.
  • Experimental Setup: Figure 5 visualizes qualitative outputs beside PF-I and PF-T error maps, with warmer colors indicating larger deviations.PF-I appears in the top panels and PF-T in the bottom panels for each method.
  • Main Results (RQ1): CameraEditor achieves state-of-the-art performance across all metrics and breaks the trade-off between precise geometric control and content preservation.Large models improve structural stability but remain below CameraEditor in content preservation and target-camera alignment; Qwen-Image reports CLIP-I 0.8723 and S_Ilat 0.2549.
  • Human Evaluation: 82.0%, 63.5%, and 69.5% of blind human comparisons prefer CameraEditor over Step1X-Edit, Qwen-Image-Editing, and Nano-Banana-Pro, respectively.The study contains 300 comparisons, with ties contributing 0.5 preference to both methods.
  • Intermediate Frame Ablation (RQ3): N = 8 provides the best balance: SSIM 0.5712, LPIPS 0.5454, and DINO-v2 0.8569, while N = 0 tears structurally and N = 12 introduces temporal drift.N = 0 reaches S_Tlat 0.1669 but has SSIM 0.5181 and LPIPS 0.5891; N = 12 reduces absolute alignment to S_Tlat 0.2476 and DINO-v2 to 0.8179.

6 Conclusions and Limitations

CameraEditor reformulates camera-controlled image editing as temporal sequence prediction and combines geometric perception, dynamic panorama cropping, and intermediate frames. Evaluations on CamEditor-Bench show strong spatial alignment and source preservation, while limitations remain in texture fidelity and pipeline efficiency.

  • CameraEditor leverages video diffusion temporal coherence, dynamic panorama cropping, and intermediate frames for camera-parameterized editing.
  • CamEditor-Bench evaluations report state-of-the-art performance balancing precise spatial alignment with source content preservation.
  • Limitations and Future Work: Video 3D-VAEs can over-smooth high-frequency details, degrading fine-grained textures during decoding.
  • Limitations and Future Work: The cascaded pipeline lacks end-to-end optimization and increases inference latency through sequential parameter extraction, reference cropping, and video denoising.

Supplementary Material

The supplementary material provides implementation details, extended analyses, and qualitative results. It covers dataset construction, CamEditor-Bench and its evaluation protocols, and additional experimental details.

  • Additional Experimental Details: It also explains additional experimental details and qualitative results supporting the main manuscript.
  • Automated Pipeline for Dataset Construction: The supplement documents the automated dataset-construction pipeline, including parameter sampling, temporal interpolation, VLM filtering prompts, and dataset comparisons.
  • CamEditor-Bench and Evaluation Protocols: It presents CamEditor-Bench and evaluation protocols, including the derivation of the Target Perspective Field and benchmark comparisons.

A.1 Training Dataset Comparison and Necessity

The training dataset is designed to combine explicit camera annotations, continuous transitions, and diverse content, which existing datasets do not provide simultaneously. Its pipeline uses stratified parameter sampling, constrained view pairing, interpolated transitions, and VLM filtering.

  • Training Dataset Comparison and Necessity: Existing datasets lack the simultaneous combination of explicit camera annotations, continuous temporal transitions, and high content diversity required for sequential camera editing.
  • Training Dataset Comparison and Necessity: The proposed dataset combines real-world panoramas with UE5 scenes, yielding 5,760 sequences with camera mappings and Chain of Frames transitions.
  • Camera Parameterization: Camera behavior is represented with yaw, pitch, roll, vertical Field of View, and radial distortion parameters.
  • Parameter Sampling: Stratified sampling divides roll, pitch, vFoV, and radial distortion into 576 parameter grids, with 10 samples per grid producing 5,760 training instances.
  • Paired View Construction: The pipeline constrains edited configurations by locking yaw and locally bounding pitch and field of view while sampling roll and distortion within controllable ranges.
  • Temporal Transition Construction: Spherical projection creates aligned base-target pairs, while linear interpolation supplies a continuous geometric trajectory for intermediate frames.
  • VLM-Based Filtering: A deterministic Qwen2.5-VL-7B-Instruct VLM scores paired views across five dimensions, retaining only pairs meeting the minimum-score criterion.

B.1 Comprehensive Benchmark Comparison: Data Properties and Evaluation Metrics

CamEditor-Bench is positioned as a source-conditioned benchmark for precise camera editing, pairing source images with parameters and evaluating geometry separately from content preservation. Its protocol uses deterministic parameter-derived references and optimization-free Perspective Field metrics, while baseline comparisons are controlled for conditioning and backbone differences.

  • Data Properties: CamEditor-Bench evaluates source-conditioned camera editing rather than text-to-image generation, using a Source Image + Params modality.This distinguishes it from video synthesis benchmarks and camera-centric T2I benchmarks that lack paired source-editing constraints.
  • Evaluation Metrics: The benchmark decouples semantic and structural fidelity from absolute camera alignment using DINO-v2, CLIP-I, SSIM, LPIPS, and Perspective Field alignments.PF-T and PF-I provide optimization-free geometric comparisons against target conditions and generated-image geometry.
  • Evaluation Metrics: PF-T analytically derives dense target latitude and up-vector fields from user-specified camera parameters for absolute geometric evaluation.The Unified Spherical camera model accommodates nonlinear distortion and large fields of view; target roll, pitch, and vFoV determine the geometry.
  • Baseline Protocol: The two Wan2.1 controls and CameraEditor share backbone, training data, and optimization settings, while retaining CoF to isolate conditioning and reference-sequence effects.The no-reference control uses deterministic prompts; the Plücker control receives GeoCalib-derived camera embeddings.
  • Baseline Protocol: CamEdit is reported only as a task-boundary reference because it controls different parameters and lacked a public implementation for numerical comparison.Warping plus inpainting and direct panorama reprojection also do not match the single-perspective-image editing evaluation setting.
  • Baseline Protocol: Baseline evaluation maps exact roll, pitch, vFoV, and distortion settings into deterministic hybrid prompts to reduce quantization and prompt-engineering differences.The unified template separates camera modifiers from semantic content, isolating intrinsic geometric alignment capacity.

C.4 Detailed Configurations of Perception Modules

The perception study compares random, VLM-based, and GeoCalib-driven routing, showing that geometric calibration quality strongly corresponds to downstream structural alignment and content preservation.

  • Perception Strategies: 54.00° E_pitch from Random Routing causes severe structural collapse, yielding DINO-v2 0.6301.Random routing samples camera parameters without visual perception, producing catastrophic geometric misalignment.
  • Perception Strategies: Puffin-base and Puffin-thinking retain identical E_ξ 0.338 to the Random baseline, exposing limited sensitivity to continuous radial distortion.Chain-of-Thought reasoning slightly reduces roll and vFoV errors but does not resolve distortion estimation.
  • Perception Strategies: GeoCalib reduces distortion error to E_ξ 0.201 and pitch error to E_pitch 6.05°, providing a stronger geometric anchor.Its calibration pipeline extracts dense Perspective Fields and is designed for nonlinear lens physics.

D.3 Ablation on Intermediate Frames (N)

The CoF ablation shows that longer intermediate-frame sequences improve structural integrity across large perspective shifts, but progressively reduce high-frequency texture clarity.

  • Sequence Length: N=0 causes severe spatial tearing and outpainting hallucinations when bridging a massive perspective gap in one step.The non-linear pixel displacement exceeds the model’s structural capacity in direct single-step editing.
  • Sequence Length: N=4, 8, and 12 improve structural integrity by decomposing the transformation into a continuous sequence with inter-frame geometric consistency.The video diffusion prior acts as a temporal buffer for large camera transformations.
  • Trade-off: Increasing N trades spatial coherence for textural clarity because cumulative temporal smoothing degrades high-frequency details.The loss becomes more noticeable as sequence length increases.

E Limitations

CameraEditor’s limitations include sequence-length-dependent texture degradation and unresolved ambiguity in 2D-to-3D camera estimation and geometric evaluation.

  • Algorithmic Limitations: Longer CoF sequences accumulate temporal smoothing and errors, progressively degrading high-frequency textures and single-frame clarity.The authors identify hybrid latent upscaling as a future mitigation direction.
  • Geometric Ambiguity: 2D-to-3D estimation remains ill-posed, so weak vanishing points, reflective surfaces, and non-rigid content can cause initial camera estimates to drift.Such drift can misguide the entire editing sequence.
  • Evaluation Limitations: PF-T’s downstream estimators can be misled by complex artifacts, occasionally conflating generative failures with metric noise.The limitation affects evaluation reliability rather than only image generation.
Loading 2609.01479v1…