PaperlayerView exact revision on arXiv

Source-linked AI summary

Stabilizing Camera-Controlled Novel View Synthesis at Inference Time

Prajwal Singh, Arjun Badola, Seema Kumari, Hajime Nagahara, Shanmuganathan Raman

arXiv:2609.03639v1cs.CV

TL;DR

Training-free camera-controlled novel view synthesis becomes unstable with large camera motions and long generation horizons because geometric and appearance errors accumulate. CamTrol++ decomposes trajectories into small autoregressive steps and adds complementary consistency and warping refinements, improving stability, reconstruction quality, and efficiency across datasets while remaining effective for 56-frame generation. Very long trajectories and imperfect monocular depth still constrain performance.

  • Problem

    Large camera motions and long autoregressive horizons cause accumulating geometric and appearance errors, while the relative importance of existing inference-time components remains unclear.

  • Method

    CamTrol++ decomposes camera trajectories into small autoregressive steps and combines this with geometry-constrained attention, appearance anchoring, and registration-free warping.

  • Results

    CamTrol++ improves temporal and geometric consistency, downstream 3D reconstruction, and generation efficiency across RealEstate10K and MegaScene, with stable 56-frame generation.

  • Takeaways & Limitations

    Small-step camera control is the central stability mechanism, while attention and appearance constraints provide complementary refinements without modifying the diffusion backbone.

  • Takeaways & Limitations

    Very long trajectories can still cause texture drift, fine-detail loss, or depth inconsistencies, and performance depends on monocular depth quality and scene conditions.

Abstract

from arXiv · show

Training-free, camera-controlled novel view synthesis from a single image using pre-trained video diffusion models often becomes unstable under large camera motion and long generation horizons. Existing approaches commonly combine several inference-time components, making it unclear which design choices are most important for stability. We show that the main source of stability is simple. Decomposing camera motion into small autoregressive steps limits per-step geometric distortion and reduces error accumulation. A controlled camera-step study shows that performance remains stable for small motions and degrades more strongly as the per-step motion approaches $18$-$20^\circ$. We further evaluate geometry-constrained spatial attention and low-frequency appearance anchoring as supporting refinements, together with an efficient registration-free warping pipeline. Across RealEstate10K and MegaScene, CamTrol++ improves temporal and geometric consistency, downstream 3D reconstruction quality, and generation efficiency over training-free baselines. The method remains effective for 56-frame generation and under substantial controlled depth corruption. These results show that careful control of camera motion at inference time can substantially improve the stability of camera-controlled novel view synthesis without retraining or modifying the diffusion backbone.

1. Introduction

Camera-controlled novel view synthesis from a single image remains unstable under large motions and long autoregressive horizons. CamTrol++ addresses this with small-step trajectory decomposition and supporting geometric, appearance, and efficiency refinements.

  • Motivation: Training-free novel view synthesis aims to generate multiple scene views from one image for 3D content creation and scene exploration.
  • Motivation: Large camera motions amplify reprojection errors and disocclusions, causing geometric distortion, perspective drift, temporal flicker, and appearance drift.
  • Approach: CamTrol++ divides camera trajectories into small autoregressive sub-paths, limiting per-step geometric change and reducing error accumulation.
  • Approach: The framework adds geometry-constrained spatial attention, low-frequency appearance anchoring, and registration-free warping as supporting components.
  • Results: Across RealEstate10K and MegaScene, CamTrol++ improves temporal and geometric consistency, downstream 3D reconstruction quality, and extends stable generation from 14 to 56 frames.

2. Related Works

Prior novel-view methods trade off scene-specific optimization, single-image consistency, and long-horizon stability. CamTrol++ focuses on stabilizing training-free generation across multiple autoregressive windows by examining small camera steps alongside geometric and appearance constraints.

  • Per-Scene Optimization: Per-scene methods such as NeRF and 3D Gaussian Splatting achieve high-quality reconstruction but require multi-view data and scene-specific optimization.
  • Zero-Shot Novel View Synthesis: Zero-shot methods synthesize new viewpoints from one image without per-scene training but often struggle with scene consistency and long-range stability.
  • Training-Free Generation: Camera-controlled and training-free approaches combine trajectory control, depth-based warping, diffusion refinement, and geometric or appearance constraints, but large motions and long trajectories remain challenging.
  • Positioning: CamTrol++ studies stabilization across multiple autoregressive windows, emphasizing small-step camera decomposition together with supporting geometric and appearance constraints.

3. Method

CamTrol++ stabilizes inference-time camera-controlled novel view synthesis by decomposing trajectories into small autoregressive steps, with attention, appearance, and warping refinements.

  • CamTrol++ generates multi-view video from a single image using a predefined camera trajectory and a pre-trained SVD backbone without retraining or weight modification.
  • Large single-step camera motions create disocclusions, geometric distortion, structural artifacts, and drift during diffusion refinement.The problem is especially pronounced for wide rotations such as 60° or long translations within one 14-frame chunk.
  • The method divides continuous 28- or 56-frame trajectories into autoregressive N-frame chunks, regenerating the point cloud at each boundary to limit distortion and error accumulation.Each chunk reprojects the previous output or the initial image through depth-estimated 3D points before diffusion conditioning.
  • Low-frequency appearance anchoring matches chunk-boundary chrominance to the source image, while registration-free warping removes CamTrol’s costly per-frame point-cloud optimization.Color correction preserves lightness and aligns CIELAB chrominance; the original registration loop takes approximately 27.4s per frame at 704 × 904 resolution.
  • Geometry-constrained spatial attention restricts feature correspondence to sampled epipolar-line locations and blends it with standard attention using α = 0.9.The constraint is applied at the final SVD decoder spatial-attention layer, where late injection offers a better quality-alignment trade-off.

4. Experiments

CamTrol++ is evaluated against training-free baselines across perceptual, temporal, motion, camera, geometric, reconstruction, efficiency, ablation, camera-step, long-horizon, and depth-error settings. The experiments identify small-step autoregression as the strongest stabilization component, while supporting components improve geometry and appearance, and degradation emerges beyond the standard horizon.

  • Efficiency and Baselines: Removing per-frame point-cloud registration reduces generation time by approximately 5×, while CamTrol++ remains less stable and substantially slower than CamTrol++ when CamTrol is applied to segmented trajectories.The latter comparison is reported for an extended 28-frame supplementary evaluation.
  • Quantitative Comparison: CamTrol++ reduces LPIPS-next from 0.2189 to 0.0888 on RealEstate10K and from 0.2408 to 0.0794 on MegaScene in 56-frame comparisons against WAVE.CLIPSIM-next also increases from 0.9692 to 0.9895 and from 0.9539 to 0.9881, respectively.
  • Qualitative Analysis: CamTrol++ preserves coherent motion and perspective in challenging 60° rotations and 56-frame sequences, unlike baselines showing blur, drift, distortions, flicker, or incorrect scene motion.Qualitative comparisons include CamTrol and WAVE on MegaScene and RealEstate10K.
  • Ablation Study: The small-step autoregressive trajectory produces the largest temporal-consistency change, while epipolar attention and LAB anchoring provide smaller geometric and appearance improvements.Removing LAB correction permits color-tone shifts, whereas removing epipolar attention can reduce perspective consistency.
  • Depth Estimation Ablation: The small-step reprojection remains effective across different monocular depth predictors, although depth quality affects overall generation quality.This robustness is evaluated on MegaScene.
  • Camera Step Size: LPIPS-next is lowest near 15° per chunk, while geometric errors increase more strongly beyond approximately 18–20°; CamTrol++ therefore uses a 15° arc.The study varies camera arcs from 10° to 40° with epipolar attention disabled to isolate reprojection effects.
  • Long-Horizon Generation: In extensions up to 84 frames, temporal consistency and motion remain stable initially, but texture and fine-detail loss becomes the most common later failure mode.Appearance drift and motion freezing follow as additional degradation modes.

5. Discussion

CamTrol++ stabilizes long-horizon camera-controlled synthesis primarily by decomposing trajectories into small autoregressive steps, while additional constraints and registration-free warping provide complementary refinements. It remains effective over 56 frames and under controlled depth corruption, though very long trajectories still lose texture and detail.

  • Discussion: Small-step trajectory decomposition is the main source of stability, with degradation increasing particularly beyond approximately 18-20° per step.Limiting per-pass camera motion reduces reprojection distortion and error accumulation.
  • Discussion: Epipolar attention and perceptual anchoring add geometric and appearance constraints, but their effects are smaller and vary across metrics.The paper identifies these components as complementary refinements rather than the central stabilizing mechanism.
  • Discussion: CamTrol++ maintains temporal and geometric consistency over 56-frame sequences and remains effective under substantial controlled depth corruption.These results extend its effectiveness beyond the standard setting.
  • Discussion: Registration-free warping improves practicality by making the framework substantially more efficient than CamTrol while preserving inference-time operation.The approach retains the original diffusion backbone without model changes or retraining.
  • Discussion: Very long trajectories remain limited by texture drift, loss of fine details, and depth inconsistencies, especially under severe occlusion or reflective surfaces.Performance depends on monocular depth quality, and more severe depth failures remain challenging.

6. Conclusion

CamTrol++ is a training-free framework that stabilizes camera-controlled novel view synthesis by decomposing trajectories into small autoregressive steps while keeping the diffusion model frozen. Experiments show improved consistency, reconstruction quality, efficiency, and robustness over longer and corrupted-depth settings.

  • Conclusion: CamTrol++ combines small-step trajectory decomposition, geometry-constrained attention, perceptual anchoring, and registration-free warping for inference-time stabilization.The framework operates without retraining or modifying the diffusion backbone.
  • Conclusion: Experiments on RealEstate10K and MegaScene show improved temporal and geometric consistency, better downstream 3D reconstruction, and stable generation over 56-frame trajectories.The method also improves generation efficiency and remains effective under challenging settings.
  • Conclusion: Camera-step analysis supports small-step control, while long-horizon and depth-robustness experiments show effectiveness under more challenging conditions.The conclusion attributes stability to careful inference-time camera-motion control with the diffusion model kept frozen.

8. Implementation Details

CamTrol++ takes an input image and predefined camera trajectory, then synthesizes camera-consistent frame chunks through depth-based 3D warping, filling, diffusion refinement, attention, and color anchoring. Chunks are generated autoregressively without training or finetuning.

  • 8. Implementation Details: The pipeline receives an input image and predefined camera trajectory and outputs a camera-consistent frame sequence without training or finetuning.Each chunk of N frames is synthesized independently before the last frame becomes the next chunk input.
  • 8. Implementation Details: Each chunk estimates depth with ZoeDepth, unprojects image-depth pairs into 3D, and reprojects them to target views using differentiable splatting.The splatting kernel has size 3 × 3.
  • 8. Implementation Details: Disocclusion gaps are filled with distance-transform inpainting, followed by Stable Diffusion outpainting for large missing regions.The resulting warped frames are encoded into the Stable Video Diffusion latent space for refinement.
  • 8. Implementation Details: Epipolar-guided spatial attention and autoregressive color anchoring enforce spatial and temporal consistency across generated chunks.The framework combines depth-based warping and epipolar priors with color-based temporal coherence.
  • 8. Implementation Details: Experiments use 14-frame chunks and generate 28- or 56-frame sequences with fixed parameters across datasets.Epipolar attention uses α = 0.9 and k = w/2 key samples on a single NVIDIA L40S GPU.

9. Additional Analysis

Additional analyses examine extended baselines, implementation choices, camera-step size, attention sensitivity, and depth corruption. They support small camera steps, identify efficient splatting and deeper attention placement as useful choices, and show that depth error matters more than missing pixels.

  • Extended CamTrol Comparison: CamTrol++ outperforms segmented CamTrol on temporal, geometric, and perceptual consistency while reducing runtime from approximately 15.4 to 4.5 minutes per scene.The comparison uses 28-frame sequences formed from two independently run 14-frame CamTrol segments.
  • Extended Component Analysis: The 23rd SVD decoder layer provides the best tested trade-off for epipolar attention and is used in the final configuration.Early insertion can over-constrain low-level features and suppress motion, whereas deeper layers better balance quality and geometric alignment.
  • Extended Component Analysis: Splatting provides comparable perceptual quality to linear interpolation while reducing runtime from 5.71 s/frame to 1.36 s/frame.This corresponds to a 4.2× speedup.
  • Arc-Size Validation: Small camera steps yield more stable generation, while larger arcs increase perceptual and geometric errors; 15° balances stability and motion per chunk.The arc-size validation varies camera arcs from 10° to 40° across 20 MegaScene scenes.
  • Epipolar Attention α-Sensitivity: Generation quality remains relatively stable across tested epipolar-attention blending weights α ∈ [0, 1].The sensitivity study uses 20 scenes.
  • Depth Corruption Sensitivity: Increasing depth error affects performance more than increasing the fraction of missing pixels across the tested corruption range.The complete study varies both depth-inaccuracy severity and missing-pixel fraction.

10. Geometric and Motion Metrics

The paper evaluates perceptual, camera, geometric, and motion-consistency metrics, alongside depth corruption, warping efficiency, and qualitative stability. Together, these analyses characterize spatial and temporal behavior without requiring ground-truth 3D geometry.

  • Perceptual and Camera Metrics: LPIPS, CLIPSIM, Frobenius Norm, Rotation Angle Difference, and Angular Consistency measure perceptual and camera-based consistency, with lower values preferred for most metrics and higher CLIPSIM preferred.The protocol evaluates temporal and reference-frame consistency together with pose stability.
  • Depth Corruption Sensitivity: Depth-inaccuracy severity affects LPIPS-next more than missing-pixel fraction, whose variation changes LPIPS-next by less than 0.003 at fixed severity.The sweep uses n = 20 scenes per severity × hole-fraction cell on MegaScene.
  • Extended CamTrol Comparison: CamTrol++ provides stronger temporal and camera trajectory consistency than CamTrol while substantially reducing runtime on 28-frame RealEstate10K and MegaScene sequences.The comparison uses two 14-frame split-trajectory segments.
  • Warping Efficiency: Splatting-based warping reduces warping time from 5.71 s/frame to 1.36 s/frame while maintaining comparable perceptual quality to linear interpolation.This corresponds to a 4.2× speedup for the warping stage.
  • Geometric Consistency: CamTrol++ produces highly consistent, parallel feature tracks with alignment scores of 96–99% and low error, compared with WAVE’s 70–73% alignment and high error.The tracks respect the scene’s perspective in the CamTrol++ visualization.
  • Metric Coverage: EAE and Sampson Distance assess epipolar geometric consistency, while FPA and ZMR assess temporal motion directly from generated videos.Together, these metrics capture spatial and temporal stability without requiring ground-truth 3D geometry.

11. Camera Parameter Visualization

The camera-trajectory visualization compares COLMAP-estimated poses with ground-truth normalized translations in the x–z plane. CamTrol++ follows the reference trajectory more closely than WAVE, supporting more stable camera motion.

  • Trajectory Encoding: The visualization plots normalized x and z translations from COLMAP-estimated poses against ground-truth orbital camera trajectories for representative MegaScene scenes.Height variation is omitted because poses are scale-normalized and vertical variation is minimal.
  • Trajectory Comparison: CamTrol++ closely follows the ground-truth trajectory, whereas WAVE shows greater deviation and irregular curvature in estimated camera poses.The qualitative trajectory comparison supports the reported Frobenius Norm, Rotation Angle Difference, and Angular Consistency metrics.

12. Additional Results and Details

Additional qualitative results test complex camera motions and stylized scenes, while also documenting failure cases. CamTrol++ maintains sharper structure and more stable motion in the reported comparisons, but depth ambiguity can still cause distortion.

  • Complex Camera Motion: Additional results cover dolly zoom-in, zoom-out, wavy, and other complex camera motions, with coherent and consistent multi-view generation reported for the method.The figures also include some failed cases.
  • Stylized and Challenging Scenes: On NeRF-LLFF scenes with complex camera motion and stylization, CamTrol++ maintains sharper structure and more stable motion than CamTrol and WAVE.A failure case shows noticeable distortion across all training-free methods under depth ambiguity.
  • Stylized Inputs: The stylized inputs are generated with Style-NeRF2NeRF before applying the same camera-controlled generation pipeline.The accompanying qualitative results are application-oriented stylized-scene demonstrations.
Loading 2609.03639v1…