Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors

1 Introduction

• Single-image 3D-aware editing remains a missing piece in AI image-editing tool chains, while constructing 3D scenes for manipulation requires considerable time.The introduction illustrates the task with moving, rotating, and scaling a handcart in a rendered alley scene.Source: §1 · pp. 1–2
• Earlier 3D-aware editing work explored GAN latent spaces and disentangled GANs, while limited GAN modeling capacity constrained applicability [38].Diffusion editors trained on synthetic manipulation pairs also faced reduced effectiveness on varied open-domain layouts and styles [34] [70].Source: §1 · pp. 2–3
• The tuning-free approach uses pre-trained large-scale diffusion models for open-domain editing without multiview-dataset training, apart from simple single-image LoRA training.The authors note that multiview datasets are usually limited or biased in distribution.Source: §1 · p. 3
• An iterative depth-warp-assisted algorithm uses diffusion priors both to complete target-view texture and to correct geometric distortions from inaccurate depth.Diffusion-based undistortion refines the original depth map and aligns it with the target-view depth.Source: §1 · p. 3
• The method produces texture-consistent, shape-consistent edits with high image quality and visual harmony, including under open-domain use.The authors report results not achieved by previous methods and an implemented interactive editing system.Source: §1 · pp. 3–4

Generative image editing and 3D-aware image editing

• GAN-based 3D-aware editing used latent-space manipulations or disentangled attributes, but limited GAN capacity restricted applicability or image categories [38].Source: §2 · p. 4
• Diffusion-based 3D-aware editors used specialized models trained on synthetic image pairs, whose domain gap from real-world images often limits applicability [34] [70].Tuning-free diffusion editing methods use pre-trained models without additional fine-tuning [35] [36] [52].Source: §2 · p. 4
• The proposed method applies tuning-free, large-scale diffusion foundation models to 3D-aware editing and therefore exhibits significant generalization across open-domain images.Source: §2 · p. 4

Novel view synthesis

• Novel-view synthesis methods trained on limited real-world multiview images or synthetic renderings have constrained generalization, while score-distilled radiance-field methods can take hours and may lose visual quality [32] [51] [65] [68].Source: §2 · pp. 4–5

3 Approach

• The approach iterates through view synthesis, undistortion, and shape alignment, gradually reducing the undistortion noise level.The three phases use appearance and geometric priors from pretrained text-to-image diffusion models.Source: §3 · pp. 5–6

3.1 View Synthesis

• Depth-assisted warping preserves visible texture under the instructed transformation, while layered inpainting completes the background and ambiguous regions.The method first fills the background occluded by the foreground object, then uses diffusion-based inpainting to complete the transformed view.Source: §3.1 · pp. 6–7

3.2 Undistortion

• Undistortion uses LoRA conditioning and DDIM inversion to correct depth-induced distortions while preserving appearance consistency.The maximum noise level controls how much content may change: higher values allow larger corrections but can reduce texture consistency.Source: §3.2 · pp. 7–8

3.3 Shape Alignment

• Shape alignment adjusts the original depth map by minimizing reprojection error between source and undistorted target-view correspondences.The pre-undistortion warped view bridges correspondence matching between the source and distortion-corrected target view.Source: §3.3 · p. 8

3.4 The Overall Algorithm

• The overall algorithm repeats view synthesis, undistortion, and shape alignment while progressively lowering the maximum noise level.The passage reports a gradual reduction in geometric distortion and an increase in consistency with the original image.Source: §3.4 · p. 9

4 Experiments

• The benchmark contains 80 manually collected 512 × 512 images spanning over 10 categories, with 160 instructed rotation and translation edits.The experiments also report about 20 seconds per editing operation on one NVIDIA Tesla A100 GPU, plus roughly 10 seconds for LoRA preparation.Source: §4 · p. 10

4.1 Visual Results

• Visualizations show depth-warping and inpainting distortions being corrected, with later iterations producing less distorted warps and greater consistency with the input.The method also produces high-quality edits under large viewpoint transformations while preserving layout plausibility and input consistency.Source: §4.1 · pp. 10–11

4.2 Comparison with Previous Methods

• Compared with prior methods, our edits show more plausible layouts, fewer visual artifacts and less blurriness, and stronger consistency in detailed textures.OBJect-3DIT fails on many test cases, while Zero123 struggles with complex scenarios; explicit warping supports detailed appearance consistency.Source: §4.2 · pp. 11–12
• Our method achieves the highest scores across all four reported appearance-consistency metrics.The metrics include perceptual similarity, LPIPS distance after flow warping, mean confidence, and confident-region area.Source: Table 1 · p. 12, §4.2 · pp. 12–13
• 71.36% of preferences from 33 participants favored our results, more than 5× the second-best performer, Stable Zero123.Participants evaluated layout plausibility, image quality, and appearance consistency.Source: Figure 6 · p. 12, §4.2 · p. 13

4.3 Ablation Study

• Appearance consistency improves across iterations of novel view synthesis and shape alignment.The text reports that this quantitative trend matches the visual assessment of progressively improved results.Source: Table 2 · p. 13, §4.3 · p. 13
• Removing LoRA harms distortion correction and appearance preservation, while removing DDIM inversion harms appearance consistency; combining both gives the best undistortion results.The ablation evaluates each modification separately under a single iteration.Source: §4.3 · p. 13

5 Limitations

• Preserving extremely fine input details is constrained by the pretrained base diffusion models.The method also does not guarantee physical correctness of lighting and shadow.Source: §5 · p. 14
• Under extremely large transformations, inpainting failure and substantial depth-estimation errors can prevent realistic layout correction.The passage links these problems to small visible areas and errors that hinder the successive undistortion phase.Source: §5 · p. 14
• Small, incremental edits can mitigate large-transformation issues, but extend editing duration.Improving speed and robustness in these challenging scenarios is identified as future work.Source: §5 · p. 14

• The method combines pre-trained image diffusion priors with iterative depth-assisted novel-view synthesis and depth alignment for single-image 3D editing.Source: §6 · p. 14
• Experiments demonstrate edits with superior quality and visual fidelity that significantly outperform prior techniques.Source: §6 · p. 14
• The authors identify graphic design, augmented reality, and video game development as potential application areas.Source: §6 · p. 14

• The top row shows horizontal rotation from −30◦ to 30◦.Source: Figure 9 · p. 20
• The bottom row shows lateral 3D translation from further left to closer right.Source: Figure 9 · p. 20
• The figure presents editing results under interpolated pose sequences.Source: Figure 9 · p. 20

• Results exhibit high shape and texture consistency under both rotation and translation sequences.Source: §A.1 · p. 20
• The sequences use interpolated transformations between target transformations in opposite directions.Source: §A.1 · p. 20
• The sequences are presented for visual examination of shape and appearance consistency.Source: §A.1 · p. 20

• Compared qualitatively with the paper’s results, Make-IT-3D and DreamGaussian show blurriness and noticeable artifacts.Source: §A.2 · p. 21
• SDS-based methods typically take several minutes to hours, which is not applicable to interactive editing requiring timely feedback.Source: §A.2 · p. 21
• The comparison evaluates Make-IT-3D and DreamGaussian at 5122 resolution.Source: §A.2 · p. 21

A.3 Results and Comparison on the Full Benchmark

• The comparison covers the whole evaluation set and all four methods used in the main paper.The passage says the results appear in a separate supplementary document.Source: §A.3 · p. 21
• A single set of hyperparameters is used to generate all benchmark results.Source: §A.3 · p. 21
• The displayed columns show the input image, 3D edit with source and target boxes, then OBJect-3DIT, Zero123, Stable-Zero123, and the proposed method.Source: §A.3 · p. 21

B.1 View Synthesis

• Depth warping converts selected object pixels into a 3D point cloud and mesh, rasterizes from the target camera, and masks pixels exceeding a stretch threshold.The stretch ratio is the inverse of the smaller singular value of the source-to-target image-coordinate Jacobian.Source: §B.1 · p. 21
• Layered inpainting uses Stable Diffusion Inpaint and depth ControlNet for background completion, then inpaints disparity gradients and solves a Poisson problem to complete disparity.Source: §B.1 · pp. 21–22

B.2 Undistortion

• Undistortion uses Stable Diffusion v1.5 with a rank-16 LoRA, DDIM inversion and sampling, and spatially masked DDIM noise to preserve background while correcting foreground.The LoRA is trained for 60 steps at learning rate 5 × 10−4 with batch size 4.Source: §B.2 · p. 22

B.3 Shape Alignment

• Shape alignment minimizes reprojection error with gradient regularization to suppress outliers, solving the sparse nonlinear least-squares problem with SciPy.Source: §B.3 · p. 22

B.4 Zero123 and Stable Zero123

• Zero123’s 2-DoF rotation input is adapted to 6-DoF edits with object centering and image-space translation, rolling, and scaling; Stable Zero123 also receives absolute source elevation.Source: §B.4 · pp. 22–23

B.5 ObJect3DIT

• OBJect3DIT’s separate pretrained translation and rotation models are applied sequentially, with BLIP-2-generated object descriptions used in prompts.Source: §B.5 · p. 23

B.6 User Study Interface

• User-study participants saw four methods in randomly shuffled order and selected the one they judged best; Fig. 11 shows the interface.Source: §B.6 · p. 23, Figure 11 · p. 23