Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models

• Single-image NVS is challenging because it requires accurate geometry transformation of observed parts and hallucination of unseen regions.The paper frames the task as a long-standing computer-vision problem with applications in content creation, robotics, and AR/VR.Source: §1 · p. 1
• Training-based methods optimize an instance-specific model, whereas training-free methods synthesize in one forward pass; Consistent 1-to-3 is training-free and is reported as uniquely combining six listed capabilities.The table's criteria include single or sparse inputs, multi-view consistency, unseen-region hallucination, open-set generalization, and avoiding instance-specific training.Source: Table 1 · p. 2, §1 · p. 2
• Zero123 produces high-quality, generalizable views but remains geometrically inconsistent, and the authors identify resource constraints in current NVS models.The authors conjecture that substantially more training data and larger pretrained models could mitigate the issue, while emphasizing architecture design.Source: §1 · p. 2
• Consistent 1-to-3 separates observed-region transformation from unseen-region generation using a deterministic Scene Representation Transformer and view-conditioned diffusion.Epipolar-guided and multi-view attention aggregate geometric and multi-view information; hierarchy generation supports long, consistent view sequences.Source: §1 · p. 2
• Across multiple datasets, Consistent 1-to-3 produces state-of-the-art synthesis quality and geometric consistency across camera viewpoints.The paper reports that its architecture significantly improves geometric consistency over the compared methods.Source: §1 · p. 2

Single Instance Training-based Reconstruction

• Instance-training methods progressively reduce required observations, but RealFusion can take more than 30 minutes to optimize a single model, limiting interactive use.Examples include RegNeRF with three images and single-image approaches such as SinNerf, RealFusion, and NeuralLift.Source: §2 · p. 3

Re-projection methods

• Re-projection methods use explicit geometry, while category-specific mesh priors can improve results but generalize poorly to open-set instances.The review also notes that PixelNeRF's reported training on 13 ShapeNet categories did not demonstrate learning from large open-set image collections.Source: §2 · p. 3

Latent methods

• Latent methods decode a learned code directly into novel views for faster rendering, but their open-set category generalization remains unshown.LFN introduced the approach, and SRT later reduced the required observations to a few shots.Source: §2 · p. 3

Diffusion Models for NVS

• Diffusion NVS conditions target-view generation on input views and camera parameters; Zero-1-to-3 [22] fine-tunes pretrained Stable Diffusion and reports relatively good generalizability.Earlier methods in the review were limited to their training data and had not demonstrated generalization to arbitrary image input.Source: §2 · p. 3

3D Diffusion Models

• 3D diffusion work generates point clouds and other 3D representations, with recent open-set efforts supported by the release of Objaverse [7].The review lists meshes, volumes, and neural implicit representations among the explored forms.Source: §2 · p. 3

3.1. Geometry-guided Novel View Synthesis

• The pipeline separates geometry-guided photometric warping of observed regions from diffusion-based hallucination of unseen regions.The Scene Representation Transformer handles warping, while a view-conditioned diffusion model generates unseen content.Source: §3.1 · pp. 3–4
• The SRT encodes posed input images into a latent scene representation and renders queried pixel colors by attending to that representation.The model replaces standard attention with epipolar attention and trains the SRT in image latent space for lower computation costs and compatibility with latent diffusion.Source: §3.1 · p. 4
• View-conditioned diffusion estimates the target-view distribution to recover fine detail blurred by the SRT’s pixel-level reconstruction objective.It is conditioned on an SRT-predicted target latent and the scene representation.Source: §3.1 · p. 4
• Epipolar attention reweights cross-view affinities using epipolar distance, incorporating relative-camera geometry into attention between source and target views.The method replaces cross-attention layers between source and target views in both the SRT and diffusion model.Source: §3.1 · p. 5

3.2. Improving Multi-view Consistency

• Multi-view attention jointly processes noisy views to aggregate information across viewpoints, while hierarchy generation conditions new views on the input and previously generated views.Hierarchical generation addresses unreliable epipolar matching when directly generating views far from the input viewpoint.Source: §3.2 · p. 5

3.3. 3D Reconstruction

• For 3D reconstruction, the method applies Score Distillation Sampling to optimize a NeRF-like representation using rendered views from random camera poses.The paper motivates reconstruction for AR/VR scenarios requiring accurate 3D objects for occlusion and lighting reasoning.Source: §3.3 · p. 5
• On Objaverse, the paper reports that its NVS results outperform competitive baselines in both realism and consistency.Figure 4 visualizes results across a range of methods.Source: Figure 4 · p. 6

4.1. Experimental Setting

• The evaluation uses Objaverse test views and out-of-distribution Google Scanned Objects, measuring PSNR, SSIM, LPIPS, and flow-warping error across 32-frame camera-circle videos.Baselines include Zero123, Image Variation, and an adapted Pose-Guided Diffusion model.Source: §4.1 · pp. 6–7

4.2. Novel View Synthesis

• On Objaverse, the method significantly improves across all metrics; on out-of-distribution Google Scanned Objects, it is best across datasets and metrics.The qualitative comparison also reports that it is the only approach achieving both fidelity and consistency.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 7
• Geometry guidance and joint multi-view modeling alleviate inconsistency across independently sampled views, while nearby views align with the input and unseen regions show greater variation.The Figure 5 comparison contrasts SRT's consistent but blurry views with diffusion baselines' sharper but inconsistent views.Source: §4.2 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 8

4.3. 3D Reconstruction

• Combining view-conditioned diffusion with NeRF-style reconstruction yields NeRF renderings and meshes extracted using DiffRast.The NeRF is optimized with the SDS loss.Source: §4.3 · p. 8, Figure 7 · p. 8

4.4. Ablation Study

• Epipolar guidance and input-output attention support similarity metrics, whereas multi-output attention and hierarchy generation improve consistency but slightly reduce reconstruction performance.The full model balances fidelity against consistency.Source: §4.4 · p. 8, Table 4 · p. 8

• Consistent 1-to-3 targets high-quality, 3D-consistent novel-view synthesis from a single image or a few images.It first uses a transformer to generate a blurry but geometry-correct image, then a diffusion model to finish the details.Source: §5 · p. 8
• The authors report significantly improved generated results from this generation progression and hope geometry constraints and representations will improve practical usage.Source: §5 · p. 8

• The supplementary material provides technical details, experiment details including metric computation and baseline reproduction, and additional qualitative results.It also directs readers to the video for an overview and more visualizations.Source: §6 · p. 12

Epipolar Attention

• For a target-view point, the method back-projects its 3D ray and projects two points on that ray into the source view to obtain the corresponding epipolar line.Source: §7 · p. 12

• Combining view-conditioned diffusion with NeRF-style reconstruction is described as enabling high-fidelity 3D reconstruction from single images.Source: Figure 8 · p. 12

Epipolar Attention

• The method inverts distances from patch centers to the epipolar line into a weight map and adds it to the original attention weights.Source: §7 · p. 12

UNet Block

• The UNet block applies self-attention across tokens from multiple output views, then epipolar cross-attention between output views and SRT input tokens.The block also includes a convolution layer and concatenates patch tokens with corresponding ray embeddings.Source: §7 · p. 12

Dataset

• Evaluation uses Objaverse renderings and splits supplied by Zero123, with 800k objects and 7k test objects, plus Google Scanned Object renderings for zero-shot evaluation.Source: §8 · pp. 12–13

Metrics

• For each object, evaluation generates a 32-frame video spanning 360 degrees at 10-degree elevation and radius 1.5, using the first GSO frame as source and subsequent frames as targets.Source: §8 · p. 13
• LPIPS is computed with ImageNet-pretrained VGG, while RAFT optical flow between consecutive frames is used for flow-warping error.Source: §8 · p. 13

Reproducing Baselines

• Zero123 is run from its official Objaverse-pretrained checkpoint at guidance scale 3.0 and 256 × 256 resolution; Point-E uses its official implementation and pretrained models.PoseDiff is reimplemented from its paper description with hyperparameters matched to the proposed approach.Source: §8 · p. 13