RealFusion: 360° Reconstruction of Any Object from a Single Image

1. Introduction

• Single-image reconstruction is ill-posed because an image provides weak 3D shape evidence for only one side, motivating a statistical prior to fill in missing information.Although 2D diffusion models have strong image priors, training a 3D diffusion model is infeasible given the scarcity of 3D data relative to 2D images.Source: §1 · pp. 1–2
• Earlier 2D-generator approaches sampled views for reconstruction, but recent methods such as DreamFusion [33] generate from text rather than reconstructing a specified input image.Image conditioning is challenging because generators may fail to cover the requested image space; face GANs, for example, can generate high-quality images yet be difficult to invert [1].Source: §1 · p. 2
• RealFusion fits a neural radiance field to the input view while constraining random novel views with a diffusion prior prompted using random augmentations of the image.It adds geometry shading, texture dropout, surface-normal smoothing, and coarse-to-fine fitting; using InstantNGP enables reconstruction in hours rather than days with traditional MLP-based NeRFs.Source: §1 · p. 2
• RealFusion uses a pretrained 2D generator as an image-by-image prior, without 3D supervision or restrictions to specific object categories.The paper presents the method as extracting a 360° photographic reconstruction from a single object image and introduces a single-image variant of textual inversion.Source: §1 · p. 2
• RealFusion reports state-of-the-art results and surpasses previous single-image reconstructors quantitatively and qualitatively, including Shelf-Supervised Mesh Prediction.Its reconstructions plausibly match the input image, though the diffusion prior may not capture every detail.Source: §1 · p. 2

Image-based reconstruction of appearance and geometry

• Classical multi-view methods discard photometry, while NeRF-like models jointly represent appearance and geometry but typically assume dozens or hundreds of views; this work uses diffusion to dream up missing views.Some NeRF variants use SDFs to recover cleaner geometry [24] [48] [50].Source: §2 · p. 3

Few-view reconstruction

• Few-view methods reduce input needs through semantic view selection [17] or learned NeRF priors, but generally still require multiple test views, multi-view training data, or category-specific optimization.NeRF-on-a-Diet [17] generates random views and scores semantic compatibility using CLIP embeddings [35].Source: §2 · p. 3

Single-view reconstruction

• Single-image methods such as 3D-R2N2 [5], Pix2Vox, LegoFormer, CodeNeRF [19], and AutoRF rely on multi-view training and often target specific object categories.The earlier volumetric methods focus mainly on simple objects and synthetic data such as ShapeNet [4]; CodeNeRF predicts radiance fields including object photometry, while AutoRF focuses on cars.Source: §2 · p. 3

Extracting 3D models from 2D generators

• Prior 2D-generator-to-3D methods include GAN-based extraction [9] [30] [31] [54], CLIP-conditioned CLIP-Mesh [20] and Dream Fields [16], and diffusion-prior DreamFusion [33]; this work applies related ideas to single-view reconstruction.Unlike [53], which requires multi-view training, synthetic testing, and explicit view sampling, this approach keeps novel views implicit.Source: §2 · p. 3

3.1. Radiance fields and DreamFusion

• A radiance field maps 3D points to opacity and color, and differentiable rendering enables image fitting; fitting only one view severely overfits.RealFusion uses a diffusion prior to supply information missing from the single image, while DreamFusion’s high guidance weight limits output diversity and conflicts with reconstructing arbitrary objects.Source: §3.1 · pp. 4–5
• DreamFusion uses a prompt-conditioned diffusion model to score randomly rendered views and updates the radiance field with Score Distillation Sampling (SDS).It uses the denoiser as a frozen critic when guiding updates to the rendered view.Source: §3.1 · pp. 4–5

3.2. RealFusion

• RealFusion jointly optimizes a radiance field for reconstruction from the fixed input viewpoint and an SDS prior on randomly sampled novel views.The two objectives combine the observed image with diffusion-prior information about views absent from the input.Source: §3.2 · p. 5
• Single-image textual inversion learns an input-specific token from random image augmentations, used as pseudo-views while other text embeddings and model parameters remain frozen.The token is inserted into prompts such as “an image of a ⟨e⟩”; the procedure generalizes textual inversion [10] to single-image training.Source: §3.2 · p. 5
• RealFusion trains its InstantNGP radiance field coarse-to-fine, optimizing lower-resolution feature grids in the first half and all grids in the second.This schedule helps alleviate small surface irregularities while retaining efficient training and high-quality results.Source: §3.2 · p. 5
• RealFusion regularizes smoothly varying normals in 2D to reduce surface artifacts, and adds an L2 loss between rendered opacity and the input mask.The authors report that 2D normal regularization reduces variance and gives superior results to operating in 3D.Source: §3.2 · pp. 5–6

4.1. Implementation details

• Experiments largely use shared hyperparameters without per-scene optimization, with a few higher-angle camera cases assigned 30° or 40°.The stated exception applies to images whose camera angle was clearly above 15°.Source: §4.1 · p. 6

4.2. Quantitative results

• RealFusion outperforms Shelf-Supervised in aggregate across seven object categories, despite Shelf-Supervised using category-specific models.The comparison reports F-score for reconstruction shape and CLIP similarity for appearance, averaged over three images per category; no individual metric values are provided here.Source: Table 1 · p. 7, §4.2 · p. 7
• Shape evaluation aligns sampled reconstruction-mesh points to CO3D point clouds with scaling and ICP, while appearance evaluation compares generated views with the closest available CO3D view using CLIP similarity.The shape F-score uses a 0.05 threshold; appearance similarity is measured against the CO3D view with maximum similarity.Source: §4.2 · pp. 7–8

4.3. Qualitative results

• Repeated runs from one input preserve little variation on the object’s front but allow substantial variation on its unseen back.Figure 6 illustrates different backside textures among reconstructions that match the same reference view.Source: Figure 6 · p. 7, §4.3 · p. 8
• RealFusion sometimes fails to converge, producing ill-defined geometry, or copies the front appearance onto the back even when semantically incorrect.The latter failure is illustrated by a reconstruction with two Pikachu faces, one on each side.Source: §4.3 · p. 8, Figure 11 · p. 9

4.4. Analysis and Ablations

• Single-image textual inversion improves novel-view reconstruction by making the backside resemble the specific input object rather than a generic category instance.Without textual inversion, the backside may resemble a different, generic-looking object from the same category.Source: Figure 7 · p. 8, §4.4 · p. 9
• Normal smoothness regularization reduces surface artifacts, while coarse-to-fine optimization reduces low-level artifacts and produces smoother surfaces.The normal regularizer is also reported to improve surface appearance and, in most cases, overall shape realism.Source: Figure 9 · p. 8, Figure 8 · p. 8, §4.4 · p. 9
• Stable Diffusion produces higher-quality, more plausible object shapes than the alternative CLIP prior.The comparison is shown for the two priors used in reconstruction.Source: §4.4 · p. 9, Figure 10 · p. 9

Conclusions

• RealFusion produces plausible 360° photographic reconstructions of in-the-wild objects that remain faithful to the single input image.Source: §5 · p. 9
• The method prompts an off-the-shelf diffusion model trained only on 2D images to imagine other views, then fits a multi-scale radiance field with surface smoothing.The model requires no special 3D reconstruction supervision.Source: §5 · p. 9
• Future work could specialize the diffusion model for new-view synthesis and incorporate dynamics to reconstruct animated 3D scenes.Source: §5 · p. 9

Shading

• Reconstruction views use albedo shading; prior views use albedo for the first 1,000 steps, then stochastically use albedo, diffuse, and textureless shading with probabilities 0.2, 0.4, and 0.4.After the first 1,000 steps, ambient and point-light colors are set to 0.1 and 0.9, respectively.Source: A. Implementation Details · p. 12

Camera, lighting, and view-dependent prompt

• Prior cameras are sampled around the object with randomized distance, azimuth, elevation, and field of view; every tenth iteration, the camera is placed near the reconstruction view.Reconstruction views use a camera distance of 1.8 and elevation of 15° (or 40° for some overhead scenes); point-light position is perturbed from the prior camera, and prompt suffixes reflect the prior view direction.Source: A. Implementation Details · p. 12

InstantNGP and rendering

• InstantNGP uses a 16-level, 2-dimensional feature grid in a 0.75-side-length bounding box, with only its eight coarsest levels active during the first half of training.The stacked grid features feed a three-layer MLP with 64 hidden units; renders are produced at 96px and upsampled to 512px for the Stable Diffusion [41] latent encoder.Source: A. Implementation Details · p. 12

Rendering and diffusion prior

• The diffusion prior uses latent noise sampled from t ∼ U(0.02, 0.98) and classifier-free guidance strength 100; gradients pass through the latent encoder but not the Stable Diffusion UNet.Guidance strengths above 30 produced good results, while strengths below 30 led to more geometric deformities.Source: A. Implementation Details · p. 12

Optimization and regularization

• Optimization runs for 5,000 iterations with Adam [21] at learning rate 1e−3, while a direction-conditioned two-layer background MLP is kept deliberately weak.Density regularization includes Ref-NeRF’s orientation loss [49] (λorient = 0.01) and an entropy loss encouraging points to be fully transparent or opaque.Source: A. Implementation Details · p. 12

Single-image textual inversion

• Single-image textual inversion optimizes an object token for 3,000 steps using heavy augmentations, 512px images, batch size 16, Adam, and learning rate 5 · 10−4.The token can be initialized randomly, manually, or automatically; the automated method selects the noun token whose “An image of a ⟨token⟩” caption best matches the input image embedding.Source: A. Implementation Details · p. 13

B. Method diagram

• Figure 2 illustrates the method.Source: B. Method diagram · p. 13

C. Additional Qualitative Examples

• Additional examples in Figure 13 show that the method often reconstructs plausible geometries and object backsides.Source: C. Additional Qualitative Examples · p. 13

D. Additional Comparisons

• Unlike methods designed for this setting, RealFusion is not tailored to near-reference views, which favor depth-supervised methods such as DS-NeRF and SinNeRF.Some competing methods warp the input image, an approach that performs well only for views close to the reference.Source: D. Additional Comparisons · p. 13, Table 2 · p. 13
• The comparison uses SinNeRF’s 60-view test set of views close to the reference image on the synthetic NeRF lego scene [27].The table describes this as a novel-view-synthesis comparison.Source: Table 2 · p. 13, D. Additional Comparisons · p. 13

E. Text-to-Image-to-3D

• The method generates an image from a text prompt and reconstructs that image in 3D with RealFusion.The section shows examples of text-to-image-to-3D generation in Fig. 14.Source: E. Text-to-Image-to-3D · p. 13
• Compared with one-step text-to-3D [33], users may have more control by sampling images from a 2D diffusion model, selecting one, and lifting it to 3D with RealFusion.The authors suggest this workflow could help address the diversity-of-generation issue discussed in [33].Source: E. Text-to-Image-to-3D · pp. 13–14
• Single-image textual inversion is usually unnecessary because first-stage sampled images are already extremely well-aligned with their prompts.Source: E. Text-to-Image-to-3D · p. 14

• Single-image textual inversion, normals smoothing, and coarse-to-fine training make these failure cases less frequent and less severe, but do not prevent them across images.Source: F. Analysis of Failure Cases · p. 14

Neural fields lacking well-defined geometry

• Semi-transparent neural fields can match the input from its reference view yet lack plausible geometry from other views; this is extremely common with CLIP and occasional with Stable Diffusion and LSDS.Source: F. Analysis of Failure Cases · p. 14

Floaters

• Floaters—disconnected scene parts near the camera—can mimic the input in front of the reference view; they occur in most reconstructions without image-specific prompts, which greatly but incompletely reduce them.Source: F. Analysis of Failure Cases · p. 14

The Janus Problem

• Janus failures—reconstructions with multiple faces—arise because the loss encourages every view to resemble the input; view-specific prompts mitigate but do not eliminate them, and [33] also exhibits them.Source: F. Analysis of Failure Cases · p. 14

G. Unsuccessful Experiments and Regularization Losses

• A diffusion-model reconstruction loss recovered the input image only partly, failing to match its exact colors or textures.The modified LSDS compared predicted noise for a noisy rendered image with predicted noise for a noisy input image.Source: G. Unsuccessful Experiments and… · p. 14
• 3D normal-smoothing regularizers were noisy and harmed reconstruction quality, possibly because of the grid-based radiance field or finite-difference normals.The final method instead smooths normals in 2D using normals rendered via the NeRF equation.Source: G. Unsuccessful Experiments and… · p. 14
• MiDaS-based depth matching [37] usually did not noticeably improve reconstruction and sometimes harmed it, though the authors considered these results inconclusive.The loss matched depth rendered from the reference view to depth predicted for the input image.Source: G. Unsuccessful Experiments and… · p. 14
• LPIPS performed similarly to L2 but required more computation and memory; SSIM alone worsened quality, while adding it to L2 or LPIPS yielded fine results.SSIM was excluded from the final objective for simplicity.Source: G. Unsuccessful Experiments and… · p. 14
• Rendering above 96px or 128px showed no noticeable quality difference, so the main-paper experiments used 96px for computational efficiency.Source: G. Unsuccessful Experiments and… · p. 14
• DINO feature losses added to the Stable Diffusion prior neither noticeably improved nor degraded performance, so the authors omitted them for simplicity [2].They maximized cosine similarity between DINO embeddings of novel-view renders and the reference image.Source: G. Unsuccessful Experiments and… · p. 15

• Images for qualitative results come primarily from Co3D, with a small number sourced from the web to demonstrate performance on uncurated data; links are provided on the project website.Source: H. Links to Images for Qualitative… · p. 15
• Additional examples show input images, reference-view reconstructions, and novel-view renders that demonstrate plausible reconstructed object shapes.Source: Figure 13 · p. 16
• Images generated from text prompts with Stable Diffusion [41] also yield plausible object-shape reconstructions.Prompts included animals and a “pencil drawing of a horse.”Source: Figure 14 · p. 17
• Additional failure cases are exposed by novel-view renders, with some examples, including a panda bear and hamster, suffering from the Janus problem.Source: Figure 15 · p. 18
• Single-image textual inversion produces less-generic back-side novel views than standard text prompts in the illustrated comparisons.Source: Figure 16 · p. 19
• For a challenging Stable Diffusion image of an astronaut riding a horse, two random seeds yield different outcomes: one reasonable shape and one failure.The example illustrates reconstruction variability and motivates improving consistency and quality.Source: Figure 17 · p. 20