Source-linked AI summary
RealFusion: 360° Reconstruction of Any Object from a Single Image
Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, Andrea Vedaldi
TL;DR
Reconstructing a complete 360° object model from one image is severely ill-posed because the unseen geometry and appearance are underdetermined. RealFusion fits a neural radiance field using the input view, a diffusion prior conditioned through single-image textual inversion, and regularization, producing plausible reconstructions that match the input and achieve state-of-the-art comparisons.
Problem
A single image provides insufficient information for full 360° reconstruction, offering weak evidence about 3D shape and only one side of the object.
Method
RealFusion optimizes a neural radiance field with the input-image reconstruction loss and a diffusion prior on novel views, using a custom token learned through single-image textual inversion.
Results
RealFusion achieves state-of-the-art reconstruction results compared with prior monocular methods and produces plausible 3D models faithful to the input image.
Takeaways & Limitations
A pretrained 2D diffusion model can provide a conditional prior for plausible single-image 360° reconstruction without 3D-specific supervision.
Takeaways & Limitations
The approach does not work well out of the box and diffusion-based reconstructions cannot always match all input details.
Abstract
from arXiv · showhide
We consider the problem of reconstructing a full 360° photographic model of an object from a single image of it. We do so by fitting a neural radiance field to the image, but find this problem to be severely ill-posed. We thus take an off-the-self conditional image generator based on diffusion and engineer a prompt that encourages it to "dream up" novel views of the object. Using an approach inspired by DreamFields and DreamFusion, we fuse the given input view, the conditional prior, and other regularizers in a final, consistent reconstruction. We demonstrate state-of-the-art reconstruction results on benchmark images when compared to prior methods for monocular 3D reconstruction of objects. Qualitatively, our reconstructions provide a faithful match of the input view and a plausible extrapolation of its appearance and 3D shape, including to the side of the object not visible in the image.
1. Introduction
RealFusion addresses the ill-posed task of reconstructing a full 360° photographic model from one image by combining a neural radiance field with a conditioned diffusion prior. The method produces plausible reconstructions matching the input view and outperforms prior single-image reconstructors across in-the-wild and benchmark images.
- Motivation: A single image provides weak evidence about an object's 3D shape and reveals only one side, making full reconstruction severely ill-posed.The paper motivates using prior knowledge to supply information absent from the image.
- Method: RealFusion optimizes a neural radiance field to reconstruct the input image while constraining randomly sampled novel views with a diffusion prior.The approach combines the usual rendering loss with a DreamFusion-like prior objective.
- Method: Conditioning the diffusion model with a prompt engineered from random augmentations of the input image makes its imagined views sufficiently strong for meaningful 3D reconstruction.The prompt is designed to make the prior sample views that plausibly belong to the given object.
- Method: Regularizers for shading, texture dropout, normal smoothing, and coarse-to-fine optimization improve qualitative results, while InstantNGP enables reconstruction in hours instead of days.The paper also emphasizes an efficient implementation based on InstantNGP.
- Results: RealFusion surpasses previous single-image reconstructors quantitatively and qualitatively on random in-the-wild images and existing benchmark datasets.The comparison includes Shelf-Supervised Mesh Prediction, despite RealFusion using no category-specific 3D supervision.
- Results: The reconstructions are plausible and match the provided input image, although the diffusion prior cannot always reproduce every detail.The authors characterize the results as demonstrating the approach's viability while leaving room for future improvements.
2. Related work
Prior reconstruction methods often rely on multiple views, category-specific training, synthetic data, or vague conditioning. RealFusion instead extracts 3D information from a general 2D diffusion generator for single-view reconstruction without category restrictions.
- Image-based reconstruction: Classical photometric and geometric reconstruction methods typically rely on many views, whereas RealFusion targets reconstruction from one image.Neural radiance fields jointly model appearance and geometry but commonly assume dozens or hundreds of views.
- Few-view reconstruction: Few-view NeRF methods reduce image requirements using semantic compatibility or learned priors, but still require several input views, multi-view training, or category-specific models.These constraints apply across approaches such as NeRF-on-a-Diet and learned NeRF inference networks.
- Single-view reconstruction: Single-view reconstruction systems generally depend on multi-view training and object-category-specific models, often targeting simple synthetic objects or domains such as cars.Examples include volumetric methods trained mainly on ShapeNet, CodeNeRF, and AutoRF.
- RealFusion: Its method jointly optimizes fixed-view reconstruction and a diffusion-based prior over randomly sampled novel views.The method diagram highlights these two simultaneous objectives and custom-token conditioning from single-image textual inversion.
- RealFusion: Unlike related multi-view-trained methods tested on synthetic data, RealFusion keeps novel views implicit during reconstruction.The paper contrasts this design with methods that explicitly sample multiple views.
3. Method
RealFusion fits a neural radiance field to one input image while using a diffusion prior, optimized through single-image textual inversion, to constrain plausible novel views. Regularization and coarse-to-fine training improve the reconstructed geometry and appearance.
- Radiance fields: A radiance field maps each 3D point to opacity and color, and differentiable rendering enables optimization against image reconstruction error.The field represents both object shape and appearance; standard fitting minimizes L2 error over rendered and observed images.
- RealFusion: RealFusion jointly optimizes fixed-view reconstruction and an SDS-based diffusion-prior objective on randomly sampled novel views.The radiance field is trained to match the input image while the diffusion model constrains rendered views through a DreamFusion-like objective.
- Single-image textual inversion: Single-image textual inversion optimizes a new token from random augmentations of the input, using it as a proxy for unavailable multi-view information.The prompt is automatically instantiated in templates such as “an image of a ⟨e⟩”, while other text embeddings and model parameters remain frozen.
- Prompt conditioning: Using the learned token instead of a generic object prompt can preserve the input object across viewpoints, whereas generic prompting may produce a different backside instance.The paper illustrates this contrast with a fish reconstruction and reports that the optimized embedding captures detailed information.
- Regularization and efficiency: The method adds normal smoothing, texture dropout, geometry shading, and coarse-to-fine optimization to reduce artifacts and capture structure before fine detail.Normal regularization is performed in 2D because the authors found it reduced variance and produced superior results; InstantNGP supports training in hours instead of days.
4. Experiments
Experiments evaluate RealFusion across benchmark and in-the-wild images, ablations, qualitative comparisons, and failure cases. The method outperforms a category-specific baseline in aggregate while producing diverse but sometimes inconsistent unseen views.
- Qualitative comparison: RealFusion reconstructs CO3D objects at higher detail than the category-level baseline without using category-specific information.
- Quantitative comparison: RealFusion outperforms Shelf-Supervised in aggregate on F-score and CLIP-similarity across seven CO3D categories.Metrics are averaged over three images per category; higher values indicate better shape and appearance reconstruction.
- Ablations: Single-image textual inversion dramatically improves reconstruction by making unseen views resemble the specific input object rather than a generic category instance.The effect is illustrated by comparing standard prompts with prompts optimized for the input image.
- Qualitative results: Repeated reconstructions match the front view consistently but vary substantially on the unseen backside, reflecting multiple plausible solutions.
- Ablations: Normal smoothness and coarse-to-fine optimization reduce surface irregularities and low-level artifacts while improving visual realism.The normal regularizer also produces smoother, more realistic meshes; Stable Diffusion performs better than CLIP as the prior.
- Failure cases: Failure cases include non-convergent geometry, semi-transparent scenes, and copying the front view onto the back, sometimes producing the Janus problem.
5. Conclusions
RealFusion reconstructs plausible, faithful 360° object models from one image using a pretrained 2D diffusion prior and an efficient radiance-field representation. The authors identify specialized new-view synthesis and animated-scene reconstruction as future directions.
- RealFusion produces plausible 360° photographic reconstructions from single in-the-wild images that remain faithful to the input view.
- The method combines a prompt-conditioned diffusion prior with an efficient multi-scale radiance field and surface-smoothing regularization.
- Future work includes specializing the diffusion model for new-view synthesis and incorporating dynamics for animated 3D scenes.
A. Implementation Details
Implementation uses controlled rendering, camera sampling, shading, density initialization, and InstantNGP, with single-image textual inversion driven by heavy augmentations.
- Shading: The reconstruction view uses albedo shading, while randomly sampled prior views use stochastic albedo, diffuse, and textureless shading after an albedo-only warmup.
- Density bias: The implementation adds a small Gaussian density blob at the scene origin to assist early optimization.
- Camera: The reconstruction camera is generally placed 1.8 units from the origin at 15° elevation, with higher elevation used for some overhead scenes.
- Camera: Prior cameras are randomly sampled around the object across azimuth, elevation, and distance, with occasional samples near the reconstruction camera.
- InstantNGP: InstantNGP represents density and albedo with a 16-level multi-resolution feature grid trained coarse-to-fine.
- Single-image textual inversion: Single-image textual inversion optimizes a new diffusion text-encoder token to match the input image using heavy image augmentations.
B. Method diagram
The supplementary method diagram is accompanied by additional reconstruction examples showing plausible object geometries and backsides.
- The paper provides a diagram illustrating the RealFusion method.
- Additional examples show that the method often reconstructs plausible geometries and object backsides.
D. Additional Comparisons
RealFusion is compared with recent single-view reconstruction methods on synthetic and real-world evaluation settings. The comparisons include a test set favorable to methods designed for views near the reference image.
- Additional Comparisons: The authors also compare RealFusion with recent single-view methods on the synthetic NeRF lego scene using SinNeRF’s specialized test set.This comparison is described as additional evaluation against recent reconstruction methods.
- Additional Comparisons: The lego comparison uses 60 test views very close to the reference view, a setting favorable to depth-supervised methods such as DS-NeRF and SinNeRF.The test set is rendered by SinNeRF from the synthetic NeRF lego scene.
E. Text-to-Image-to-3D
The paper explores a two-step text-to-image-to-3D pipeline: users generate and select an image, then reconstruct it with RealFusion. This may offer easier control than direct text-to-3D generation.
- Text-to-Image-to-3D: The text-to-image-to-3D procedure first uses a text prompt to generate an image and then reconstructs that image with RealFusion.The paper presents examples of this process in Fig. 14.
- Text-to-Image-to-3D: Users can select a desired generated image and then lift it to 3D using RealFusion.The first-stage samples are already well aligned with their prompts, so single-image textual inversion is usually unnecessary in this setting.
F. Analysis of Failure Cases
The paper identifies three recurring failure cases: ill-defined neural-field geometry, disconnected floaters, and duplicated faces. Its techniques reduce but do not eliminate these failures.
- Failure cases: RealFusion’s textual inversion, normal smoothing, and coarse-to-fine training make the three failure cases less frequent and less severe, but they still occur on various images.The paper illustrates additional failure cases in Fig. 15.
- Neural fields lacking well-defined geometry: Semi-transparent neural fields can match the reference view while lacking well-defined geometry and producing implausible appearances from other viewpoints.This failure occurs occasionally with Stable Diffusion and LSDS and is extremely common with CLIP as a prior.
- Floaters: Floaters are disconnected scene parts near the camera that can make a render resemble the input image from the reference view.Image-specific prompts greatly alleviate floaters, but do not eliminate them entirely.
- The Janus Problem: The Janus problem produces reconstructions with two or more faces because the loss encourages renders from every view to resemble the input image.View-specific prompting partially alleviates this issue, but it still occurs.
G. Unsuccessful Experiments and Regularization Losses
The authors report several experiments that did not justify inclusion in the final method. These alternatives were ineffective, harmful, computationally costly, or unnecessary for the chosen objective and resolution.
- Unsuccessful Experiments and Regularization Losses: A diffusion-model reconstruction loss matched the input image only to a certain degree and did not match the evidence beyond that image.The attempted loss modified LSDS by comparing diffusion noise predictions for rendered and input images.
- Unsuccessful Experiments and Regularization Losses: 3D normal-smoothing regularizers were noisy and harmful to reconstruction quality, possibly because of the grid-based radiance field or finite-difference normal computation.The final normal-smoothing term instead operates in 2D using normals rendered via the NeRF equation.
- Unsuccessful Experiments and Regularization Losses: Monocular depth losses usually did not noticeably improve reconstruction quality and were sometimes harmful.The authors state that these results are not conclusive and could merit future work.
- Unsuccessful Experiments and Regularization Losses: LPIPS performed similarly to L2 while requiring more computation and memory, whereas SSIM alone worsened quality but worked well when combined with them.SSIM was omitted from the final objective for simplicity.
- Unsuccessful Experiments and Regularization Losses: Rendering above 96px or 128px produced no noticeable quality difference, so the main experiments used 96px resolution.The comparison reflects the resolution used for the Stable Diffusion prior.
- Unsuccessful Experiments and Regularization Losses: Adding DINO feature losses to the Stable Diffusion prior was explored as a way to encourage semantic similarity between novel-view renders and the input image.The passage describes maximizing cosine similarity between DINO image embeddings.
H. Links to Images for Qualitative Results
The qualitative results use primarily Co3D images, supplemented by web-sourced examples, and show reconstructions across reference and novel viewpoints, including synthetic-image inputs and failure cases.
- Qualitative results primarily use images from datasets such as Co3D, with a small number of uncurated web images also included.Links to all images are provided on the project website.
- Additional examples compare the input image, the reference-view reconstruction, and renders from novel viewpoints to demonstrate plausible object shapes.
- The method also reconstructs plausible object shapes from synthetic images generated directly from Stable Diffusion text prompts.
- Failure cases are shown using the same input, reference-view, and novel-view layout, with some examples exhibiting the Janus problem.
- Single-image textual inversion is evaluated by comparing standard text prompts with textual-inversion reconstructions on novel views of object backsides.
- Different random seeds can produce multiple reconstructions for one challenging input image, with outcomes ranging from reasonable shapes to failures.