LRM: Large Reconstruction Model for Single Image to 3D

Yicong Hong; Kai Zhang; Jiuxiang Gu; Sai Bi; Yang Zhou; Difan Liu; Feng Liu; Kalyan Sunkavalli; Trung Bui; Hao Tan

Source-linked AI summary

LRM: Large Reconstruction Model for Single Image to 3D

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, Hao Tan

arXiv:2311.04400v2Nov 8, 2023cs.CVcs.AIcs.GRcs.LG

TL;DR

Single-image 3D reconstruction is difficult because unseen geometry is ambiguous and prior methods often relied on category-specific data or pretrained generative guidance. LRM uses a large transformer encoder-decoder to predict a triplane NeRF from one image, trained end-to-end on roughly one million 3D data items. It produces high-fidelity reconstructions across diverse inputs in five seconds, while remaining limited by blurry occluded textures and camera-parameter mismatch on cropped or resized images.

Problem
Single-image 3D reconstruction requires a generic approach because unseen geometry is ambiguous and many prior methods use limited or category-specific data.
Method
LRM uses a large transformer-based encoder-decoder with DINO image features to predict a triplane NeRF from a single image, trained end-to-end on large multi-view data.
Results
LRM reconstructs high-fidelity 3D shapes from diverse real-world, generated, and rendered images and produces a shape in five seconds without post-optimization.
Takeaways & Limitations
LRM demonstrates a scalable, data-driven route to single-image 3D reconstruction for diverse in-the-wild inputs.
Takeaways & Limitations
LRM can produce blurry textures in occluded regions and may suffer camera-parameter mismatch when test images are cropped or resized.

Abstract

from arXiv · show

We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds. In contrast to many previous methods that are trained on small-scale datasets such as ShapeNet in a category-specific fashion, LRM adopts a highly scalable transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field (NeRF) from the input image. We train our model in an end-to-end manner on massive multi-view data containing around 1 million objects, including both synthetic renderings from Objaverse and real captures from MVImgNet. This combination of a high-capacity model and large-scale training data empowers our model to be highly generalizable and produce high-quality 3D reconstructions from various testing inputs, including real-world in-the-wild captures and images created by generative models. Video demos and interactable 3D meshes can be found on our LRM project webpage: https://yiconghong.me/LRM.

1 INTRODUCTION

LRM addresses single-image 3D reconstruction with a scalable transformer architecture trained on large multi-view data. It predicts triplane NeRF representations efficiently and reconstructs diverse real, generated, and rendered inputs.

Motivation: Single-image 3D reconstruction is ambiguous, motivating generic and efficient methods beyond category-specific shape priors.The ambiguity of unseen geometry makes single-view reconstruction difficult, while earlier learning-based methods often relied on category data.
Approach: LRM uses a large transformer-based encoder-decoder to learn 3D object representations from a single image.The method regresses a NeRF in triplane form from image features produced by a pretrained DINO image encoder.
Approach: LRM combines a fully transformer-based pipeline, compact triplane representation, and simple novel-view image reconstruction losses for scalable training.The model avoids excessive 3D-aware regularization and delicate hyperparameter tuning, supporting varied multi-view datasets.
Scale and results: More than 500 million parameters and approximately one million diverse 3D shapes and video data support large-scale reconstruction.The training scale is substantially larger than that of recent methods using shallower networks and smaller datasets.
Scale and results: Five seconds is sufficient for LRM to produce a 3D shape without post-optimization.The paper reports high-fidelity reconstructions across real-world and generative-model images.

2 RELATED WORK

Prior single-image-to-3D methods span explicit and implicit representations and increasingly use pretrained models for semantic or multi-view guidance. LRM instead learns a data-driven reconstruction model from extensible rendered or captured image supervision without pretrained vision-language guidance.

Single-image reconstruction: Earlier reconstruction methods explored point clouds, voxels, meshes, SDFs, occupancy networks, and NeRFs.The field has used both explicit geometric representations and implicit functions for single-image reconstruction.
Pretrained-model guidance: Recent approaches use pretrained image or language models to introduce semantics and multi-view guidance for image-to-3D reconstruction.Examples include novel-view generation with diffusion models and text- or image-based guidance during shape creation.
LRM distinction: LRM learns to reconstruct arbitrary objects in the wild using minimal, extensible supervision from rendered or captured 2D images of 3D objects.Unlike methods in the related-work discussion, it does not rely on pretrained vision-language contrastive or generative models.
LRM distinction: LRM does not rely on pretrained vision-language contrastive or generative guidance.Its supervision consists of rendered or captured multi-view images associated with 3D objects.
Representation learning: Single-image reconstruction is ill-posed, so many encoder-decoder methods model the image-to-3D data distribution with compact latent codes.Expressive representations generally require capable networks and abundant 3D data, which has limited many methods to a few categories and coarse results.

3 METHOD

LRM encodes an input image, projects its features into a triplane through a transformer decoder, and decodes queried 3D features into NeRF color and density. Camera conditioning controls global orientation and distortion, while image features provide fine-grained geometric and appearance information.

Image encoder and decoder: LRM encodes the input image into patch-wise feature tokens before an image-to-triplane decoder projects them into 3D features.The decoder output is upsampled and reshaped into a triplane representation used to query 3D point features.
Image encoder: DINO features preserve structural and texture information useful for reconstructing 3D geometry and color.LRM uses the full feature sequence rather than only the ViT class token.
Camera conditioning: Camera features encode extrinsics, focal length, and principal point to condition the decoder on input-view geometry.The camera controls the orientation and distortion of the reconstructed shape, while image features carry fine-grained geometry and color.
Triplane-NeRF: A triplane stores three axis-aligned feature planes whose interpolated features are decoded into NeRF outputs.The three planes are queried at 3D points, and the resulting features are passed to an MLP for volumetric rendering.
Image-to-triplane decoder: Learnable spatial-positional embeddings are progressively updated by the image-to-triplane transformer through modulation and cross-attention.The decoder conditions on camera and image features, then upsamples its output to the final triplane.
Triplane-NeRF: The NeRF MLP predicts four outputs: three RGB color values and one density value.These outputs define the field used for volumetric rendering.

4 EXPERIMENTS

LRM is evaluated on diverse synthetic, real, rendered, and generated images, with visual comparisons, implementation details, and documented failure cases. It produces high-fidelity reconstructions and sharper, more consistent surfaces than One-2-3-45, but remains sensitive to occlusion and camera assumptions.

Evaluation: LRM is evaluated on Objaverse, MvImgNet, ImageNet, Google Scanned Objects, Amazon Berkeley Objects, real-world captures, and Adobe Firefly images.The evaluation includes arbitrary images from multiple datasets, newly captured images, and generated images.
Implementation details: LRM uses normalized camera poses, a ViT-B/16 DINO encoder, a transformer decoder, triplane dimension 80, and 128 sampled points per ray.The implementation specifies normalized cameras and the main encoder, decoder, triplane, and rendering dimensions.
Visualization: Figure 2 shows high-fidelity reconstructions across real, generated, and rendered inputs, including complex geometry, high-frequency texture, and semantically reasonable occluded regions.The examples include asymmetric objects whose unseen portions are inferred plausibly.
Comparison: LRM produces sharper details and more consistent surfaces than One-2-3-45 on examples selected from that method’s paper or demo page.The comparison also reports worse reconstructions from One-2-3-45 on two examples used in Figure 2.
Limitations: The method produces blurry textures in occluded regions and distorted shapes when its fixed camera assumptions are inaccurate.The paper attributes these failures to ambiguity in unseen regions and mismatches in camera intrinsics and extrinsics.
Training: Training uses 128 A100 GPUs, batch size 1024, 30 epochs, and three side views per sample for supervision.Each epoch balances one copy of Objaverse renderings with three copies of MvImgNet video frames.

5 CONCLUSION

The conclusion presents LRM as a large transformer-based model that learns an expressive 3D prior from million-scale data for single-image object reconstruction. It emphasizes efficient end-to-end learning and five-second high-fidelity reconstruction while identifying scaling as a future direction.

Conclusion: LRM learns an expressive 3D prior from a million 3D data to reconstruct objects from single images.The conclusion characterizes it as a large transformer-based framework.
Conclusion: LRM is fully differentiable, trains end-to-end with simple image reconstruction losses, and renders a high-fidelity 3D shape in five seconds.The conclusion links this efficiency to potential real-world applications.
Future directions: Future work includes scaling the model and training data through larger encoders, deeper decoders, and higher-resolution triplane representations.The paper presents these as directions enabled by the simple transformer-based design.

ETHICS STATEMENT

The ethics statement notes that LRM deterministically maps an input image to one 3D shape and could amplify unethical or misleading visual content into more convincing 3D disinformation.

Ethics: LRM deterministically infers the identical 3D shape from the same input image.This contrasts with models that synthesize varied content from language inputs.
Risks: Given an unethical or misleading image, LRM could produce unethical 3D objects or 3D disinformation more convincing than the input image.The statement notes that these reconstructions remain less realistic than real-world objects.
Training data: The training data mostly contain ethical content from Objaverse and MvImgNet.The statement identifies the datasets used for training.

REPRODUCIBILITY STATEMENT

The reproducibility statement describes LRM as assembled from publicly available codebases and trained on publicly available datasets, with detailed preprocessing, architecture, and training information.

Reproducibility: LRM integrates the public threestudio, x-transformers, and DINO codebases and uses public Objaverse and MvImgNet data.The statement names the principal software and datasets used.
Reproducibility: The paper provides comprehensive preprocessing, network architecture, and training details to facilitate reproducing LRM.

APPENDICES

The appendix describes NeRF and transformer components used in LRM, including triplane spatial encoding, differentiable volume rendering, and attention-based feature processing.

NeRF Representation: LRM predicts a compact triplane NeRF representation, whose spatial encoding has O(N^2) tokenization complexity instead of a voxel grid’s O(N^3).The triplane representation is used to encode spatially varying color and density fields.
NeRF Rendering: Differentiable volume rendering casts a ray through the NeRF and combines sampled point colors and densities to produce each rendered pixel.The rendering process supports optimization with image reconstruction losses.
Attention Mechanisms: The attention operator computes normalized relationships between input features and condition features, then returns a weighted combination of value vectors.Multi-head attention splits features into multiple heads, processes them independently, and concatenates their outputs.
Transformer Layers: Self-attention derives queries, keys, and values from the same input feature sequence, whereas cross-attention derives keys and values from condition vectors such as image features.Both layers apply multi-head attention followed by an output linear layer.
Transformer Components: The transformer decoder uses MLP layers for channel mixing, LayerNorm in a Pre-LN arrangement, and upsampled positional embeddings for higher-resolution inputs.The MLP uses two linear layers with GELU activation and an intermediate dimension four times the model dimension.

B TRAINING SETUP

This section covers training details and comparisons involving LRM’s large-scale architecture, compact triplane representation, and analysis protocol.

Training Setup: LRM training uses mixed-precision optimization with cosine scheduling, warm-up iterations, gradient clipping, and selective weight decay.The setup uses AdamW with β2=0.95, gradient clipping of 1.0, and weight decay of 0.05.
Comparison with SOTA: LRM is compared quantitatively with Point-E, Shap-E, and One-2-3-45 on Google Scanned Objects using rendered-view metrics.The compared methods generate point clouds, implicit-function parameters, or reconstructed multi-view images through different pipelines.
Comparison with SOTA: LRM uses 500 million learnable parameters and 1 million publicly accessible 3D data items, while Point-E and Shap-E use hundreds of millions of parameters and several million assets.The authors state that LRM does not have an advantage in network or dataset size over those methods.
Analyses: Analytic experiments evaluate data, model, and training choices with PSNR, CLIP-Similarity, SSIM, and LPIPS, using a smaller baseline model because the final model is costly to train.Table 2 compares the final model with this analysis baseline.

D.1 SYNTHETIC VS. REAL DATA

Combining synthetic Objaverse data with real MvImgNet video data produces better reconstruction results than training on either dataset alone.

Synthetic vs. Real Data: Removing real MvImgNet data causes an obvious drop across all evaluated metrics despite Objaverse containing three times more shapes.The authors suggest real data provide greater variation in lighting, target size, and camera poses.
Synthetic vs. Real Data: Combining Objaverse and MvImgNet leads to substantially better results than training on either dataset alone.The two datasets contribute synthetic 3D assets and real-world object videos, respectively.

D.2 NUMBER OF VIEWS IN TRAINING DATA

Increasing the number of training views generally improves reconstruction results, with gains saturating at 16 views.

Number of Views in Training Data: More training views lead to better results, while improvement saturates at 16 views and additional views do not worsen performance.The authors associate the gains with increased data diversity.

D.3 MODEL HYPER-PARAMETERS

The experiments examine how decoder depth, NeRF MLP size, triplane resolution, camera-pose processing, training views, rendering resolution, and the LPIPS objective affect reconstruction quality. They identify deeper decoders, moderate MLP depth, higher triplane resolution, normalized poses, more views, higher training resolution, and LPIPS supervision as beneficial choices.

Decoder depth: Deeper image-to-triplane decoders slightly improve scores, especially CLIP and LPIPS, indicating better representations for reconstructing higher-quality images.The experiment varies the number of cross-attention layers in the decoder.
NeRF MLP depth: Two to four NeRF MLP layers provide a practical sweet spot, while making the network very large is unnecessary.The triplane encodes shape information, leaving the MLP as a shallow projector to color and density.
Triplane resolution: Higher triplane resolution improves image quality, although querying finer image details would increase computational cost.The reported experiment upsamples the decoder’s 32^3 triplane with deconvolution layers.
Camera pose: Normalizing camera poses improves generalization by projecting images onto the triplane from a consistent direction.Unmodified poses produce the worst results, while random rotation improves results but is harder to learn than normalized poses.
Training views and resolution: More side-view supervision and higher training-image resolution improve reconstruction quality by supporting multi-view consistency and high-frequency detail learning.The experiments vary both the number of side views per sample and the rendering resolution used during training.
Training objective: Removing the LPIPS objective reduces CLIP-Similarity, SSIM, and LPIPS scores to 74.7, 76.4, and 29.4, respectively.The ablation demonstrates that LPIPS supervision substantially affects the reported results.