Source-linked AI summary
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation
Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, Gordon Wetzstein
TL;DR
GRM targets efficient, high-quality 3D asset reconstruction and generation, addressing slow optimization and resolution-constrained triplane methods. It uses a transformer-based sparse-view reconstructor that converts multi-view pixels into pixel-aligned 3D Gaussians, achieving state-of-the-art quality and speed while supporting text-to-3D and image-to-3D workflows.
Problem
Existing 3D asset creation is laborious, while optimization-based generation is slow and triplane-based feed-forward models rely on inefficient volume rendering and reduced 3D resolution.
Method
GRM uses a pure transformer architecture to translate sparse multi-view pixels into pixel-aligned Gaussians, which are unprojected into densely distributed 3D Gaussians.
Results
GRM achieves state-of-the-art quality and speed for sparse-view 3D reconstruction and supports text-to-3D and image-to-3D generation with multi-view diffusion models.
Takeaways & Limitations
The framework provides an efficient reconstruction core that can be combined with existing multi-view generators for text- and image-conditioned 3D generation.
Takeaways & Limitations
The framework is limited to object-centric scenes because large-scale 3D scene datasets are unavailable, and reconstruction quality suffers with inconsistent input views.
Abstract
from arXiv · showhide
We introduce GRM, a large-scale reconstructor capable of recovering a 3D asset from sparse-view images in around 0.1s. GRM is a feed-forward transformer-based model that efficiently incorporates multi-view information to translate the input pixels into pixel-aligned Gaussians, which are unprojected to create a set of densely distributed 3D Gaussians representing a scene. Together, our transformer architecture and the use of 3D Gaussians unlock a scalable and efficient reconstruction framework. Extensive experimental results demonstrate the superiority of our method over alternatives regarding both reconstruction quality and efficiency. We also showcase the potential of GRM in generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with existing multi-view diffusion models. Our project website is at: https://justimyhxu.github.io/projects/grm/.
1 Introduction
GRM addresses the slow, resolution-limited reconstruction and generation of 3D assets with a fast feed-forward model based on pixel-aligned 3D Gaussians and transformers. It supports sparse-view reconstruction and, with multi-view generation tools, text-to-3D and image-to-3D generation.
- Motivation: High-quality 3D assets remain costly to create manually, despite their importance across robotics, gaming, and architecture.Traditional creation requires expertise in difficult-to-use computer graphics tools.
- GRM: ∼0.1s: GRM is trained for fast sparse-view reconstruction while producing high-fidelity 3D assets.The model is designed to use sparse but well-distributed views rather than synthesize all missing regions from incomplete observations.
- Motivation: Triplane-based feed-forward models require inefficient volume rendering, hindering inference speed and often limiting 3D resolution.Optimization-based methods can also require hours to produce one asset.
- GRM: GRM replaces triplanes with 3D Gaussians and uses a pure transformer to translate input pixels into pixel-aligned Gaussians for 3D reconstruction.Its upsampler uses windowed self-attention to pass non-local cues efficiently and recover high-frequency appearance details.
2 Related Work
Related work spans sparse-view neural reconstruction, feed-forward and optimization-based 3D generation, and generalizable Gaussian representations. GRM differs from concurrent Gaussian methods through its transformer-based encoder and upsampler.
- Sparse-view Reconstruction: Sparse-view neural rendering methods improve generalization to unseen scenes but face challenges in capturing multiple modes within 3D scenes.
- 3D Generation: Feed-forward 3D generation is faster than optimization-based approaches, whereas score-distillation methods can require hours to converge.Directly trained 3D diffusion models may lack pretrained 2D diffusion priors or suffer from 3D inconsistency.
- Generalizable Gaussians: Concurrent Gaussian feed-forward models use 3D Gaussians, but GRM uses a purely transformer-based encoder and efficient upsampler instead of a convolutional U-Net.The architecture generates many pixel-aligned 3D Gaussians and is described as offering superior reconstruction quality.
- Generalizable Gaussians: 3D Gaussians support efficient high-fidelity reconstruction from posed images using a moderate number of Gaussians and have been applied to image-, text-, avatar-, dynamic-scene, and 4D generation.Prior work in this area focuses on single-scene optimization.
3 Method
GRM reconstructs a 3D scene from four views by combining a pure transformer architecture with pixel-aligned 3D Gaussians. Its encoder aggregates information across views, while the upsampler restores resolution for dense Gaussian prediction and rendering.
- Overview: GRM uses four input images to infer the underlying 3D scene and can integrate with multi-view image generators for text-to-3D or image-to-3D generation.Instant3D and Zero123++ provide multi-view images from text prompts or single images, respectively.
- Pixel-aligned Gaussians: Pixel-aligned Gaussian attribute maps are predicted per view and unprojected along viewing rays into V × H × W densely distributed 3D Gaussians.Each map has 12 channels covering depth, rotation, scale, opacity, and the DC spherical-harmonics term.
- Transformer-based Encoder: The transformer encoder tokenizes all views, concatenates their features, and applies self-attention across every input-view token to exchange mutual information.Camera information and learnable image-position encodings are injected before the attention layers.
- Transformer-based Upsampler: A transformer-based upsampler progressively doubles spatial resolution using PixelShuffle and shifted window attention to balance detail reconstruction with feasible computation.Shifted windows improve non-local information flow while keeping attention computation manageable.
- Rendering and Training: Separate linear heads predict Gaussian attributes from upsampled features, after which Gaussian splatting renders arbitrary-view images and alpha masks for supervision.Training samples four sufficiently covering input views, supervises additional views, and uses object masks to remove floaters.
- Training and Gaussian Constraints: The scale output is constrained by interpolating between predefined s_min and s_max values through a sigmoid activation.This activation is applied to the scale head output before Gaussian rendering.
4 Experiments
Experiments evaluate GRM on sparse-view reconstruction, single image-to-3D generation, text-to-3D generation, and component ablations. GRM achieves strong reconstruction quality and fast inference across these settings, while ablations identify important design choices.
- Sparse-view Reconstruction: GRM significantly outperforms all sparse-view reconstruction baselines across PSNR, SSIM, and LPIPS, despite SparseNeuS and IBRNet using four times more input views.GRM is among the two fastest methods and predicts 16 times more Gaussians than concurrent LGM, yielding higher reconstruction fidelity.
- Sparse-view Reconstruction: GRM reconstructs geometric structures without visible floaters and captures finer appearance details than competing methods in novel-view renderings.The comparison uses the same sparse-view inputs for the visualized results.
- Single Image-to-3D Generation: GRM outperforms all single-image-to-3D baselines across the reported metrics while generating 3D Gaussians in 5 seconds.This runtime includes the generation head; TriplaneGaussian is faster but produces lower reconstruction quality.
- Single Image-to-3D Generation: GRM produces sharper details, more accurate geometry, and better consistency with the input image than Gaussian- and non-Gaussian-based single-image methods.Competing methods exhibit blurry textures, geometry artifacts, or inconsistencies with the input.
- Ablation Study: Linear interpolation for Gaussian scales improves appearance metrics over exponential activation, which can produce overly large Gaussians and unstable training.The exponential activation is associated with blurry images in the ablation discussion.
- Ablation Study: Increasing upsampler blocks from 0 to 3 improves performance, benefiting from detailed spatial features produced by the transformer-based upsampler.The growing number of blocks also increases Gaussian density and modeling capability.
- Ablation Study: Alpha regularization removes floaters around objects and improves reconstruction metrics.The regularization supervises the alpha map using object masks during training.
5 Discussion
GRM is presented as a feed-forward model for efficient, high-quality sparse-view 3D reconstruction and generation. The discussion identifies limitations involving inconsistent views, deterministic inference, object-centric scope, and misuse risks.
- Discussion: GRM combines a transformer-based sparse-view reconstructor with pixel-aligned Gaussians to reconstruct 3D objects efficiently.The model reconstructs objects from sparse views and represents outputs with pixel-aligned Gaussians.
- Discussion: The authors report state-of-the-art quality and speed for 3D reconstruction and generation.
- Limitations and Future Work: Input-view inconsistency reduces the output quality of the sparse-view reconstructor.
- Limitations and Future Work: GRM is deterministic and currently limited to object-centric scenes because large-scale 3D scene datasets are unavailable.Future work could investigate probabilistic reconstruction and larger, more complicated scenes.
- Ethics: The authors caution that generative models can pose societal threats when used to create deep fakes intended to spread misinformation.
A Implementation Details
The implementation uses memory-saving training techniques and evaluates alternatives for Gaussian scale activation, upsampling, and perceptual loss. These choices address memory use, image quality, and training stability.
- Implementation Details: Deferred backpropagation increases the training batch size from 2 to 8 on 80GB A100 GPUs while using 38GB per GPU.The model generates 4 × 512 × 512 Gaussians, creating substantial GPU-memory demand.
- Perceptual loss: LPIPS causes severe training instability, preventing the model from converging well.
B Geometry Evaluation
The geometry evaluation compares GRM with reconstruction and generation baselines using Chamfer Distance and F-score. The reported results favor GRM across the evaluated geometry metrics and settings.
- Evaluation Protocol: The evaluation reports Chamfer Distance and F-score, using multiple F-score thresholds to reduce evaluation uncertainty.ICP alignment registers all shapes into a common canonical space, and metrics are evaluated at the original GSO scale.
- Sparse-view Reconstruction: GRM achieves better geometry scores than SparseNeuS and LGM across all metrics in sparse-view reconstruction.The improvement is particularly evident for F-score at small thresholds; SparseNeuS has far-away floaters that inflate CD.
- Single-Image-to-3D Generation: GRM outperforms all baseline methods across all geometry metrics for single-image-to-3D generation.The original One-2-3-45++ implementation can generate only a single component in multi-object scenes.
- Comparison with Optimization-based Methods: Compared with optimization-based methods such as DreamGaussian, GRM provides runtime advantages together with superior geometry quality.
C Mesh Extraction from 3D Gaussians
GRM meshes are extracted by rendering the Gaussian scene from many cameras, fusing RGB-D data into a TSDF volume, and removing small external floaters.
- Mesh Extraction: The extraction process renders RGB images and depth maps from 200 uniformly distributed spherical cameras.The cameras are sampled using the Fibonacci method.
- Mesh Extraction: The rendered RGB-D data are fused with TSDFVolume to generate a mesh.
- Mesh Extraction: Clustering removes very small floaters outside the object surface caused by the Gaussian distribution.This post-processing step smooths the extracted geometry.
D Additional Visual Results
The paper includes additional visual results covering sparse-view reconstruction, text-to-3D generation, and image-to-3D generation.
- The extracted texture mesh is assembled in Blender to construct a 3D scene.
- Figure 1 presents the constructed scene image.
- Figures 2, 3, and 4 show qualitative results for sparse-view reconstruction, text-to-3D generation, and image-to-3D generation, respectively.
E Limitations
GRM relies on information present in the input views and lacks hallucination capability, so unobserved regions may have blurry textures.
- GRM may produce blurry textures in regions not observed by any input image.