PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, Zexiang Xu

arXiv:2404.19702v1cs.CV

TL;DR

GS-LRM addresses sparse-view 3D reconstruction with a scalable model that handles both objects and scenes. It predicts pixel-aligned Gaussian primitives from posed images using a transformer, achieving state-of-the-art reconstruction quality and fast inference, while remaining limited by resolution, camera-pose requirements, and unseen regions.

  • Problem

    Existing reconstruction methods require dense views or per-scene optimization, while recent LRMs primarily target objects and use fixed-resolution triplanes.

  • Method

    GS-LRM patchifies posed images, processes concatenated multi-view tokens with transformer blocks, and decodes per-pixel Gaussian parameters for differentiable rendering.

  • Results

    GS-LRM achieves state-of-the-art reconstruction for both objects and scenes, with 4dB PSNR and 2.2dB PSNR improvements over previous methods, and runs in ∼0.23 seconds on one A100 GPU.

  • Takeaways & Limitations

    Pixel-aligned Gaussian prediction enables fast feed-forward reconstruction across object-level and scene-level captures while preserving high-frequency details.

  • Takeaways & Limitations

    The model currently operates at about 512 × 904 resolution, requires known camera parameters, and has limited, non-guaranteed reconstruction of unseen regions.

Abstract

from arXiv · show

We propose GS-LRM, a scalable large reconstruction model that can predict high-quality 3D Gaussian primitives from 2-4 posed sparse images in 0.23 seconds on single A100 GPU. Our model features a very simple transformer-based architecture; we patchify input posed images, pass the concatenated multi-view image tokens through a sequence of transformer blocks, and decode final per-pixel Gaussian parameters directly from these tokens for differentiable rendering. In contrast to previous LRMs that can only reconstruct objects, by predicting per-pixel Gaussians, GS-LRM naturally handles scenes with large variations in scale and complexity. We show that our model can work on both object and scene captures by training it on Objaverse and RealEstate10K respectively. In both scenarios, the models outperform state-of-the-art baselines by a wide margin. We also demonstrate applications of our model in downstream 3D generation tasks. Our project webpage is available at: https://sai-bi.github.io/project/gs-lrm/ .

1 Introduction

GS-LRM is a scalable transformer-based model for fast, high-quality sparse-view reconstruction of both objects and scenes. Its pixel-aligned Gaussian prediction supports high-resolution details and achieves strong object- and scene-level results.

  • Traditional high-quality 3D reconstruction requires complex photogrammetry and dense multi-view images, while optimized neural methods remain slow and view-intensive.
  • GS-LRM produces novel-view renderings for object captures, generated objects, scene captures, and generated scenes, with rendered scene depth.
  • GS-LRM predicts per-view, per-pixel 3D Gaussian primitives directly from posed image tokens using a simple transformer architecture.The model aligns input pixels with output Gaussians instead of using additional triplane NeRF tokens.
  • Pixel-aligned Gaussian prediction preserves high-frequency input details and supports inputs with higher resolution than fixed-resolution triplanes.
  • 4dB PSNR for objects and 2.2dB PSNR for scenes: GS-LRM outperforms previous methods by large margins across both reconstruction scenarios.The reported 300M-parameter model supports up to 16K transformer tokens and is trained separately on Objaverse and RealEstate10K.

2 Related Work

GS-LRM builds on feed-forward reconstruction, radiance-field representations, and transformer-based LRMs, replacing their typical designs with pixel-aligned Gaussian prediction. The resulting model is designed to scale across object and scene reconstruction.

  • Multi-view 3D Reconstruction: Traditional and learning-based multi-view stereo methods reconstruct sparse or dense geometry using calibrated views and often 3D cost volumes.
  • Radiance Field Reconstruction: Radiance-field reconstruction uses differentiable rendering across coordinate-, voxel-, factorization-, hash-grid-, and point-based representations.
  • Feed-forward Reconstruction: Feed-forward reconstruction methods estimate neural points, multi-plane images, or generalizable radiance fields for efficient reconstruction and rendering.
  • Large Reconstruction Models: Prior transformer-based LRMs use triplane NeRF representations and mainly focus on object reconstruction.
  • Large Reconstruction Models: GS-LRM uses a simplified transformer with per-pixel Gaussian prediction to improve quality, rendering speed, and scalability to large scenes.
  • Concurrent Gaussian Methods: Compared with concurrent LGM and PixelSplat, GS-LRM targets both object- and scene-level reconstruction with a clean transformer architecture and reports significantly better results in both settings.LGM focuses on object generation, whereas PixelSplat focuses on scene reconstruction.

3 Method

GS-LRM converts posed multi-view images into pixel-aligned 3D Gaussians through patchification, transformer processing, and linear decoding. It merges per-view outputs and trains them through rendered-image reconstruction losses.

  • Processing image tokens with transformer: Concatenated multi-view tokens pass through transformer blocks containing self-attention and MLP layers with residual connections and Pre-LayerNorm.
  • Tokenizing posed images: The model receives multi-view RGB images and camera parameters, concatenates RGBs with Plücker ray coordinates, and patchifies each posed feature map.Each flattened patch is mapped by a linear layer to a transformer token.
  • Tokenizing posed images: Plücker coordinates provide per-pixel pose conditioning and spatial information, eliminating the need for additional positional or view embeddings.
  • Decoding output tokens to per-pixel Gaussians: Each output token is linearly decoded into Gaussian parameters, then unpatchified into p^2 Gaussians so every image pixel corresponds to one Gaussian.The representation uses q=12 parameters per Gaussian: RGB, scale, rotation quaternion, opacity, and ray distance.
  • Merging Gaussian outputs: The final representation merges Gaussians from all N input views, producing N · HW Gaussians whose count increases with image resolution.This differs from fixed-resolution triplanes and supports high-frequency details and large-scale scenes.
  • Loss functions: Training renders predicted Gaussian splats at supervision views and minimizes a combination of MSE and perceptual image losses.The perceptual term uses a VGG-19-based loss weighted by λ.

4 Experiments

GS-LRM is evaluated on object and scene reconstruction using matched datasets, baselines, resolutions, and qualitative comparisons. It substantially improves reconstruction quality while supporting high-resolution details and downstream 3D-generation workflows, with limitations in resolution, camera-pose requirements, and unseen regions.

  • Evaluation against baselines: Table 1 compares object-level and scene-level reconstruction against Instant3D’s Triplane-LRM, LGM, and other baselines under matched input and rendering resolutions.Scene-level comparisons use 256-resolution inputs and renderings, while Instant3D’s Triplane-LRM and LGM are compared at 512 and 256 resolution, respectively.
  • Datasets and setup: The experiments train separate object- and scene-level models on Objaverse and RealEstate10K, respectively, using the same architecture with minimal domain-specific changes.The models share nearly the same training recipe, differing mainly in training data and view selection or normalization.
  • Evaluation against baselines: 3.98dB PSNR on GSO and 1.59dB on ABO are the object-level novel-view-rendering improvements over Instant3D’s Triplane-LRM.GS-LRM also produces sharper renderings and better preserves high-frequency details such as text, while its predicted Gaussians render faster than the baseline’s predicted NeRF.
  • Evaluation against baselines: 8dB higher PSNR on both GSO and ABO is achieved by GS-LRM over LGM in an approximately equal-compute comparison.The comparison uses GS-LRM’s low-resolution model and evaluates both methods with 256×256 renderings.
  • Evaluation against baselines: 2.2db PSNR, 0.034 SSIM, and 0.028 LPIPS improvements over pixelSplat are reported for scene-level reconstruction.The visual comparisons show sharper results, fewer floaters, and greater faithfulness to the original scenes.
  • Qualitative results and applications: High-resolution examples preserve readable text, thin and transparent structures, large outdoor depth variations, and complex structures such as trees, while enabling text-to-3D and image-to-3D workflows.The workflows chain multi-view generators with GS-LRM, and the model’s limitations include an operating resolution of about 512 × 904, known camera parameters, and limited guaranteed reconstruction of unseen regions.

5 Conclusion

GS-LRM is a simple, scalable transformer model for Gaussian Splatting that rapidly predicts high-resolution reconstructions from sparse posed images. It supports both object- and scene-level captures and achieves state-of-the-art performance in both scenarios.

  • GS-LRM predicts Gaussian Splatting representations with a simple, scalable transformer-based model.
  • ∼0.23 seconds on a single A100 GPU enables fast feed-forward high-resolution Gaussian prediction from sparse posed images.
  • The model works on both object-level and scene-level captures and achieves state-of-the-art performance in both scenarios.

A.1 Pseudo Code

The appendix provides pseudocode for GS-LRM and includes the Gaussian parameterization described in the method section.

  • Algorithm 1 presents pseudocode for the GS-LRM method.
  • The pseudocode implements the method discussed in the main method section.
  • The appendix also details the Gaussian parameterization used by the model.

A.2 Additional Model Details

GS-LRM omits bias terms throughout its Linear and LayerNorm layers and initializes weights with a zero-mean normal distribution.

  • The model uses no bias terms in its Linear and LayerNorm layers.
  • Weights are initialized from a normal distribution with zero mean.
  • The initialization distribution uses a standard deviation of 0.02.

A.3 Additional Training Details

GS-LRM is trained with AdamW, scheduled learning-rate warmup and decay, and staged resolution training from 256-resolution pretraining to 512-resolution fine-tuning.

  • GS-LRM uses AdamW with β1=0.9, β2=0.95, and 0.05 weight decay except for LayerNorm parameters.
  • The learning rate follows linear warmup and cosine decay, with a peak learning rate of 4e-4.
  • Training runs for 80K iterations at 256-resolution and another 20K iterations of 512-resolution fine-tuning.

A.4 3D Gaussian Parameterization

GS-LRM parameterizes per-pixel Gaussian primitives by transforming posed-image, camera-ray, and geometric tokens into Gaussian parameters. Separate activations and depth mappings constrain outputs for stable differentiable rendering across object and scene settings.

  • Representation considerations: Because Gaussian Splatting is an unstructured explicit representation, output parameterization can substantially affect convergence, unlike structural implicit representations.Here, structurality refers to whether each token has a deterministic spatial meaning.
  • Transformer parameterization: The transformer patchifies concatenated image and camera-ray features, processes the tokens sequentially, and unpatchifies them into per-pixel Gaussian parameters.Inputs include RGB images, ray origins, and cross products of ray origins and directions; outputs are merged across views.
  • Scale and opacity: Gaussian scale uses exponential activation, opacity uses Sigmoid activation, and both are shifted toward an initial value of approximately 0.1.The scale output is additionally capped at 0.3 through the stated parameterization.
  • Rotation: Rotation is predicted as an unnormalized quaternion and L2-normalized to produce a unit quaternion.This normalization is applied directly to the transformer output.
  • RGB: RGB outputs are interpreted as zero-order Spherical Harmonics coefficients, while higher-order coefficients are omitted for simplicity.The paper leaves improved view-dependent modeling to future work.
  • XYZ and distance: Gaussian centers use xyz = rayo + t · rayd, with t mapped between empirical near and far distances through a sigmoid-weighted interpolation.Object models use dnear = 0.1 and dfar = 4.5 with XYZ clipped to [−1, 1]3; scene models use dnear = 0.0 and dfar = 500.

A.5 Camera Pose Normalization

Camera pose normalization differs by capture type: object inputs retain their coordinates, while scene inputs are centered on the mean camera pose and rescaled into a bounded world-space box.

  • Object-level normalization: Object-level GS-LRM applies no camera normalization because object sizes are pre-normalized before data generation.
  • Scene-level normalization: Scene-level GS-LRM defines world space from the mean input camera pose and scales camera locations into the bounding box ([−1, 1]3).
Loading 2404.19702v1…