PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

Chen Ziwen, Hao Tan, Kai Zhang, Sai Bi, Fujun Luan, Yicong Hong, Li Fuxin, Zexiang Xu

arXiv:2410.12781v2cs.CV

TL;DR

Long-LRM targets the gap between high-quality but slow per-scene optimization and fast feed-forward methods limited to few, narrow-view inputs. It combines Mamba2 and transformer blocks with token merging for long-sequence Gaussian reconstruction, achieving comparable quality to optimization-based 3D GS with much faster inference and stronger sparse-view performance.

  • Problem

    Existing radiance-field and Gaussian methods are either slow because they optimize each scene independently or limited to a few input images, preventing fast wide-coverage reconstruction.

  • Method

    Long-LRM combines Mamba2 and transformer blocks with lightweight token merging for feed-forward Gaussian reconstruction from long, high-resolution multi-view sequences.

  • Results

    Long-LRM reconstructs 32 960×540 views in 1 second with quality comparable to optimization-based 3D GS, while outperforming prior feed-forward methods in sparse-view settings.

  • Takeaways & Limitations

    The model enables instant wide-coverage, high-resolution scene-level reconstruction and extends feed-forward Gaussian reconstruction beyond the input scales of previous methods.

  • Takeaways & Limitations

    Increasing the input from 32 to 64 views yields less than 1 dB PSNR improvement, while 64 high-resolution views exceed 500K context tokens.

Abstract

from arXiv · show

We propose Long-LRM, a feed-forward 3D Gaussian reconstruction model for instant, high-resolution, 360° wide-coverage, scene-level reconstruction. Specifically, it takes in 32 input images at a resolution of 960x540 and produces the Gaussian reconstruction in just 1 second on a single A100 GPU. To handle the long sequence of 250K tokens brought by the large input size, Long-LRM features a mixture of the recent Mamba2 blocks and the classical transformer blocks, enhanced by a light-weight token merging module and Gaussian pruning steps that balance between quality and efficiency. We evaluate Long-LRM on the large-scale DL3DV benchmark and Tanks&Temples, demonstrating reconstruction quality comparable to the optimization-based methods while achieving an 800x speedup w.r.t. the optimization-based approaches and an input size at least 60x larger than the previous feed-forward approaches. We conduct extensive ablation studies on our model design choices for both rendering quality and computation efficiency. We also explore Long-LRM's compatibility with other Gaussian variants such as 2D GS, which enhances Long-LRM's ability in geometry reconstruction. Project page: https://arthurhero.github.io/projects/llrm

1. Introduction

Long-LRM addresses the slow, narrow-coverage nature of prior 3D reconstruction by using a feed-forward model for high-resolution, 360° scene reconstruction from 32 images. Its hybrid long-sequence architecture enables rapid reconstruction while retaining quality comparable to optimization-based methods.

  • Motivation and contribution: Prior feed-forward Gaussian methods generally handle only 1–4 low-resolution or narrow-coverage views, limiting reconstruction of large real-world scenes.Wide-view scenes require at least dozens of images, while earlier methods were designed for small input sets.
  • Motivation and contribution: 1 second versus over 10 minutes enables comparable-quality reconstruction from 32 high-resolution views, addressing the speed and coverage gap in prior methods.Long-LRM processes 32 images at 960×540 resolution and targets wide-coverage scene-level reconstruction without per-scene optimization.
  • Architecture: A 32-view 960×540 input produces roughly 250K tokens, making standard transformer processing difficult because of quadratic time complexity.The token sequence is longer than the 128K context length cited for Llama3 [13].
  • Architecture: Long-LRM combines Mamba2 and transformer blocks with token merging to support efficient sequential long-context reasoning while preserving global context.The model uses state space models for linear-complexity long-context processing and reduces intermediate token counts with a lightweight merging module.
  • Results: Evaluation on DL3DV140 and Tanks&Temples reports comparable novel-view quality to optimization-based 3D GS while reducing reconstruction time from 13 minutes to 1 second.The study also includes ablations of rendering quality and computation efficiency and tests compatibility with 2D GS.

2. Related Work

Prior reconstruction methods target geometry or rendering through explicit geometry, differentiable radiance fields, or feed-forward Gaussian representations. Long-sequence transformer approaches offer flexibility but become inefficient as input sequences grow.

  • 3D Reconstruction: Traditional reconstruction methods predict meshes or depth maps, often using epipolar feature matching followed by TSDF or neural depth prediction.These approaches primarily target explicit geometry rather than jointly reconstructing geometry and color.
  • Neural reconstruction and rendering: NeRF uses differentiable volume rendering to reconstruct novel views without ground-truth 3D supervision, but requires scene-specific optimization that can take hours or days.Its implicit representation produces visually pleasing results while sacrificing reconstruction speed and generalizability.
  • Generalizable NeRF and 3D GS: Generalizable NeRF and Gaussian methods aggregate multi-view features with epipolar lines or plane-sweep cost volumes for feed-forward reconstruction.These geometric designs support nearby-view aggregation but are associated with local estimation strategies.
  • Efficient models for long sequences: Transformer-based large reconstruction models scale flexibly across modalities, but quadratic complexity makes very long sequences extremely slow and computationally demanding.The related-work discussion notes that long-sequence processing can require thousands of GPUs for parallel computing [13].

3. Method

Long-LRM reconstructs wide-coverage Gaussian scenes by combining Mamba2 and transformer processing with token merging, Gaussian pruning, and regularized training objectives. Its design addresses long sequences, dense per-pixel Gaussian predictions, training instability, and inference efficiency.

  • Overview: Long-LRM tokenizes concatenated RGB images and Plücker rays, then processes them with hybrid Mamba2-transformer blocks before decoding per-pixel Gaussian parameters and pruning Gaussians.Each hybrid block contains seven Mamba blocks and one global-self-attention transformer block, with token merging between hybrid blocks.
  • Mamba2 Block: 250K-token inputs make transformer-only processing costly, so Mamba reduces sequence complexity from O(L^2) to O(L) while transformer blocks preserve global context.The architecture uses bidirectional Mamba2 scans because one-directional scanning is suboptimal for images.
  • Token Merging and Gaussian Pruning: Token merging reduces the sequence length to one quarter through a channel-wise 2×2 stride-2 convolution, balancing computation efficiency against reconstruction quality.The module reshapes tokens into a spatial representation, applies convolution, and reshapes the result back into a shorter sequence with an effective patch size of 2p.
  • Token Merging and Gaussian Pruning: Gaussian pruning is necessary because dense per-pixel prediction produces approximately 17 million Gaussians for 32 high-resolution images, creating substantial memory and speed pressure.Opacity regularization encourages a compact Gaussian set, after which low-opacity Gaussians can be pruned without the incomplete opaque objects caused by naive pruning.
  • Training Objectives: Training combines rendering, depth, and opacity objectives: depth regularization stabilizes Gaussian positions, while opacity regularization improves computational efficiency.The total loss is L_image + λ_opacity·L_opacity + λ_depth·L_depth, with λ_opacity = 0.1 and λ_depth = 0.01.
  • Training Objectives: Opacity regularization reduces the share of Gaussians with opacity above 0.001 from 99% to around 40%, enabling pruning and reducing Gaussian rendering and backpropagation time.The rendering loss combines MSE and perceptual loss, with λ = 0.5.

4. Experiments

Experiments evaluate Long-LRM on wide-coverage high-resolution reconstruction, sparse-view reconstruction, and compatibility with 2D Gaussian splatting. Across these settings, the model combines fast inference with competitive or leading rendering quality.

  • Implementation and training: The architecture uses repeated Mamba2 and transformer blocks, token merging, Gaussian pruning, and a low-to-high-resolution curriculum to manage high-resolution inputs efficiently.Training progresses through 256×256, 512×512, and 960×540 stages; the final stage expands sampling to entire sequences and applies pruning.
  • Evaluation settings: Evaluation selects scene-covering input views by K-means clustering and retains the top 50% of Gaussians by opacity with negligible reported quality loss.The evaluation uses 960 × 540 images and uniformly samples every eighth frame for testing.
  • Datasets and evaluation: The model is evaluated on DL3DV-140 and Tanks&Temples, including 16- and 32-view full-scene settings and zero-shot inference on Tanks&Temples.DL3DV-10K is used for training, while DL3DV-140 and Tanks&Temples provide evaluation scenes.
  • High-resolution, wide-coverage reconstruction: An 800× speedup reduces full-scene reconstruction to 1 second versus 13 minutes, while quality remains comparable to 30K-step optimization and PSNR leads by +2dB over 3D GS30k.The comparison uses 960 × 540 inputs on wide-span scenes; LPIPS remains better for optimization-based methods.
  • Low-resolution, sparse-view reconstruction: Long-LRM achieves the best overall quality in the 256×256 two-view RealEstate10K setting, exceeding pixelSplat and MVSplat by over 2dB PSNR and slightly surpassing GS-LRM.This setting compares Long-LRM with prior feed-forward GS methods under their commonly used sparse-view protocol.
  • Compatibility with other GS variants: A 2D GS fine-tuning variant preserves color quality comparable to optimization-based 2D GS and supports depth and geometry evaluation on ScanNetv2.The 2D GS variant is assessed on DL3DV-140 for color reconstruction and on ScanNetv2 for zero-shot depth and mesh quality.

5. Analysis

The analysis evaluates architecture, token merging, regularization, and Gaussian usage across input scales. It finds hybrid designs and merging improve the efficiency–quality trade-off, while opacity and depth supervision affect Gaussian sparsity and rendering behavior.

  • 5.1. Ablation Studies of Model Designs: Token merging is essential for dense ultra-resolution training, reducing token length to one-quarter and enabling training where other variants run out of memory.With constant token dimensionality, training time falls to two-thirds; reducing the initial dimensionality further saves an additional one-third of training time and memory.
  • 5.1. Ablation Studies of Model Designs: The hybrid Mamba2–transformer model combines transformer-level reconstruction quality with Mamba2-level speed, while balancing model size and quality through merging placement.The architecture ablation covers sparse and dense inputs at low, high, and ultra resolution.
  • 5.2. Ablation Studies of Training Objectives: Depth supervision improves rendering quality and prevents gradient explosions by guiding floater Gaussians toward true surfaces.Opacity loss substantially reduces visible Gaussians with negligible rendering impact, while depth supervision encourages view-dependent color representation through spherical harmonics.
  • 5.3. Gaussian Usage: Higher input resolution decreases Gaussian usage because more predicted per-pixel Gaussians increase the chance that multiple pixels share one Gaussian.Gaussian usage is measured as the percentage of Gaussians with opacity above 0.001; Table 7 examines this alongside opacity loss and input size.

6. Conclusions

Long-LRM provides fast, scalable wide-span Gaussian reconstruction by combining hybrid sequence modeling with token merging and Gaussian pruning. It reconstructs high-resolution scenes from 32 images in one second while maintaining quality comparable to optimization-based methods.

  • 6. Conclusions: Long-LRM reconstructs 360° wide-span scenes from 32 960 × 540 images in 1 second, with rendering quality comparable to optimization-based 3D GS methods.The model combines Mamba2 and transformer blocks with token merging and Gaussian pruning, and also supports 2D GS variants.

A. More Qualitative Results

The paper provides additional qualitative Long-LRM results and points readers to a project page containing visual comparisons with optimization-based 3D GS methods.

  • A. More Qualitative Results: Additional qualitative results include a video comparison between Long-LRM and optimization-based 3D GS methods.The comparisons are provided in Fig. 6 and on the project page.

C. Experiment Details for Ablation Studies on Model Architecture

The architecture ablations compare model behavior across progressively denser and higher-resolution input settings. Evaluation uses consistent training and testing views, with convergence-based training schedules and controlled preprocessing.

  • C. Experiment Details for Ablation Studies on Model Architecture: Architecture variants are evaluated at 4 views with 256×256 input, 32 views with 256×256 or 512×512 input, and 32 views at 960×540.The variants are trained on DL3DV-10K and evaluated on DL3DV-140, with training steps selected empirically according to convergence.
  • C. Experiment Details for Ablation Studies on Model Architecture: Ablation comparisons keep training and testing views consistent by sampling eight test views from the first 96 frames and selecting non-overlapping training views.Input images are resized and center-cropped to square for the controlled comparisons.

D.1. More results on RealEstate10K.

On RealEstate10K, the training-objective ablation finds depth and opacity losses less essential for the two-view setup, while Long-LRM renders details better than baselines. The supplied quantitative table passages additionally describe evaluation protocols for novel-view, mesh, and depth reconstruction.

  • Training objectives: Depth and opacity losses are less essential for the two-view RealEstate10K setup.The opacity loss targets Gaussian-count reduction, while the depth loss stabilizes training for the long-input setting.
  • Evaluation scope: Table 8 compares novel-view PSNR, mesh Chamfer, and F-Score between Long-LRM’s 2D GS variant and optimization-based 2D GS on ScanNetv2.
  • Evaluation scope: Table 9 evaluates zero-shot full-scene geometry by comparing rendered median-depth maps against ground-truth depth across ScanNetv2 sequences.
  • Qualitative comparison: Long-LRM demonstrates better rendering of details than baselines in qualitative RealEstate10K comparisons.

D.2. Zero-shot results on ScanNetv2.

Long-LRM is evaluated zero-shot on ScanNetv2 using 32 sampled views, with 2D Gaussian renderings assessed for novel-view synthesis, mesh reconstruction, and depth quality. Qualitative results show fine details and multi-perspective scene reconstruction, while hybrid-block placement affects training stability.

  • Evaluation setup: 32 sampled ScanNet views are resized to height 540 for zero-shot inference after training solely on DL3DV-10K.Aspect-ratio augmentation is applied during stage 3 training.
  • Quantitative reconstruction results: Long-LRM’s 2D GS variant is compared with optimization-based 2D GS for novel-view synthesis and mesh reconstruction on ScanNetv2.The sparse 32-view setup leads to poor optimization-based 2D GS performance even with COLMAP initialization, highlighting the value of learned prior knowledge.
  • Qualitative geometry: Predicted 2D Gaussians provide color and depth renderings, with depth maps showing fine object details for geometry reconstruction.
  • Qualitative reconstruction: Long-LRM reconstructs diverse complex scenes with detailed, photorealistic novel views from multiple perspectives.The qualitative examples present reconstructed Gaussian overviews alongside synthesized views.
  • Architecture ablation: Evenly distributing transformer blocks among Mamba2 blocks makes training more stable than concentrating them at the beginning or end.PSNR curves are similar across configurations, including configurations that diverge late in training.

E. Limitations

Long-LRM has limited gains from adding more views and struggles with substantial field-of-view variation. These constraints reflect both ultra-long sequence processing difficulty and the fixed-FOV training distribution.

  • Scaling views: Increasing input from 32 to 64 views yields less than 1 dB PSNR improvement.At 64 high-resolution views, the sequence exceeds 500K tokens, challenging current sequence-processing models.
  • Field-of-view generalization: The model struggles to generalize to test sets with substantial FOV variation, such as MipNeRF360’s smaller FOV.The DL3DV training set uses images with a fixed wide FOV, and the authors suspect Mamba2 contributes to position-dependent token-meaning changes across FOVs.
  • Future scope: Future work requires better ultra-long-sequence handling and more diverse training data spanning FOVs at DL3DV-like scale.
Loading 2410.12781v2…