PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse auto-regressive modeling for scene generation from multi-view images

Thomas Lucas, Maxime Pietrantoni, Philippe Weinzaepfel, Wonjune Cho, Bardienus Pieter Duisterhof, Vincent Leroy, Jerome Revaud

arXiv:2609.03931v1cs.CVcs.LG

TL;DR

SPAR3S tackles the problem of completing 3D scenes from sparse, unconstrained views despite limited visible content, costly dense representations, and scarce 3D supervision. It learns a sparse voxel-aligned latent space from multi-view photometric supervision and uses an occupancy-aware autoregressive model to complete missing regions. The method improves novel-view quality on synthetic indoor scenes and generalizes to RealEstate10K.

  • Problem

    Sparse-view 3D scene completion must infer unobserved geometry and appearance, while existing reconstruction is visibility-limited and large-scale 3D supervision is scarce.

  • Method

    SPAR3S learns a sparse voxel-aligned 3D latent space from multi-view photometric supervision and autoregressively predicts missing voxel occupancy and latent tokens.

  • Results

    SPAR3S achieves improved novel-view synthesis quality over prior feed-forward 3DGS regression and 3D generative methods on synthetic indoor scenes, with validation on RealEstate10K.

  • Takeaways & Limitations

    The results support autoregressive latent modeling as a viable approach for sparse-view 3D scene generation and completion.

  • Takeaways & Limitations

    The method assumes that randomly placed cameras make the unobserved voxel set random, so training learns to cover full latent representations on average.

Abstract

from arXiv · show

Generating complete 3D scenes from sparse, unconstrained views is a fundamental challenge in 3D vision which requires reasoning beyond observed content while remaining computationally tractable. Existing feed-forward reconstruction methods are inherently limited to content visible in the input images, while 3D generative modeling is hindered by the high computational cost of dense volumetric representations and the scarcity of large-scale 3D supervision. We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion without requiring ground-truth 3D data for supervision. Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space where only occupied voxels are represented. We learn this sparse latent space directly from multi-view images using photometric supervision via differentiable 3D Gaussian Splatting. Given a partial set of observed voxels encoded from sparse input views, scene completion reduces to predicting the missing latent tokens and their spatial support within the voxel grid. To this end, we train a masked autoregressive transformer that jointly models voxel occupancy and latent token values, enabling efficient and spatially consistent generation of unseen regions. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.

1. Introduction

SPAR3S addresses sparse-view scene completion by learning a compact voxel-aligned 3D latent space from images and autoregressively generating unobserved content. It reports improved novel-view synthesis on synthetic scenes and generalization to RealEstate10K.

  • Pixel-aligned reconstruction methods are limited to content visible in input images, while image-space completion can require known cameras and lack geometric consistency.
  • SPAR3S learns a sparse voxel-aligned 3D latent space directly from multi-view images using photometric supervision through differentiable 3D Gaussian Splatting, without ground-truth 3D supervision.
  • SPAR3S improves novel-view synthesis quality over prior feed-forward 3DGS regression and 3D generative methods on synthetic indoor scenes, with validation on RealEstate10K.
  • Its occupancy-aware masked autoregressive transformer jointly predicts voxel occupancy and latent token values for conditional scene completion.

2. Related work

Prior work reconstructs or generates 3D content using feed-forward Gaussian prediction, image-space diffusion, iterative optimization, or latent generative models. SPAR3S differs by operating directly in a sparse voxel-aligned 3D latent space for geometrically consistent scene generation.

  • Feed-forward 3D Gaussian methods regress scene parameters from sparse views but lack structured 3D representations, limiting geometric consistency across views.
  • Image-space diffusion methods generate novel views rather than explicit 3D scenes, requiring repeated rendering procedures and often camera conditioning or simplifying assumptions.
  • Iterative 3DGS optimization with diffusion priors can require significant time because scene reconstruction is optimization-based.
  • Some 3D generative methods require teacher models, ground-truth splats, dense views, or slow per-scene optimization, limiting scalability.
  • SPAR3S adapts autoregressive latent modeling to sparse 3D voxel grids, using arbitrary token orderings while enforcing geometric consistency across views.

3. 3D latent representation from multiview data

The scene encoder-decoder learns a sparse, compact voxel-aligned latent representation from multi-view images and reconstructs them through a single differentiable 3DGS scene. Hierarchical compression and occupancy prediction reduce computation while preserving sparse structure.

  • The encoder-decoder constrains information to pass through one 3DGS representation per scene, which is decoded to reconstruct all input views.
  • Only content-bearing voxels are processed, and hierarchical downsampling and upsampling make the latent space more compact and reduce computational load.
  • Multi-view pointmaps initialize sparse voxel tokens, while image patches provide 2D tokens that interact with 3D tokens through bidirectional cross-attention.
  • The decoder maps the latent representation to 3D Gaussians associated with occupied voxels, excluding empty voxels throughout computation.
  • Occupancy is predicted before upsampled token values, with false-positive noise during training improving robustness to inference-time classification errors.
  • Dense views provide latent targets and sparse views provide conditioning, allowing the encoder to handle variable input-set sizes.

4. Sparse Auto-Regressive Modeling in 3D Latent Space

SPAR3S performs sparse 3D scene completion by predicting occupancy and latent content in a voxel-aligned representation, rather than modeling all positions sequentially. Its masked autoregressive design supports iterative, spatially consistent completion while retaining computational efficiency.

  • Sparse 3D latent representation: The sparse representation materializes only occupied voxels, reducing the computation required compared with raster-order autoregression over the entire possible volume.Raster-order models consider all tokens sequentially, whereas SPAR3S exploits sparsity in its voxel-aligned latent space.
  • Predicting occupancy and latent values: SPAR3S predicts both voxel occupancy and latent values for unknown positions, allowing scene completion despite spatially sparse and variable-length voxel sequences.The model takes conditioning voxel tokens and unknown voxel positions, then predicts occupancy and latent content for occupied voxels.
  • Training procedure: Training uses randomly masked target positions, keeps unobserved positions in the prediction target, and masks those unknown positions from the loss.This procedure trains completion without requiring ground-truth labels for unobserved voxels.
  • Inference: At inference, the model predicts occupancy scores and latent tokens over successive voxel subsets, retaining predictions only when occupancy exceeds threshold τ.Predicted subsets are added to the conditioning set iteratively until the volume is processed.
  • Inference: Region-growing mask order uses a k-NN graph and breadth-first depth levels, promoting spatial consistency during autoregressive inference.The resulting ordering expands from observed voxels through neighboring regions.
  • Training details: Tokenwise diffusion refines latent predictions, and this lightweight formulation empirically improves over a fully autoregressive formulation while maintaining scalability.The diffusion component uses a few token-wise MLP layers, and the authors retain it by default after observing an empirical improvement.

5. Experiments

Experiments evaluate SPAR3S on challenging sparse-view novel-view synthesis using synthetic and real-world data, varied conditioning views, occupancy thresholds, and architectural ablations. SPAR3S substantially outperforms competing 3D generative baselines in wide two-view reconstruction, while performance improves with more views and depends on its sparse autoregressive design choices.

  • 5.1. Experimental details: The evaluation uses random conditioning and novel-view sampling on 3DFront and RealEstate10K, spanning easy overlapping views to difficult views with little or no overlap.RealEstate10K provides real-world validation but has smoother camera paths and more overlap, making its novel-view task comparatively easier.
  • 5.3. Conditioning and thresholds: More conditioning views consistently improve reconstruction, while lower occupancy thresholds generally help and form a diagonal PSNR/LPIPS pattern with increasing view count.With more views, geometry prediction becomes more accurate, allowing higher occupancy thresholds.
  • 5.2. Two-view NVS: SPAR3S substantially outperforms 3D generative baselines for challenging two-view reconstruction with wide viewpoint variation.Optimization-based and feed-forward 3DGS methods fail to extrapolate unobserved areas, while SPAR3S handles unconstrained cameras through autoregressive generation in a sparse voxelized latent space.
  • 5.4. Ablations: Encoder-decoder ablations select 63 splats per voxel, approximately 25% false positives, and a latent dimension of 32 with KL coefficient 0.1.More splats show diminishing returns; stronger KL coefficients or latent dimensions below 32 significantly degrade performance.
  • 5.4. Ablations: Removing diffusion decoding, 3D relative positional encoding, BFS ordering, or occupancy refinement degrades latent coherence, spatial reasoning, or rendered-scene quality.Using ground-truth occupancies improves quality, isolating the effect of occupancy prediction from latent-token reconstruction.

6. Conclusion

SPAR3S completes sparse-view 3D scenes in a voxel-aligned latent space using autoregressive inference of geometry and latent content. Its results support autoregressive latent modeling as a promising direction for sparse-view 3D scene generation.

  • 6. Conclusion: SPAR3S encodes sparse, unconstrained views into a voxel-aligned 3DGS latent space and autoregressively completes geometry and latent content.The model jointly infers scene structure and appearance while exploiting sparsity.
  • 6. Conclusion: The results demonstrate the potential of autoregressive latent modeling for 3D scene generation and sparse-view generative reconstruction.The paper identifies masked autoregressive models as a promising avenue for applying generative modeling to 3D data.

A. Additional implementation details

The implementation uses voxel-wise autoregressive prediction with occupancy and diffusion heads, positional encodings, and coarse-to-fine occupancy inference. Training proceeds through a separately trained scene encoder-decoder and latent prediction model.

  • Autoregressive architecture: The autoregressive architecture uses bidirectional attention blocks with occupancy and denoising diffusion heads to predict masked voxel outcomes.Conditioning and unmasked tokens are aggregated before masked voxel embeddings are refined and decoded.
  • Positional encoding: Sinusoidal voxel-coordinate encodings and 3D RoPE provide positional information, while masked tokens begin from a learnable mask token.The internal attention dimension is 512.
  • Prediction heads: The occupancy head is a three-layer GELU MLP trained with focal loss, while the diffusion head uses SNR-weighted denoising and adaptive layer-normalization conditioning.These heads separately predict occupancy logits and latent diffusion outputs.
  • Occupancy prediction: Coarse-to-fine occupancy prediction evaluates all voxels in a lightweight coarse stage before refining selected regions with latent prediction disabled initially.Inference uses a high-recall coarse threshold to avoid missing potentially occupied regions.
  • Training: Training first fits the scene encoder, then trains the autoregressive latent model using encoder-produced conditioning and target tokens.The latent dimension is 32, and group normalization with eight groups stabilizes the encoded latent space.

B. Visualizations

Visualizations show SPAR3S extrapolating coherent geometry beyond conditioning-camera frustums and producing more consistent, detailed reconstructions as conditioning views increase.

  • Novel-view visualizations: With two conditioning images, SPAR3S extrapolates coherent scene content and geometry far outside the conditioning-camera frustums.The visualization samples rendered novel views from randomly selected camera viewpoints.
  • Conditioning-view scaling: Increasing conditioning views from 4 to 8 to 12 reduces ambiguity and yields more consistent, detailed reconstructions.The visual comparison presents one conditioning-view count per row for a single scene.

C. Method limitations

SPAR3S’s sparse occupancy modeling introduces quality limitations when voxels are missed or resolution and Gaussian density are constrained. Its evaluation also focuses on novel-view synthesis from sparse conditioning views on 3D-Front and RealEstate10k.

  • Missed occupancy predictions can create visually salient holes inside walls or objects, currently the main degradation to visual quality.The authors identify occupancy errors as a key limitation of sparsifying computation.
  • Voxel resolution and the number of Gaussians allocated per voxel can produce localized blurriness in rendered scenes.The paper attributes these artifacts to representation constraints.
  • The reported novel-view evaluation uses two conditioning views on 3D-Front and RealEstate10k, while more views reduce ambiguity and improve renderings.The figures indicate that contextual information affects reconstruction quality and that unseen areas can still yield plausible modes.
Loading 2609.03931v1…