PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation

Nicolas von Lützow, Barbara Rössle, Katharina Schmid, Matthias Nießner

arXiv:2603.26661v2cs.CV

TL;DR

Structured 3D scene generation lacks a natural sequence, while existing approaches commonly rely on holistic diffusion or flow-matching formulations. GaussianGPT instead compresses Gaussian scenes into discrete latent grids and generates them autoregressively with spatially aware tokens. The paper reports improved visual and layout quality in shape synthesis, scene generation, and completion, while retaining a trade-off in geometric diversity.

  • Problem

    Structured 3D scene generation must capture geometry, appearance, semantics, and spatial consistency despite the lack of a canonical sequence.

  • Method

    GaussianGPT uses vector-quantized Gaussian representations, structured tokenization, and a causal transformer with 3D-aware positional modeling for next-token scene generation.

  • Results

    The method demonstrates improved visual quality and stronger perceptual and layout quality in 3D shape synthesis, scene generation, and completion, with a trade-off in geometric diversity.

  • Takeaways & Limitations

    Sequential next-token prediction is a viable complementary alternative for structured 3D synthesis, supporting incremental generation, completion, and outpainting.

Abstract

from arXiv · show

Most recent advances in 3D generative modeling rely on diffusion or flow-matching formulations. We instead explore a fully autoregressive alternative and introduce GaussianGPT, a transformer-based model that directly generates 3D Gaussians via next-token prediction, thus facilitating full 3D scene generation. We first compress Gaussian primitives into a discrete latent grid using a sparse 3D convolutional autoencoder with vector quantization. The resulting tokens are serialized and modeled using a causal transformer with 3D rotary positional embedding, enabling sequential generation of spatial structure and appearance. Unlike diffusion-based methods that refine scenes holistically, our formulation constructs scenes step-by-step, naturally supporting completion, outpainting, controllable sampling via temperature, and flexible generation horizons. This formulation leverages the compositional inductive biases and scalability of autoregressive modeling while operating on explicit representations compatible with modern neural rendering pipelines, positioning autoregressive transformers as a complementary paradigm for controllable and context-aware 3D generation.

1 Introduction

GaussianGPT frames structured 3D scene generation as sequential prediction over discrete scene primitives, addressing the difficulty of serialization and spatially coherent modeling. Its structured tokenization and autoregressive transformer support incremental generation, completion, and outpainting.

  • Structured 3D scenes jointly require geometry, appearance, semantics, multi-view consistency, and coherent layouts despite lacking a canonical sequence.
  • GaussianGPT represents 3D synthesis as sequential prediction over structured scene primitives.
  • The method uses structured tokenization of Gaussian-based scene representations to enable autoregressive modeling of complex indoor environments.
  • Its autoregressive transformer framework supports incremental 3D Gaussian scene generation, completion, and outpainting.

2 Related Works

Related work spans multiple 3D representations and generation paradigms, while full scene synthesis remains difficult because of scale, dependencies, compositionality, and layout constraints. GaussianGPT builds on structured Gaussian representations and autoregressive modeling to address these challenges.

  • 3D generative models use voxels, implicit geometry, point clouds, meshes, NeRFs, and Gaussian splats as underlying representations.
  • NeRF-based representations provide high-fidelity modeling but require dense volumetric sampling, limiting training and inference scalability.
  • 3D Gaussian Splatting offers an efficient, expressive representation with high visual quality and real-time rendering, but unconstrained Gaussian sets lack structural regularity.
  • Recent 3D generation is dominated by diffusion and flow-matching, while autoregressive approaches provide a parallel sequence-modeling direction.
  • Full scene generation involves large spatial scale, long-range dependencies, multiobject compositionality, coherent layout, and physical plausibility.

3 Methodology

GaussianGPT compresses Gaussian scenes into sparse discrete latent grids, serializes them as structured token sequences, and models them with a causal transformer using spatial and token-type cues. The same autoregressive mechanism supports generation, completion, and larger-scene synthesis.

  • Scene Compression: A sparse 3D convolutional autoencoder converts Gaussian scenes into compact discrete latent grids through feature-grid encoding and vector quantization.
  • Scene Compression: Gaussian primitives are assigned to world-coordinate voxels, represented with relative offsets, subsampled per voxel, and encoded into unified sparse feature vectors.
  • Autoregressive Modeling: The latent grid is serialized with a fixed xyz traversal and interleaved position and feature tokens.
  • Autoregressive Modeling: Chunk-relative voxel indices limit context requirements while allowing the model to operate across positions, scenes, and layouts.
  • Autoregressive Modeling: Separate position and feature vocabularies decouple geometric structure from appearance modeling.
  • Autoregressive Modeling: 3D RoPE encodes relative spatial offsets, while an additional rotary dimension identifies position versus feature tokens.
  • Generation and Completion: The transformer continues a serialized partial scene as a prefix, enabling completion and repeated sliding-window synthesis beyond fixed training chunks.

4 Experiments

GaussianGPT is evaluated on unconditional shape and scene generation, scene completion, and large-scale outpainting. Results show strong visual, appearance, and layout quality, while revealing task-dependent trade-offs in geometric diversity and distribution matching.

  • Shape Generation: GaussianGPT achieves the best FID, KID, and COV among evaluated unconditional shape-generation methods while remaining competitive in MMD.These metrics indicate strong alignment with the target distribution in appearance and geometry.
  • Shape Generation: GaussianGPT produces sharper chair structures and cleaner renderings by avoiding noisy outlier primitives while retaining substantial shape and style variation.Both GaussianGPT and L3DG generate high-quality chairs with plausible geometry and appearance.
  • Scene Synthesis: GaussianGPT generates semantically meaningful and diverse scene completions from partial inputs, using arbitrary prefixes without architectural changes or specialized conditioning.The causal formulation supports completion from observed scene prefixes.
  • Scene Synthesis: In unconditional scene generation, L3DG provides greater sample variety and better ground-truth geometry matching, whereas GaussianGPT delivers cleaner geometry and stronger appearance and layout metrics.The comparison uses matched chunked 3D-FRONT training settings for quantitative evaluation.
  • Large Scene Generation: GaussianGPT extends scenes beyond the fixed training horizon by iteratively appending latent-grid columns conditioned on previously generated regions.A backtracking resampling strategy retries empty predicted columns up to five times to promote occupancy.
  • Ablations: Alternative Z-order and Hilbert serializations do not improve performance over simple xyz traversal, suggesting that 3D RoPE supplies sufficient spatial information.Transposed Z-order achieves nearly identical validation performance, while the simple xyz traversal performs best overall.
  • Ablations: Separating position and feature vocabularies and using 3D RoPE improve autoregressive scene modeling relative to shared vocabularies, learned positional embeddings, or 1D RoPE.The ablation attributes the benefit to explicit spatial-coordinate encoding and separate spatial and feature prediction.

5 Conclusion

The paper presents GaussianGPT as a fully autoregressive framework that generates and completes 3D scenes from vector-quantized Gaussian representations. It reports improved visual, perceptual, and layout quality while retaining a trade-off in geometric diversity.

  • Conclusion: GaussianGPT combines sparse 3D latent compression, structured tokenization, and 3D-aware transformer modeling for sequential 3D scene generation and completion.The framework operates directly on vector-quantized Gaussian representations.
  • Conclusion: The same autoregressive mechanism supports incremental generation, completion, outpainting, explicit sampling control, and reasoning over partially generated geometry.Scenes are represented as sequences of discrete spatial decisions.
  • Conclusion: GaussianGPT demonstrates improved visual quality in shape synthesis and scene generation and completion, with stronger perceptual and layout quality but a trade-off in geometric diversity.The conclusion positions structured token sequences as a basis for controllable and compositional 3D generation.

A Real-World Results

On ScanNet++ v2, GaussianGPT retains useful synthetic-data priors and produces diverse, plausible completions despite greater real-world complexity and a smaller dataset. However, autoencoder fidelity and unobserved scan regions limit reconstruction quality.

  • Real-World Setting: ScanNet++ v2 introduces higher visual and geometric complexity, greater scene variability, inconsistent global axis alignment, and substantially less data than the synthetic datasets.The models are fine-tuned on this more challenging real-world setting.
  • Real-World Completion: GaussianGPT produces diverse and plausible ScanNet++ v2 completions in geometry, texture, and semantic layout despite the small real-world dataset.The model retains useful priors learned from synthetic data.
  • Limitations: Autoencoder fidelity limits real-world performance because high-frequency details are incompletely reconstructed, producing noisier geometry and Gaussian predictions.Missing or unobserved scan regions also cannot be faithfully modeled by the current pipeline.
  • Future Work: Autoregressive modeling is presented as suitable for uncertain partial scans because uncertain regions can be masked while conditioning remains available for probabilistic reasoning.This is identified as a direction for future work rather than a demonstrated capability of the current evaluation.
  • Real-World Completion: Temperature 0.6 with nucleus sampling p = 0.7 yields varied and feasible ScanNet++ v2 completions.These settings are reported for the qualitative completion results.

B Inference Efficiency Analysis

GaussianGPT’s autoregressive sampling trades slower unconditional generation for faster completion from partial context, while its memory use remains nearly constant as a conservative implementation bound. Decode acceleration opportunities include cache optimization, speculative decoding, auxiliary heads, and locality-aware attention.

  • Autoregressive sampling is slower than diffusion-based L3DG for unconditional generation because next-token sampling is sequential while diffusion updates representations in parallel.The comparison disables resampling for GaussianGPT and backward repainting for L3DG completion.
  • Completion from partial context drastically reduces GaussianGPT’s sampling time, unlike the diffusion baseline, which continues equivalent update steps.
  • GaussianGPT’s peak memory is dominated by a key-value cache sized for the full context window and remains nearly constant across unconditional generation and completion.This is a conservative implementation-level bound rather than memory required by the realized sequence.
  • L3DG requires approximately 1.25 GB of peak memory during decoding because it uses a fixed dense grid during generation.
  • Decode-optimized kernels, speculative decoding, auxiliary decoding heads, and sparse or locality-aware attention could further accelerate autoregressive sampling.These proposals target cache access, parallel verification, multiple future-token proposals, and the spatial structure of the sequence.

C Large Scene Generation

GaussianGPT generates large scenes through autoregressive outpainting with sliding-window local context, staged expansion, and occupancy-focused resampling. Its quality degrades with distance from the initial tile, especially along x because of the xyz serialization order.

  • Large Scene Generation: Large scenes are extended by predicting new columns from previously generated columns within a sliding-window local coordinate frame.The xyz ordering preserves relative ordering across the global sequence and local chunks.
  • Sampling Efficiency: Scene outpainting uses single-chunk sampling, shifted x-direction bootstrapping with 20-column slices, and a y-direction loop until the target size is reached.Bootstrapping reduces the number of sampling loop calls by generating multiple columns per step.
  • Chunk Window Positioning: The model generates up to 5 columns at a time from the chunk center and predicts around x = 5 to match training-chunk distributions.These choices are intended to yield a stable, well-conditioned generation frontier.
  • Resampling and degradation: Empty predicted columns are retried up to a fixed number of times to encourage meaningful occupancy in larger scenes.The strategy’s quality and degradation effects are analyzed in Fig. 9.
  • Resampling and degradation: Quality gradually deteriorates with distance from the initial region, with substantially stronger degradation along x than y.The reported analysis uses KID per 4 m × 4 m tile across a 12 m × 12 m scene, normalized to the top-left initial tile.
  • Resampling and degradation: The x–y degradation asymmetry is attributed to serialization: y extensions remain relatively local in the token sequence, whereas x extensions require larger spatial jumps.A hard occupancy constraint performed poorly because it prevents the model from modeling natural scene structure.

E GPT Training Hyperparameters

GPT training uses a nanochat backend with per-module optimization settings selected for fast, stable convergence. The configuration combines AdamW and Muon-related parameters with specified initialization and regularization choices.

  • GPT Training Hyperparameters: The nanochat backend provides per-module optimization hyperparameters and is reported to converge quickly and stably in this setting.The implementation is based on commit e527521, with parameters summarized in Table 6.
  • Optimizers: AdamW modules use ϵ = 10^-10 for stability and no weight decay.
  • Optimizers: Muon uses β2 for variance reduction, n for Newton-Schulz iterations, and Nesterov momentum increasing from 0.85 to 0.95 over the first 300 iterations.
  • Optimizers: Muon applies weight decay of 0.025 for scenes and 0.1 for objects, scaled linearly to zero during training.
  • Weight Initialization: Token embeddings are initialized from N(0, 1), output heads from N(0, 10^-3), attention and MLP output projections at zero, and residual weights at 1.0.Input skip weights are initialized to 0.1; attention and MLP input layers use uniform bounds based on embedding dimension.

F Autoencoder Ablations

The autoencoder ablations expose a quality–sequence-length trade-off: less downsampling improves reconstruction but produces sequences unsuitable for autoregressive modeling. Codebook changes have smaller quality effects, while LFQ avoids the training cost of traditional vector quantization.

  • Downsampling: 1×, 2×, and 3× downsampling produce approximately 53k, 14k, and 3.2k latent voxels per scene, respectively.The 1× and 2× variants have higher reconstruction fidelity after 20 epochs but substantially longer sequences.
  • Downsampling: The 1× and 2× downsampling variants are infeasible for downstream autoregressive modeling because their latent sequences are substantially longer.
  • Codebook Size and Type: Increasing the codebook size to 16384 slightly improves reconstruction quality but reduces code usage to 86.4%.The larger codebook is not utilized efficiently, making its limited reconstruction gain difficult to justify downstream.
  • Codebook Size and Type: Traditional vector quantization provides slightly stronger early reconstruction but requires approximately one-third longer training because of nearest-neighbor searches.The performance gap narrows with continued optimization.
  • Codebook Size and Type: LFQ is used in the final model for improved computational efficiency, while extended training substantially improves final reconstruction metrics beyond the 20-epoch results.

G Chunked L3DG Results

L3DG produces unconditional scene chunks and scene completions, but respects partial completion conditions less than GaussianGPT at condition boundaries.

  • G Chunked L3DG Results: L3DG is evaluated on unconditional scene chunk generation and scene completion using chunked 3D-FRONT data.Figures 10 and 11 show the two qualitative settings.
  • G Chunked L3DG Results: Compared with GaussianGPT, L3DG changes style and geometry at the completion-condition boundary.This indicates weaker adherence to partial conditions in the completion setting.

H Additional Results

Additional results show unconditional scene chunk generation with 3D-FRONT training and increased diversity after ASE pretraining, alongside diverse, stable, high-quality completions.

  • H Additional Results: The model generates unconditional scene chunks when trained on 3D-FRONT alone.Additional qualitative examples are presented in Fig. 12.
  • H Additional Results: ASE pretraining improves the diversity of generated scenes.These results are shown in Fig. 13.
  • H Additional Results: Additional scene chunk completions demonstrate diversity, stability, and quality across many completions per chunk.These examples are presented in Fig. 14.
Loading 2603.26661v2…