PaperlayerView exact revision on arXiv

Source-linked AI summary

ATSplat: Compact Feed-forward 3D Gaussian Splatting with Adaptive Token Expansion

In Cho, Jeonghwan Cho, Mijin Yoo, Gim Hee Lee, Seon Joo Kim

arXiv:2607.20417v2cs.CV

TL;DR

Feed-forward 3DGS often ties Gaussian placement and budget to input pixels rather than scene complexity, producing dense, redundant representations. ATSplat uses adaptive 3D anchor tokens and uncertainty-guided expansion to allocate primitives selectively, achieving state-of-the-art quality with more than 5.7× fewer Gaussians.

  • Problem

    Pixel-aligned feed-forward 3DGS ties primitive budgets to input image grids rather than reconstruction difficulty, causing redundancy in trivial regions and insufficient capacity in challenging ones.

  • Method

    ATSplat builds sparse 3D anchor tokens, decodes local Gaussians with relative 3D offsets, and adaptively expands high-uncertainty tokens in challenging regions.

  • Results

    More than 5.7× fewer Gaussians are used while ATSplat achieves state-of-the-art rendering quality across RealEstate10K and DL3DV.

  • Takeaways & Limitations

    ATSplat demonstrates that feed-forward 3DGS can preserve high rendering quality with a compact representation by allocating capacity according to scene complexity.

  • Takeaways & Limitations

    ATSplat does not prune redundant tokens during decoding and remains limited in scalability to larger scenes, more views, higher resolutions, and unposed images.

Abstract

from arXiv · show

3D Gaussian Splatting (3DGS) achieves high-quality novel-view synthesis by optimizing freely placed primitives in 3D and adaptively densifying them in under-reconstructed regions. However, this scene-adaptive capacity allocation is largely lost in existing feed-forward 3DGS methods, which commonly regress Gaussians at input pixels and lift them along camera rays. Such pixel-aligned formulations make the number and placement of primitives depend on image resolution and input viewpoints rather than scene complexity, resulting in dense and often redundant Gaussian sets. We present ATSplat, a feed-forward 3DGS framework that restores the adaptive allocation capability of 3DGS optimization through Adaptive 3D Tokens. ATSplat first lifts coarse patch-level depth and camera cues into sparse 3D anchor tokens, forming a compact scaffold of the scene. Each token is then regressed into local Gaussians with learnable 3D offsets, decoupling primitive placement from input image grids. An Adaptive Token Expansion module predicts a token-level uncertainty score, supervised by rendering error maps, and selectively expands high-uncertainty tokens through learnable expansion layers. This sparse-to-adaptive formulation enables ATSplat to concentrate primitives in challenging regions while maintaining a compact representation. Experiments on two representative datasets, RealEstate10K and DL3DV, show that ATSplat achieves state-of-the-art rendering quality while reducing the number of Gaussians by more than $5.7\times$ compared with dense feed-forward 3DGS methods. From 12 input images at $512 \times 960$ resolution, ATSplat completes reconstruction in less than a second using a single commercial GPU, and renders high-quality novel views at 1136 FPS ($512 \times 960$) with only 311K Gaussians.

1 Introduction

ATSplat restores scene-adaptive capacity allocation to feed-forward 3DGS by using adaptive 3D anchor tokens and uncertainty-guided expansion. It achieves state-of-the-art rendering quality with over 5.7× fewer Gaussians than dense pixel-aligned methods.

  • Motivation: Pixel-aligned feed-forward 3DGS ties Gaussian placement to input image grids and camera sampling, sacrificing adaptive allocation based on reconstruction difficulty.This produces dense, often redundant representations whose capacity depends on input viewpoints rather than scene complexity.
  • Method: ATSplat lifts coarse patch-level depth and camera information into sparse 3D anchor tokens, creating a compact scaffold with flexible 3D placement.The design reinterprets sparse initialization and free 3D placement as feed-forward operations centered on adaptive anchor tokens.
  • Method: Adaptive Token Expansion predicts per-token uncertainty from rendered uncertainty maps and progressively expands tokens in challenging regions without directly accessing rendering errors.This enables representational capacity to concentrate where reconstruction is difficult during a single forward pass.
  • Results: More than 5.7× fewer Gaussians are used while achieving state-of-the-art rendering quality on RealEstate10K and DL3DV.The comparison is against previous dense pixel-aligned feed-forward 3DGS methods.

2 Related Work

Related work progresses from neural radiance fields and optimized 3D Gaussian Splatting toward feed-forward Gaussian prediction that avoids per-scene optimization. ATSplat addresses the redundancy and inflexibility of dense pixel-aligned methods with sparse, scene-conditioned 3D anchor tokens and adaptive capacity allocation.

  • Neural Fields and 3D Gaussian Splatting: NeRF represents scenes as continuous radiance-density fields, while 3DGS uses anisotropic Gaussian primitives with differentiable rasterization for high-quality real-time rendering.NeRF research has targeted anti-aliasing, training speed, explicit factorization, and generalizable radiance fields.
  • Feed-Forward 3DGS: Feed-forward 3DGS methods predict Gaussian primitives directly from multi-view captures, avoiding expensive per-scene optimization through pixel-aligned lifting, cost volumes, and depth estimation.Examples include Splatter Image, pixelSplat, MVSplat, and DepthSplat.
  • Limitations of Pixel Alignment: Dense pixel-aligned formulations tie primitive counts to image resolution and view number, producing redundant Gaussians in simple regions while limiting capacity in challenging regions.This motivates sparse-to-adaptive capacity allocation.
  • ATSplat: ATSplat builds a scene-conditioned scaffold from sparse 3D anchor tokens lifted from coarse depth and cameras, then predicts local Gaussians with learnable 3D offsets beyond camera rays.It differs from approaches based on dense pixel-aligned sets, fixed voxel lattices, or purely learnable global tokens.

3 Problem Formulation

The problem is to reconstruct a 3D Gaussian scene representation from posed multi-view images in one feed-forward pass, avoiding per-scene optimization. Existing pixel-aligned methods tie Gaussian allocation to input pixels, causing redundancy in trivial regions and insufficient capacity in challenging ones.

  • Problem setting: The task is single-pass reconstruction of a 3D Gaussian representation from V posed multi-view images, without costly per-scene optimization.The model receives multi-view images and camera poses and predicts Gaussian attributes in a feed-forward manner.
  • Gaussian representation: Each Gaussian is parameterized by center μ_g, rotation quaternion q_g, scale s_g, opacity α_g, and spherical harmonics SH_g.
  • Pixel-aligned formulation: Pixel-aligned methods determine each Gaussian center from predicted depth and the corresponding camera ray, tying every primitive to an input pixel.This reduces center prediction to per-pixel depth estimation, with o_v and r_v(x) defining the camera origin and ray direction.
  • Pixel-aligned formulation: Pixel grids determine the Gaussian budget and spatial distribution, producing redundant primitives in trivial regions while under-allocating capacity to challenging regions.

4 ATSplat Framework

ATSplat restores scene-adaptive Gaussian placement in feed-forward 3DGS by organizing sparse 3D anchor tokens around sparse initialization, free 3D placement, and adaptive capacity allocation. Its pipeline extracts multi-view features and depths, refines anchors with adaptive expansion, and decodes them into locally placed Gaussians.

  • Framework overview: ATSplat recasts sparse initialization, free 3D placement, and adaptive capacity allocation as feed-forward operations centered on 3D anchor tokens.Sparse initialization unprojects coarse patches into 3D, while local primitive offsets enable free placement and adaptive expansion allocates capacity where needed.
  • Image-to-3D decoder: The image-to-3D decoder refines anchors through cross-attention to fine-grained image features and selectively expands tokens associated with under-reconstructed regions.Its blocks combine an Adaptive Token Expansion module with cross-attention layers, increasing representational capacity only where needed.
  • Multi-view image encoder: The multi-view encoder extracts coarse cross-view patch features, injects Plücker raymap geometry, and applies global self-attention across views.Each view is tokenized into coarse patches using a frozen DINO backbone before cross-view processing.
  • Initializing sparse anchor tokens: Patch-level depths and features are unprojected along camera rays to initialize a sparse 3D anchor scaffold.Each anchor is positioned at the unprojected 3D coordinate and carries its encoded patch feature, with local context optionally aggregated from k-nearest neighbors.
  • Anchors to local 3D Gaussians: Each refined anchor is decoded into K local Gaussian primitives whose centers are predicted relative to the anchor, freeing placement from the input pixel grid.A lightweight Gaussian head based on 2-layer MLPs maps anchor features to Gaussian attributes and local centers.

5 Adaptive Token Expansion

ATSplat predicts anchor-level uncertainty with a lightweight MLP and selectively expands high-uncertainty anchors into child tokens. The uncertainty is supervised by 2D reconstruction errors, enabling adaptive allocation in one forward pass without inference-time renderings or back-propagation.

  • Uncertainty Prediction: A lightweight MLP estimates each anchor token’s uncertainty to identify regions requiring additional representational capacity.Feed-forward decoding cannot directly access rendering errors or gradients, so the MLP approximates each anchor’s rendering error.
  • Token Expansion: Top-ρ_l anchors are expanded through a linear projection into M child tokens sharing the parent anchor coordinate.Each child feature is formed as f_i,m = f_i + Δf_i,m, and expanded and unselected tokens continue to the next decoder block.
  • Inference: The expansion mechanism identifies under-reconstructed regions in a single forward pass without auxiliary renderings or back-propagation at inference.The expanded and unselected tokens are concatenated and passed onward through the decoder.
  • 2D Uncertainty Supervision: 2D reconstruction errors supervise predicted uncertainty maps generated by rasterizing anchor scores across their K Gaussians.Intermediate Gaussian sets are produced after each decoder block, and D-SSIM is used empirically to compute the error maps.

6 Training

ATSplat is trained end-to-end with final and intermediate rendering losses plus uncertainty supervision. The final loss combines MSE with a perceptual term, while intermediate losses use LD-SSIM for efficiency.

  • Training: ATSplat is trained end-to-end using a final rendering loss, intermediate rendering losses, and uncertainty supervision.The final rendering loss combines MSE with a perceptual term.
  • Training: λ_p = 0.5 weights the perceptual term in the final rendering loss.
  • Training: Each intermediate Gaussian set G(l) is supervised with an auxiliary rendering loss to train uncertainty heads at intermediate decoder blocks.
  • Training: LD-SSIM replaces the perceptual term in intermediate rendering losses for efficiency.
  • Training: λ_interm = 0.5 and λ_unc = 0.1 are used in all experiments.

7 Experiments

Experiments on RealEstate10K and DL3DV show that ATSplat preserves rendering quality while using substantially fewer Gaussians than dense feed-forward baselines. Analyses further attribute this efficiency to sparse 3D anchors, adaptive expansion in uncertain regions, and local Gaussian offsets.

  • Benchmark results: ATSplat achieves comparable rendering quality on RealEstate10K using only 23K Gaussians, a 5.7× reduction versus dense pixel-aligned formulations.The evaluation uses two input views at 256 × 256 resolution.
  • Benchmark results: ATSplat outperforms prior feed-forward baselines on DL3DV across 2, 4, and 6 input views while using substantially fewer Gaussians.DL3DV contains more challenging scenes with complex geometry, larger spatial extents, and more diverse layouts.
  • High-resolution evaluation: At 512 × 960 resolution, ATSplat reconstructs scenes in a single forward pass in less than one second using far fewer Gaussians than dense pixel-aligned methods.Pixel-aligned methods typically produce more than a million Gaussians at high resolution.
  • Component analysis: Expanded-token children spread around parent anchors and cover distinct nearby regions through local Gaussian offsets and features.ATE does not explicitly update anchor coordinates during expansion.
  • Component analysis: ATE expands tokens in uncertain regions, where predicted uncertainty is high around larger rendering errors, fine structures, and visually complex areas.Selected tokens concentrate in these regions, placing additional Gaussian capacity where it is needed.
  • Ablations: The anchor-offset design gives the best reconstruction quality, while random selection, FPS, and STE fail to improve adaptive expansion as effectively.Pixel-aligned anchors limit expansion flexibility, whereas coarse 3D anchors provide scene structure and local offsets preserve placement flexibility.

8 Conclusion and Future Works · A Additional Implementation Details

ATSplat restores scene-adaptive capacity allocation in feed-forward 3DGS through sparse adaptive 3D anchor tokens, local Gaussian decoding, and progressive expansion of challenging regions. Future work includes pruning redundant tokens and improving architectural scalability, while implementation details document the model and training configuration for reproducibility.

  • 8 Conclusion and Future Works: ATSplat restores scene-adaptive capacity allocation in feed-forward 3DGS.It replaces dense Gaussian assignment to input image grids with a sparse-to-adaptive formulation.
  • 8 Conclusion and Future Works: ATSplat builds a sparse set of adaptive 3D anchor tokens instead of densely assigning Gaussians to input image grids.This design decouples Gaussian placement from input pixels.
  • 8 Conclusion and Future Works: ATSplat decodes anchor tokens into local Gaussians with 3D relative offsets.The offsets support Gaussian placement independently of input pixels.
  • 8 Conclusion and Future Works: ATSplat progressively selects and expands tokens associated with challenging regions.This mechanism implements adaptive capacity allocation within the sparse-to-adaptive formulation.
  • 8 Conclusion and Future Works: ATSplat does not prune tokens that become redundant during decoding.A pruning mechanism analogous to 3DGS is proposed as a direction for improving decoder efficiency and capacity allocation.
  • 8 Conclusion and Future Works: Architectural improvements remain open for more scalable designs and extension to larger-scale scenes.The passage identifies scalability as a future-work direction beyond adaptive token expansion.
  • A Additional Implementation Details: Additional implementation details describe the model architecture and training configuration used in the experiments.These details are included to facilitate reproducibility.

A.1 Training Details … B.1 Results with Varying Viewpoint Overlaps

ATSplat uses memory-efficient BF16 training and a multi-branch architecture with normalized camera inputs and adaptive token-to-Gaussian budgets. Across viewpoint-overlap regimes, it performs best at small overlap, matches pixel-aligned methods at medium overlap, and lags at large overlap.

  • A.1 Training Details: BF16 precision reduces training memory while maintaining stable optimization, using PyTorch scaled dot-product attention and AdamW with weight decay 0.05.The initial learning rate is 2 × 10−4 with cosine annealing.
  • A.2 Architecture Details: 768-dimensional features combine DINO-based coarse and fine branches, using patch sizes 14 and 8 respectively.The fine patch grid has twice the spatial resolution of the coarse grid.
  • A.2 Architecture Details: Raw-image patches with patch size 16 are concatenated with projected DINO features, while fine-resolution features receive coarse-level encoder context.The fused features are passed through a linear projection layer to the model dimension.
  • A.2 Architecture Details: Two-layer MLP heads predict coarse depth and Gaussians after LayerNorm, while the uncertainty head uses softplus and normalized camera poses.Camera poses are normalized using the average camera pose and a transformed camera coordinate system.
  • A.3 Total Gaussian Budgets: Fixed selection ratios determine reported experimental budgets, while uncertainty thresholding also enables per-scene dynamic budgets at inference.The number of tokens after decoding is mainly determined by the input configuration under fixed ratios.
  • A.3 Total Gaussian Budgets: 16 Gaussians are decoded from each anchor token, giving a total budget of G = K N_final.The first ATE block uses selection ratio 0.8 instead of 0.5 in the marked configuration.
  • B.1 Results with Varying Viewpoint Overlaps: ATSplat is strongest under small viewpoint overlap, comparable to pixel-aligned baselines at medium overlap, and weaker at large overlap.Small overlap indicates wider camera baselines; large overlap indicates narrow baselines with target views close to inputs.

B.2 Results with Square-Cropped Inputs … C.3 Expansion at each decoder block

ATSplat maintains compact, high-quality reconstruction across square-cropped, cross-dataset, extrapolated-view, and unified-model settings. Ablations show that uncertainty-based expansion enables scene-dependent budgets, while more per-token Gaussians and later decoder-block expansion progressively improve detail.

  • B.2 Results with Square-Cropped Inputs: ATSplat achieves higher quality across all metrics than YoNoSplat on square-cropped DL3DV inputs while using 4.1× fewer Gaussians on average.The comparison uses 224×224 resolution and 6 input views, with both input and target images square-cropped.
  • B.3 Cross-Dataset Generalization: ATSplat attains the best PSNR and comparable SSIM and LPIPS in zero-shot transfer from RealEstate10K to DL3DV, using significantly fewer Gaussians without fine-tuning.All models are trained on RealEstate10K and evaluated directly on DL3DV at 256×256 resolution with 2 input views.
  • B.4 Extrapolation: ATSplat achieves better results as extrapolated target viewpoints move farther from the input views by freely placing primitives in 3D space.The setting renders the 10 frames before and after the input-view window on DL3DV with 6 input views.
  • B.5 A Unified Model across Datasets and View Counts: ATSplat is evaluated with a single unified model jointly trained across both datasets and all input-view counts, rather than separate models for each resolution.This verifies that its components are not tied to a specific resolution or number of input views.
  • C.3 Expansion at each decoder block: Progressively enabling token expansion from earlier to later decoder blocks refines renderings by recovering fine-grained texture, thin structures, and high-frequency detail.With expansion disabled, limited representation capacity leaves some fine details missing.
  • C.1 Dynamic Gaussian Budgets via Uncertainty Thresholds: Uncertainty-threshold expansion produces scene-dependent Gaussian budgets at inference without retraining or architectural changes, with budgets varying by up to 6.9× across scenes.Lower thresholds expand more tokens and improve quality; the smallest threshold reaches state-of-the-art quality while using 3.4× fewer Gaussians than iLRM.
  • C.2 Scaling the Gaussian Budget: Rendering quality improves consistently as the number of Gaussians per anchor token increases from 4 to 64, with diminishing returns beyond the default K = 16.The experiment is conducted on RealEstate10K with 2 input views while keeping all other settings identical.

C.4 Ablations on DL3DV

DL3DV ablations with 6 input views test whether ATSplat’s core components remain effective on larger, more complex scenes. Absolute Gaussian-center regression without anchor initialization fails to converge, underscoring the importance of coarse 3D anchors as scene extent grows.

  • Ablation setup: 6 input views are used on DL3DV to evaluate the 3D anchor designs and Adaptive Token Expansion module on larger, more complex scenes.The corresponding results are reported in Tables 14 and 15.
  • 3D anchor designs: Absolute Gaussian-center regression without anchor initialization fails to converge on DL3DV.This result indicates that coarse 3D anchors become more critical as the spatial extent of the scene grows.

D Additional Qualitative Results

Additional qualitative comparisons evaluate ATSplat on RealEstate10K and DL3DV across multiple resolutions and input-view settings, with further rendered sequences provided in the supplementary video.

  • Qualitative comparisons: Additional comparisons cover RealEstate10K and DL3DV, with results shown at 256 × 256, 256 × 448, and 512 × 960 resolutions.RealEstate10K uses 2 input views, while DL3DV results use 6 or 12 input views depending on resolution.
  • Supplementary results: The supplementary video provides more comprehensive rendered sequences of ATSplat and side-by-side comparisons.These supplementary results extend the qualitative evidence beyond the figures.
Loading 2607.20417v2…