Source-linked AI summary
TokenGS: Decoupling 3D Gaussian Prediction from Pixels with Learnable Tokens
Jiawei Ren, Michal Jan Tyszkiewicz, Jiahui Huang, Zan Gojcic
TL;DR
Feed-forward 3DGS methods commonly predict ray-depth means with pixel-aligned primitives, limiting flexibility under noisy poses, inconsistent views, and changing input sizes. TokenGS directly predicts 3D coordinates with learnable Gaussian tokens and self-supervised rendering, achieving strong reconstruction performance while enabling token-space test-time scaling.
Problem
Pixel-aligned ray-depth prediction couples Gaussian count to image resolution and view number and limits robustness to noisy poses, multiview inconsistencies, and dynamic scenes.
Method
TokenGS directly regresses Gaussian 3D coordinates using self-supervised rendering loss and an encoder-decoder with learnable tokens, then tunes token embeddings at test time while preserving network parameters.
Results
TokenGS achieves state-of-the-art feed-forward reconstruction on static and dynamic scenes, with improved robustness, geometric fidelity, and balanced Gaussian distributions.
Takeaways & Limitations
Decoupling Gaussian prediction from pixels enables independent control of primitive count and input views, scene completion, dynamic modeling, and efficient test-time scaling.
Takeaways & Limitations
The method can struggle with large-scale environments and fine-grained geometric detail, while test-time token tuning typically requires several dozen optimization steps.
Abstract
from arXiv · showhide
In this work, we revisit several key design choices of modern Transformer-based approaches for feed-forward 3D Gaussian Splatting (3DGS) prediction. We argue that the common practice of regressing Gaussian means as depths along camera rays is suboptimal, and instead propose to directly regress 3D mean coordinates using only a self-supervised rendering loss. This formulation allows us to move from the standard encoder-only design to an encoder-decoder architecture with learnable Gaussian tokens, thereby unbinding the number of predicted primitives from input image resolution and number of views. Our resulting method, TokenGS, demonstrates improved robustness to pose noise and multiview inconsistencies, while naturally supporting efficient test-time optimization in token space without degrading learned priors. TokenGS achieves state-of-the-art feed-forward reconstruction performance on both static and dynamic scenes, producing more regularized geometry and more balanced 3DGS distribution, while seamlessly recovering emergent scene attributes such as static-dynamic decomposition and scene flow.
NVIDIA
TokenGS is presented as a feed-forward 3DGS reconstruction framework from posed input images. The project credits equal contributions and equal advising.
- TokenGS outputs a 3D Gaussian Splatting representation from posed input images.
- The framework uses an encoder-decoder architecture that detaches 3D Gaussians from input pixels.
- The paper marks equal contribution and equal advising.
1. Introduction
The introduction identifies limitations in pixel-aligned, encoder-only Gaussian prediction and presents TokenGS as a decoupled alternative using direct 3D coordinates and learnable tokens.
- Pixel-aligned Gaussian prediction ties primitive count to image resolution and view number, potentially producing over 8 million particles from 32 views at 512 × 512 resolution.
- Predicting Gaussian means as depths along camera rays limits correction of noisy poses and multiview inconsistencies and complicates dynamic scenes.
- TokenGS directly regresses 3D Gaussian coordinates with self-supervised rendering loss and uses learnable tokens that cross-attend to compact image features.
- This design makes Gaussian count independent of input resolution and image number while supporting scene completion, noisy-camera robustness, and dynamic content.
2. Related works
The related work situates TokenGS within 3D scene representations, feed-forward reconstruction, and emerging test-time scaling approaches for 3D vision.
- 3D scene representations include NeRF encodings, explicit 3D Gaussian Splatting, and extensions for dynamic scenes such as deformation fields and keyframe interpolation.
- Feed-forward reconstruction infers scenes in one pass from images, generalizes through large multiview datasets, and has expanded from static to dynamic and long-horizon settings.
- Test-time scaling allocates additional inference compute through adaptation, refinement, or related strategies to improve performance and robustness on challenging data.
- For 3D reconstruction, test-time scaling has mainly used per-scene optimization or geometry-model online finetuning, while scaling large 3DGS reconstruction models remains largely unexplored.
3. Method
TokenGS directly regresses Gaussian means in a shared 3D coordinate frame and uses learnable Gaussian tokens in an encoder–decoder architecture. This decouples Gaussian placement and count from camera rays, image resolution, and view count while enabling visibility-regularized training and test-time token tuning.
- 3.2. Model Architecture: Learnable Gaussian tokens decouple Gaussian positions from camera rays and the number of primitives from input resolution and view count.The encoder produces multiview image tokens, while decoder tokens cross-attend to them and decode independently of the number of input pixels.
- 3.1. Directly Regressing Gaussian Means: TokenGS directly regresses Gaussian means in a global coordinate frame instead of predicting depths along camera rays, improving representation of occluded geometry.A preliminary single-view study reports that pixel-aligned parametrization cannot recover occluded geometry, while direct coordinates avoid this limitation.
- 3.1. Directly Regressing Gaussian Means: Direct mean prediction enables view extrapolation, pose-noise robustness, and fewer spiky artifacts, but requires visibility regularization because inactive Gaussians otherwise waste capacity.Gaussians outside all camera frusta receive no rendering gradients and can become floating, noisy points.
- 3.1. Directly Regressing Gaussian Means: A visibility loss keeps Gaussians visible in at least one supervision view, addressing zero gradients from particles outside every camera frustum.The loss projects Gaussian centroids into all supervision views and penalizes their distance outside the nearest visible boundary; rendering supervision combines MSE and SSIM with visibility regularization.
- 3.2. Model Architecture: The encoder–decoder predicts Gaussian attributes from learnable tokens, with shared cross-attention projections reducing decoder memory dependence on image-token count.Each output embedding produces Gaussian attributes, and sharing image-token key–value projections changes the allocation from O(N_I D_dec) to O(N_I).
- 3.3. Test-Time Scaling: Test-time scaling combines context extension with token-tuning, keeping Gaussian count fixed while adapting only token embeddings to improve scene-specific reconstruction.Token-tuning freezes network parameters and image features, using self-supervision and a small number of gradient steps while preserving learned priors.
4. Experiments
TokenGS is evaluated across static, extrapolation, noisy-pose, dynamic, and ablation settings, showing competitive or improved reconstruction while supporting scalable token-based inference.
- Static Scene Reconstruction: TokenGS wins clearly on RE10K with 4096 Gaussians and improves reconstruction through token-tuning without overfitting from only two views.The base model is on par with baselines using 50% of their Gaussians.
- Static Scene Reconstruction: On DL3DV, TokenGS remains competitive across 2-, 4-, and 6-view settings, and token-tuning surpasses the baseline with 72% fewer Gaussians at six views.
- View Extrapolation: TokenGS outperforms baselines on view extrapolation without pointmap supervision and reconstructs more complete geometry beyond the input cameras.
- Robustness to Camera Noise: As camera-pose noise increases to 10 degrees, TokenGS gains a larger PSNR and LPIPS advantage over GS-LRM because prediction is decoupled from input rays.
- Dynamic Reconstruction: On dynamic Kubric scenes, continuous 3D-coordinate prediction yields better motion reconstruction and scene tracking than BTimer, with higher PSNR.
- Ablation Study: Visibility regularization removes floaters and improves PSNR by 0.4, while token-tuning improves quality and preserves geometry better than extensive Gaussian-parameter tuning.
- Ablation Study: Test-time context extension benefits from up to four times more inputs before saturation, and token-tuning improves PSNR from as few as 10 optimization steps.
- Ablation Study: Varying token count provides independent control over Gaussian count and input views, while tokens tend to specialize spatially and allocate more primitives to high-frequency regions.
5. Conclusion
TokenGS decouples Gaussian prediction from pixels through direct 3D-coordinate regression and learnable tokens, improving reconstruction flexibility across static and dynamic settings. Its main limitations are large-scale scenes, fine geometry, and the cost of test-time tuning.
- TokenGS directly regresses Gaussian centroids in 3D and uses learnable tokens to decouple reconstruction from input pixels.
- The design improves robustness and geometric fidelity, supports dynamic-scene modeling, and enables test-time scaling through context extension and token tuning.
- The method can struggle with large-scale environments and fine-grained geometry, while token tuning typically requires several dozen optimization steps.
A. More Implementation Details
Implementation details specify token initialization, training regularization, augmentation, camera scaling, and baseline reproduction procedures.
- New Gaussian tokens are initialized from existing tokens with small random perturbations, and the decoder uses patch size 8 so each token produces 64 Gaussians.
- Training uses gradient clipping, weight decay, random image-and-pose flips, view sampling following DepthSplat, and dataset-specific translation rescaling.
- GS-LRM and BTimer baselines were reproduced using their original training details, with published numbers used when available.