PaperlayerView exact revision on arXiv

Source-linked AI summary

TokenSplat: Token-aligned 3D Gaussian Splatting for Feed-forward Pose-free Reconstruction

Yihui Li, Chengxin Lv, Zichen Tang, Hongyu Yang, Di Huang

arXiv:2603.00697v1cs.CV

TL;DR

Feed-forward 3D reconstruction remains constrained by accurate pose inputs, while pose-free methods can entangle viewpoint cues with scene semantics and produce redundant Gaussians. TokenSplat addresses these issues with token-aligned multi-view fusion, learnable camera tokens, and an Asymmetric Dual-Flow Decoder. It reports improved reconstruction and pose accuracy, including stronger novel-view synthesis and lower pose errors than pose-free baselines.

  • Problem

    Feed-forward reconstruction methods remain broadly limited by precise pose requirements, while pose-free approaches can entangle scene and viewpoint features and create redundant Gaussians.

  • Method

    TokenSplat performs token-level semantic multi-view aggregation and uses learnable camera tokens with an Asymmetric Dual-Flow Decoder to separate pose reasoning from scene encoding.

  • Results

    TokenSplat consistently improves reconstruction and pose accuracy over pose-free methods, including 0.335 and 0.147 lower RPE-R than VicaSplat and AnySplat under 8 views.

  • Takeaways & Limitations

    The framework supports coherent feed-forward reconstruction and stable camera estimation from unposed multi-view images without iterative refinement.

  • Takeaways & Limitations

    Broader applicability of related feed-forward methods remains limited because they typically support sparse views and require precise input poses.

Abstract

from arXiv · show

We present TokenSplat, a feed-forward framework for joint 3D Gaussian reconstruction and camera pose estimation from unposed multi-view images. At its core, TokenSplat introduces a Token-aligned Gaussian Prediction module that aligns semantically corresponding information across views directly in the feature space. Guided by coarse token positions and fusion confidence, it aggregates multi-scale contextual features to enable long-range cross-view reasoning and reduce redundancy from overlapping Gaussians. To further enhance pose robustness and disentangle viewpoint cues from scene semantics, TokenSplat employs learnable camera tokens and an Asymmetric Dual-Flow Decoder (ADF-Decoder) that enforces directionally constrained communication between camera and image tokens. This maintains clean factorization within a feed-forward architecture, enabling coherent reconstruction and stable pose estimation without iterative refinement. Extensive experiments demonstrate that TokenSplat achieves higher reconstruction fidelity and novel-view synthesis quality in pose-free settings, and significantly improves pose estimation accuracy compared to prior pose-free methods. Project page: https://kidleyh.github.io/tokensplat/.

1. Introduction

TokenSplat addresses the limited scalability, pose dependence, and redundancy of feed-forward 3D reconstruction by combining token-level cross-view aggregation with pose-aware decoding. Its design targets coherent reconstruction and disentangled camera estimation from unposed multi-view images.

  • Motivation: Existing 3DGS pipelines often optimize per scene, limiting scalability and generalization, while feed-forward methods commonly require accurate camera poses.SfM-based pose estimation is computationally expensive and can fail in challenging environments, affecting reconstruction stability.
  • Motivation: Pose-free methods jointly infer cameras and structure but often entangle scene information with viewpoint cues in shared feature embeddings.Alternating optimization can also lose reconstruction fidelity with many or densely captured images.
  • Motivation: Pixel-aligned Gaussian prediction creates redundant, overlapping primitives as views increase, causing geometric blurring, color inconsistency, and reduced fidelity.Local multi-view fusion may suppress some redundancy but remains limited in forming globally coherent structure.
  • TokenSplat: TokenSplat aligns semantically corresponding features across views and aggregates multi-scale context using coarse token positions and fusion confidence.This enables long-range cross-view reasoning and reduces redundancy before Gaussian prediction.
  • TokenSplat: Its one-to-many token-to-Gaussian mapping decouples Gaussian density from pixel resolution, producing denser splats while preserving structural and semantic coherence.The prediction head maps each aggregated token to multiple Gaussians rather than generating them strictly at pixel-level granularity.
  • TokenSplat: Learnable camera tokens and the Asymmetric Dual-Flow Decoder regulate pose–scene interaction to maintain cleaner factorization without iterative refinement loops.The decoder is designed to avoid the viewpoint–semantic entanglement risk of symmetric attention.

3. Method

TokenSplat is a feed-forward architecture that jointly predicts canonical 3D Gaussians and per-view camera poses from unposed images. It combines canonical scene decoding, asymmetric camera–image token interaction, token fusion, and supervised reconstruction and pose losses.

  • Problem Formulation: TokenSplat maps a sequence of unposed images to canonical 3D Gaussian attributes and per-view poses relative to a reference view.The predicted Gaussian attributes include centers, opacity, rotations, scales, and spherical-harmonic coefficients.
  • Architecture: A shared ViT encoder produces image tokens, while the Canonical Scene Decoder uses cross-attention to establish a consistent reference-view scene representation.The reference view is decoded using information from the other views.
  • Asymmetric Dual-Flow Decoder: The ADF-Decoder asymmetrically refines image and camera tokens so image tokens aggregate scene context while camera tokens extract geometric cues for pose estimation.Camera and image representations communicate directionally to separate viewpoint information from scene features.
  • Asymmetric Dual-Flow Decoder: Cross-view attention excludes each view’s own tokens and can restrict attention to pnv − 1 neighbors to balance complementary context with computational efficiency.Camera tokens also interact with image and camera tokens from other views to aggregate multi-view geometric cues.
  • Token Fusion for Scene Reconstruction: Token-aligned Gaussian Prediction fuses multi-scale decoder features through projection, upsampling, residual fusion, and prediction layers to produce continuous Gaussian tensors.Progressive fusion combines fine-grained details with semantic context before the final Gaussian prediction block.
  • Loss Functions: The camera pose head linearly projects decoder camera tokens, while joint pose and reconstruction supervision encourages geometrically consistent features.Rendering supervision combines L2 and LPIPS losses; pose supervision combines MSE with Unit Dual Quaternion alignment.

4. Experiment

TokenSplat is evaluated for novel-view synthesis and camera pose estimation on RE10K and ScanNet across varying view counts, including cross-dataset and long-sequence settings. It consistently outperforms pose-free baselines, preserves reconstruction quality as views increase, and benefits from token-based fusion and asymmetric decoding.

  • Evaluation Setup: Experiments evaluate novel-view synthesis and camera pose estimation on RE10K and ScanNet under sparse, diverse, cross-dataset, and 28-view settings.NVS uses PSNR, SSIM, and LPIPS; pose estimation uses ATE, RPE-t, and RPE-r.
  • Novel-View Synthesis: TokenSplat consistently outperforms pose-free NVS baselines and achieves 0.95 dB higher PSNR than FreeSplat on RE10K with 8 views.It maintains stable reconstruction quality in the 28-view setting while competing methods degrade.
  • Pose Estimation: TokenSplat reduces RPE-R by 0.335 and 0.147 versus VicaSplat and AnySplat on RE10K with 8 views, respectively.On ScanNet with 28 views, it also reduces ATE by 0.018 versus AnySplat.
  • Qualitative Analysis: Scene-level visualizations show that TokenSplat preserves coherent, geometrically consistent reconstructions from viewpoints beyond the training range.Compared methods exhibit misalignment artifacts, fusion cracks, scattered Gaussians, or poor scalability.
  • Ablation Analysis: Replacing token-aligned prediction with a pixel-aligned head lowers SSIM by 0.026 and raises RPE-r by 0.038, while AnySplat-style fusion remains 0.38 dB lower in PSNR.Replacing the ADF-Decoder raises RPE-r by 0.046 and LPIPS by 0.011, supporting the token-aligned and directionally constrained designs.

5. Conclusion

TokenSplat jointly reconstructs 3D scenes and estimates camera poses from unposed multi-view images using token-based fusion and the ADF-Decoder. It improves reconstruction and pose accuracy while maintaining stable aggregation without iterative refinement.

  • TokenSplat jointly estimates camera poses and reconstructs 3D scenes from unposed multi-view images.
  • Token-based multi-view fusion and the ADF-Decoder support clean feature disentanglement and stable aggregation.
  • TokenSplat consistently improves reconstruction and pose accuracy and generalizes well across settings.
  • The method achieves competitive zero-shot performance when trained on a single RE10K-scale dataset.

A1. More Implementation Details of Experiment

The implementation uses specified training settings and compares AnySplat under zero-shot and fine-tuned conditions because its training data differ substantially. Visual inspection reports distortions in AnySplat reconstructions that become clearer after alignment with ground-truth views.

  • Training uses a batch size of 16, backbone learning rate 2 × 10−5, other-component learning rate 2 × 10−4, and 30,000 iterations.
  • AnySplat results are reported both zero-shot and after fine-tuning on RE10K and ScanNet because its dataset is substantially larger.AnySplat’s dataset is approximately four times larger than the RE10K training scenes and 95 times larger than ScanNet images.
  • AnySplat reconstructions show distortions and deformations that become more apparent when test views are aligned with ground-truth views for evaluation.

A2. More Implementation Details of Modulation in Asymmetric Dual-Flow Decoder

The modulation design derives parameters from continuously updated camera tokens at multiple decoder stages. These parameters modulate image-token processing after normalization and attention.

  • An MLP predicts scale, shift, and gate modulation parameters from the camera token at each decoder step.
  • After self-attention, a separate MLP predicts new modulation parameters from the updated camera token.
  • The normalization operation applies scale and shift to image-token features before attention processing.The passage specifies the form i × (1 + scale) + shift.

A3. More Implementation Details of Gaussian Prediction Heads

The Gaussian Prediction heads process multi-scale decoder features, unify their channels, and fuse them from deep to shallow layers before producing the final feature output. The appendix also identifies figures describing the head structure, network details, Gaussian counts, and inference time.

  • The Gaussian Prediction heads take fused-token decoder features from decoder layers 0, 6, 9, and 12 as multi-scale inputs.
  • Each multi-scale feature is projected with channel transformation, upsampling, and fully connected layers, then unified to 256 channels.
  • Features are fused from deep to shallow layers using residual modules, upsampling, and a final fully connected layer.
  • The appendix figures cover Gaussian Prediction head structure and network details, along with Gaussian counts and inference time as input views increase.

A4. More Implementation Details of Pose Heads

The pose head converts decoder camera-token features into camera rotation and translation outputs for pose supervision. Its compact design uses a ReLU and a single fully connected layer before dual-quaternion loss computation.

  • The pose head consumes the camera token output from the Asymmetric Dual-Flow Decoder.
  • A ReLU and fully connected layer with dimensions (768, 7) produce the pose-head output.
  • The head predicts quaternion rotation and translation, which are converted to unit dual quaternions for loss computation.

A5. More Experimental Analysis

The experiments analyze how TokenSplat scales with more input views in Gaussian count, inference time, and novel-view synthesis. Token Fusion slows Gaussian growth, stabilizes inference, and progressively improves scene completeness and detail.

  • Visualization Results with Increasing Number of Views: The increasing-view visualization evaluates how reconstruction quality changes as intermediate views are added between the first and last views.
  • Efficiency: Token Fusion produces a much slower increase in Gaussian count than pixel-aligned prediction as input views increase.The Gaussian count becomes more stable than with direct Gaussian fusion.
  • Efficiency: TokenSplat maintains more stable inference time as the number of input views increases.Fusion occurs before Gaussian prediction, decoupling Gaussian count from the number of input images.
  • Visualization Results with Increasing Number of Views: Additional input views progressively improve scene completeness and visual details in novel-view synthesis, especially around railings.

A6. More Visualization Results

Additional visual comparisons show TokenSplat across RE10K and ScanNet with varied numbers of input images. The method is reported to maintain stable, superior performance and produce more complete structures than existing methods in challenging settings.

  • TokenSplat achieves stable and superior performance across varying numbers of input images and diverse RE10K and ScanNet data.
  • Rendered videos show more complete and coherent structures, while existing methods exhibit blurring, defocus-like artifacts, or severe structural issues.
  • Qualitative comparisons cover RE10K with 4 and 8 input images and ScanNet with 3, 10, and 28 input images.
Loading 2603.00697v1…