Source-linked AI summary
AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Views
Lihan Jiang, Yucheng Mao, Linning Xu, Tao Lu, Kerui Ren, Yichen Jin, Xudong Xu, Mulin Yu, Jiangmiao Pang, Feng Zhao, Dahua Lin, Bo Dai
TL;DR
Novel-view synthesis from uncalibrated images must avoid costly pose estimation and per-scene optimization while remaining effective for dense views. AnySplat predicts Gaussian scene primitives and camera parameters in one feed-forward pass, using voxelization and pseudo-label distillation; it reports competitive sparse- and dense-view results, with seconds-level inference, while retaining known artifacts and scaling limits.
Problem
Existing high-fidelity pipelines require camera poses and costly per-scene optimization, while feed-forward methods face rising complexity with dense views.
Method
AnySplat predicts 3D Gaussians and camera parameters from uncalibrated images, applies differentiable voxelization, and trains with pseudo-label distillation.
Results
AnySplat demonstrates robust, competitive reconstruction and rendering on sparse and dense multiview benchmarks, with high-fidelity rendering in seconds at inference.
Takeaways & Limitations
The unified feed-forward pipeline supports unconstrained inputs while reducing Gaussian processing by 30–70% without sacrificing comparable rendering quality.
Takeaways & Limitations
AnySplat exhibits artifacts in skies, specular highlights, and thin structures, and can slow for very high-resolution inputs or many views.
Abstract
from arXiv · showhide
We introduce AnySplat, a feed forward network for novel view synthesis from uncalibrated image collections. In contrast to traditional neural rendering pipelines that demand known camera poses and per scene optimization, or recent feed forward methods that buckle under the computational weight of dense views, our model predicts everything in one shot. A single forward pass yields a set of 3D Gaussian primitives encoding both scene geometry and appearance, and the corresponding camera intrinsics and extrinsics for each input image. This unified design scales effortlessly to casually captured, multi view datasets without any pose annotations. In extensive zero shot evaluations, AnySplat matches the quality of pose aware baselines in both sparse and dense view scenarios while surpassing existing pose free approaches. Moreover, it greatly reduce rendering latency compared to optimization based neural fields, bringing real time novel view synthesis within reach for unconstrained capture settings.Project page: https://city-super.github.io/anysplat/
1 Introduction
AnySplat addresses the tension between geometry-prior models and high-fidelity, efficient novel-view synthesis from uncalibrated multiview images. It combines feed-forward prediction with pseudo-label distillation and voxel-guided Gaussian processing.
- Foundation models accelerate 3D reconstruction but can lose fine detail, photorealism, and geometric consistency on highly overlapping inputs.
- NeRF-style pipelines provide high rendering fidelity but require pose estimation and costly per-scene optimization, limiting practical use as input views grow.
- AnySplat augments a foundation model with a lightweight rendering head trained through pseudo-label distillation to refine appearance and synthesize views.
- AnySplat predicts Gaussian parameters and camera poses from unposed multiview images, while voxelization removes 30–70% of redundant primitives without comparable rendering-quality loss.
- The model uses only RGB images and trains across nine diverse datasets, targeting scalable zero-shot generalization to unconstrained scenes.
2 Related Work
Prior work spans high-fidelity but optimization-heavy neural rendering, pose-aware feed-forward reconstruction, and pose-free models that jointly infer scene geometry and cameras. AnySplat is positioned within the latter effort toward efficient end-to-end reconstruction.
- NeRF pioneered high-quality novel-view synthesis through continuous volumetric fields, but expensive volume rendering prevents real-time performance.
- 3D Gaussian Splatting represents scenes with anisotropic Gaussians and differentiable rasterization, enabling photorealistic 1080p rendering above 30 FPS.
- Most view-synthesis methods require tens of minutes or hours to optimize densely captured data.
- Pose-aware generalizable methods assume calibrated images and corresponding camera poses when rapidly reconstructing 3D models.
- Pose-free methods use images alone and commonly predict camera poses alongside reconstructed geometry in end-to-end models.
3 Method
AnySplat encodes uncalibrated views with a transformer, predicts Gaussian scene representations and camera parameters, and voxelizes primitives for efficient rendering. Geometry and camera supervision combine rendered consistency with pseudo-label distillation.
- 3.1 Problem Setup: Given one to hundreds of uncalibrated images, AnySplat directly predicts 3D Gaussian primitives representing the reconstructed scene.
- 3.2 Pipeline: Decoder heads predict camera parameters, depth maps, and Gaussian attributes, which are back-projected and rendered into images and depth maps.
- 3.2 Pipeline: The geometry transformer patchifies all views and processes their tokens using alternating frame and global attention.
- 3.2 Pipeline: Differentiable voxelization clusters Gaussian centers into voxels to address complexity beyond 32 views and reduce the primitives processed during rendering.
- 3.3 Training and Inference: Distillation improves training stability, while the model achieves strong novel-view rendering despite training only on context views.
- 3.3 Training and Inference: The training objective combines RGB, geometry, pose, and depth losses, using confidence-selected pixels and pretrained-model pseudo-labels for supervision.
4 Experiments
AnySplat is evaluated on zero-shot novel-view synthesis across sparse and dense inputs, with comparisons covering quality, speed, pose estimation, geometric consistency, and component ablations.
- Novel-view synthesis: AnySplat improves sparse-view zero-shot rendering over NoPoSplat and Flare, with faster inference as the number of input views increases.The reported advantages are attributed to diverse training data, random view selection, and more accurate geometry and pose estimation.
- Novel-view synthesis: In dense-view settings above 32 views, AnySplat outperforms 3D-GS and Mip-Splatting while reconstructing an order of magnitude faster.The paper reports finer, cleaner geometry and more detailed novel-view rendering than these optimization-based methods.
- Pose estimation and geometry consistency: Rendering-based supervision slightly outperforms VGGT for relative pose estimation and improves alignment between rendered and predicted depth maps during training.The pose evaluation uses ten random frames on RealEstate10K and CO3Dv2, while depth alignment is measured on Hypersim.
- Ablation study: Removing distillation losses degrades rendering quality, depth, and pose, while geometry consistency loss reduces AbsRel by 1.7% and improves δ1 accuracy by 1.6%.The distillation losses counteract overfitting to input views, and the consistency loss encourages coherent multi-view geometry.
- Ablation study: Differentiable voxelization reduces Gaussian redundancy, limits growth with additional context views, and lowers rendering GPU memory consumption despite slightly changing performance.The module merges pixel-wise primitives into voxel-wise Gaussians and eventually reaches saturation as context views increase.
- Ablation study: The third training configuration achieves PSNR gains of 0.41 dB and 0.35 dB over configurations 1 and 2, balancing pretrained representation preservation with attention adaptation.These results support retaining pretrained visual representations while adapting the attention mechanism.
5 Conclusion and Future Works
AnySplat is presented as a feed-forward 3D reconstruction model for unconstrained, uncalibrated inputs, combining a rendering head, geometry-consistency enhancement, and pseudo-label distillation. It achieves competitive sparse- and dense-view results with second-scale inference, while retaining artifacts and scaling limitations in challenging settings.
- AnySplat integrates a lightweight rendering head, geometry-consistency enhancement, and pseudo-label knowledge distillation into feed-forward 3D reconstruction.
- The model achieves robust and competitive results on sparse- and dense-view reconstruction and rendering benchmarks using unconstrained, uncalibrated inputs.
- AnySplat produces feed-forward Gaussian reconstructions and high-fidelity renderings in just seconds at inference time.
- AnySplat still exhibits artifacts in skies, specular highlights, and thin structures.
- Dynamic scenes, varying illumination, and very high-resolution or large-view inputs expose stability and compute–resolution limitations.The number of Gaussians scales with input and voxel resolution, which can slow performance.
3DGS MipSplatting AnySplat
This section identifies the comparison methods and supporting material for evaluating AnySplat against sparse- and dense-view baselines. It also points to appendices containing additional technical details and experimental results.
- Sparse-view comparisons use FLARE and NoPoSplat, while dense-view comparisons use 3DGS and MipSplatting.
- The qualitative comparison figure notes slight misalignment between rendered novel views and ground truth, likely from imperfect estimated poses.
- The appendices provide additional technical details and experimental results supporting the main findings.
A Experiment Details
The experiments train AnySplat on a heterogeneous nine-dataset mixture and evaluate it across varied indoor scenes. View counts and selection procedures define sparse- and dense-view test conditions, while VGGT initialization supplies a geometric prior.
- Training Setting: Training uses a heterogeneous mixture of nine synthetic and real-world datasets, with one dataset randomly sampled each iteration.The datasets span synthetic indoor scenes and include Hypersim, ARKitScenes, BlendedMVS, ScanNet++, CO3D-v2, Objaverse, Unreal4K, WildRGBD, and DL3DV.
- Training Setting: Dataset sampling follows probabilities listed in Table 6, determining each dataset’s relative training frequency.
- Model Initialization: The geometry-transformer backbone is initialized from VGGT-pretrained weights, while the Gaussian-prediction head learns appearance and density from scratch.Gaussian-head weights use a zero-mean Gaussian initialization with standard deviation 0.02, and biases are zero.
- Evaluation Setting: Evaluation uses seven representative scenes from VR-NeRF and Mip-NeRF360 to stress-test sparse- and dense-view reconstruction and rendering.The scenes cover varied indoor layouts, camera configurations, densities, and appearance variations.
- Evaluation Setting: Dense-view experiments use 32, 48, or 64 input images after holding out one of every eight images as test views.Sparse-view experiments hold out one of every two images, using the same view-selection procedure.
B More Results
Additional results show that AnySplat remains competitive with per-scene optimization under fixed test views and produces cleaner qualitative renderings across sparse and dense settings. Failure cases concentrate around illumination changes, occlusions, dynamics, specularities, and fine geometry.
- Same Test Views: With fixed test views, more input views improve rendering quality for both feed-forward and per-scene optimization methods.
- Same Test Views: AnySplat’s rendering quality is consistently competitive with per-scene optimization methods across the fixed-test-view dense-view setup.
- Failure Case: Failure cases include variable illumination, transient occlusions, specular highlights, dynamic scenes, and fine-grained geometry.The figure examples include the Brandenburg Gate, a sedan, a bus scene, and bicycle thin structures.
- More Comparisons: AnySplat delivers higher visual quality, reliable geometry, and finer details than NoPoSplat and FLARE for sparse-view inputs.
- More Comparisons: For dense-view inputs, AnySplat produces cleaner renderings with fewer artifacts than 3D-GS and Mip-Splatting, which tend to overfit training views.
- More Comparisons: Figures 11 and 12 visualize AnySplat results on Mip-NeRF360 and VR-NeRF scenes, respectively.