Source-linked AI summary
Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers
Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, Song-Hai Zhang
TL;DR
Single-view images provide insufficient information for complete, high-quality 3D reconstruction, while diffusion-based methods can require time-intensive optimization. TGS uses transformer decoders to construct a hybrid Triplane-Gaussian representation, and experiments report higher quality and faster runtime than prior state-of-the-art techniques. Its limitations include dependence on initial geometry and camera parameters, with potentially blurry backsides.
Problem
Single-view images provide insufficient information for recovering complete, high-quality object shape and texture, while diffusion-based methods can require time-intensive optimization.
Method
TGS uses transformer-based point and triplane decoders to construct a hybrid representation combining explicit point-cloud geometry with implicit triplane features for Gaussian decoding.
Results
TGS outperforms existing baselines in reconstruction quality and speed, achieving object reconstruction in a matter of seconds through feed-forward inference.
Takeaways & Limitations
The hybrid representation supports fast, high-resolution rendering and efficient end-to-end training while combining explicit geometry with implicit Gaussian-property encoding.
Takeaways & Limitations
Rendering quality depends on initial geometry, backsides tend to be blurry, and the method relies on camera parameters, restricting applications.
Abstract
from arXiv · showhide
Recent advancements in 3D reconstruction from single images have been driven by the evolution of generative models. Prominent among these are methods based on Score Distillation Sampling (SDS) and the adaptation of diffusion models in the 3D domain. Despite their progress, these techniques often face limitations due to slow optimization or rendering processes, leading to extensive training and optimization times. In this paper, we introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation. This hybrid representation strikes a balance, achieving a faster rendering speed compared to implicit representations while simultaneously delivering superior rendering quality than explicit representations. The point decoder is designed for generating point clouds from single images, offering an explicit representation which is then utilized by the triplane decoder to query Gaussian features for each point. This design choice addresses the challenges associated with directly regressing explicit 3D Gaussian attributes characterized by their non-structural nature. Subsequently, the 3D Gaussians are decoded by an MLP to enable rapid rendering through splatting. Both decoders are built upon a scalable, transformer-based architecture and have been efficiently trained on large-scale 3D datasets. The evaluations conducted on both synthetic datasets and real-world images demonstrate that our method not only achieves higher quality but also ensures a faster runtime in comparison to previous state-of-the-art techniques. Please see our project page at https://zouzx.github.io/TriplaneGaussian/.
1. Introduction
Single-view 3D reconstruction is difficult because one image lacks sufficient information for complete, high-quality shape and texture recovery. TGS addresses limitations of diffusion-based optimization and representation choices with a hybrid Triplane-Gaussian approach for fast, high-quality reconstruction and novel-view synthesis.
- Motivation: Single-image reconstruction is challenging because the input provides ambiguous and incomplete information about an object’s full shape and texture.The problem is important for applications including augmented reality and virtual reality.
- Limitations of prior methods: Diffusion-based novel-view methods struggle with view consistency because they lack 3D structural constraints and often require time-intensive, object-specific optimization.These methods distill diffusion priors and monocular cues into neural 3D representations.
- Representation trade-offs: Triplane representations are compact and expressive, but their implicit volume rendering introduces substantial runtime and memory costs.This limits training efficiency and rendering practicality.
- Proposed approach: TGS combines an explicit point cloud with an implicit triplane field, separating object position from Gaussian properties such as opacity and spherical harmonics.The point cloud captures rough geometry, while the triplane refines geometry and encodes Gaussian attributes.
- Results: Experiments show that TGS surpasses existing baselines in reconstruction quality and speed, producing objects in a matter of seconds.The method uses feed-forward inference and supports geometry reconstruction and novel-view synthesis.
2. Related Works
Prior work spans feed-forward reconstruction from learned 3D priors and conditional generation with GANs or diffusion models. Neural implicit representations offer strong rendering quality but remain less efficient than point-based rendering, motivating hybrid alternatives.
- Single-view reconstruction: Single-view reconstruction is ill-posed because images provide insufficient geometry cues, leading researchers to learn priors from synthetic models or real scans.Encoder-decoder methods reconstruct several 3D representations but are often category-specific and weak on texture.
- Framework context: The framework overview combines image and camera features with point-cloud and triplane decoders before decoding 3D Gaussians for novel-view rendering.Point upsampling densifies geometry, while geometry-aware encoding transfers point-cloud features into the triplane latent representation.
- Image-conditioned 3D generation: Conditional generation methods use GANs or diffusion models to sample 3D content from pretrained distributions rather than directly regressing a representation.Examples include methods generating point clouds or parameters of implicit functions.
- Diffusion-based methods: Score-distillation approaches combine 2D diffusion and CLIP priors with per-shape optimization, but off-the-shelf diffusion models may produce inconsistent novel views.View-conditioned diffusion methods seek to address this issue.
- Neural representations and rendering: Neural implicit representations such as NeRF achieve strong novel-view synthesis through differentiable volume rendering, but their efficiency remains below point-based rendering.Point-based methods benefit from fast GPU/CUDA rasterization, although some rely on 2D CNNs.
3. Method
The method reconstructs a hybrid Triplane-Gaussian representation from one image using transformer decoders for explicit geometry and implicit features. Gaussian attributes are then decoded and rendered with efficient splatting, while projection- and geometry-aware conditioning improve consistency and detail.
- Hybrid representation: The method first predicts coarse point-cloud geometry and then constructs a hybrid representation combining explicit points with implicit triplane features.The design supports efficient rendering without discarding feature-based representation capacity.
- Hybrid Triplane-Gaussian: The point cloud provides explicit geometry, while triplane queries produce features from which Gaussian attributes can be decoded.The triplane consists of three orthogonal feature planes whose interpolated features are concatenated for each queried position.
- Gaussian decoding: An MLP decodes opacity, anisotropic covariance parameters, spherical-harmonics coefficients, and a position offset for each point.The offset allows Gaussian positions to differ from surface-point locations.
- Projection-aware conditioning: Projected local image features are concatenated with triplane features to condition Gaussian decoding on image evidence and explicit geometry.This projection-aware conditioning uses the input camera pose and point cloud to calculate local features.
- Rendering: Gaussian Splatting renders anisotropic splats with tile-based rasterization, enabling fast alpha blending and efficient backward passes.The rendering procedure supports end-to-end training with higher-resolution images and lower GPU memory cost.
- Transformer decoders: Two transformer decoders reconstruct the point cloud and triplane from image tokens, with camera modulation and geometry-aware encoding guiding their viewpoint-conditioned predictions.The point decoder predicts coarse geometry, while the triplane decoder predicts the implicit field used for Gaussian-property decoding.
- Point upsampling: The point cloud is densified from 2048 to 16384 points to improve the number of Gaussians available for novel-view synthesis.The upsampling module uses two-step Snowflake point deconvolution with shape conditioning.
- Training: Training combines 3D point-cloud supervision with 2D rendering losses for the reconstructed images.The losses include Chamfer Distance, Earth Mover’s Distance, pixel-wise MSE, SSIM, and perceptual LPIPS loss.
4. Experiments
Experiments evaluate TGS against reconstruction, novel-view synthesis, representation, ablation, and runtime baselines on GSO. The method achieves strong geometry and view consistency, with fast feed-forward reconstruction and rendering, while remaining limited by initial geometry, backside blur, and camera-parameter dependence.
- Evaluation setup: The evaluation compares TGS with Point-E, Shap-E, Zero-1-2-3, and One-2-3-45 on geometry reconstruction and novel-view synthesis.GSO provides the evaluation images, while Objaverse-LVIS supplies 46K training models across 1,156 categories.
- Novel-view synthesis: TGS delivers higher-quality and more consistent novel views than the compared baselines while preserving detailed textures and generalizing to unseen objects.Zero-1-2-3 can produce plausible but inconsistent views, whereas One-2-3-45 is consistent but limited in image quality; TGS combines quality and consistency.
- Runtime efficiency: Feed-forward inference and efficient rasterization make TGS faster for both reconstruction and rendering than the evaluated baselines.Runtime comparisons include diffusion sampling for Zero-1-2-3 and volume rendering from One-2-3-45’s implicit feature volume.
- Representation comparison: The TriplaneGaussian representation achieves the best quantitative novel-view-synthesis results among the compared representations.Directly decoded generalizable 3D Gaussians perform worst across metrics, while projection-aware conditioning with explicit geometry improves texture detail.
- Ablation study: Projection-aware conditioning improves point-cloud geometry and novel-view synthesis, while geometry-aware encoding further improves backside texture and explicit–implicit alignment.Image-based shape codes improve Volume IoU over point-cloud tokens, and projection-aware conditioning improves CD and all reported view-synthesis metrics.
- Limitations: Rendering quality depends strongly on the initial point-cloud geometry; severe prediction errors hinder recovery of missing regions, while backside results can remain blurry and camera parameters restrict applications.The method is not probabilistic, and its use of camera parameters constrains its application scope.
5. Conclusion
TGS reconstructs 3D objects from a single image using feed-forward inference and a hybrid Triplane-Gaussian representation. It combines transformer decoders with Gaussian decoding to achieve high-quality reconstruction and novel-view synthesis while maintaining fast reconstruction and rendering.
- TGS uses point-cloud and triplane transformer decoders to construct a hybrid Triplane-Gaussian representation from one image.A 3D Gaussian decoder then generates properties such as opacity and spherical harmonics.
- Projection-aware conditioning and geometry-aware encoding are applied to improve consistency with the input observation.
- The method achieves high-quality geometry reconstruction and novel-view synthesis while maintaining fast reconstruction and rendering speeds.