Source-linked AI summary
AsySplat: Efficient Asymmetric 3D Gaussian Splatting for Long-Sequence Scene Modeling
Yingji Zhong, Dave Zhenyu Chen, Fuzhao Ou, Youyu Chen, Zhihao Li, Lanqing Hong, Dan Xu
TL;DR
Long-sequence, high-resolution novel view synthesis in generalizable 3D Gaussian Splatting incurs redundant computation from fine-grained inter-frame processing. AsySplat decouples geometry and appearance with asymmetric branches, matching optimization-based performance at nearly 800× speedup while using fewer parameters and lower overhead.
Problem
Generalizable 3D Gaussian Splatting methods face redundant, costly token interactions when modeling long-sequence, high-resolution novel views.
Method
AsySplat decouples geometry and appearance into asymmetric branches, using coarse-grained inter-frame processing for geometry and fine-grained intra-frame processing for appearance.
Results
Nearly 800× faster inference matches optimization-based performance, while zero-shot performance surpasses prior generalizable methods with 30% fewer parameters and lower overhead.
Takeaways & Limitations
Task-aware computation allocation enables smaller generalizable 3DGS models to deliver strong long-sequence novel view synthesis performance more efficiently.
Takeaways & Limitations
AsySplat can lag behind optimization-based methods in visual fidelity, including higher LPIPS, because it lacks error-driven densification for recovering fine details.
Abstract
from arXiv · showhide
Recent generalizable 3D Gaussian Splatting models have advanced long-sequence novel view synthesis (NVS), but at the cost of substantial redundant computation. We identify that the redundancy can be mitigated based on two observations: (i) high-precision geometry is not strictly required for high-quality NVS; (ii) appearance learning is generally easier than geometry recovery. Motivated by these insights, we propose an asymmetric architecture that decouples geometry and appearance modeling. The geometry branch processes coarse-grained tokens with most of the parameters for multi-view reconstruction, while the appearance branch operates on fine-grained tokens to capture details using significantly fewer parameters. The two branches interact through bilateral connections, enabling mutual guidance for their respective tasks. This task-aware asymmetry reduces the computational redundancy and allocates the computation more judiciously, thereby increasing parameter efficiency and enabling smaller models to achieve strong performance. On 32-view 960P inputs, our model matches optimization-based methods while delivering nearly 800x speedup, and surpasses the zero-shot performance of state-of-the-art generalizable models with markedly fewer parameters and reduced training/inference overhead, achieving an overall efficiency improvement.
1 Introduction
AsySplat addresses computational redundancy in generalizable 3DGS for high-resolution, long-sequence inputs by decoupling geometry and appearance into asymmetric branches. It allocates coarse-grained, computation-heavy modeling to geometry and fine-grained, lighter modeling to appearance, while using sparse attention to preserve performance without large training overhead.
- Motivation: The design targets redundancy in methods that use small patchified tokens and numerous layers, which concentrate computation on inter-frame token interactions.Small patches support high-quality rendering but increase computational cost and parameter demand.
- Motivation: High-precision geometry is unnecessary for high-quality rendering, whereas high-precision appearance is necessary but generally easier to model after geometry estimation.Rendering quality largely arises from alpha compositing, while appearance modeling mainly assigns pixel-aligned RGB evidence to Gaussians once multi-view depth is established.
- Asymmetric architecture: AsySplat decouples geometry and appearance modeling into asymmetric branches with different token granularities, computation, and parameter allocations.The geometry branch uses coarse-grained tokens, while the appearance branch captures details with finer-grained tokens and fewer parameters.
- Geometry branch: AsySplat reinvests computation saved by asymmetry into additional attention layers because explicit multi-view matching is important for depth estimation and excessive Mamba reliance degrades performance.The geometry branch uses a hybrid structure, while attention captures contextual cues more effectively for compact models.
2 Related Work
3D Gaussian Splatting offers strong novel-view synthesis quality, optimization efficiency, and real-time rendering, motivating feed-forward generalizable extensions that predict scene representations from multi-view images. Recent work further targets high-resolution, long-sequence inputs, including up to 32 views at 960P with known camera poses.
- 3D Gaussian Splatting for Novel View Synthesis: 3D Gaussian Splatting surpasses NeRFs in rendering fidelity, optimization efficiency, and real-time rendering performance.
- Generalizable 3D Gaussian Splatting: Generalizable 3DGS predicts scene representations directly from multi-view images in a feed-forward manner, avoiding conventional per-scene optimization.
- Generalizable 3D Gaussian Splatting: Recent methods extend generalizable 3DGS beyond sparse-view settings to long-sequence inputs of up to 32 views at 960P resolution under known camera poses.
3 Methodology
AsySplat reduces redundancy by asymmetrically decoupling geometry and appearance modeling, using coarse tokens for inter-frame geometric reasoning and fine tokens for intra-frame appearance processing. Bilateral connections exchange geometric and high-frequency information while allocating most parameters to geometry and substantially fewer to appearance.
- Asymmetric Tokenization: AsySplat uses different token granularities: large patches produce coarse geometry tokens, while smaller patches produce fine appearance tokens that preserve rendering details.The geometry and appearance branches process these token sets separately before asymmetric fusion.
- Geometry Branch: The geometry branch performs exhaustive multi-view matching on coarse tokens through three hierarchical stages combining bidirectional Mamba and sparse attention.Its token dimension progressively increases across stages, while sparse attention improves computational efficiency.
- Parameter Allocation: 10% of the total parameters are allocated to the appearance branch, while most parameters support the more difficult multi-view geometry task.This allocation follows the observation that appearance attributes are generally easier to model than geometry.
- Appearance Branch: The appearance branch restricts interactions to intra-frame processing, keeping its width constant and using MambaVision for efficient global modeling.This bounds the effective sequence length by the per-frame token count rather than the total across frames.
- Bilateral Connection Module: Bilateral connections let coarse tokens absorb high-frequency cues and fine tokens receive geometric context, including depth-related information useful for predicting Gaussian attributes.The two branches exchange information at the end of each stage through cross-attention-based communication.
- Computational Allocation: Inter-frame interactions are confined to coarse tokens, while fine-grained tokens remain intra-frame, reducing redundancy compared with entangled fine-grained inter-frame computation.Applying attention to a reduced-length sequence significantly shortens training time, while convolutional refinement preserves context integrity.
4 Experiments
Experiments evaluate AsySplat for 32-view long-sequence generalizable 3DGS across established real and synthetic benchmarks, showing major speed and quality advantages over optimization-based and generalizable baselines. Ablations support the asymmetric architecture, bilateral interactions, and task-aware computation allocation.
- Experimental Setup: 32-view experiments at 540×960 train on DL3DV and evaluate on DL3DV-140, Tanks&Temples, MipNeRF-360, Deep Blending, and Replica.DL3DV contains over 10k training scenes and 140 benchmark scenes.
- Optimization-Based Comparisons: <1s inference is nearly 800× faster than 13min for 3DGS on DL3DV-140, while PSNR increases from 23.6 to 24.0.With 10 post-optimization iterations completed in 18s, PSNR reaches 25.2 versus 24.8 for Scaffold-GS.
- Generalizable 3DGS Comparisons: 18s for AsySplat10 surpasses LongLRM3 at 15s, 36s for AsySplat20 matches LongLRM10 at 50s, and 50s for AsySplat28 outperforms LongLRM10 at 50s.AsySplat predicts fewer Gaussians, allowing more post-optimization iterations under similar time budgets.
- Zero-Shot Generalization: AsySplat delivers higher scores than LongLRM across all metrics on Tanks&Temples in a single feed-forward pass and remains superior on Deep Blending under all settings.Under equal post-optimization iterations, it also provides higher quality with less total runtime.
- Ablations: Using only coarse- or fine-grained tokens degrades performance, while removing bilateral connections or MambaVision confirms their contributions to cross-branch interaction and appearance modeling.The ablations also show that more accurate geometry does not improve NVS quality, and fine appearance can rely on intra-frame-only interaction.
5 Conclusion
The paper addresses computation redundancy in generalizable 3D Gaussian Splatting for long-sequence, high-resolution novel view synthesis with an asymmetric architecture that separates geometry and appearance modeling. By allocating computation and parameters according to precision needs and learning difficulty, the design improves parameter efficiency and enables smaller models to achieve strong performance.
- The work targets computation redundancy in generalizable 3D Gaussian Splatting for long-sequence, high-resolution novel view synthesis.
- An asymmetric architecture decouples geometry and appearance modeling into two branches.
- Computation and parameters are allocated according to the respective precision needs and learning difficulty of geometry and appearance.
- The design improves parameter efficiency, enabling smaller models to achieve strong performance.
AsySplat: Efficient Asymmetric 3D Gaussian Splatting
AsySplat presents an efficient asymmetric approach to 3D Gaussian splatting for long-sequence scene modeling.
- AsySplat targets efficient 3D Gaussian splatting for long-sequence scene modeling.
- The section concerns long-sequence scene modeling.
Supplementary Material · A Implementation Details
The supplementary implementation details specify AsySplat’s asymmetric architecture, progressive training schedule, overhead measurement protocols, evaluation setup, post-optimization procedure, and combined image-depth supervision.
- A Implementation Details: The geometry branch uses bidirectional Mamba layers with state dimension 256, while the appearance branch uses MambaVision layers with state dimension 16.All transformer blocks apply RMSNorm after query and key projections, and spherical harmonics use degree 3 for Gaussian attribute prediction.
- A Implementation Details: Training progressively increases resolution from 256×256 to 512×512 and 540×960 using AdamW with weight decay 0.05 and cosine learning-rate decay.Peak learning rates are 4e-4, 4e-5, and 4e-5 across the three stages, with 2k iterations of initial-stage linear warmup.
- A Implementation Details: Training-overhead comparisons with LongLRM at 256×256 and 512×512 average the first 20 iterations from randomly initialized parameters.LongLRM checkpoints were unavailable at these resolutions, and predicting 4× Gaussians causes its GPU-memory overhead to rise more steeply during training.
- A Implementation Details: Inference overhead is compared with LongLRM at 540×960 using inference time, TFLOPs, GPU memory usage, and number of Gaussians.Inference time averages DL3DV-140 results after removing the longest time as an outlier, while TFLOPs are measured with thops.
- A Implementation Details: Evaluation follows LongLRM’s input and test splits, with every 8th image used for testing and K-means selecting input views by camera position and viewing direction.The clustering aims to provide optimal scene coverage, with the number of clusters determined by the available evaluation setting.
- A Implementation Details: Post-optimization matches LongLRM’s procedure by rendering input views, minimizing MSE against ground-truth images, and backpropagating updates to Gaussian parameters.Learning rates are 5e-4 for centers, 1e-3 for spherical harmonics, and separately specified for opacity, scale, and quaternion parameters.
B More Comparisons
The method targets pose-given generalizable 3DGS modeling, so pose-free methods are excluded from the main-paper comparisons because the mismatched settings can create misleading performance gaps. Comparisons with AnySplat in Tab. A1 demonstrate that directly comparing the settings is unfair.
- Comparison with pose-free methods: The method focuses on pose-given generalizable 3DGS modeling and therefore omits pose-free generalizable 3DGS comparisons from the main paper.The authors identify the differing settings as a reason not to report these comparisons in the main paper.
- Comparison with pose-free methods: Setting mismatch can cause large performance gaps and potentially misleading comparisons between pose-given and pose-free methods.The passage warns that direct comparison across the mismatched settings may misrepresent relative performance.
- Comparison with pose-free methods: The comparison with the state-of-the-art pose-free method AnySplat in Tab. A1 demonstrates that directly comparing the settings is not fair.The reported performance gap is presented as evidence against direct comparisons.
C More Ablations
The ablations identify 2m6a as the Mamba-ratio sweet spot for novel view synthesis, with higher ratios degrading depth accuracy and cross-view matching. Allocating 90% of parameters to geometry achieves decent performance across benchmarks at roughly constant total parameter count.
- Mamba ratio ablations: 2m6a is the Mamba-ratio sweet spot for novel view synthesis, beyond which increasing Mamba reliance degrades performance.The experiments vary each stage’s Mamba ratio while keeping the overall parameter count similar.
- Mamba ratio ablations: Beyond 2m6a, depth accuracy declines alongside NVS metrics, indicating impaired cross-view matching in the geometry branch.Depth accuracy is used as a reliable indicator of matching quality.
- Parameter allocation ablations: 90% of parameters allocated to the geometry branch achieves decent performance across benchmarks with roughly unchanged total parameters.The parameter-allocation ablation varies geometry and appearance proportions while keeping total parameter count roughly unchanged.
D Discussion
AsySplat achieves comparable or superior performance to state-of-the-art generalizable 3DGS methods with substantially fewer parameters and lower overhead. However, its visual fidelity can lag behind optimization-based methods, including higher LPIPS.
- AsySplat matches or surpasses state-of-the-art generalizable 3DGS approaches while using substantially fewer parameters and lower overhead.
- Higher LPIPS indicates that AsySplat’s visual fidelity can still lag behind optimization-based methods.