Source-linked AI summary
2Xplat: Decoupling Geometry and Appearance Modeling for Feed-Forward 3D Gaussian Splatting
Hwasik Jeong, Seungryong Lee, Gyeongjin Kang, Seungkwon Yang, Xiangyu Sun, Seungtae Nam, Eunbyung Park
TL;DR
Pose-free feed-forward 3DGS often entangles camera-pose estimation and appearance synthesis in monolithic networks, limiting modular use of pose-conditioned architectures. 2Xplat separates these roles with geometry and appearance experts, outperforming prior pose-free methods and matching state-of-the-art posed approaches in fewer than 5K iterations.
Problem
Most pose-free feed-forward 3DGS methods jointly estimate camera poses and Gaussian parameters in monolithic networks, complicating high-quality appearance modeling and pose-conditioned architectural integration.
Method
2Xplat uses a two-stage architecture in which a geometry expert predicts camera poses and a pose-conditioned appearance expert synthesizes 3D Gaussians.
Results
Fewer than 5K training iterations yield consistent improvements over prior pose-free methods and performance on par with state-of-the-art posed approaches.
Takeaways & Limitations
These results highlight the potential of modular expert-decomposed architectures for pose-free 3D reconstruction and appearance synthesis.
Takeaways & Limitations
The appearance model’s global attention lacks 3D consistency, which can produce unrealistic geometry in generated 3DGS.
Abstract
from arXiv · showhide
Pose-free feed-forward 3D Gaussian Splatting (3DGS) has opened a new frontier for rapid 3D modeling, enabling high-quality Gaussian representations to be generated from uncalibrated multi-view images in a single forward pass. The dominant approach adopts unified monolithic architectures, often built on geometry-centric 3D foundation models, to jointly estimate camera poses and synthesize 3DGS representations within a single network, entangling geometric reasoning and appearance modeling within a shared representation. In this work, we introduce 2Xplat, a pose-free feed-forward 3DGS framework based on a two-experts design that explicitly separates geometry estimation from Gaussian generation: a dedicated geometry expert first predicts camera poses, which are then passed to an appearance expert that synthesizes 3D Gaussians. Despite its conceptual simplicity, and being largely underexplored in prior works, our two-experts pipeline outperforms prior pose-free feed-forward 3DGS approaches in fewer than 5K training iterations, achieving performance on par with state-of-the-art posed methods. These results challenge the prevailing unified paradigm and suggest the potential advantages of modular design for complex 3D geometric estimation and appearance synthesis tasks.
Introduction
The introduction identifies a limitation in monolithic pose-free feed-forward 3DGS architectures and presents 2Xplat’s two-experts design, which separates geometry estimation from appearance synthesis. Lightweight end-to-end optimization reuses pretrained experts, converges in fewer than 5K iterations, improves over prior pose-free methods, and performs on par with state-of-the-art posed models.
- Motivation: Pose-free feed-forward 3DGS methods reconstruct Gaussian representations from uncalibrated images, addressing settings where reliable camera poses are unavailable.Many feed-forward approaches assume accurate camera poses, limiting their use in unconstrained settings.
- Problem: Most existing pose-free approaches use one network to jointly predict camera poses and 3DGS parameters from shared features.The introduction argues this monolithic design may be suboptimal because appearance depends on lighting and view-dependent reflectance as well as geometry.
- Method: 2Xplat decomposes pose-free feed-forward 3DGS into a dedicated geometry expert and a pose-conditioned appearance expert.The geometry estimator predicts poses first, which condition the subsequent 3DGS generator.
- Optimization: Fewer than 5K iterations are required when two mature pretrained experts are reused and optimized through lightweight end-to-end fine-tuning.Joint optimization also makes the appearance expert robust to noisy camera pose estimates, mitigating sensitivity to geometric errors.
- Results and novelty: The approach improves consistently over prior pose-free feed-forward 3DGS methods and performs on par with state-of-the-art posed models in novel-view synthesis.The work revisits geometry-first, appearance-second decomposition by leveraging high-capacity pose-conditioned appearance models rather than focusing primarily on self-supervised training.
Related Works
Related work progresses from optimization-heavy reconstruction toward feed-forward methods, including pose-free systems that jointly infer geometry, appearance, and cameras. 2Xplat instead motivates specialized geometry and appearance experts connected through sequential processing and joint fine-tuning.
- Feed-forward 3D reconstruction: Traditional Structure-from-Motion and Multi-View Stereo pipelines are computationally expensive and brittle for sparse or unstructured inputs.These limitations motivate feed-forward reconstruction, which amortizes reconstruction cost through large-scale training.
- Feed-forward 3D reconstruction: DUSt3R and MASt3R pioneered feed-forward pairwise reconstruction by jointly predicting camera poses and geometry through dense pointmap regression.Their paradigm builds on Vision Transformers and replaces per-scene optimization with learned inference.
- Known-pose reconstruction: Known-pose feed-forward methods use external systems such as SfM and map images to neural fields or explicit 3D primitives in a single pass.LRM established this paradigm with a large-scale transformer mapping images to a neural radiance field.
- Pose-free reconstruction: Pose-free feed-forward reconstruction removes test-time pose dependence by jointly inferring scene geometry, appearance, and camera parameters from unposed images.This line of work spans neural fields and 3D Gaussian Splatting representations.
- Two-experts architecture: 2Xplat decomposes pose-free 3DGS into sequential geometry and appearance experts rather than a monolithic shared backbone with task-specific heads.The geometry expert estimates camera poses first, while the posed feed-forward 3DGS appearance expert generates 3D Gaussian representations.
- Two-experts architecture: Pretrained geometry and appearance experts are combined through lightweight end-to-end fine-tuning, keeping specialized objectives coupled through joint optimization.This design assigns each objective to a specialized module while retaining coordinated training.
Method
2Xplat decouples pose estimation from 3D Gaussian generation through sequential geometry and appearance experts. The system jointly fine-tunes both experts using rendered-view reconstruction and camera-pose supervision, with optional opacity regularization.
- Two-experts architecture: 2Xplat uses sequential experts: Fpose estimates camera parameters from all input views, then F3dgs generates pixel-aligned 3D Gaussians from context images and predicted poses.The framework explicitly decomposes geometry estimation and 3DGS generation into separate modules.
- Appearance expert: The appearance expert adopts MVP, integrating PRoPE pose conditioning, register tokens, and Alternating Attention within a dual hierarchical architecture.MVP is selected for reconstruction quality and inference efficiency among posed feed-forward 3DGS models.
- Appearance expert: Geometry and appearance decoupling permits a smaller appearance-expert patch size, such as 4 × 4, while leaving the geometry expert unchanged.Monolithic models typically retain 14 × 14 or 16 × 16 patches because changing them breaks pretrained-backbone compatibility.
- Training objective: Both pretrained experts are fine-tuned jointly end-to-end, with target-view rendering loss and ground-truth camera supervision forming the core training objective.Differentiable 3DGS rendering compares predicted target views with corresponding ground-truth images, while pose supervision regularizes prediction and limits geometric drift.
- Training objective: The rendering loss combines weighted ℓ2 reconstruction and perceptual losses, while camera supervision uses a relative pose loss that addresses global-reference-frame ambiguity.These terms balance pixel-wise accuracy with perceptual fidelity and stabilize pose learning.
- Training objective: When view-dependent opacity is used, an additional regularizer suppresses viewpoint-inconsistent opacity artifacts with weight 0.001.The regularization applies to predicted Gaussian opacities evaluated through spherical-harmonic basis functions at randomly sampled viewing directions.
Experiments
Experiments evaluate 2Xplat on RE10K and DL3DV using official splits, varying input-view counts and resolution settings. Comparisons cover pose-dependent and pose-free novel-view-synthesis methods using PSNR, SSIM, and LPIPS, with additional EPA reporting for pose-free baselines and cross-dataset testing on ScanNet++.
- Dataset: The model is trained on RealEstate10K and DL3DV using their official data splits.RE10K benchmarking retains test sequences with at least 200 frames, yielding 1,580 sequences.
- Dataset: DL3DV evaluation uses 6, 12, and 24 input views at low resolution, with farthest-point sampling over camera centers and 8 views held out for validation.Maximum frame intervals are 50, 100, and 150, respectively, following the stated evaluation protocol.
- Evaluation Protocol: High-resolution DL3DV novel-view synthesis is evaluated at 960 × 540 with 16, 32, 64, and 128 input views.The high-resolution evaluation uses the undistorted DL3DV version, while the table reports varying input-view counts.
- Baselines: Novel-view-synthesis comparisons include pose-dependent methods MVSplat, DepthSplat, Long-LRM, iLRM, and MVP, plus pose-free methods NoPoSplat, AnySplat, and YonoSplat.The comparisons are defined in the paper’s baseline list for novel view synthesis.
- Evaluation Protocol: Evaluation reports PSNR, SSIM, and LPIPS, with additional EPA results for prior pose-free methods that use evaluation-time pose alignment.Unless otherwise specified, results use low resolution and exclude view-dependent opacity.
- Evaluation Protocol: Cross-dataset generalization is evaluated from DL3DV to ScanNet++ using PSNR.The experiments also include qualitative comparisons on DL3DV, high-resolution DL3DV, RE10K, and ScanNet++.
Results
2Xplat consistently outperforms baselines in low-resolution novel view synthesis across DL3DV and RE10K, including methods using ground-truth cameras or intrinsics. It remains competitive at high resolution and generalizes from DL3DV to ScanNet++ without fine-tuning.
- Novel View Synthesis: 2Xplat consistently outperforms all baselines on low-resolution DL3DV and RE10K novel view synthesis, including methods using ground-truth camera poses or intrinsics.The evaluations vary input views on DL3DV and use a fixed-view setting on RE10K.
- Novel View Synthesis: Across 16 to 128 input views, 2Xplat is the only pose-free method in the high-resolution comparison and achieves competitive performance.The comparison includes 30K-iteration optimization-based 3D Gaussian Splatting and feed-forward reconstruction methods.
- Cross-Dataset Generalization: Without fine-tuning, training on DL3DV and testing on ScanNet++ yields consistent improvement over baselines across all evaluation metrics.Qualitative comparisons further report sharper results, supporting cross-dataset generalization across different scene distributions.
Analyses
The analyses compare 2Xplat’s two-experts design with monolithic alternatives and examine its robustness across different geometry backbones. The results indicate that the modular framework remains competitive even with a smaller backbone and offers faster inference.
- Monolithic vs. Two-Experts Architectures: The study compares the two-experts framework against three monolithic architectures: DA3-G, DA3-L, and YoNoSplat.DA3-L was equipped with DA3-G’s Gaussian head and trained for 150K iterations using RGB and camera losses.
- Backbone Robustness: 2Xplat performs consistently well with DA3-G, DA3-L, and π3 used as geometry-expert backbones, demonstrating robustness across backbone choices.The analysis identifies the geometry backbone paired with the framework as a variable in the comparison.
- Backbone Robustness: With DA3-L’s fewer parameters, 2Xplat still achieves competitive performance with faster inference speed, indicating limited dependence on backbone capacity.This result supports the framework’s robustness when using a smaller geometry expert.
Conclusion
2Xplat is a two-experts framework for pose-free feed-forward 3DGS that decouples pose estimation from appearance synthesis. It outperforms prior pose-free methods and matches state-of-the-art posed approaches in fewer than 5K training iterations.
- 2Xplat decouples pose estimation from appearance synthesis through a two-experts framework for pose-free feed-forward 3DGS.
- Fewer than 5K training iterations deliver performance on par with state-of-the-art posed approaches.The framework also outperforms prior pose-free methods while maintaining strong reconstruction quality and training efficiency.
- These findings challenge the assumption that geometric reasoning and appearance synthesis should be entangled.
Appendix Additional Details
The appendix specifies the joint rendering and relative-pose training losses, implementation choices, datasets, baselines, and evaluation protocols. It also reports the training setup and weighting values used for the model.
- Joint Training Loss Formulation: The rendering loss combines weighted ℓ2 reconstruction and perceptual losses to balance pixel accuracy and perceptual fidelity.The perceptual contribution is controlled by λperc.
- Joint Training Loss Formulation: Relative pose supervision uses predicted view-to-view transformations, with separate rotation and translation losses and Huber-based terms.The formulation addresses ambiguity in the global reference frame between predicted and ground-truth poses.
- Joint Training Loss Formulation: The pose objective includes camera-intrinsics l2 loss with weighting factors λR, λt, and λK, and may optionally include view-dependent opacity regularization.The appendix identifies LK as the l2 loss between predicted and ground-truth camera intrinsics.
- Baselines and Evaluation Protocol: Evaluation covers novel-view PSNR, SSIM, and LPIPS, plus pose AUC at 5°, 10°, and 20° thresholds, using comparisons with pose-dependent, pose-free, and listed pose-estimation baselines.The pose-estimation baselines are MASt3R, VGGT, π3, and DA3.
- Implementation Details: The geometry and appearance experts use pretrained Depth Anything 3 and Multi-view Pyramid Transformer models, respectively, with DA3-Giant adopted unless otherwise specified.Inputs are resized by shorter-side matching and center square cropping; RE10K training uses 224×224 resolution.
Additional Analyses
Additional analyses show that 2Xplat generalizes across appearance experts, estimates camera poses competitively, and balances rendering quality against pose accuracy through relative pose supervision. They further motivate dynamic grouping while showing that the benefit of 3D consistency depends on input-view sparsity.
- Generality Across Appearance Experts: 2Xplat generalizes across appearance experts, including iLRM and Long-LRM, with all variants jointly trained within 5K iterations.These experts lose nearly 1–2 PSNR relative to MVP because of limited tolerance to pose noise, but remain effective pose-free.
- Pose Estimation: The framework also provides competitive camera-pose AUC despite fine-tuning on only a small dataset subset.Prior state-of-the-art approaches cited in the passage rely on extensive pose supervision.
- Pose Supervision: Relative pose loss offers the best balance between rendering quality and pose accuracy, whereas removing pose loss slightly improves rendering but substantially harms pose estimation.The comparison includes relative, absolute, and no-pose-loss configurations.
- MVP Variants: A stronger MVP variant combines dynamic grouping, increased capacity, and approximately 34M training samples without increasing training cost.The variant achieves highly 3D-consistent global attention while retaining equivalent training cost to the original model.
- MVP Variants: Improved 3D consistency does not always improve novel-view synthesis, because its value changes with the amount of available input information.The passage identifies 4, 8, and 16 views as challenging sparse-view settings where 3D consistency is especially important.
Limitations
2Xplat’s two-experts design has two limitations: rendering quality is bottlenecked by the appearance expert’s capacity, while pose accuracy is slightly lower because pose supervision primarily regularizes rather than directly optimizes prediction.
- Limitations: Rendering quality is bottlenecked by the appearance expert’s capacity, regardless of improvements elsewhere in the framework.The appearance expert carries most of the rendering quality.
- Limitations: Pose estimation accuracy is slightly lower than in methods specifically designed for pose prediction.Pose supervision mainly acts as a regularization signal rather than a primary objective.
Additional Qualitative Results
Additional qualitative comparisons on RE10K, DL3DV, high-resolution DL3DV, and ScanNet++ show strong reconstruction quality across datasets and resolutions. The visualizations cover varying numbers of context views and cross-dataset generalization settings.
- Additional Qualitative Results: Additional comparisons on RE10K, DL3DV, high-resolution DL3DV, and ScanNet++ demonstrate strong reconstruction quality across datasets and resolutions.The comparisons are presented in Figures 9–12.
- DL3DV: DL3DV qualitative comparisons evaluate 6, 12, and 24 context views at 224 × 224 resolution.These settings correspond to rows 1–3, 4–6, and 7–9, respectively, in Figure 9.
- RE10K: RE10K qualitative comparisons use 6 context views at 224 × 224 resolution.This comparison is shown in Figure 10.
- Cross-dataset generalization: Cross-dataset qualitative results from DL3DV to ScanNet++ evaluate 32, 64, and 128 context views at 224 × 224 resolution.These settings correspond to rows 1–2, 3–4, and 5–6, respectively, in Figure 11.
- High-resolution DL3DV: High-resolution DL3DV comparisons evaluate 16, 32, 64, and 128 context views at 960 × 540 resolution.These settings correspond to rows 1–2, 3–4, 5–6, and 7–8, respectively, in Figure 12.