Source-linked AI summary
COLMAP-Free 3D Gaussian Splatting
Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, Xiaolong Wang
TL;DR
Unknown camera poses make neural scene reconstruction difficult, especially for implicit NeRF representations and large camera motions. CF-3DGS instead uses explicit 3D Gaussians, temporal continuity, and sequential progressive growth to jointly estimate poses and synthesize views without SfM preprocessing. It reports stronger view synthesis and camera pose estimation under large motion changes, with rapid training and inference speeds.
Problem
Pre-computed camera poses are costly and failure-prone, while implicit NeRF representations make joint optimization of scene structure and camera poses challenging, particularly under large motion.
Method
CF-3DGS sequentially processes video frames, using local 3DGS for nearby-frame relative poses and global 3DGS to progressively grow an explicit scene representation.
Results
CF-3DGS significantly improves view synthesis and outperforms prior approaches in camera pose estimation when camera motion is large, including 360-degree videos.
Takeaways & Limitations
Explicit Gaussian representations combined with video continuity support robust pose estimation and novel view synthesis without SfM preprocessing, while retaining rapid training and inference.
Takeaways & Limitations
The sequential joint optimization primarily restricts CF-3DGS to video streams or ordered image collections.
Abstract
from arXiv · showhide
While neural rendering has led to impressive advances in scene reconstruction and novel view synthesis, it relies heavily on accurately pre-computed camera poses. To relax this constraint, multiple efforts have been made to train Neural Radiance Fields (NeRFs) without pre-processed camera poses. However, the implicit representations of NeRFs provide extra challenges to optimize the 3D structure and camera poses at the same time. On the other hand, the recently proposed 3D Gaussian Splatting provides new opportunities given its explicit point cloud representations. This paper leverages both the explicit geometric representation and the continuity of the input video stream to perform novel view synthesis without any SfM preprocessing. We process the input frames in a sequential manner and progressively grow the 3D Gaussians set by taking one input frame at a time, without the need to pre-compute the camera poses. Our method significantly improves over previous approaches in view synthesis and camera pose estimation under large motion changes. Our project page is https://oasisyang.github.io/colmap-free-3dgs
1. Introduction
CF-3DGS removes SfM preprocessing by jointly estimating camera poses and reconstructing scenes with explicit 3D Gaussians, using video continuity and sequential optimization. It improves view synthesis and handles large camera motions more robustly than prior approaches.
- COLMAP-based pose preprocessing is time-consuming and can fail with feature-extraction errors, textureless regions, or repetitive regions.
- NeRF-based joint reconstruction and camera registration remains difficult because implicit representations optimize camera parameters indirectly through ray casting.
- CF-3DGS exploits explicit point-cloud representations and temporal continuity to perform view synthesis without SfM preprocessing.
- The method progressively grows global 3D Gaussians one frame at a time while using local Gaussians to estimate relative poses between nearby frames.
- CF-3DGS significantly improves view synthesis on Tanks and Temples and CO3D, while outperforming prior methods under large camera motions.For camera pose estimation, it is on par with Nope-NeRF under small motion and outperforms all approaches by a large margin in 360-degree CO3D videos.
2. Related Work
Prior unposed-NeRF methods commonly depend on restricted motion, prior knowledge, or lengthy optimization, while explicit point-cloud methods improve rendering efficiency but usually retain SfM pose requirements. CF-3DGS addresses these limitations with local and global 3DGS strategies for joint pose estimation and scene reconstruction.
- NeRF-based methods have achieved photorealistic rendering but commonly rely on pre-computed camera parameters from SfM algorithms.
- Existing unposed-video methods can provide generalizable scene representations, but their view synthesis is unsatisfactory without per-scene optimization.
- Prior unknown-pose NeRF approaches assume small perturbations, narrow camera-motion ranges, or prior knowledge, limiting performance on trajectories such as 360-degree scenes.
- CF-3DGS combines local and global 3DGS strategies to jointly optimize camera parameters and 3D Gaussians.
3. Method
CF-3DGS jointly estimates camera poses and reconstructs scenes by combining explicit 3D Gaussian representations with sequential local-to-global optimization.
- 3D Gaussian Splatting: 3DGS represents scenes explicitly as 3D Gaussians, each defined by a center and covariance matrix.The covariance describes an ellipsoid through scaling and rotation parameters.
- 3D Gaussian Splatting: The differentiable renderer projects Gaussian covariances into camera coordinates and combines overlapping points through alpha blending.Projected covariance uses the viewing transform and the Jacobian of an affine projective approximation.
- Local 3DGS for Relative Pose Estimation: Unlike NeRF methods that optimize camera parameters indirectly through ray casting, CF-3DGS directly exploits explicit Gaussian geometry for pose estimation.The method is motivated by the ability of explicit point clouds to undergo straightforward deformation and movement.
- Local 3DGS for Relative Pose Estimation: CF-3DGS initializes Gaussians from monocular depth, fits them to one frame, and estimates relative pose by transforming the pretrained Gaussian set toward the next frame.The relative transformation is optimized using photometric loss while freezing the pretrained Gaussian attributes; adjacent frames make the transformation relatively small.
- Global 3DGS: The global model grows sequentially by updating Gaussian attributes with estimated poses and densifying under-reconstructed regions as new views arrive.Noisy local relative poses motivate the progressive global optimization strategy.
4. Experiments
The experiments evaluate CF-3DGS on Tanks and Temples and CO3D V2 for novel view synthesis and camera pose estimation, including challenging camera motion and ablations of key design choices.
- Experimental Setup: Experiments use Tanks and Temples and CO3D-V2, evaluating novel view synthesis and camera pose estimation against pose-unknown baselines.Tanks and Temples covers eight indoor and outdoor scenes; CO3D-V2 provides more complex object-centric video motions.
- Experimental Setup: The test-view camera poses are optimized photometrically while the trained 3DGS model remains frozen.Each test pose is initialized from the closest learned training-view camera position to accelerate convergence.
- Comparing with Pose-Unknown Methods: 25 hrs vs. 1.5 hrs: CF-3DGS outperforms competing methods on Tanks and Temples within a shorter training duration.The paper reports consistently better results across all novel-view-synthesis metrics, with sharper and clearer synthesized images.
- Comparing with Pose-Unknown Methods: CF-3DGS achieves comparable pose-estimation performance on Tanks and Temples, while its relative and absolute pose errors may reflect reliance on photometric loss alone.Nope-NeRF uses additional relative-pose constraints, including point-cloud Chamfer distance.
- Ablation Study: Progressive growth is essential for both tasks because removing it prevents 3DGS from using video continuity and destabilizes global optimization.Using both RGB and depth losses is less effective than using photometric loss alone for novel view synthesis.
- Ablation Study: Joint optimization achieves performance comparable to original 3DGS trained with COLMAP-assisted poses on Tanks and Temples.This comparison evaluates whether pose-free training can approach a COLMAP-pose reference.
5. Conclusion
CF-3DGS jointly estimates camera poses and reconstructs scenes from image sequences using explicit point clouds and sequential Gaussian growth. Its main scope is ordered imagery, since the sequential optimization restricts application primarily to video streams or ordered image collections.
- CF-3DGS jointly estimates camera poses and performs novel view synthesis from a sequence of images.
- The method uses explicit point clouds and progressively expands 3D Gaussians as it processes video frames sequentially.
- CF-3DGS demonstrates effectiveness and robustness on challenging scenes including 360° videos, with rapid training and inference speeds.
- Sequential joint optimization restricts the method primarily to video streams or ordered image collections.
A.1. Dataset
The evaluation uses sequences with dramatic camera motion from Tanks and Temples and CO3D-V2. Training and test images are sampled from these sequences under specified temporal and category-selection procedures.
- The study selects sequences containing dramatic camera motions from Tanks and Temples and CO3D-V2.
- Test images are sampled starting from the fifth image, taking every eighth image in each sequence.
- For the Family scene, an additional setting trains on odd frame IDs and tests on even frame IDs.
- CO3D-V2 evaluation randomly selects 10 scenes from six categories, including apple, bench, hydrant, plant, skateboard, and teddybear.
- Table 8 reports FPS and maximum relative rotation angle between frames, with large maximum rotation angles indicating dramatic camera motion.
A.2. Training Details.
Training initializes local Gaussian scenes from monocular depth and optimizes them before sequentially updating a global Gaussian scene. Camera poses are estimated alongside global-scene updates as new frames arrive.
- Local 3DGS: Local 3DGS begins with monocular depth estimation from an input image and initializes a local Gaussian set.
- Local 3DGS: The local Gaussian set is rasterized, compared with the input image using a loss, and updated with Adam until convergence.
- Local 3DGS: The downsampled point cloud initializes local 3DGS, which is optimized on the input view using photometric loss for 500 iterations.
- Global 3DGS: Global 3DGS initializes from the first frame and its monocular depth estimate, then updates using all observed images to date.
- Global 3DGS: As each new frame is introduced, the global Gaussian representation progressively grows while camera poses are estimated sequentially using local 3DGS.
A.3. Evaluation Metrics
The evaluation measures novel view synthesis with image-quality metrics and camera pose estimation with visual-odometry metrics.
- Novel View Synthesis: Novel view synthesis quality is measured using PSNR, SSIM, and LPIPS with a VGG architecture for LPIPS.
- Pose Accuracy: Camera pose accuracy is evaluated using Absolute Trajectory Error and Relative Pose Error.
B. Additional Experiments
Additional experiments evaluate novel view synthesis and camera pose estimation on Tanks and Temples and CO3D-V2, including five additional CO3D-V2 scenes.
- Experiments cover novel view synthesis and camera pose estimation on Tanks and Temples and CO3D-V2.
- Five additional CO3D-V2 scenes are evaluated for camera pose estimation, with results reported in Table 9.
- On the additional CO3D-V2 scenes, the method outperforms Nope-NeRF in pose accuracy and synthesis quality.
B.2. Novel View Synthesis.
The novel-view-synthesis experiments assess interpolated trajectories, camera-intrinsic settings, depth estimators, and qualitative results across Tanks and Temples and CO3D-V2.
- Interpolated poses sampled from fitted Bezier curves provide more novel viewpoints for rendering beyond views close to training images.This evaluation addresses possible overfitting to test views sampled near the training views.
- Compared with Nope-NeRF, the approach renders photo-realistic images with more details in highlighted regions.
- Camera-intrinsic sensitivity is evaluated on Tanks and Temples using both unknown and heuristic intrinsic settings.The heuristic setting fixes every scene's field of view at 79° and centers the principal point.
- Depth-estimator ablations on Tanks and Temples report that more accurate monocular depth estimates consistently yield better performance.
- Additional qualitative comparisons on Tanks and Temples and CO3D-V2 report more realistic rendering than other baselines.
- Camera-pose qualitative comparisons on CO3D-V2 show ground-truth and estimated trajectories as blue and red, respectively.