Source-linked AI summary
Reliability-Regulated Trajectory Optimization for Progressive COLMAP-Free 3D Gaussian Splatting
Zijian Wu, Jinliang Wang, Zidian Lin, Ying Song, Ziqian Lu, Hanjie Ma, Zhen Ye, Mingfeng Jiang
TL;DR
Progressive COLMAP-free 3DGS can accumulate camera-tracking errors that drift through a sequence and become coupled to scene geometry. This paper regulates forward warm-starting and retrospective sliding-window pose correction with bidirectional cycle consistency. Evaluations report improved trajectory accuracy and novel-view rendering quality over existing unposed baselines.
Problem
Progressive camera tracking in COLMAP-free 3DGS is vulnerable to accumulating pose errors, and photometric alignment can be misleading in textureless or repetitive regions.
Source: §1 · p. 2Method
An intrinsic bidirectional cycle-consistency signal gates forward kinematic warm-starts and weights relative-pose consistency in fixed-geometry sliding-window pose refinement.
Source: §1 · pp. 2–3Results
The framework substantially outperforms existing unposed baselines in trajectory recovery accuracy and novel-view rendering fidelity.
Source: §1 · pp. 3–4Takeaways & Limitations
Progressive COLMAP-free 3DGS trajectory estimation can be regulated across forward initialization and retrospective correction without external neural priors.
Source: §1 · p. 2, §5 · p. 16Jointly optimizing Gaussian geometry and camera poses can let the geometry absorb pose errors and permanently bake drift into the scene.
Source: §3.4 · p. 9
Abstract
from arXiv · showhide
COLMAP-free 3D Gaussian Splatting (3DGS) bypasses computationally expensive structure-from-motion (SfM) pipelines, yet progressive camera pose tracking remains fundamentally vulnerable to error compounding---early pairwise tracking inaccuracies both corrupt subsequent frame initializations and remain permanently frozen in the scene representation. Rather than relying on heavyweight external neural priors or treating progressive tracking through isolated heuristic fixes, we propose a unified reliability-regulated trajectory optimization framework for progressive COLMAP-free 3DGS. At its core, our framework establishes an intrinsic, self-supervised bidirectional cycle-consistency mechanism that systematically regulates progressive camera trajectory estimation across two complementary temporal horizons: (1) Forward Motion Propagation, where the online reliability signal adaptively gates first-order kinematic warm-starts of rigid motion into upcoming pairwise registrations, supplying informed directional search priors while safely intercepting untrusted transitions; and (2) Retrospective Trajectory Correction, where the same reliability signal dynamically weights relative-pose consistency constraints within a sliding window of neighboring camera poses. By governing both prospective state initialization and retrospective trajectory consolidation through a unified reliability regulator, our self-contained framework resolves progressive drift without external priors or offline preprocessing. Extensive evaluations on Tanks and Temples and CO3D-V2 benchmarks show that our method substantially improves camera trajectory accuracy and novel-view rendering quality, outperforming existing unposed baselines. Code is available at https://github.com/Zijian1026/RRTO-CF3DGS.
1 Introduction
Progressive COLMAP-free 3DGS avoids SfM preprocessing but remains vulnerable to error accumulation, drift, and geometry-pose coupling. The paper uses bidirectional cycle consistency to regulate both forward motion initialization and retrospective trajectory correction, improving trajectory recovery and novel-view rendering against unposed baselines.
1 Introduction
- Conventional 3DGS depends on SfM-derived poses, whose computation is costly and may fail under weak textures, repetitive patterns, or large-baseline motion.When SfM fails to establish correspondences, downstream 3DGS cannot be initialized; noisy poses can also degrade scene optimization.Source: §1 · p. 2
- Progressive sequential registration accumulates localized pose errors into trajectory drift, while misleading photometric minima can couple camera errors to deformed Gaussian geometry.Sparse observations and texture-scarce regions can also leave photometric guidance insufficient to bound optimization.Source: §1 · p. 2
- The framework targets progressive drift without external neural priors, using an intrinsic reliability regulator rather than disconnected heuristic modifications.Its two horizons are prospective forward propagation and retrospective trajectory correction.Source: §1 · pp. 2, 4
- A shared bidirectional cycle-consistency score gates forward kinematic warm-starts and weights relative-pose constraints during fixed-geometry sliding-window correction.The forward gate intercepts untrusted transitions, while retrospective optimization jointly refines neighboring camera poses under fixed Gaussian geometry.Source: §1 · pp. 2–4
- The framework substantially outperforms existing unposed baselines in trajectory recovery accuracy and novel-view rendering fidelity.Evaluations cover Tanks and Temples and CO3D-V2, where it yields smoother, highly accurate trajectories and superior rendering quality.Source: §1 · pp. 3–4, Figure 1 · p. 3
2 Related Work
3DGS offers fast, explicit rendering but remains dependent on reliable camera poses, motivating pose-free reconstruction. Existing approaches include implicit NeRF optimization, sparse-view feed-forward models, and progressive test-time tracking; the proposed framework targets drift in the last setting without external priors or offline preprocessing.
2.1 Novel View Synthesis
- Explicit 3DGS uses learnable Gaussians and differentiable rasterization for real-time rendering and faster optimization, but remains dependent on SfM poses and point clouds.SfM failure or noisy poses can severely degrade optimization.Source: §2.1 · p. 5
2.2 Pose-Free 3D Reconstruction and Neural Rendering
- NeRF-based pose-free methods jointly optimize implicit scene representations and camera parameters, but remain computationally expensive and prone to local minima under wide-baseline motion.The cited approaches include NeRFmm, BARF, and Nope-NeRF, with scene and initialization constraints described for individual methods.Source: §2.2 · p. 5
- Pose-free 3DGS spans sparse-view feed-forward reconstruction and progressive test-time tracking, with the latter still vulnerable to cumulative drift and often relying on external models or offline preprocessing.The proposed pipeline addresses continuous-sequence tracking through kinematic warm-starting, fixed-geometry local-window consistency, and intrinsic forward–backward cycle consistency.Source: §2.2 · pp. 5–6
3 Method
The framework alternates forward initialization and retrospective local refinement, coupling both phases through a shared bidirectional tracking-confidence signal.
3 Method
- A shared bidirectional tracking confidence connects two alternating phases of progressive COLMAP-free 3DGS trajectory optimization.The passage identifies these as forward warm-starting and retrospective sliding-window refinement.Source: §3 · p. 6
- Forward propagation uses short-term SE(3) kinematic continuity to warm-start new frames, with preceding-transition confidence gating the extrapolation.The gate prevents untrusted motion priors from corrupting upcoming registration.Source: §3 · p. 6
- Retrospective correction periodically jointly updates poses in a sliding temporal window under fixed Gaussian geometry, using multi-view photometric alignment and confidence-weighted relative-pose consistency.The confidence scores are cached from sequential tracking measurements.Source: §3 · p. 6
3.1 Preliminaries
3DGS represents scenes with explicit Gaussians rendered by differentiable rasterization, making rendering efficient but sensitive to camera-pose errors. In the COLMAP-free setup, new camera poses are estimated from local Gaussian models through differentiable rendering and then committed to the scene.
3.1.1 3D Gaussian Splatting
- A 3DGS scene consists of explicit Gaussian primitives with centers, covariances, opacities, and appearance features, projected and rendered through differentiable alpha compositing.Camera-pose errors can impair geometric consistency and rendering quality.Source: §3.1.1 · p. 6
3.1.2 COLMAP-Free 3D Gaussian Splatting
- CF-3DGS progressively estimates camera motion and reconstructs the Gaussian scene instead of relying on externally estimated camera poses.It constructs a local Gaussian representation from a reference image using monocular depth prediction, then optimizes the target pose by differentiable rendering.Source: §3.1.2 · p. 6
- For each newly observed frame, CF-3DGS estimates a relative transformation through differentiable rendering, updates the global camera pose, and replays the frame to update the Gaussian scene.The relative transformation is optimized against the target image using a photometric reconstruction loss.Source: §3.1.2 · p. 7
3.2 Bidirectional Cycle-Consistency Evaluation
The method estimates tracking reliability online by independently registering motion in both directions and checking cycle consistency. The resulting confidence is used as a safety gate and soft anchor, not as a guarantee of global accuracy.
3.2 Bidirectional Cycle-Consistency Evaluation
- Forward and backward Gaussian registrations are independently optimized, and their composed motion is evaluated for cycle consistency.The confidence proxy is computed online without ground-truth poses or external sensors.Source: §3.2 · p. 7
- The translational cycle residual is normalized by forward translation magnitude to reduce dependence on monocular translation scale.A small positive constant prevents denominator singularities near static or pure-rotation motion, where rotational residual predominantly steers the metric.Source: §3.2 · p. 7
- Bidirectional error metrics are mapped to a motion-tracking confidence score through a smooth exponential decay formulation.The exponential mapping provides a smooth, bounded gradient that attenuates uncertain frames.Source: §3.2 · p. 8
- Forward–backward agreement is a self-supervised reliability proxy, not an absolute guarantee of global accuracy, since ambiguous or repetitive textures can yield mutually consistent biased tracking.Confidence gates motion propagation and acts as an adaptive soft anchor during window refinement rather than replacing photometric loss.Source: §3.2 · p. 8
3.3 Forward Motion Propagation with Kinematic Priors
The method uses short-term continuity in camera motion to initialize incoming registrations, but applies extrapolation only when the preceding transition is trusted. Untrusted transitions instead use identity initialization.
3.3 Forward Motion Propagation with Kinematic Priors
- A first-order continuity assumption in the camera’s relative-motion tangent space makes the preceding frame-to-frame transformation an extrapolation prior for the incoming motion.Standard progressive pipelines instead initialize each new frame with the identity, leaving no directional search prior under texture-scarce or wide-baseline conditions.Source: §3.3 · p. 8
- The preceding transition’s confidence gates extrapolation: trusted motion uses the kinematic warm-start, while untrusted motion falls back to identity.This makes the initialization fail-safe against abrupt trajectory perturbations while guiding convergence in well-behaved segments.Source: §3.3 · p. 8
- On Tanks and Temples, the figure contrasts baseline blurring, missing structures, and distortion with the method’s detailed textures and sharp edges.Red and green insets highlight detailed regions.Source: Figure 3 · p. 9
3.4 Retrospective Trajectory Correction via Joint Refinement
The method corrects progressive trajectory drift by periodically refining neighboring camera poses while freezing scene geometry. It combines multi-view photometric alignment with confidence-weighted relative-pose consistency in overlapping windows.
3.4 Retrospective Trajectory Correction via Joint Refinement
- Freezing Gaussian parameters during local window refinement directs rendering-error gradients into camera poses, avoiding the geometry–pose degeneracy of jointly optimizing scene and poses.The window advances by one newly registered frame, repeatedly co-optimizing poses across overlapping configurations while fixing an anchor pose as the gauge reference.Source: §3.4 · p. 9
- The sliding-window poses are jointly refined using multi-view photometric alignment and scale-normalized, confidence-weighted relative-pose consistency.Each pose is optimized against multiple views rather than independently against a single reference.Source: §3.4 · p. 9, §3.4.1 · pp. 9–10
3.4.1 Photometric Joint Alignment
- Window-level photometric alignment renders the current Gaussian representation from multiple camera poses to supply local photometric evidence.This evidence can resolve ambiguities not addressed by a single reference view.Source: §3.4.1 · p. 10
3.4.2 Confidence-Weighted Relative-Pose Consistency
- Relative-pose consistency treats each cached motion as a reliability-aware soft measurement, rather than enforcing uniform pose smoothness.The same forward–backward confidence weights the constraint, anchoring the trajectory to high-confidence measurements while downweighting low-consistency transitions.Source: §3.4.2 · pp. 10–11
- On displayed CO3D-V2 sequences with wide baselines and continuous rotations, competing methods show trajectory deviations, floaters, and blurriness, while the framework yields more accurate trajectories and higher-fidelity reconstructions.The figure presents a qualitative comparison on challenging 360-degree object-centric sequences.Source: Figure 4 · p. 11
3.5 Overall Optimization Objectives
The optimization objectives use photometric reconstruction during progressive registration and jointly refine neighboring poses in a retrospective window. A shared cycle-consistency confidence regulates motion propagation and weights retrospective pose consistency.
3.5 Overall Optimization Objectives
- The same confidence score governs prospective motion propagation and weights relative-pose consistency during retrospective window correction.This shared reliability signal links the two trajectory-optimization horizons.Source: §3.5 · pp. 11–12
- During progressive registration, photometric loss combines L1 and D-SSIM reconstruction terms and drives pose tracking and subsequent Gaussian-attribute updates.It compares a rendered view with the target ground-truth view.Source: §3.5 · p. 11
- The retrospective phase jointly refines camera poses within a window using a multi-task loss.The objective combines window photometric and consistency terms.Source: §3.5 · p. 12
4 Experiments
Experiments evaluate novel-view synthesis and camera-pose recovery on Tanks and Temples and CO3D-V2, alongside a check of cycle-consistency confidence against pose errors. Results report improved rendering quality and lower trajectory errors on both benchmarks.
4.1.1 Datasets
- Tanks and Temples frames are subsampled at stride 5; CO3D-V2 covers six object categories, with every eighth frame held out for testing.The remaining frames are used for training.Source: §4.1.1 · pp. 12–13
4.1.2 Baselines
- The comparison includes Nope-NeRF, CF-3DGS, HT-3DGS, and PCR-GS as pose-free baselines.Source: §4.1.2 · p. 13
4.1.3 Evaluation Metrics
- Camera recovery is measured with ATE, RPEt, and RPEr, while held-out rendering is assessed with PSNR, SSIM, and LPIPS.Source: §4.1.3 · p. 13
4.1.4 Implementation Details
- The implementation uses a five-frame sliding window with four-frame overlap and sets the motion-initialization confidence threshold to 0.85.Window refinement runs for 100 iterations on CO3D-V2 and 30 on Tanks and Temples.Source: §4.1.4 · p. 13
4.2 Novel View Synthesis Evaluation; 4.3 Camera Pose Estimation Evaluation
- On Tanks and Temples, mean PSNR rises from 21.93 dB to 24.84 dB versus CF-3DGS, while mean ATE is 0.010; on CO3D-V2, the method reports 27.71 dB PSNR and mean ATE 0.014.CO3D-V2 mean RPEt is 0.352 and mean RPEr is 0.382°; Tanks and Temples mean RPEt is 0.484 and mean RPEr is 0.347°.Source: §4.2 · p. 13, §4.3 · p. 14
4.3.1 Reliability Verification of Bidirectional Confidence
- Across representative sequences, sharp confidence drops coincide with local translation and rotation error spikes, indicating an inverse relationship between confidence and tracking inaccuracy.The confidence check requires no ground-truth trajectory supervision; it can also be conservative, including around Book frame 14 despite low instantaneous pose error.Source: §4.3.1 · pp. 14–15, Figure 5 · p. 14
4.4 Ablation Study
Ablations examine motion-guided initialization, confidence gating, and local window refinement. They report gains from gating extrapolated motion and from confidence-weighted relative-pose constraints.
Ablation Setup
- The Tanks and Temples ablation compares controlled variants across novel-view synthesis and camera-pose metrics.The variants include motion-guided initialization, confidence gating, and window joint refinement.Source: §4.4 · p. 15
Kinematic Warm-Start and Confidence Gating
- Confidence-gated motion propagation reaches 24.26 dB PSNR and 0.011 ATE, compared with 23.58 dB PSNR and 0.016 ATE for ungated extrapolation.The passage reports that ungated motion can overshoot after direction changes or noisy tracking.Source: §4.4 · p. 15
Confidence-Weighted Relative Pose Regularization
- In the full model, F–B loss weighting beats uniform weighting by 0.33 dB PSNR (24.84 dB vs. 24.51 dB) and achieves 0.010 ATE and 0.484 RPEt.Confidence weighting also reduces ATE to 0.019 within window refinement compared with uniform relative-pose weighting at 0.021.Source: §4.4 · p. 16
5 Conclusion
The conclusion presents a self-contained, reliability-regulated framework for progressive COLMAP-free 3DGS. It reports improved trajectory accuracy and novel-view rendering on Tanks and Temples and CO3D-V2 against unposed baselines.
5 Conclusion
- A shared self-supervised bidirectional cycle-consistency signal gates forward kinematic warm-starts and weights retrospective relative-pose constraints during fixed-geometry window refinement.The conclusion reports improved camera trajectory accuracy and novel-view rendering on Tanks and Temples and CO3D-V2, outperforming unposed NeRF and 3DGS baselines.Source: §5 · p. 16