Source-linked AI summary
Triangulation-Free Bundle Adjustment with Graduated Non-Convexity for Camera Pose Refinement from Coarse Priors
Nikolaos Kyriazis
TL;DR
Metric AR pose priors must be refined for reconstruction without losing their accuracy or frame, yet triangulation-based bundle adjustment can damage good priors and has a narrow basin. The paper replaces triangulation with per-observation ray depths and symmetric cross-projection residuals, then applies GNC to traverse coarse-prior basins. It preserves accurate room-scale priors, runs on CPU, and recovers all feasible perturbations through 16°/80 mm, though degenerate texture remains a failure case.
Problem
Triangulation-based bundle adjustment can worsen accurate metric AR priors and collapses when prior errors exceed a narrow basin, limiting cheap CPU refinement for reconstruction.
Method
The paper replaces triangulated landmarks with per-observation ray depths and symmetric cross-projection residuals, enabling structure re-expression and graduated non-convexity.
Results
425/425: GNC recovered every feasible run through 16°/80 mm and 85% at 32°/160 mm, while the plain solve held accurate room priors at 0.57°.
Takeaways & Limitations
The structureless solver preserves the prior’s metric frame and provides CPU refinement across substantially coarser priors than triangulation-based bundle adjustment within the tested envelopes.
Takeaways & Limitations
Degenerate repeated texture can defeat filter-free sparse refinement: one scene failed for every solver at zero perturbation, and the twisted solution traded pose fidelity for rendering quality.
Abstract
from arXiv · showhide
Mobile AR frameworks attach a metric pose prior to every casual phone capture, and turning it into reconstruction-grade poses cheaply on CPU is the step before novel-view synthesis. The least a refiner owes an accurate prior is not to make it worse. The workhorse refiner does. On 15 ScanNet++ iPhone room captures, COLMAP triangulation plus prior-seeded bundle adjustment degrades an accurate ARKit prior in all 15, 0.55 degrees to 0.74 degrees by scene-mean. The cause is the seeding. Structure is triangulated from the prior before anything is optimized, so the prior's error is baked into the structure the optimizer trusts. We remove the triangulation. Every keypoint owns a scalar depth along its own back-projected ray and each match contributes two symmetric cross-projection residuals, so structure is re-expressed at every iterate. The same solve holds the room prior at 0.57 degrees and never fails in 330 perturbed room runs, and at object scale reaches 0.265 degrees/1.80 mm from a prior at 0.456 degrees in a median of 10 s per scene on one CPU, against 2.5 GPU-hours for a learned refiner. Because no structure is committed, the objective also admits graduated non-convexity, which measures how deep the defect goes. Classical refinement collapses past 1-2 degrees of prior error, barely beyond a real ARKit prior, and no classical refinement arm survives 32 degrees. Ours recovers 425 of 425 runs through 16 degrees/80 mm and 85% at 32 degrees/160 mm, and perturbed rooms through 32 degrees. Nominal object-scale accuracy is on par rather than better, on a benchmark at its own noise floor, where classical bundle adjustment is a strong baseline absent from the literature. One scene fails for every solver already at zero perturbation. Re-mapping from position priors matches us in the prior's frame but discards it, so it cannot exploit a prior worth keeping or be warm-started.
1 Introduction
The paper targets cheap CPU refinement of metric AR pose priors without damaging accurate estimates. It removes triangulation by re-expressing structure per iteration and uses GNC to widen robustness to coarse priors.
- Motivation and problem: 0.55°→0.74°: Classical triangulation plus bundle adjustment worsened accurate ARKit priors in all 15 ScanNet++ room scenes.The paper attributes this failure to structure triangulated from erroneous prior poses before optimization.
- Method: Each observation receives a scalar depth on its own back-projected ray, while matches use symmetric cross-projection residuals instead of committed triangulated structure.Structure is re-expressed at every solver iterate.
- Results: 0.57°: The plain solve preserved the room-scale prior and completed all 330 perturbed room runs without solver failure.At object scale, it reached 0.265°/1.80 mm from a 0.456° prior in a median of 10 seconds per scene on one CPU with capped matches.
- Results: 425/425: GNC recovered every feasible MobileBrick run through 16°/80 mm and 85% at 32°/160 mm.Classical refinement collapsed beyond approximately 1–2° of prior error and remained conditional on a narrow basin.
- Practical scope: The approach preserves the prior’s metric frame, unlike alignment-based evaluation or prior-free remapping that can discard the prior’s gauge and scale.Its CPU implementation uses 1×1 eliminated Schur blocks and shared evaluation caching.
2 Related work
Related work spans landmark and structureless bundle adjustment, initialization-free objectives, robust optimization, and joint pose–radiance-field refinement. The paper positions its method as retaining reprojection-error bundle adjustment while using per-observation structure and GNC for sparse classical refinement.
- Point parameterizations: Per-observation depth assigns one ray scale to each observation, with match consistency enforced by symmetric residuals rather than shared landmark variables.This differs from inverse-depth and parallax parameterizations that retain one structure parameter set per landmark.
- Structureless bundle adjustment: Structureless methods eliminate landmarks through multi-view, epipolar, null-space, or scale-consistency constraints, trading away bundle adjustment’s reprojection-error form.The paper’s formulation keeps the familiar robust reprojection-error objective.
- Initialization-free optimization: Initialization-free bundle adjustment widens convergence basins by changing the objective through projective, affine, Gaussian, or related surrogates.These methods address basin size through alternative formulations rather than the paper’s per-observation structure plus GNC combination.
- Known-rotation formulations: Known-rotation L∞ formulations can optimize remaining variables globally, but they require rotations that are already good and do not refine wrong rotations.Their worst-residual objective is also outlier-sensitive on exhaustive matches.
- Robust and learned refinement: GNC sequences objectives from smooth to sharp, while related dense, learned, and neural-rendering systems use per-pixel depths, learned correspondences, or radiance-field losses.The paper applies GNC within a sparse classical CPU refiner and compares against learned systems operating on MobileBrick.
3 Method
The method refines camera poses from sparse matches without triangulation by assigning each observation a ray depth and optimizing symmetric cross-projection errors, optionally with intrinsics, distortion, and graduated loss-scale annealing.
- Each phone-video input supplies pose priors, rough shared intrinsics, and sparse feature matches; the solve outputs refined poses with optional intrinsics and distortion.
- Ray-time parameterization: Each observation back-projects to a world-space ray and owns one positive scalar depth, eliminating shared landmark variables and track building.The depth is initialized to 1 metric meter and is shared with no track.
- Cross-projection objective: Each match contributes two symmetric cross-projection residuals evaluated against undistorted pixels using shared pinhole intrinsics.Distortion enters through observation undistortion rather than per-residual forward distortion.
- Objective: The objective combines robust match errors with position and orientation priors that anchor the metric AR frame and discourage trajectory collapse.The arctangent loss saturates around 10 px of reprojection error, while the prior terms constrain global similarity and orientation.
- Efficient solve: Per-observation depths are eliminated as 1×1 Schur blocks, and each residual couples exactly two cameras for a sparse reduced camera system.A shared evaluation cache avoids repeating expensive undistortion and unprojection work across residuals.
- Graduated non-convexity: Graduated non-convexity anneals the arctangent scale from 10^4 to 10^3 to 10^2 in warm-started stages, while the final stage retains the nominal objective.The starting scale is load-bearing; lower starting scales lose runs at 8°/40 mm, whereas stage counts are largely indistinguishable.
4 Experiments
Experiments compare prior-seeded classical BA, learned and prior-free alternatives, and the proposed solver on MobileBrick and ScanNet++ room captures, emphasizing accuracy, robustness, cost, and evaluation limits.
- MobileBrick accuracy: On 18 MobileBrick scenes, classical prior-seeded BA reaches 0.218°/1.50 mm at 53 s per scene on CPU, while the proposed solver reaches 0.246°/1.61 mm with its trust region.The classical baseline is established as a strong, previously unreported benchmark comparison; the nominal differences are near the dataset’s noise floor.
- Alternative initializations: Prior-free global SfM registers all 18 MobileBrick scenes but discards the metric prior, whereas a feed-forward output can serve as another coarse-prior source for refinement.MASt3R-SfM improves from 1.26°/5.84 mm to 0.221°/1.24 mm on six scenes, but the comparison changes multiple system components.
- MobileBrick accuracy: MobileBrick’s prior-seeded refiners occupy the same accuracy band, roughly 2× better than the prior in rotation, but differences near 0.5°/2 mm are not rankable.The ground-truth poses have approximately 0.5°/2 mm residual accuracy.
- Evaluation scope: The ScanNet++ room selection is type-stratified rather than a random sample, and published ground-truth pose estimates introduce a circularity risk that cannot be bounded.The 15 scenes were frozen before solver runs, while the evaluation poses and selection protocol constrain interpretation of the room-scale gap.
- Room-scale refinement: At zero perturbation, classical BA worsens all 15 ScanNet++ room priors from 0.55° to 0.74° scene-mean rotation, while the plain proposed solve gives 0.57°/18.5 mm.The staged GNC recipe gives 0.69°/19.6 mm nominally and costs 14 CPU-minutes per scene versus 1.7 minutes for classical control.
- Robustness to coarse priors: The staged GNC solver recovers every feasible room run through 16°/80 mm and maintains a 0.52°–0.54° median, with no solver failure in 330 runs.At 32°/160 mm, the staged recipe exposes one seed-dependent apartment failure in the five-seed sweep.
4.4 Runtime
Runtime comparisons distinguish nominal costs from coarse-prior robustness: the structureless method is fast on object-scale scenes, while room-scale refinement is slower than prior-position mapping.
- CPU solve times exclude feature extraction and matching, while GPU figures cover full inference or alignment.All methods use the same precomputed per-scene database, and CPU measurements use one 32-thread machine.
- Object scale: 35 s median makes the structureless solve the cheapest object-scale method that handles coarse priors.Prior-position mapping takes 45 s, annealed re-triangulation 87 s, and default bundle adjustment 13 s but fails past 1–2°.
- Room scale: 780–920 s per scene makes the staged room-scale recipe slower than prior-position mapping at 360 s.Prior-position mapping is the only evaluated arm both faster than ours and able to handle a coarse prior.
- Learned refiners: 2.5 h per scene is PoRF’s reported full-training budget, but pose convergence typically occurs within its first 5,000 iterations.The learned and classical timings differ in hardware and by-products, so a single compute-ratio comparison is avoided.
4.5 Downstream novel-view synthesis
The paper evaluates whether refined poses improve downstream Gaussian-splatting novel-view synthesis under identical training conditions, finding gains over raw ARKit priors but near-noise-floor differences between refiners.
- Refinement recovers one half of the ARKit-to-ground-truth PSNR gap with the proposed poses and two thirds with the classical control.The mean gap between the two refiners is 0.23 dB, or 0.10 dB excluding castle.
- Held-out-view Gaussian splatting uses identical training configurations for four fixed pose sources across 18 evaluation scenes.Models use every eighth frame held out, 7,000 half-resolution steps, and identical Gaussian and hyperparameter settings.
- Interpretation: Differences below approximately 0.4 dB lie at the measured training-noise floor, limiting how strongly rendering scores distinguish pose quality.Castle is a designed diagnostic case whose returned twisted solve can out-render ground truth.
4.6 Ablation of the parameterization and match capping
The ablation compares per-observation depths with anchored tracks and match capping, finding small mean differences near the noise floor and a modest benefit from the orientation prior.
- Per-observation structure beats anchored tracks on 14 of 18 scenes in translation and 11 in rotation, while anchored tracks are slightly faster.Both formulations share the same matches, priors, robust loss, and wide GNC basin.
- Differences between parameterizations remain below the ground-truth noise floor and are rankable mainly through paired per-scene comparisons.The shared ground-truth error partially cancels in those paired comparisons.
4.7 Why triangulation narrows the basin
The mechanism study attributes classical basin collapse to triangulating structure from perturbed priors: fixed poisoned structure favors the perturbation, whereas re-expressed structure preserves the accurate solution.
- Mechanism: At 16° perturbation, prior-based triangulation retains 52,814 of 580,183 observations, or 9%, and the retained structure determines which poses appear favorable.This establishes triangulation as a commitment before pose refinement.
- Mechanism: Oracle structure moves the classical objective minimum back to the accurate prior, while perturbed structure makes cost decrease toward the perturbation.The comparison isolates the triangulation input rather than the reprojection objective form.
- Basin comparison: Oracle-structure classical BA recovers through 16° but collapses from 2°, whereas the annealed structureless objective still recovers at 32°.Thus, oracle structure widens the classical basin but does not match the structureless method at the largest perturbation.
4.8 Basin of convergence
The basin study shows that triangulation-based refinement collapses under coarse priors, whereas structureless optimization with GNC recovers substantially larger perturbations. Controls indicate the gain is tied to the formulation rather than merely favorable tuning, though nominal infeasibility and edge stochasticity remain.
- Classical BA succeeds in no run from 4°/20 mm onward under its nominal configuration, returning the perturbed prior essentially unchanged.Its triangulator retains only matches consistent with the given poses, leaving sparse, incorrect structure for bundle adjustment to polish.
- At 16°/80 mm, the best tuned classical configuration recovers 1/18 runs, while GNC recovers every feasible run; at 32°, classical recovers none.Porting the annealing schedule into classical refinement improves recovery but does not match the structureless formulation.
- 26 solver configurations were searched on the authors’ side versus 8 classical configurations, an approximately three-to-one search-budget asymmetry.Despite the larger search, the authors report that their basin is insensitive to tested variations except for a start scale a decade too low.
- 425/425 feasible runs recover through 16°/80 mm, while GNC recovers 85% at 32°/160 mm.These results span 17 scenes and five noise seeds; the 18th scene fails at zero perturbation for reasons unrelated to the basin.
- The structureless solve removes seed-dependent failures through 16°/80 mm, while its edge failures at 32° concentrate in four scenes.Additional seeds preserve the 32° recovery rate, but the objective remains stochastic near the perturbation edge.
- Rough seeds with the correct topology can still reach the ARKit-primed fixed point when match evidence admits the true solution.This was observed with look-at and translation-averaging seeds, including every feasible MobileBrick scene.
4.9 Refinement in the prior’s own frame
Evaluating poses in the prior’s metric frame separates shape recovery from frame preservation. Classical refinement recovers shape but loses the frame, while remapping anchors the frame without exploiting the prior’s quality; the proposed method and remapping are nearly indistinguishable in absolute-frame accuracy.
- Classical refinement inherits essentially the whole prior error in the absolute frame because bundle adjustment is gauge-free.At a 16° prior it reports 16.04° absolute rotation error despite a 0.25° Sim(3)-aligned score.
- The control table compares matched prior, noise, schedule and parameterization variants using successes per perturbation level.The noise block substitutes correlated random walks for i.i.d. perturbations, while the parameterization block swaps per-observation depths for anchored-track depths.
- Re-mapping’s Sim(3) score remains 0.209° across 1°, 8°, 16° and 32° priors, showing that its reconstructed shape does not depend on prior quality.Its absolute error grows because the prior-independent reconstruction is registered onto an increasingly wrong anchor.
- At coarse levels, in-place refinement and prior-position mapping have absolute translation-error ratios of 0.91, 0.96 and 1.03 at 8°, 16° and 32°.At the nominal 1° prior, the mapper is better: 1.35 mm versus 3.03 mm.
- Room-scale staging preserves more absolute-frame accuracy at 32°/160 mm, reaching 1.96° and 50 mm versus the object-scale configuration’s wider-basin trade-off.The reported comparison makes explicit that releasing the orientation prior buys basin width at a cost in absolute rotation.
4.10 Real drift on LaMAR
Real LaMAR ARKit drift is mainly translation-dominated and remains within the plain solver’s rotational basin over five minutes. On seven resampled 60-second chunks, refinement changes performance mainly when drift has moved beyond the prior’s accuracy floor.
- The LaMAR evaluation measures per-chunk error after rigid alignment on the first ten frames and reports refinement separately on the resampled CAB sequences.This protocol distinguishes real drift behavior from the synthetic perturbation experiments.
- 178 LaMAR chunks show rotationally small, translation-dominated ARKit drift, with worst frames reaching 70 cm.The captures were split into 10-second and 60-second windows anchored using the first ten ground-truth frames.
- On seven resampled 60-second CAB chunks, GNC leaves the median result unchanged at 8.4 cm.The raw 10 Hz streams were used because the benchmark’s 1 Hz sampling provides too few pairwise matches for a full solve.
- After five minutes, mean rotation drift reaches 3.5° and worst-frame rotation 5.9°, while mean translation reaches 2.8 m.The authors conclude that smooth accumulated drift does not populate the wide-basin regime tested by coarse relocalization or registration errors.
4.11 Failure analysis on repeated texture and degenerate geometry
The castle scene defeats sparse prior-seeded refinement because repeated texture and near-planar geometry drive multiple systems toward the same coherent twist. A trust-region fallback detects the implausible departure, but the underlying wrong optimum is not exposed by internal match consistency.
- Failure characterization: 5° orientation-field twist defeats the refinement, with 4.2–5.2° per-frame rotation errors and 2.0–56.5 mm position drift after Sim(3) alignment.The raw solver frame instead keeps rotations within 0.7° of the prior and transfers the twist into positions.
- Failure characterization: The twist persists from ground-truth initialization and across prior weights, filters, match sources, GNC, and solver families.GLOMAP and VGGT converge to the same twist, while GLOMAP’s mean error-axis agreement with the proposed solver is 0.99997.
- Scene degeneracy: Repeated-texture LEGO facade geometry, weak observation sharing, and 34% singleton observations characterize the failure, although weak connectivity alone does not predict it.Castle’s mean match-graph degree is 5.2, while bridge is lower at 2.35; the authors identify repeated texture as essential to the failure hypothesis.
- Guardrail and trade-off: Table 10 compares where systems land on castle and which interventions leave the shared 5° twist unchanged.The table reports axis coherence and agreement of the mean rotation-error axis under identical Sim(3) alignment.
- Guardrail and trade-off: 18.35 versus 15.08 dB PSNR favors the raw twisted solve over the returned prior, creating a reconstruction-versus-localization trade-off.The trust region returns the prior when the refined solution leaves its plausibility envelope, preserving ground-truth-frame pose fidelity at a rendering cost.
5 Conclusion
The paper concludes that triangulation-free refinement preserves accurate metric priors while extending the usable basin for coarse priors, at CPU-scale cost. Its strongest claims are bounded by benchmark noise, synthetic perturbations, initialization requirements, and a degenerate repeated-texture failure.
- Conclusion: 0.218°/1.50 mm at 53 s per scene establishes triangulation-based BA as a strong nominal classical baseline, but no classical arm survives 32° prior error.The proposed method reaches 0.265°/1.80 mm from a 0.456° prior, while the benchmark’s roughly 0.5°/2 mm ground-truth residual limits fine-grained rankings.
- Conclusion: 0.52°–0.59° median error through 32° perturbations shows the annealed variant retains a broad basin across the tested scales.The room-scale result extends through 32°/160 mm, while the conclusion notes that the lifted formulation re-expresses the cost landscape at every iterate.
- Conclusion: No-alignment evaluation shows that classical refinement fails to anchor the prior frame, whereas the proposed solve preserves that frame while re-mapping discards it.Re-mapping matches the proposed method within ten percent at 8°, 16°, and 32°, but is better at a nominal prior and does not refine the prior in place.
- Practical implications: The practical policy combines pre-solve fragility checks, trust-region choices, and posterior checks to select between nominal safety and wide-basin recovery.The unified recipe reproduces the oracle-configured arm through 16° at object scale and 32° at room scale, with 5–23 s pre-solve cost against a 50 s solve.
- Limitations: Near-unanimous wrong matching, as in castle, is invisible to pre-solve disagreement statistics and appears only through posterior checks.The conclusion explicitly treats this as an exception to automatic prior-quality gating.
- Limitations: The evaluation uses synthetic perturbations, remains unevaluated for retrieval or GPS-plus-compass priors, and still needs basin-selecting initialization.At 64°/320 mm, GNC recovers only some scenes; MobileBrick also saturates near its approximate 0.5°/2 mm ground-truth accuracy.
A Per-scene results
The per-scene-results section points readers to a detailed breakdown of errors and solve times for the full method and the prior-seeded COLMAP control.
- Table 11 lists per-scene rotation and translation errors and solve times for the full method and the prior-seeded COLMAP control.
B Per-scene downstream rendering
The downstream-rendering section reports held-out Gaussian-splatting PSNR by pose source under identical training settings, with castle scored using the trust-region fallback.
- Table 12 compares held-out 3D Gaussian splatting PSNR across pose sources under identical training configurations, where higher is better.Castle’s proposed-method entry uses the returned prior after trust-region fallback; the raw twisted solve is analyzed separately.