Source-linked AI summary
Direct Sparse Mapping
Jon Zubizarreta, Iker Aguinaga, J. M. M. Montiel
TL;DR
Current PBA systems use temporary maps and cannot reuse map points when scenes are revisited. DSM adds a persistent map and processes reobservations in photometric bundle adjustment, reporting the most accurate EuRoC results among direct monocular methods.
Problem
Current PBA systems use temporary maps that cannot reuse marginalized map points when scenes are revisited.
Source: §I · p. 2Method
DSM builds a persistent map and uses covisibility-based local-window selection and coarse-to-fine photometric bundle adjustment to process map-point reobservations.
Source: §I · p. 2, B. Photometric Bundle Adjustment (PBA) · p. 4, §IV · p. 4Results
DSM reports the most accurate results among direct monocular methods on EuRoC and completes 10 of 11 challenging sequences with RMS ATE below 0.1 m without loop-closure detection and correction.
Source: §I · p. 2, §VIII · p. 8Takeaways & Limitations
Reusing map information in the photometric model improves both camera-trajectory accuracy and reconstructed maps.
Source: §VIII · p. 8, §IX · p. 9During long-term sequences in the same environment, reliable estimates depend on successfully processing point reobservations.
Source: §VIII · p. 8
Abstract
from arXiv · showhide
Photometric bundle adjustment, PBA, accurately estimates geometry from video. However, current PBA systems have a temporary map that cannot manage scene reobservations. We present, DSM, a full monocular visual SLAM based on PBA. Its persistent map handles reobservations, yielding the most accurate results up to date on EuRoC for a direct method.
I. INTRODUCTION
Current photometric bundle-adjustment systems use temporary maps and cannot reuse points when a scene is revisited. DSM adds a persistent map and methods for processing reobservations within photometric bundle adjustment.
I. INTRODUCTION
- Temporary-map PBA systems marginalize points outside the active window, so revisited areas force point duplication and lose informative reobservations.The paper links this limitation to motion drift and structure inconsistencies.Source: §I · p. 2
- DSM is a fully direct monocular VSLAM system with a persistent map that reuses existing map information in the photometric formulation.The system is named Direct Sparse Mapping.Source: §I · p. 2
- The Local Map Covisibility Window selects keyframes observing the same scene region, including keyframes far apart in time, to include point reobservations.The selection combines temporal and covisibility criteria.Source: §I · p. 2
- A coarse-to-fine PBA scheme processes high-parallax point reobservations and supports convergence.The authors state that point reobservations provide rich geometric information.Source: §I · p. 2
- A t-distribution robust influence function and pixel-wise outlier management increase PBA consistency against outliers from distant keyframes.Source: §I · p. 2
- DSM reports the most accurate EuRoC results among direct monocular methods and shows consistent maps without duplicate points when managing reobservations.The paper reports quantitative results for both camera trajectories and reconstructed maps.Source: §I · p. 2, Figure 1 · p. 2
II. RELATED WORK
Prior visual SLAM and odometry approaches use feature correspondences, sliding windows, or loop-closure corrections; DSM transfers persistent-map ideas to direct photometric optimization. Its mapping system separates tracking from local-window optimization and maintains a globally consistent map.
II. RELATED WORK
- Feature-based VSLAM uses repeatable image features and reprojection error, while DSM builds covisibility connections geometrically because it has no explicit point matches.The related-work discussion describes ORB-SLAM as a reference indirect VSLAM approach and notes DSM uses a smaller covisible-keyframe window to control computation.Source: §II · p. 3
- DSO performs direct photometric bundle adjustment in a sliding window, while DSM extends this approach with a persistent map that reuses marginalized scene information.DSO's temporary-map design cannot reuse points after marginalization, which the paper associates with drift and map inconsistencies.Source: §II · p. 3
- Loop-closure extensions often correct trajectories with a pose graph but do not reuse map points in optimization; persistent-map systems process reobservations in bundle adjustment for more accurate estimates.Source: §II · p. 3
- DSM incorporates a t-distribution robust influence function and an outlier-management strategy to handle spurious observations from widely separated keyframes.Source: §II · p. 3
III. DIRECT MAPPING
- Tracking estimates frame poses and marks keyframes, while mapping tracks points, recalculates the local window, activates points, optimizes motion and structure, and maintains global consistency.The front-end and back-end run in parallel threads.Source: §III · p. 3
- The persistent map stores keyframe images and associated map points; active keyframe poses, affine-light parameters, and point inverse depths are optimized, while inactive parameters remain fixed.The mapping model assumes undistorted images and uses a pinhole camera model for projection and inverse projection.Source: §III · p. 3
A. Photometric Model
The photometric model compares small image patches across host and target frames, using affine brightness transfer and weighted intensity differences. Its cost depends on both geometric and photometric parameters.
A. Photometric Model
- The system uses the photometric function proposed in for geometry initialization, local-window PBA, and map reuse.The point-observation residual sums squared intensity differences over a small patch and includes affine brightness transfer for automatic gain control and illumination changes.Source: A. Photometric Model · p. 4
- Each patch pixel is projected into the target frame using its inverse depth to compare host and target intensities.The projection and residual are defined over pixels in a small patch around each point.Source: A. Photometric Model · p. 4
- The photometric cost depends on camera poses and point inverse depths, as well as each frame’s affine brightness parameters.Pixel weights combine a robust influence function with a gradient-dependent weight that reduces the influence of high-gradient pixels due to noise.Source: A. Photometric Model · p. 4
B. Photometric Bundle Adjustment (PBA)
When a keyframe is created, PBA optimizes the model over the active LMCW, using coarse-to-fine optimization to handle larger increments. The method solves the weighted photometric objective with iteratively re-weighted Levenberg–Marquardt.
B. Photometric Bundle Adjustment (PBA)
- Each new keyframe triggers minimization of the photometric error over the active LMCW, including the observations of points in active keyframes.The total error sums point-observation errors over keyframes, points, and observations.Source: B. Photometric Bundle Adjustment (PBA) · p. 4
- Coarse-to-fine optimization across active keyframes lets PBA handle larger camera and point increments when reobserved points start outside the convergence radius.Each level iterates to convergence, initializes the next level, and makes outlier decisions independently.Source: B. Photometric Bundle Adjustment (PBA) · p. 4
- The objective is minimized with iteratively re-weighted Levenberg–Marquardt, which estimates an increment from a second-order approximation with fixed weights.The implementation uses analytic derivatives and the Schur complement to solve the reduced problem.Source: B. Photometric Bundle Adjustment (PBA) · p. 4
- The illustrated LMCW has seven active keyframes: four temporal and three covisible, with the latest keyframe being created.The figure distinguishes temporal keyframes in blue and covisible ones in orange.Source: Figure 2 · p. 4
IV. LMCW: LOCAL MAP COVISIBILITY WINDOW
The LMCW combines a fixed-size temporal window with covisible keyframes so optimization can use reobservations from distant-in-time views. Its selection strategy also seeks coverage of depleted image areas while avoiding duplicated map points.
IV. LMCW: LOCAL MAP COVISIBILITY WINDOW
- The LMCW combines temporal and covisible keyframes relative to the latest keyframe being created.Its two parts are selected using temporal and covisibility criteria.Source: §IV · p. 4
- The fixed-size temporal part supports exploration by initializing new points and maintaining odometry accuracy.A newly created keyframe enters this part and another keyframe is removed.Source: §IV · p. 4
- The temporal selection retains the two latest keyframes and distributes the rest evenly in space, favoring high-parallax observations.The removal strategy avoids premature fixation of keyframe locations while they are being optimized.Source: §IV · p. 4
- The covisible selection favors old keyframes that project points into depleted areas of the latest keyframe, excluding points above a viewing-angle threshold.The distance map is updated after each selection until Nc keyframes are chosen or no suitable ones remain.Source: §IV · p. 4
- Adding covisible keyframes before activating new map points incorporates mapped areas into the LMCW and avoids point duplication.The values of Nt and Nc are tuned experimentally.Source: §IV · p. 5
V. ROBUST NON-LINEAR PBA
Because covisible reobservations can contain photometric outliers, the method derives PBA weights from an error distribution. It evaluates Gaussian, Student’s t, and Huber models, with t-distribution fitting and per-keyframe adaptation for varying error conditions.
V. ROBUST NON-LINEAR PBA
- The proposed outlier strategy derives PBA weights from the photometric error distribution.The probabilistic formulation treats minimizing the photometric objective as minimizing the negative log-likelihood of model parameters given the errors.Source: §V · p. 5
- A Gaussian error model gives constant weights, so it treats all points equally and cannot neutralize outliers.The model assumes errors are normally distributed around zero.Source: §V · p. 5
- The t-distribution explains sparse photometric errors and quickly lowers weights for tail errors, while its degrees of freedom and scale are fitted together.The fitting uses Nelder–Mead after rejecting gross outliers using a MAD-based scale estimate.Source: §V · p. 5
- The t-distribution fits challenging reobservation errors better than the normal model when occlusions introduce many outliers.For mostly photo-consistent points, both models fit well.Source: Figure 3 · p. 5
- Huber weighting reduces the influence of high-error measurements without removing them, a property described as crucial for reobservation processing.For normally distributed errors, the Huber model uses a threshold λ = 1.345σn and gives outliers linear influence.Source: §V · p. 5
A. Implementation of the probabilistic model into the PBA
The model fits photometric-error distributions separately for each keyframe, accommodating differences such as motion blur, occlusions, and noise. It holds the resulting weights fixed during each pyramid level’s optimization, then recomputes the distribution from the best geometry solution.
A. Implementation of the probabilistic model into the PBA
- Photometric-error distributions are fitted separately for each keyframe, allowing different error patterns across frames to be represented.The observed variation may come from motion blur, occlusions, or noise.Source: A. Implementation of the probabilistic… · p. 5
- Keyframe-specific distributions let the PBA treat an error value as an outlier in a regular frame but an inlier in a challenging one.The threshold adapts to the keyframe’s error distribution.Source: A. Implementation of the probabilistic… · p. 5
- The error distribution is computed at each pyramid level’s start, held fixed during optimization, then recomputed from the best geometry solution.Re-estimating weights every iteration would change the objective and could degrade optimization.Source: A. Implementation of the probabilistic… · p. 5
B. Outlier management
The system tracks frames against an updated local map and expands it with selectively initialized points. It manages outliers at pixel, observation, and map-point levels, while retaining candidate points through the temporal part of the LMCW for delayed initialization.
B. Outlier management
- Pixel-level masks classify each of an observation’s eight measurements using the target keyframe’s 95th-percentile error threshold.Observations with more than 30% outlier pixels are removed after local PBA; those above 60% are discarded during optimization.Source: B. Outlier management · pp. 5–6
- Map-point retention depends on observation history: new points must appear in every subsequent keyframe, and mature points are removed below three observations.A point becomes mature after being observed in three keyframes.Source: B. Outlier management · p. 6
VI. FRONT-END
- Each frame is tracked against a local map formed from active LMCW points, with map points and the latest keyframe fixed during pose and brightness estimation.Tracking uses a velocity-model initialization, coarse-to-fine optimization, a robust influence function, and inverse compositional updates.Source: §VI · p. 6
- Keyframe selection combines map-point visibility, tracked-frame parallax, and relative brightness change in a weighted score.The tracked frame is selected when the weighted combination exceeds 1.Source: §VI · p. 6
- New map points are initialized from consecutive tracked frames by searching along epipolar lines, and only distinctive, low-uncertainty points are activated.Candidates persist until they leave the temporal part of the LMCW; points in already mapped areas are mostly reused.Source: §VI · p. 6
VII. RESULTS
The parameter analysis evaluates DSM across EuRoC sequences and finds that coarse-to-fine PBA helps DSM benefit from reobservations. Two pyramid levels balance accuracy and runtime, while additional levels increase cost without improving accuracy.
VII. RESULTS
- EuRoC evaluation covers three scenarios, trajectory and reconstruction accuracy, and challenging motions and illumination changes.The parameter study runs each sequence five times across camera and travel-direction variants, totaling 220 experiments.Source: §VII · p. 6, A. Parameter analysis and tuning · p. 6
1) Coarse-to-fine PBA:
- About 80% of DSM runs achieve RMS ATE below 0.1 m, compared with 40% for DSM-SW, when using coarse-to-fine PBA.Without coarse-to-fine optimization, DSM-SW performs better because DSM cannot benefit from reobservations amid accumulated drift.Source: 1) Coarse-to-fine PBA · p. 6
- Np = 2 is the default balance: adding pyramid levels beyond two raises runtime significantly without improving DSM accuracy.The number of pyramids increases runtime by approximately 50% per level, while reobservations have little effect on processing time.Source: 1) Coarse-to-fine PBA · pp. 6–7
2) Robust Influence Function:
The parameter study favors the t-distribution and a mix of temporal and covisible keyframes. In trajectory comparisons, DSM is more accurate than other direct approaches in almost all sequences, with its strongest reported sequence result at 7.6 cm RMS ATE.
2) Robust Influence Function:
- The t-distribution provides greater robustness than Huber in challenging situations, while fixed and dynamic Huber settings show no significant difference.The t-distribution quickly reduces weights for errors in the tails; the authors use the complete t-distribution model by default.Source: 2) Robust Influence Function · p. 7
3) Number of covisible keyframes in the LMCW:
- More covisible keyframes improve trajectory accuracy, but too few temporal keyframes sharply reduce it; Nt = 4 and Nc = 3 are the default balance.Temporal keyframes support odometry robustness, optimize new keyframes, and help initialize points during exploration.Source: 3) Number of covisible keyframes in… · p. 7
B. Quantitative results
- The benchmark compares DSM with ORB-SLAM, DSO, and LDSO using sequential runs and seven active keyframes for direct methods.Reported runs cover both directions, both cameras, and ten repetitions per sequence.Source: B. Quantitative results · p. 7
1) Trajectory error:
- 7.6 cm RMS ATE on V1 03 difficult is DSM’s best reported sequence result; DSM is more accurate than other direct approaches in almost all sequences.ORB-SLAM performs better in V1 and V2, while DSM performs best on MH sequences; all compared approaches fail on V2 03 difficult.Source: 1) Trajectory error · p. 7, Table I · p. 7
- Global PBA only slightly improves RMS ATE at substantially higher computational cost; the local mapping result is very close at a small fraction of that cost.For V2 02 medium, global PBA optimizes fifty times more parameters and takes two orders of magnitude longer.Source: 1) Trajectory error · p. 7
2) Mapping vs Pose-Graph:
DSM’s persistent map improves accuracy and sequence completion over LDSO while supporting more precise reconstructions with fewer duplicated points; its initial implementation remains processing-limited.
2) Mapping vs Pose-Graph:
- DSM processes 80% of sequences with RMS ATE below 0.1m, versus 50% of LDSO runs, and achieves better overall accuracy.The comparison is between DSM and LDSO.Source: Figure 8 · p. 8, 2) Mapping vs Pose-Graph · p. 8
- Although final RMS ATE can be similar, DSM maintains more stable trajectory accuracy over time as its persistent map reuses information continuously.LDSO sometimes misses loop closures for lack of feature matches, allowing drift until a larger correction is detected.Source: Figure 9 · p. 8, 2) Mapping vs Pose-Graph · p. 8
3) Map error:
- DSM’s reconstructed map is more accurate than LDSO’s in the compared sequences, while LDSO creates ten times more points because DSM reuses map points.The LDSO comparison excludes V2 03 difficult, where LDSO fails.Source: Figure 10 · p. 8, 3) Map error · p. 8
4) Processing time:
- PBA is the initial implementation’s processing bottleneck; reaching the keyframe creation rate requires twice the speed, with SIMD and point-level parallelism proposed as upgrades.The authors suggest these upgrades could support real-time applications because mapping need only run at keyframe rate.Source: 4) Processing time · p. 8
C. Qualitative results
DSM produces accurate trajectories and reconstructions on difficult EuRoC sequences, and completes 10 of 11 sequences below the stated ATE threshold. Its long-term operation depends on successful reobservations, and the authors identify map maintenance and place recognition as future needs.
VIII. DISCUSSION & FUTURE WORK
- DSM completes 10 of 11 challenging EuRoC sequences with RMS ATE below 0.1m without loop-closure detection and correction; long-term reliability depends on successfully processing reobservations.The authors identify removal of redundant keyframes and points as a possible map-maintenance strategy for long-term efficiency.Source: §VIII · p. 8
C. Qualitative results
- DSM estimates precise trajectories and 3D reconstructions in difficult V1 03 and MH 05 sequences, with qualitative maps shown for both.The trajectory is displayed in red in Fig. 11.Source: C. Qualitative results · p. 8, Figure 11 · p. 9
IX. CONCLUSION
- The conclusion identifies covisibility-based window selection, coarse-to-fine PBA, and a t-distribution influence function as components of DSM’s persistent-map approach.On EuRoC, the method reduces trajectory and map error while avoiding inconsistent map-point duplications.Source: §IX · p. 9