PaperlayerView exact revision on arXiv

Source-linked AI summary

DefSLAM: Tracking and Mapping of Deforming Scenes from Monocular Sequences

Jose Lamarca, Shaifali Parashar, Adrien Bartoli, J. M. M. Montiel

arXiv:1908.08918v2cs.CVeess.IV

TL;DR

Monocular SLAM relies on rigidity, limiting its use in deforming scenes such as medical endoscopy. DefSLAM combines frame-rate SfT tracking with keyframe-rate NRSfM mapping to handle exploratory monocular sequences. Experiments report accurate scene estimates, while direct comparison with other systems is not possible.

  • Problem

    Monocular SLAM’s rigidity assumption is a limitation for deforming scenes, including medical endoscopy.

    Source: §I · p. 2

  • Method

    DefSLAM runs SfT-based camera and deformation tracking in parallel with isometric NRSfM mapping that incrementally extends and refines templates.

    Source: B. Non-Rigid Monocular Techniques · p. 4

  • Results

    Experiments on deforming exploratory sequences report that DefSLAM robustly initializes, adapts its map to deformation, and produces accurate scene estimates.

    Source: §VIII · p. 12

  • Takeaways & Limitations

    Perspective SfT and NRSfM models let DefSLAM handle close-up exploratory scenes where perspective effects are prevalent.

    Source: §VIII · p. 12

    The authors state that direct comparison with other systems is not possible.

    Source: §VIII · p. 12

Abstract

from arXiv · show

Monocular SLAM algorithms perform robustly when observing rigid scenes, however, they fail when the observed scene deforms, for example, in medical endoscopy applications. We present DefSLAM, the first monocular SLAM capable of operating in deforming scenes in real-time. Our approach intertwines Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM) techniques to deal with the exploratory sequences typical of SLAM. A deformation tracking thread recovers the pose of the camera and the deformation of the observed map, at frame rate, by means of SfT processing a template that models the scene shape-at-rest. A deformation mapping thread runs in parallel with the tracking to update the template, at keyframe rate, by means of an isometric NRSfM processing a batch of full perspective keyframes. In our experiments, DefSLAM processes close-up sequences of deforming scenes, both in a laboratory controlled experiment and in medical endoscopy sequences, producing accurate 3D models of the scene with respect to the moving camera.

I. INTRODUCTION

DefSLAM addresses monocular SLAM’s rigidity assumption by combining frame-rate Shape-from-Template tracking with keyframe-rate Non-Rigid Structure-from-Motion mapping. It targets exploratory deforming scenes, including medical endoscopy, and reports accurate scene estimates while optimizing only the observed map zone.

A. SLAM

  • Monocular SLAM’s rigidity assumption limits its use when deformation predominates, including in medical endoscopy.The introduction identifies monocular visual SLAM as relevant to augmented reality and autonomous medical robotics in endoscopy.Source: §I · p. 2

I. INTRODUCTION

  • DefSLAM combines frame-rate SfT tracking with keyframe-rate NRSfM mapping to estimate camera pose, scene deformation, and templates for newly explored regions.The mapping thread incrementally refines NRSfM solutions, while tracking uses the resulting shape-at-rest template.Source: §I · pp. 2–3, B. Non-Rigid Monocular Techniques · p. 4
  • DefSLAM’s experiments cover exploratory deforming-scene sequences in a new kerchief dataset and the in-vivo Hamlyn endoscopy dataset.The authors compare with rigid monocular ORBSLAM and stereo MISSLAM, which has medical-scene SLAM accuracy results.Source: §I · p. 3

A. SLAM

  • Earlier deformable SLAM systems use depth sensors, and optimizing the whole map each time scales poorly with map size.DefSLAM instead optimizes the observed map zone; the authors state this supports scalability and CPU operation.Source: A. SLAM · p. 3

B. Non-Rigid Monocular Techniques

  • Orthographic NRSfM assumes a distant scene, whereas perspective is needed for close-up sequences; isometric NRSfM has also shown accuracy and occlusion-handling benefits.DefSLAM uses local isometric NRSfM and extends its incremental optimization to keyframe-rate mapping.Source: B. Non-Rigid Monocular Techniques · pp. 3–4
  • DefSLAM also contributes sequential active matching and surface alignment to build a global map in the monocular case.Its tracking and mapping algorithms can run in parallel.Source: B. Non-Rigid Monocular Techniques · p. 4

III. DEFSLAM SYSTEM OVERVIEW

DefSLAM represents the deforming scene as map points embedded in a template and runs separate tracking and mapping threads. Tracking estimates camera pose and point positions with SfT, while mapping uses NRSfM to estimate the surface at keyframe rate.

III. DEFSLAM SYSTEM OVERVIEW

  • The map stores time-varying 3D map points and keyframes; a reference keyframe supplies the template used to track incoming frames.Map points are initialized in anchor keyframes, and one anchor is selected as the reference after each new keyframe is processed.Source: §III · p. 4

IV. DEFORMATION TRACKING

  • The frame-rate tracking thread uses SfT to estimate map-point positions and camera pose from a template whose shape-at-rest is the reference-keyframe surface.Map points are embedded in the template; the caption describes tracking as estimating camera pose and viewed-map deformation.Source: §III · p. 4, Figure 2 · p. 5

III. DEFSLAM SYSTEM OVERVIEW

  • The keyframe-rate mapping thread uses NRSfM to estimate the surface observed in each keyframe.That surface defines the shape-at-rest template used by tracking for subsequent frames.Source: §III · p. 4

A. Template

  • The template is a triangular mesh whose map points are embedded in facets using barycentric coordinates.The mesh comprises nodes and edges, and its deformation is represented through the poses of its nodes.Source: A. Template · p. 4

B. Camera Model

  • The camera model is a calibrated pinhole model that projects 3D map points into image observations.The camera pose comprises rotation and translation, and focal lengths and principal point come from calibration.Source: B. Camera Model · p. 4

C. Camera Pose and Template Deformation

Tracking jointly estimates the observed template region’s deformation and camera pose by minimizing image reprojection error and template deformation energy. It optimizes only the observed local zone and fixes its boundary nodes to constrain camera-pose gauge freedom.

C. Camera Pose and Template Deformation

  • Only the observed template zone and its vicinity are optimized, limiting computation to the portion relevant to the current view.Tracking minimizes reprojection error together with a deformation energy formed from stretching, bending, and reference regularization terms.Source: C. Camera Pose and Template Deformation · p. 5
  • The stretching term measures changes in local mesh-edge lengths relative to the template’s shape-at-rest.The bending term uses mean curvature estimated through a discrete Laplacian; reference regularization keeps the template near its reference-keyframe position to alleviate weak pose observability.Source: C. Camera Pose and Template Deformation · p. 5
  • Fixing local-zone boundary nodes makes absolute camera pose observable, but observability may remain weak when boundary geometry is unsuitable.If the camera observes the entire template, there are no boundary points to fix and absolute camera pose is fully non-observable.Source: C. Camera Pose and Template Deformation · p. 5

D. Data Association

Data association matches current-frame ORB keypoints to map points by searching around predicted image locations. Figure 3 illustrates how matched points and the NRSfM-estimated surface relate to the reference-keyframe template during map extension.

D. Data Association

  • Active matching searches near predicted map-point projections and accepts the most similar ORB descriptor only when its Hamming distance is below a threshold.Predictions use the camera motion model, the latest template shape, and barycentric coordinates; map-point descriptors come from their initialization keyframes.Source: D. Data Association · p. 5
  • Figure 3 distinguishes the local area, matched points, the up-to-scale NRSfM surface, and the template computed from its scaled reference-keyframe surface.Compare the estimated surface with the resulting template as the map is extended.Source: Figure 3 · p. 6

E. New Keyframe Selection

Keyframe selection chooses a reference and template according to whether a new keyframe covers a new map region. Deformation mapping uses covisible keyframes to estimate and scale a surface, then builds a template for tracking.

E. New Keyframe Selection

  • A new-region keyframe becomes an anchor and reference with a new template; otherwise, the most covisible anchor is selected as reference and its template is refined.A keyframe is selected once mapping finishes processing.Source: E. New Keyframe Selection · p. 6

V. DEFORMATION MAPPING

  • Mapping refines observed map points and creates new ones on a reference-keyframe surface that defines the next tracking template's shape at rest.The surface contains map points observed during tracking.Source: §V · p. 6

A. NRSfM

  • Mapping estimates an up-to-scale surface from covisible keyframes using isometric NRSfM, aligns it to the previous map for scale, then meshes it and embeds map points.The local NRSfM model assumes isometry and infinitesimal planarity.Source: §V · p. 6, A. NRSfM · p. 6

B. Incremental Surface Normals Refinement

Surface normals are refined incrementally from points matched across keyframes, then transferred to the reference keyframe to recover an up-to-scale surface.

B. Incremental Surface Normals Refinement

  • Points matched in at least two keyframes provide normals in their anchor keyframe through nonlinear optimization.The normals are defined by Kk_x and Kk_y.Source: B. Incremental Surface Normals… · p. 7
  • Unlike, the optimization is incremental and initialized from the last estimate for fast convergence; refined normals transfer to the new reference keyframe.The transfer uses equation 12.Source: B. Incremental Surface Normals… · p. 7
  • Shape-from-Normals recovers the up-to-scale surface, whose bicubic B-spline depth is fit to estimated normals with a bending-energy regularizer.Control nodes lie on a regular image mesh.Source: B. Incremental Surface Normals… · p. 7

C. Surface Alignment

Surface alignment reconciles the up-to-scale estimate with the existing map before constructing a template. Template substitution and warp-guided matching handle changing shape and keyframe correspondences.

C. Surface Alignment

  • A Sim(3) transformation aligns the estimated surface with the previous map to recover coherent scale, rotation, and translation.The alignment uses map points.Source: C. Surface Alignment · p. 7

D. Template Substitution

  • Directly replacing the template can cause data-association failures under the small-deformation assumption, so matches transfer to the new template before the current shape is optimized.The transferred matches are from the previous template.Source: D. Template Substitution · p. 7

E. Warp Estimation and Non-Rigid Guided Matching

  • The system estimates a warp from tracking matches, uses it to guide ORB matching by search region and Hamming distance, then refines the warp with accepted matches.Schwarps are used because their Schwarzian regularizers are equivalent to infinitesimal planarity in the NRSfM formulation,.Source: E. Warp Estimation and Non-Rigid… · pp. 7–8

F. SLAM Initialization

Initialization supplies a planar first-frame template, which mapping replaces after the second keyframe. The experiments evaluate map quality on controlled Mandala and medical Hamlyn sequences.

F. SLAM Initialization

  • The initial template assumes the first frame's surface is a plane perpendicular to the camera optical axis.The mapping thread starts computing a replacement after the second keyframe.Source: F. SLAM Initialization · p. 8
  • Initial template accuracy depends strongly on the number of keyframes and their parallax, while tracking from an inaccurate template can still yield long tracks and low false-positive rates.As more high-parallax keyframes arrive, the estimated template eventually converges to the actual scene shape.Source: F. SLAM Initialization · p. 8

VI. IMPLEMENTATION DETAILS

  • For a 10x10-node mesh on an i7-4700HQ CPU with 7.7 GB RAM, tracking takes about 50 ms and mapping about 400 ms.The implementation runs entirely on the CPU; runtime depends on mesh resolution.Source: §VI · p. 8

VII. EXPERIMENTS

  • Evaluation uses Mandala and Hamlyn sequences with per-frame ground-truth depth, reporting scale-aligned 3D RMS error and the fraction of in-frustum map points matched.Mandala contains five 640x480, 30-fps exploratory kerchief sequences with progressively harder near-isometric deformation; Fig. 8 orders scenarios from less to more deformation.Source: §VII · p. 8, A. Mandala Dataset · p. 9, Figure 8 · p. 9

A. Mandala Dataset

  • Mandala compares DefSLAM with ORBSLAM across five sequences, while Hamlyn phantom-heart results are also compared with stereo MISSLAM.ORBSLAM was retuned for deforming sequences, and MISSLAM is stereo rather than monocular.Source: §VII · pp. 8–9

1) Overall quality experiment:

DefSLAM tracks deforming kerchiefs more accurately than rigid ORBSLAM on low-amplitude sequences, while the most challenging deformations produce tracking delay and larger errors. Scale drift and normal-estimation results show how re-observation and the mapping refinement stages affect reconstruction quality.

1) Overall quality experiment:

  • DefSLAM’s error stayed similar to rigid mandala0 on mandala1 and mandala2, while ORBSLAM’s 3D RMS error tripled under deformation.DefSLAM also recovered more accurately the observed deformation in RMS error and matched map-point fraction.Source: 1) Overall quality experiment · p. 10
  • In mandala3 and mandala4, ORBSLAM could not process either sequence entirely, while DefSLAM averaged around 4 cm despite convergence delay and RMS-error peaks.DefSLAM’s matched-map-point fraction was also smaller in these more extreme conditions.Source: 1) Overall quality experiment · p. 10

2) Scale drift analysis:

  • Scale drift grows as deformation becomes more challenging, but map re-observations can reduce it; the alignment between successive scaled templates is its main source.Drift was close to null on mandala0.Source: 2) Scale drift analysis · p. 10, Figure 11 · p. 10

3) Sensitivity Analysis:

  • Across λs values 1600–100000, λb values 100–1000, and λr values 0–0.1, final mandala3 error remained 20–40 m, but extreme regularizer weights caused failures.Lower λs and λb left the system unconstrained; higher values imposed rigidity, and the rigid fixed-template case failed under strong deformation.Source: 3) Sensitivity Analysis · p. 10, Figure 12 · p. 10
  • The reference regularizer reduced scale drift in Hamlyn SeqHeart from 36% to 2% at λr = 0.02.The sequence repeatedly observes the same zone and has few boundary point constraints.Source: 3) Sensitivity Analysis · p. 10

4) Deformation mapping normal estimation accuracy:

B. Hamlyn Dataset

The Hamlyn evaluation covers six intracorporeal sequences, including ex-vivo phantom and in-vivo laparoscopic scenes. DefSLAM processes several exploratory sequences and adapts to deformation, though the paper identifies medical-imaging challenges and broader mapping capabilities as future work.

B. Hamlyn Dataset

  • The six-sequence evaluation combines two ex-vivo phantom-heart sequences registered to CT ground truth with four in-vivo laparoscopic sequences.The in-vivo scenes include an almost-rigid abdomen, low-texture exploration, a beating heart, and tool-induced organ deformation.Source: B. Hamlyn Dataset · p. 11
  • On ex-vivo phantom heart sequences, DefSLAM reported mean accuracies of 3 mm and 4 mm, versus MISSLAM’s reported 0.28 mm and 0.35 mm.DefSLAM ran on CPU with monocular input, whereas MISSLAM used GPU and stereo input; runtimes per frame were similar.Source: B. Hamlyn Dataset · p. 11
  • DefSLAM processed SeqAbdomen and SeqExploration entirely, with mean 3D RMS errors of 17 mm and 10 mm, while ORBSLAM could not complete them.Re-observing previously mapped regions reduced scale drift as the camera returned to the same zone.Source: B. Hamlyn Dataset · p. 11
  • SeqHeart’s nearly static camera still allowed monocular initialization with short parallax, and DefSLAM achieved approximately 3 mm 3D RMS error.ORBSLAM initialized with ground truth processed the entire sequence with 5 mm error.Source: B. Hamlyn Dataset · pp. 11–12
  • DefSLAM reconstructed SeqOrgans with 8 mm accuracy through tool clutter and organ deformation, correcting scale drift.The tool covered almost the entire image at one point, and deformation at the sequence end did not increase the reported error.Source: B. Hamlyn Dataset · p. 12

VIII. CONCLUSIONS AND FUTURE WORK

  • DefSLAM’s two parallel threads use SfT for camera pose and scene deformation, and periodically update the shape-at-rest template through mapping.Both SfT and NRSfM use perspective camera models, supporting close-up scenes with prevalent perspective effects.Source: §VIII · p. 12
  • The authors report handling real exploratory deforming-scene trajectories and accurate scene estimates, but direct comparison with other systems was not possible.Their comparison used rigid monocular ORBSLAM retuned for non-rigid scenes.Source: §VIII · p. 12
  • Medical-image adaptation remains future work for uneven illumination, poor texture, non-isometric deformation, and ultra-close endoluminal views; extended mapping features are also future work.The latter include multiple maps, relocalization, loop closure, and long-term place recognition for extended periods or multiple deforming bodies.Source: §VIII · pp. 12–13

A. Derivatives of Regularizers

The supplied appendix passages identify stretching, bending, and reference regularizers and state that their Jacobians have no singularities.

A. Derivatives of Regularizers

  • The appendix presents Jacobian terms for the regularizers to show that they have no singularities.The supplied passages identify stretching, bending, and reference terms, but do not provide readable derivative expressions.Source: A. Derivatives of Regularizers · pp. 13–14
  • The bending term uses a node’s mean curvature, estimated from that node and its neighboring nodes.The appendix labels the regularizer derivatives with respect to the relevant map nodes.Source: A. Derivatives of Regularizers · p. 14
  • For a planar surface, mean curvature and its derivative tend toward zero.Source: A. Derivatives of Regularizers · p. 14

Related papers

Loading 1908.08918v2…