PaperlayerView exact revision on arXiv

Source-linked AI summary

gradSLAM: Automagically differentiable SLAM

Krishna Murthy Jatavallabhula, Soroush Saryazdi, Ganesh Iyer, Liam Paull

arXiv:1910.10672v3cs.ROcs.CVcs.LG

TL;DR

Dense SLAM contains non-differentiable components, limiting the use of task-based error signals to learn representations through the full system. gradSLAM expresses SLAM as differentiable computational graphs and proposes differentiable optimization, mapping, fusion, and raycasting. Its approaches maintain similar performance to non-differentiable counterparts while allowing gradients to flow through the system.

  • Problem

    Several components of dense SLAM are non-differentiable, limiting the ability to propagate task-based error signals through the system to learn representations.

    Source: §1 · p. 1

  • Method

    gradSLAM poses SLAM as a differentiable computational graph, with differentiable trust-region optimization, surface measurement and fusion, and raycasting.

    Source: §1 · p. 2, §3.6 · p. 5, §4 · p. 6

  • Results

    The differentiable approaches maintain similar performance to their non-differentiable counterparts while allowing gradients to flow through SLAM systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    gradSLAM provides explicit gradients with respect to input images and depth maps, and the authors identify downstream-task optimization and self-supervised learning as potential applications.

    Source: §6 · p. 8

    The performance impact of choosing among gating functions is left for future work.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

Blending representation learning approaches with simultaneous localization and mapping (SLAM) systems is an open question, because of their highly modular and complex nature. Functionally, SLAM is an operation that transforms raw sensor inputs into a distribution over the state(s) of the robot and the environment. If this transformation (SLAM) were expressible as a differentiable function, we could leverage task-based error signals to learn representations that optimize task performance. However, several components of a typical dense SLAM system are non-differentiable. In this work, we propose gradSLAM, a methodology for posing SLAM systems as differentiable computational graphs, which unifies gradient-based learning and SLAM. We propose differentiable trust-region optimizers, surface measurement and fusion schemes, and raycasting, without sacrificing accuracy. This amalgamation of dense SLAM with computational graphs enables us to backprop all the way from 3D maps to 2D pixels, opening up new possibilities in gradient-based learning for SLAM. TL;DR: We leverage the power of automatic differentiation frameworks to make dense SLAM differentiable.

1. Introduction

Dense SLAM lacks consensus on map representation, and its non-differentiable components limit gradient-based learning. gradSLAM recasts SLAM as a differentiable computational graph, with similar performance to non-differentiable counterparts and gradients flowing through the system.

1. Introduction

  • A differentiable SLAM transformation could propagate task-performance error signals through the system to raw sensor observations, supporting task-driven representation learning.Map representation affects processing blocks and downstream tasks that depend on SLAM outputs.Source: §1 · p. 1
  • Dense RGB-D mapping lacks consensus on representation, while existing methods do not answer how much a specific pixel measurement contributes to the resulting 3D map.The proposed mapping function relates sensor measurements to a map, and its gradient indicates how perturbing a measurement changes that map.Source: §1 · pp. 1–2
  • gradSLAM makes SLAM a differentiable computational graph by replacing non-differentiable functions with smooth mappings, including optimization, mapping, raycasting, and global measurement fusion.The framework is presented as extensible to dense SLAM systems.Source: §1 · p. 2
  • The differentiable approaches maintain similar performance to their non-differentiable counterparts while allowing gradients to flow through SLAM systems.The examples include implicit-surface mapping, surfel-based mapping, and ICP mapping.Source: §1 · p. 2
  • The authors provide gradSLAM as an open-source PyTorch framework.The passage directs readers to the project page and code.Source: §1 · p. 2

2. Related Work

Prior work addresses learned SLAM components and differentiable subproblems, but the paper identifies no single framework modeling an entire SLAM pipeline as a differentiable graph. It reviews learning-based SLAM, differentiable visual odometry, and differentiable optimization as related strands.

2.1. Learning-based SLAM approaches

  • Learning-based SLAM has explored scene codes, learned tracking and mapping, monocular depth prediction, object-level features, and differentiable localization, but no single framework models the entire pipeline as a differentiable graph.The passage also describes differentiable least-squares residuals for lane detection.Source: §2.1 · p. 2

2.2. Differentiable visual odometry

  • Differentiable visual odometry extends from Lucas-Kanade matching and its use in self-supervised depth and motion estimation, but global measurement fusion is not trivially differentiable.The challenge arises when fusing live-frame measurements into a global model for model-frame alignment.Source: §2.2 · p. 2

2.3. Differentiable optimization

  • Prior differentiable optimization approaches learn damping or update steps, while OptNet provides optimization layers for quadratic programs without learnable parameters.The paper describes its differentiable Levenberg-Marquardt optimizer as similar in spirit to optimizer unrolling, with gating functions added for better gradient flows.Source: §2.3 · pp. 2–3

2. Related Work

  • The authors identify the lack of a single approach modeling the entire SLAM pipeline as a differentiable model as the motivation for gradSLAM.The related-work overview notes that prior work reformulates subsets of full SLAM systems differentiably.Source: §2 · p. 2, §2.3 · p. 3

3. ∇SLAM

∇SLAM represents dense SLAM as a composition of differentiable functions, addressing non-differentiable operations in optimization, mapping, and raycasting. It adapts these components while retaining similar performance to their non-differentiable counterparts.

3.1. Preliminaries: Computational graphs

  • Dense SLAM includes non-differentiable components, including nonlinear least-squares modules, raycasting, and discretizations, which impede gradient flow.Computational graphs represent operations and variables with rules for differentiating outputs with respect to inputs, and can be composed while preserving differentiability.Source: §3.1 · p. 3

3.2. Method Overview

  • ∇SLAM makes dense SLAM computations differentiable, with a soft reparameterization of LM damping and iterate updates to address discrete damp-or-undamp decisions.Smooth gating functions use the error difference between current and lookahead iterates to update the iterate and damping coefficient.Source: §3.2 · p. 3, §3.3 · p. 4

3.4. Differentiable mapping

  • Mapping operations such as clipping, indexing, thresholding, and discrete decisions can have undefined gradients almost everywhere; ∇SLAM smooths them through pixel-neighborhood measurements and soft associations.Each valid pixel's surface measurement depends on its active or inactive neighbors, and measurements can be associated with multiple map elements rather than one.Source: §3.4 · pp. 4–5

3.5. Differentiable map fusion

  • A logistic falloff eases gradient flow through fusion truncation points, while fusion of redundant observations helps control map size and improve reconstruction quality.The differentiable mapping strategy otherwise increases map elements in proportion to exploration time.Source: §3.5 · p. 5

3.6. Differentiable ray backprojection

  • Raycasting aggregates voxel values along a ray with Gaussian falloff around the pixel's measured depth, then uses finite differences from neighboring rays to estimate pixel-neighborhood derivatives.The ray value is computed from voxels pierced by the ray; neighboring rays originate from pixels left, right, above, and below the target pixel.Source: §3.6 · p. 5, Figure 6 · p. 5

4. Case Studies: KinectFusion, PointFusion, and ICP-SLAM

The section demonstrates gradSLAM by composing its differentiable computation graphs into three practical SLAM solutions: KinectFusion, PointFusion, and ICP-SLAM.

4. Case Studies: KinectFusion, PointFusion, and ICP-SLAM

  • The framework is composed into differentiable KinectFusion, PointFusion, and ICP-SLAM solutions.Source: §4 · p. 6

4.1. KinectFusion

  • KinectFusion alternates tracking, which raycasts the TSDF volume for point-to-plane ICP, and mapping, which fuses current-frame measurements into that volume.A generalized logistic function smooths truncation to improve gradients at the boundary; TSDF fusion uses weighted averaging.Source: §4.1 · p. 6

4.2. PointFusion

  • PointFusion incrementally fuses surface measurements into a global surfel map using the paper’s differentiable mapping formulation and fusion rules from.Surfel maps are described as using less memory than volumetric maps.Source: §4.2 · p. 6

4.3. ICP-SLAM

  • ICP-SLAM aligns each incoming frame to the accumulated pointcloud, unlike ICP-Odometry, which aligns consecutive frame pairs.Source: §4.3 · p. 6

5. Experiments and results

The experiments compare differentiable SLAM components with their non-differentiable counterparts and examine gradients through reconstruction. Results show similar tracking and optimization performance, while gradients support pixel-level analysis and completion tasks.

5.1. Differentiable optimization

  • ∇LM performs near-identically to LM across nonlinear curve-fitting problems and optimization stages.The test suite compares nonlinear least-squares solvers, and Table 1 describes results across functions and stages.Source: §5.1 · pp. 6–7, Table 1 · p. 7

5.2. Comparitive analysis of case studies

  • Differentiable ICP-Odometry, ICP-SLAM, and PointFusion show no virtual change in trajectory tracking performance versus their non-differentiable versions.The comparison is reported over split subsets of the living_room_traj0 sequence.Source: §5.2 · p. 7, Table 2 · p. 7
  • 21.301 for differentiable KinectFusion versus 18.625 for Kintinuous in surface reconstruction quality, though this comparison is misleading.Kintinuous retains only high-confidence extracted-mesh points, while differentiable KinectFusion has noisy artifacts attributed to smooth truncation functions.Source: §5.2 · p. 7

5.3. Qualitative results

  • Qualitative runs cover TUM RGB-D, ScanNet, and an in-house RealSense sequence; the systems run fully on GPU and compute gradients for intermediate variables.Examples include camera poses, pixel intensities and depths, optimization parameters, and camera intrinsics.Source: §5.3 · p. 7

5.4. Analysis of Gradients

  • Backpropagated volumetric reconstruction error identifies each pixel’s contribution, including the contribution of an introduced 40 × 40 occluder.A separate example reports near-identical reconstructions to non-differentiable counterparts while gradients flow from a 3D map element to its pixel-space measurements.Source: §5.4 · pp. 7–8, Figure 8 · p. 8

5.5. Application: RGB and depth completion

  • Minimizing chamfer distance between noisy and true surfel maps targets recovery of noisy and occluded regions, and the paper also demonstrates RGB-D completion from uniform noise.The experiments introduce occluders and pixel noise in depth maps before reconstructing with ∇PointFusion.Source: §5.5 · p. 8

6. Conclusion

The conclusion presents ∇SLAM as a differentiable framework for localization and mapping tasks, with gradients to input images and depth maps. It reports case studies across tracking, mapping, and fusion, and identifies downstream-task optimization and self-supervised learning as possible future applications.

6. Conclusion

  • ∇SLAM provides explicit gradients with respect to input images and depth maps for a large set of localization and mapping tasks.The case studies demonstrate gradient propagation through tracking, mapping, and fusion.Source: §6 · p. 8
  • Future efforts may plug ∇SLAM into downstream tasks for joint optimization and may use its gradients in self-supervised learning applications.The conclusion describes these as future possibilities.Source: §6 · p. 8

A. ∇SLAM: Library

The library implements dense SLAM systems as differentiable computations and provides PyTorch-based operations for optimization, mapping, and related tasks. Its stated operating example is a low-resolution TSDF reconstruction running at about 10 Hz on a medium-end laptop GPU.

A. ∇SLAM: Library

  • KinectFusion, PointFusion, and ICP-SLAM are presented as classical dense SLAM systems that can be realized as differentiable computations.The differentiable modules can also be used to construct other differentiable SLAM systems.Source: A. ∇SLAM: Library · p. 12
  • The PyTorch library lists nonlinear least-squares optimization, depth-based perspective warping, TSDF volumetric fusion, PointFusion map building, and boilerplate operations among its functionality.PyTorch is described as a reverse-mode automatic differentiation library.Source: A. ∇SLAM: Library · p. 12
  • A 128 × 128 × 128 TSDF volume runs at about 10 Hz on a medium-end laptop GPU, with interfaces to ScanNet, TUM RGB-D, and ICL-NUIM.The stated performance is for low-resolution reconstructions.Source: A. ∇SLAM: Library · p. 12

B. Frequently asked questions (FAQ)

The FAQ distinguishes meaningful gradients from merely running classical dense SLAM in an autograd-compatible library. It also notes memory limits and potential applications in spatially grounded and task-oriented learning.

B. Frequently asked questions (FAQ)

  • Merely implementing dense SLAM in an autograd-compatible library may yield gradients that are zero almost everywhere, whereas gradSLAM addresses their usefulness for gradient-based learning.The FAQ says the resulting gradients must be meaningful enough to use in a gradient-based learning pipeline.Source: B. Frequently asked questions (FAQ) · p. 12
  • A differentiable KinectFusion run at 128 × 128 × 128 voxel resolution averages 6GB of GPU memory, restricting reconstructable scene size.The authors also identify adding robust differentiable filters and M-estimators as ongoing work.Source: B. Frequently asked questions (FAQ) · p. 13
  • The authors envisage spatially grounded learning, self-supervision, and task-oriented learning, with modularity potentially enabling learning only in selected submodules.Source: B. Frequently asked questions (FAQ) · p. 13

D. Contributions

The contributions section credits the authors with originating the idea, developing the library and pipelines, conducting experiments, and advising and writing the paper.

D. Contributions

  • Krishna Murthy Jatavallabhula conceived the idea, built initial prototypes, and wrote large parts of the paper.Source: D. Contributions · p. 13
  • Soroush Saryazdi led gradSLAM library development and was instrumental in its release after joining following the ICRA 2020 submission.Source: D. Contributions · p. 13
  • Ganesh Iyer ran experiments, implemented differentiable ICP and PointFusion pipelines, and helped shape the core ideas.He also led discussions on follow-up work.Source: D. Contributions · p. 13
  • Liam Paull advised the project and wrote a majority of the paper.Source: D. Contributions · p. 13
Loading 1910.10672v3…