PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Dense Mapping for Large-Scale Dynamic Environments

Ioan Andrei Bârsan, Peidong Liu, Marc Pollefeys, Andreas Geiger

arXiv:1905.02781v1cs.CVcs.RO

TL;DR

Large-scale mapping in dynamic environments must contend with objects that can corrupt static maps, while discarding them loses potentially useful information. This paper uses stereo, semantics, and sparse scene flow to reconstruct background and objects separately, with pruning to reduce map memory use. The authors report improved static-map accuracy over a robust visual-odometry baseline and lower memory use with light regularization.

  • Problem

    Dynamic objects can corrupt maps built under static-environment assumptions, while ignoring them discards information that may support tasks such as path planning.

    Source: §I · p. 1

  • Method

    Stereo depth, instance-aware semantics, and sparse scene flow support separate reconstruction of the static background and rigid moving or potentially moving objects, with voxel-map pruning.

    Source: §I · p. 2, E. Static map and individual object… · p. 4, F. Fixed-lag map pruning · p. 5

  • Results

    The system outperforms robust VO InfiniTAM in accuracy, while its completeness is slightly lower because object removal can also remove small background areas.

    Source: B. Quantitative accuracy evaluation · p. 6

  • Takeaways & Limitations

    The authors report high-quality dense models of static backgrounds and dynamic objects for robotic tasks such as path planning, and light regularization can reduce memory footprint by more than 30% with small costs in discarded useful information.

    Source: §V · p. 7, C. Effectiveness of fixed-lag map… · p. 7

    The reconstruction focus is rigid objects such as cars and trucks; articulated objects such as pedestrians are more computationally demanding to reconstruct from multiple observations.

    Source: §I · p. 2

Abstract

from arXiv · show

We present a stereo-based dense mapping algorithm for large-scale dynamic urban environments. In contrast to other existing methods, we simultaneously reconstruct the static background, the moving objects, and the potentially moving but currently stationary objects separately, which is desirable for high-level mobile robotic tasks such as path planning in crowded environments. We use both instance-aware semantic segmentation and sparse scene flow to classify objects as either background, moving, or potentially moving, thereby ensuring that the system is able to model objects with the potential to transition from static to dynamic, such as parked cars. Given camera poses estimated from visual odometry, both the background and the (potentially) moving objects are reconstructed separately by fusing the depth maps computed from the stereo input. In addition to visual odometry, sparse scene flow is also used to estimate the 3D motions of the detected moving objects, in order to reconstruct them accurately. A map pruning technique is further developed to improve reconstruction accuracy and reduce memory consumption, leading to increased scalability. We evaluate our system thoroughly on the well-known KITTI dataset. Our system is capable of running on a PC at approximately 2.5Hz, with the primary bottleneck being the instance-aware semantic segmentation, which is a limitation we hope to address in future work. The source code is available from the project website (http://andreibarsan.github.io/dynslam).

I. INTRODUCTION

Real-time large-scale mapping in dynamic outdoor environments remains an open problem: dynamic objects can corrupt static maps, while ignoring them discards information useful for robotic tasks. The paper separately reconstructs the static background and rigid moving or potentially moving objects, and proposes map pruning to improve accuracy and reduce memory use.

I. INTRODUCTION

  • Dynamic objects can corrupt maps built under static-environment assumptions, while systems that ignore them discard information useful for path planning and traffic understanding.Large-scale, complex settings such as crowded cities make this a practical challenge.Source: §I · pp. 1–2
  • The system classifies objects as background, moving, or potentially moving, then separately reconstructs the static background and rigid moving or potentially moving objects.It uses stereo, semantic cues, sparse scene flow, and visual-odometry camera poses; articulated objects are outside its reconstruction focus because they are more computationally demanding.Source: §I · p. 2
  • Map pruning is proposed to improve reconstruction accuracy, reduce memory consumption, and increase scalability.The paper uses a volumetric representation based on voxel block hashing for large-scale environments.Source: §I · p. 2

II. RELATED WORK

Prior dynamic-mapping systems vary in scale, speed, and whether they reconstruct moving objects separately. The cited approaches also use different sensors and representations, with several constrained by runtime or reconstruction scope.

II. RELATED WORK

  • Reddy et al. separate static background and moving objects, but noisy depth yields inconsistent object reconstructions, and expensive optimization prevents near-real-time, large-scale use.Their stereo-based SLAM uses joint CRF-based semantic-motion segmentation and bundle adjustment.Source: §II · p. 2
  • Vineet et al. [26] run near real time at large scale, but reconstruct moving objects together with the background and use object-class-dependent depth-fusion weights to limit map corruption.Areas corresponding to moving objects receive larger weights and are updated more aggressively.Source: §II · p. 2
  • Kochanov et al. encode scene flow in a semantic voxel grid, but their pipeline is not real time and detects and tracks dynamic objects without reconstructing them.The pipeline uses dense scene flow from Vogel et al., which can take up to 300 seconds per frame.Source: §II · p. 2
  • Jiang et al. reconstruct static maps and dynamic objects using LIDAR and cameras, but require roughly 12 minutes to process a 70-frame sequence; follow-up work improves robustness to partial occlusions.Source: §II · p. 2
  • Co-Fusion reconstructs static backgrounds and dynamic objects in real time with surfels, but its Virtual KITTI [3] reconstructions are limited in quality and scale compared with its indoor results.The authors also do not address noise reduction for stereo-based, large-scale outdoor reconstruction.Source: §II · p. 2

III. DYNAMIC RECONSTRUCTION

At each frame, the system processes stereo and RGB input to estimate depth, sparse scene flow, and instance-aware semantics. It uses these inputs to estimate camera and object motion, separate background and object data, update their reconstructions, and prune spurious voxels.

III. DYNAMIC RECONSTRUCTION

  • Each frame begins with dense depth and sparse scene-flow computation from stereo, alongside instance-aware semantic segmentation of the left-camera RGB image.Source: §III · p. 2
  • Sparse scene flow supplies visual odometry; semantic segmentation and motion relative to camera egomotion classify detections as static, dynamic, or uncertain.Source: §III · p. 3
  • The input is split into a background virtual frame and one for each potentially dynamic object, then rigid moving or potentially moving objects and the static map are reconstructed separately.Source: §III · p. 3
  • Voxel garbage collection removes voxels allocated spuriously because of depth-map artifacts.Source: §III · p. 3

A. Dense depth map computation

The system derives dense depth from stereo and uses instance-aware detection with inter-frame association to track rigid objects for reconstruction. Depth methods differ in sharpness and completeness, while non-rigid detections are excluded from 3D reconstruction.

A. Dense depth map computation

  • ELAS and DispNet provide alternative stereo depth maps: ELAS uses geometry-based matching, while DispNet directly regresses disparity with a neural network.ELAS maps tend to be sharper but have gaps on reflective and transparent areas; DispNet maps are denser and more robust to non-Lambertian surfaces, but less sharp, especially at object boundaries.Source: A. Dense depth map computation · p. 3

B. Object segmentation and 2D tracking

  • Multi-task Network Cascades detects and classifies object instances; greedy Intersection-over-Union association links detections across frames for rigid-object reconstruction.Potentially dynamic non-rigid objects such as pedestrians are removed from the frame to protect background reconstruction, but are not passed to 3D motion estimation or reconstruction.Source: B. Object segmentation and 2D tracking · p. 3

C. Sparse scene flow and visual odometry

Sparse scene flow supports both camera egomotion estimation and object-motion classification, while separate volumetric models preserve the static background and reconstruct moving or potentially moving objects individually.

C. Sparse scene flow and visual odometry

  • Four-way stereo feature matches yield sparse 3D scene flow used to estimate visual odometry and tracked-object motion.RANSAC estimates camera motion by minimizing reprojection error of triangulated points.Source: C. Sparse scene flow and visual odometry · pp. 3–4

D. 3D object tracking

  • Masked scene flow estimates each object's motion by treating the object as static and inverting the resulting virtual-camera motion.Comparing the estimated object motion with camera egomotion distinguishes static from dynamic objects.Source: D. 3D object tracking · p. 4

E. Static map and individual object reconstructions

  • The system fuses camera-motion-aligned background data separately from individual volumetric models for moving and potentially moving objects.Object RGB and depth are extracted using segmentation masks, and object motion guides their volumetric fusion.Source: E. Static map and individual object… · p. 4

F. Fixed-lag map pruning

Fixed-lag pruning addresses stereo-depth noise that creates streak artifacts and consumes memory, extending voxel-block garbage collection with finer-grained cleanup and deferred processing.

F. Fixed-lag map pruning

  • Stereo-depth noise creates streak-like artifacts that lower map quality and increase memory use by occupying large volumes.The streaks trail away from the viewing direction; small-scale RGB-D noise is less significant than stereo depth error.Source: F. Fixed-lag map pruning · p. 4, Figure 3 · p. 5
  • Prior garbage collection removes voxel blocks with distant TSDF values or no measurement weight, reducing artifacts and freeing memory.The criteria are evaluated for each 8×8×8 voxel block.Source: F. Fixed-lag map pruning · p. 4
  • Per-voxel pruning can delete noisy voxels even when their containing voxel block must remain allocated.InfiniTAM's voxel-block hashing prevents individual voxels from being deallocated, so memory is freed only when whole blocks are deleted.Source: F. Fixed-lag map pruning · p. 4
  • Fixed-lag collection processes blocks visible kminAge frames earlier, and cleaned areas need no reprocessing until revisited.The method uses stored visible-block history and runs in lockstep with fusion.Source: F. Fixed-lag map pruning · p. 4
  • 10-fold lower computational costs result from pruning visible-block lists rather than the entire static map at every step.Visible lists typically contain 8,000–12,000 blocks; on a GTX 1080, pruning adds less than 1 ms per frame.Source: F. Fixed-lag map pruning · pp. 4–5

IV. EXPERIMENTS

The experiments assess reconstruction accuracy on KITTI and test the fixed-lag map regularizer, while leaving localization accuracy unevaluated.

IV. EXPERIMENTS

  • The evaluation uses the KITTI Vision Benchmark Suite for qualitative and quantitative assessment of 3D reconstruction accuracy.Source: §IV · p. 5
  • The experiments also evaluate the effectiveness of the fixed-lag map regularizer.Source: §IV · p. 5
  • Localization accuracy is not evaluated because mapping is the paper's primary focus.Source: §IV · p. 5

A. Qualitative accuracy evaluation

On a KITTI sequence, dynamic-aware fusion avoids the environment-map corruption caused by ignoring individually moving vehicles.

A. Qualitative accuracy evaluation

B. Quantitative accuracy evaluation

The evaluation uses KITTI LIDAR ground truth to assess frame-averaged accuracy and completeness. Results show depth-source trade-offs on odometry sequences and compare dynamic fusion with a robust-VO baseline on tracking sequences.

B. Quantitative accuracy evaluation

  • Accuracy averages scores across all frames in a sequence; per-frame input and active-reconstruction disparities are compared with LIDAR-derived ground truth, with errors defined by deltas greater than 3 px and 5% of ground-truth disparity.Completeness is the percentage of ground-truth points in the left camera’s field of view with corresponding values in either the input or fused disparity maps.Source: B. Quantitative accuracy evaluation · p. 5
  • Across 11 KITTI odometry training sequences, ELAS yields more accurate static maps but less accurate object reconstructions than DispNet; fusion improves ELAS accuracy, while DispNet remains more complete.Volumetric fusion over multiple frames reduces the completeness gap between the depth sources.Source: B. Quantitative accuracy evaluation · pp. 5–6
  • ELAS completeness varies substantially in challenging lighting, such as after tunnel exits, but multi-frame fusion significantly reduces this effect.Source: B. Quantitative accuracy evaluation · p. 6
  • Across the first 10 KITTI tracking sequences, dynamic fusion is more accurate than robust-VO InfiniTAM, while that baseline is slightly more complete; imperfect object removal can lower fusion completeness.The removed background fragments are identified as candidates for voxel garbage collection.Source: Table I · p. 6, B. Quantitative accuracy evaluation · p. 6

C. Effectiveness of fixed-lag map regularizer

The fixed-lag regularizer reduces map memory use while retaining most useful information, but evaluation under garbage collection covers static reconstructions only. Results compare ELAS and DispNet maps on the first 1,000 frames of KITTI sequence 9.

C. Effectiveness of fixed-lag map regularizer

  • The evaluation delays both garbage collection and accuracy measurement to account for its fixed kminAge-frame processing lag.At time t, garbage collection processes blocks visible at t − kminAge; accuracy is evaluated using the depth map and pose from t − kminAge − τ, with τ = kminAge.Source: C. Effectiveness of fixed-lag map… · pp. 6–7
  • Garbage-collection accuracy is evaluated only for static reconstructions because per-frame LIDAR ground truth cannot evaluate past positions of dynamic objects.The delayed scheme evaluates objects kminAge + τ frames in the past, although the system knows their positions only while observing them.Source: C. Effectiveness of fixed-lag map… · p. 7
  • More than 30% less memory with Δweight = 1 or 2, at only small costs in discarded useful information.The vanilla InfiniTAM system corresponds to Δweight = 0.Source: C. Effectiveness of fixed-lag map… · p. 7
  • ELAS yields more accurate but less complete reconstructions than DispNet, and that trend persists as Δweight increases.DispNet maps are denser; the evaluation focuses on the static map over the first 1,000 frames of KITTI odometry sequence 9.Source: C. Effectiveness of fixed-lag map… · p. 7
  • The system runs at approximately 2.5 Hz on a PC equipped with an NVIDIA GeForce GTX 1080 GPU.Source: C. Effectiveness of fixed-lag map… · p. 7

V. CONCLUSIONS

The paper presents a robust dense mapping algorithm for large-scale dynamic environments, separately reconstructing static backgrounds and dynamic objects. Its map-pruning technique improves accuracy while reducing memory consumption, with dense models intended to support high-level robotic tasks such as path planning.

V. CONCLUSIONS

  • The algorithm uses semantic segmentation and sparse scene flow to separate static background from dynamic and potentially dynamic objects, which it reconstructs separately.Source: §V · p. 7
  • Voxel garbage collection improves map accuracy while substantially reducing memory consumption.Source: §V · p. 7
  • The resulting dense models of static backgrounds and dynamic objects are important for high-level robotic decision-making tasks such as path planning.Source: §V · p. 7
Loading 1905.02781v1…