PaperlayerView exact revision on arXiv

Source-linked AI summary

SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

John McCormac, Ankur Handa, Andrew Davison, Stefan Leutenegger

arXiv:1609.05130v2cs.CV

TL;DR

Dense maps need semantic information in addition to geometry for richer robotic and user interaction. The paper combines CNN semantic predictions with ElasticFusion correspondences and Bayesian fusion, reporting improved segmentation and real-time interactive operation. Benefits are larger with wider viewpoint variation, while the CNN’s depth integration was constrained by GPU memory limitations.

  • Problem

    Dense geometric maps lack semantic information needed for tasks that require knowing both what an object is and where it is located.

  • Method

    SemanticFusion combines ElasticFusion’s dense SLAM correspondences, CNN per-frame semantic predictions, and Bayesian updates to build a semantic 3D map.

  • Results

    Fusing semantic predictions improved single-frame segmentation, with relative gains of 2.3% for RGBD-CNN on NYUv2 and 4.7% on the office reconstruction dataset.

  • Takeaways & Limitations

    The resulting semantic 3D map supports real-time interactive room scanning and can improve corresponding 2D segmentation accuracy.

  • Takeaways & Limitations

    The chosen depth-integration approach was constrained by GPU memory limitations, making alternative depth architectures infeasible in the system.

Abstract

from arXiv · show

Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need extend beyond geometry and appearence - they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state of the art dense Simultaneous Localisation and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondence between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of approximately 25Hz.

I. INTRODUCTION

SemanticFusion combines dense SLAM geometry with CNN predictions to create a semantically annotated 3D map, while fusing multiple viewpoints can improve 2D segmentation accuracy. The system is designed for real-time interactive use and benefits more from wider viewpoint variation.

  • Motivation: Semantic maps extend dense geometry with object and location information for robotic fetching and human queries.The paper gives examples including fetching a coffee mug and querying the number or spacing of chairs.
  • Approach: SemanticFusion combines ElasticFusion’s geometric map with CNN-based semantic segmentation.SLAM supplies the geometric information, while CNNs provide semantic predictions.
  • System capability: 25Hz real-time operation supports interactive use, although offline processing for further geometric and semantic refinement was not explored.The pipeline’s component efficiency enables real-time capability; more expensive offline refinement remains future work.
  • Evaluation: NYUv2 evaluation shows that using unlabelled raw video improves segmentation over single-frame baseline approaches.The result suggests SLAM can boost 2D semantic segmentation when linked to the mapping system.
  • Evaluation: A reconstruction-focused dataset with wider viewpoint variation shows a larger fusion benefit than NYUv2.NYUv2 often lacks substantial viewpoint variation for a scene, whereas the office reconstruction provides more thorough scanning.

II. RELATED WORK

Related approaches pursue dense semantic 3D mapping, but differ in segmentation models, mapping scope, and online operation. SemanticFusion combines CNN-based predictions with ElasticFusion and Bayesian fusion for dense, real-time-capable mapping.

  • Dense semantic mapping: Stückler et al. and Hermans et al. fuse per-pixel predictions from different viewpoints into dense semantically annotated indoor 3D maps.Their systems use Random Decision Forests and a classic Bayesian framework.
  • Object-level mapping: SLAM++ maps semantically defined objects but is limited to objects in a predefined database and does not densely label entire scenes.SemanticFusion also targets walls, floors, doors, and windows, while SLAM++ uses hand-crafted matching features.
  • Scope of prior work: Most other indoor approaches focus on offline batch mapping or single-frame 2D segmentation rather than online dense semantic 3D mapping.Valentin et al. produce a globally consistent map, but infer over the whole mesh once instead of incrementally fusing predictions online.
  • CNN background: CNNs provide the state-of-the-art accuracy and efficient test-time performance motivating SemanticFusion’s semantic labelling pipeline.The paper builds on CNN work across RGB, depth, and normal modalities.
  • SemanticFusion: SemanticFusion consists of ElasticFusion, a CNN, and Bayesian updates that connect frame correspondences to per-surfel class probabilities.ElasticFusion supplies correspondences and a globally consistent surfel map, while the CNN returns per-pixel class probabilities.

A. SLAM Mapping

The SLAM component uses ElasticFusion to track camera motion, maintain a surfel map through loop closures, and support semantic mapping. The CNN operates on image data, including a depth-augmented RGBD variant.

  • SLAM Mapping: ElasticFusion tracks each arriving camera frame using combined ICP and RGB alignment, producing a camera pose for map updates.New surfels are added using the pose, while existing surfel geometry and appearance are refined.
  • SLAM Mapping: Loop-closure checks run in parallel, and the map is immediately optimised when a loop closure is detected.This supports map consistency during scanning trajectories that revisit previously observed areas.
  • SLAM Mapping: ElasticFusion’s deformation graph and surfel representation carry probability distributions through loop closures without destroying the underlying surfels.The same representation also fuses new depth readings into surfel depth and normal information.
  • CNN Mapping: The CNN uses a deconvolutional semantic-segmentation architecture that outputs dense pixel-wise semantic probability maps.The architecture is based on VGG and adds max-unpooling and deconvolutional layers.
  • CNN Mapping: The RGB CNN is modified to accept depth as a fourth input channel despite only 795 labelled NYUv2 training images.Depth filters are initialized from the average intensity of pretrained RGB filters.
  • CNN Mapping: Input and output resolutions are rescaled between the CNN’s native resolution and full 640×480 images for surfel updates.The paper uses bilinear interpolation for RGB, nearest-neighbour interpolation for depth, and nearest-neighbour upsampling for predictions.

C. Incremental Semantic Label Fusion

Semantic label fusion maintains class-probability distributions on surfels and updates them with CNN predictions projected through SLAM correspondences. This enables Bayesian integration of evidence from multiple frames and viewpoints.

  • Probability representation: Each surfel stores a discrete probability distribution over semantic classes and starts with a uniform distribution.Uniform initialization represents the absence of prior evidence about a newly generated surfel’s class.
  • Per-frame predictions: The CNN processes camera images and interprets its output as independent per-pixel class-probability distributions.Depending on the architecture, the input may include RGB, depth, or normals.
  • Frame-to-map association: Tracked camera poses project visible map surfels into image pixels so their class distributions can receive corresponding CNN evidence.The projection associates each surfel’s 3D location with pixel coordinates in the current frame.
  • Bayesian fusion: Recursive Bayesian updates combine the current evidence with each surfel’s previous label probabilities and normalize the result.The update is applied across label probabilities for the visible surfels.
  • Bayesian fusion: SLAM correspondences enable accurate association of label hypotheses across images and their Bayesian combination into a coherent map.The method’s correspondence mechanism supports fusing semantic predictions from multiple viewpoints.

D. Map Regularisation

SemanticFusion regularises surfel predictions with a fully connected 3D CRF whose unary terms come from fused CNN probabilities and whose pairwise terms use map geometry.

  • Map Regularisation: A fully connected CRF treats each surfel as a graph node and incrementally updates its probability distribution using mean-field inference and message passing.The CRF approximately minimises the Gibbs energy of the surfel labelling.
  • Map Regularisation: The energy combines a unary data term from each surfel’s fused CNN probability distribution with a pairwise smoothness term parameterised by map geometry.The unary term penalises labels through the negative logarithm of their probabilities.
  • Map Regularisation: Pairwise smoothness is a linear combination of Gaussian edge-potential kernels with Potts-model label compatibility.The kernels operate on surfel feature vectors.
  • Map Regularisation: The model uses bilateral appearance smoothing for similarly positioned and appearing surfels, plus spatial smoothing for regions with similar surface normals.These potentials combine appearance, position, and normal information to regularise predictions.
  • Map Regularisation: The Gaussian scales were fixed at 0.05m spatially, 20 in RGB colour, and 0.1 radians angularly, with weights w1=10 and w2=3 across experiments.The parameters were not tuned for a particular dataset and used defaults from the public implementation.

A. Network Training

The network was adapted for RGB-D semantic segmentation and fine-tuned on NYUv2, while the reconstruction dataset used 3D surfel annotation projected into test-frame labels.

  • Network Training: The CNN was initialized from PASCAL VOC segmentation weights and fine-tuned on NYUv2 for 13 semantic classes.The fourth depth channel was initialized using the procedure described for depth input.
  • Network Training: Training used stochastic gradient descent with learning rate 0.01, momentum 0.9, weight decay 5 × 10^-4, and 20k iterations over two days.The learning rate was reduced to 1 × 10^-3 after 10k iterations with mini-batches of 64.
  • Reconstruction Dataset: The office dataset used a relatively complete, locally and globally loopy RGB-D trajectory intended to represent active-agent scene inspection.Its trajectory had more loopiness than the typical single back-and-forth sweep in NYUv2.
  • Reconstruction Dataset: A 3D annotation tool labelled reconstructed surfels with NYUv2 classes and automatically generated 2D labels by projecting them into input frames.Only 9 of the 13 considered NYUv2 classes were present in the office data.
  • Reconstruction Dataset: Every 100th sequence frame served as a test sample, producing 49 test frames for validation against annotated ground truth.

C. CNN and CRF Update Frequency Experiments

Processing fewer frames substantially improves runtime with only a modest accuracy reduction, while frequent CRF updates can harm performance.

  • 52.5% accuracy occurs when every frame is processed, but runtime falls to 8.2Hz.
  • Processing every 10th frame yields 49-51% accuracy at 25.3Hz and is used in subsequent evaluations.
  • Frequent CRF updates can significantly reduce accuracy by drowning out CNN predictions.
  • Updating the CRF every 500 frames provides a slight improvement and becomes the default update rate.
  • For the Eigen network, SemanticFusion improves class average accuracy from 57.1% to 60.0%, a 2.9% gain.
  • CRF effects are small: +0.5% for Eigen’s network and -0.2% for the RGBD-CNN.

E. NYU Dataset

NYUv2 evaluation shows that fusing semantic predictions improves over single-frame segmentation, although gains are smaller than on the office reconstruction dataset.

  • NYUv2 evaluation excludes sequences below 2Hz, leaving 140 of 206 test sequences and 360 of 654 labelled test images.
  • 2.3% relative gains for the RGBD-CNN are approximately half the 4.7% improvement observed on the office reconstruction dataset.
  • Rotational NYUv2 scanning provides fewer useful viewpoints for fusing independent predictions than thorough reconstruction trajectories.
  • SemanticFusion improves Eigen et al.’s class average accuracy from 59.9% to 63.2%, a +3.3% increase.
  • CRF improvements are small but positive for both CNNs: +0.4% for Eigen’s CNN and +0.3% for the RGBD-CNN.

F. Run-time Performance

The system supports real-time operation when CNN predictions are computed every 10 frames, while CRF inference remains expensive for online use.

  • ElasticFusion requires 29.3ms per frame, with an additional 1.0ms for updating the stored surfel probability table.
  • The evaluation uses reconstruction outputs alongside baseline single-frame fallback predictions when no reconstruction is present.
  • The standard scheme processes every 10 frames at an average frame-rate of 25.3Hz.
  • The CRF implementation takes 20.3s for 10 iterations because inference runs single-threaded on the CPU after GPU data transfer.
  • For online use, the CRF can be disabled or applied once after a sequence finishes.

V. CONCLUSIONS

SemanticFusion fuses CNN predictions through loop-closure-capable SLAM to produce a coherent semantic 3D map and improve corresponding 2D segmentation. Its effectiveness depends on viewpoint variation, while depth integration remains constrained by GPU memory limitations.

  • SemanticFusion uses SLAM-provided pixel-wise correspondences to fuse per-frame segmentations into a coherent 3D semantic map.ElasticFusion supports this fusion through a loop-closure-capable surfel map.
  • The semantic map significantly improves corresponding 2D segmentation accuracy.The conclusion reports this improvement as an outcome of incorporating the map.
  • The approach uses an additional depth channel to improve a pretrained RGB network, while alternative depth-integration designs were infeasible because of GPU memory limitations.The authors identify CNN compression as a possible way to enable richer modalities and real-time segmentation on low-memory devices.
  • Viewpoint variation is important: a reconstruction-focused dataset showed much larger labelling improvement via fusion than the NYU dataset.The authors connect this difference to the more varied trajectories in the reconstruction-focused dataset.
  • SemanticFusion is positioned as a basis for future systems that combine semantic recognition with reconstruction and SLAM.The authors propose class-specific smoothing and replacing surfel elements with 3D object models when recognition confidence is sufficient.
Loading 1609.05130v2…