PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D Cameras

Lingni Ma, Jörg Stückler, Christian Kerl, Daniel Cremers

arXiv:1703.08866v2cs.CV

TL;DR

The paper addresses the challenge of producing consistent semantic maps from multiple RGB-D views of a scene. It trains an RGB-D CNN with SLAM-based view warping and multi-view consistency, then fuses predictions into keyframes. Multi-view consistency training and fusion outperform single-view baselines on NYUDv2, with state-of-the-art performance for single-view segmentation and multi-view semantic fusion.

  • Problem

    Semantic segmentation systems commonly focus on single views, while robotic and reconstruction applications require consistent aggregation of multiple RGB-D views.

  • Method

    The method trains a FuseNet-based RGB-D CNN with multi-scale supervision, warps sequence frames into an annotated reference view using SLAM poses, and fuses test predictions with Bayesian fusion.

  • Results

    All multi-view consistency methods outperform single-view baselines on NYUDv2, while Bayesian fusion improves segmentation by approximately 2% across all evaluation measures.

  • Takeaways & Limitations

    Multi-view max-pooling provides the best overall gains, supporting state-of-the-art single-view segmentation and multi-view fused semantic mapping on NYUDv2.

Abstract

from arXiv · show

Visual scene understanding is an important capability that enables robots to purposefully act in their environment. In this paper, we propose a novel approach to object-class segmentation from multiple RGB-D views using deep learning. We train a deep neural network to predict object-class semantics that is consistent from several view points in a semi-supervised way. At test time, the semantics predictions of our network can be fused more consistently in semantic keyframe maps than predictions of a network trained on individual views. We base our network architecture on a recent single-view deep learning approach to RGB and depth fusion for semantic object-class segmentation and enhance it with multi-scale loss minimization. We obtain the camera trajectory using RGB-D SLAM and warp the predictions of RGB-D images into ground-truth annotated frames in order to enforce multi-view consistency during training. At test time, predictions from multiple views are fused into keyframes. We propose and analyze several methods for enforcing multi-view consistency during training and testing. We evaluate the benefit of multi-view consistency training and demonstrate that pooling of deep features and fusion over multiple views outperforms single-view baselines on the NYUDv2 benchmark for semantic segmentation. Our end-to-end trained network achieves state-of-the-art performance on the NYUDv2 dataset in single-view segmentation as well as multi-view semantic fusion.

I. INTRODUCTION

The paper addresses multi-view semantic segmentation for RGB-D scenes, where moving cameras provide multiple views that must be fused consistently. It trains a CNN with multi-view consistency constraints and evaluates feature pooling and fusion against single-view approaches.

  • Semantic SLAM aggregates multiple views into a consistent 3D geometric and semantic reconstruction, but multi-view RGB-D segmentation is less explored.
  • The approach extends an RGB-D CNN with multi-scale deep supervision and SLAM-based multi-view consistency constraints.
  • Multiple views are associated by warping network outputs into a ground-truth-annotated reference view using the estimated SLAM trajectory.
  • Multi-view max-pooling during training best supports multi-view fusion at test time.
  • The end-to-end network achieves state-of-the-art NYUDv2 performance for both single-view segmentation and multi-view semantic fusion.

II. RELATED WORK

Prior work advanced CNN-based image segmentation and semantic mapping, but multi-view semantic segmentation remained comparatively uncommon. The paper differs by imposing multi-view consistency during CNN training rather than only fusing independently predicted views.

  • Deep neural networks, especially CNNs, achieved major progress in semantic image segmentation.
  • Earlier RGB-D methods combined appearance and depth using approaches such as HHA encoding, superpixel classifiers, and RGB-D CNN architectures.
  • Semantic SLAM systems projected or fused frame-level segmentations into 3D maps using visual odometry, random forests, CRFs, or voxel representations.
  • Existing multi-view methods used volumetric CNNs, surfel maps, superpixels, optical flow, or dense CRFs without imposing multi-view consistency during CNN training.
  • The proposed work builds on an RGB-D encoder-decoder and adds multi-scale loss minimization for segmentation.

A. RGB-D Semantic Encoder-Decoder

The RGB-D semantic model uses a FuseNet-inspired encoder-decoder with separate RGB and depth branches whose features are fused across scales. It predicts per-pixel class probabilities and learns them with cross-entropy against ground-truth labels.

  • The network uses an encoder-decoder architecture that extracts hierarchical features and upsamples low-resolution representations through unpooling and deconvolution.
  • Separate RGB and depth branches learn modality-specific features, with depth features fused into the RGB branch at each scale.
  • Softmax maps classification scores at each pixel to probabilities for the K semantic classes given an RGB-D image and network parameters.
  • Cross-entropy trains the network from ground-truth annotations and minimizes KL divergence under a one-hot ground-truth distribution.

B. Multi-Scale Deep Supervision

The model applies deep supervision throughout decoder refinement by computing losses at multiple output resolutions. This multi-scale design is also integrated with multi-view consistency learning through warped feature maps.

  • B. Multi-Scale Deep Supervision: The encoder downsamples input resolution by a factor of 32 through five pooling layers before decoder refinement.
  • B. Multi-Scale Deep Supervision: Classification layers at every deconvolution scale compute losses against ground-truth annotations downsampled by stochastic pooling.
  • B. Multi-Scale Deep Supervision: Multi-view consistency extends this supervision by warping multiple frames into a common reference view, enabling constraints during training and prediction.

A. Multi-view Data Association Through Warping

The method trains on RGB-D sequences by using SLAM-estimated poses and depth to associate neighboring views with a common reference frame. Differentiable warping synthesizes aligned features or outputs across resolutions for multi-view consistency.

  • Sequence construction: Training sequences contain one annotated reference view and overlapping tracking views whose relative poses are estimated with DVO-SLAM.This replaces isolated single-view training with pose-aware RGB-D sequence training.
  • Multi-scale supervision: The ground-truth and predictions span five coarse-to-fine resolutions: 20 × 15, 40 × 30, 80 × 60, 160 × 120, and 320 × 240.The figure shows stochastic pooling for ground truth and CNN predictions at each scale.
  • Differentiable warping: Warping layers synthesize reference-view CNN outputs from another view at arbitrary resolutions using fixed pose and depth transformations.They function as fixed-parameter spatial-transformer variants.
  • Geometric association: Warping transforms pixels between camera views using depth and the SLAM pose estimate.The mapping uses projection and inverse projection around a pose-derived homogeneous transformation.
  • Differentiable warping: Neighboring feature maps are sampled into the reference view with bilinear interpolation, allowing gradients to backpropagate through the warping operation.The operation is denoted Fω := F(ω(x, ξ)).
  • Multi-scale supervision: Decoder feature maps at every resolution are warped into the common reference view for multi-view deep supervision.A single input-resolution grid is normalized and pooled to generate lower-resolution grids efficiently.

B. Consistency Through Warp Augmentation

The paper enforces consistency by aligning neighboring-frame predictions with an annotated keyframe and applying supervised losses. It also describes Bayesian fusion as a probability-space alternative based on accumulated frame likelihoods.

  • Warp augmentation: Warping neighboring predictions into an annotated keyframe provides a direct supervised constraint for multi-view segmentation consistency.The approach can also be interpreted as data augmentation using nearby frames.
  • Warp augmentation: The implemented warp-augmentation method synthesizes nearby-frame classification scores from the keyframe viewpoint and applies cross-entropy loss.This exposes the network to changes in scale, location, perspective, lighting, and shape across views.
  • Bayesian fusion: Bayesian fusion aggregates semantic segmentations from multiple views in probability space.The notation treats y as a pixel label and z_i as measurements through frame i.
  • Bayesian fusion: Under i.i.d. measurements and equal class priors, the Bayesian update simplifies relative to the general formulation.These assumptions are explicitly required for the simplification.
  • Bayesian fusion: Bayesian fusion can be implemented by multiplying per-frame semantic-label likelihoods and normalizing the product into a probability distribution.The computation can also be applied recursively over a frame sequence.
  • Bayesian fusion: During consistency training, warped neighboring predictions are fused at keyframe pixels by summing unnormalized log-likelihoods before applying softmax.This produces the fused labeling distribution described by the Bayesian formulation.

D. Consistency Through Multi-View Max-Pooling

The feature-space alternative warps decoder features from multiple views into a keyframe and pools corresponding activations before classification. The resulting segmentation is supervised against keyframe ground truth.

  • Feature-space fusion: Multi-view max-pooling fuses warped feature maps directly in feature space rather than combining probabilities.Feature maps preceding classification layers are pooled at corresponding warped locations.
  • Feature-space fusion: The method warps decoder feature maps at each scale into the keyframe and applies max-pooling across corresponding feature activations.The pooled feature map is formed in the keyframe coordinate system.
  • Training objective: The fused feature maps are classified, and the resulting semantic segmentation is compared with keyframe ground truth for loss calculation.Supervision is therefore applied after multi-view feature aggregation.

V. EVALUATION

Evaluation uses NYUDv2 RGB-D data with annotated single frames and available source sequences, alongside implementation and training details. The supplied passages identify the single-view comparison table but do not report its numerical entries.

  • Dataset: NYUDv2 provides 1449 pixelwise annotated RGB-D images, split into 795 trainval frames and 654 test frames.The original sequences are available, enabling sequence-based evaluation with camera poses estimated by DVO-SLAM.
  • Implementation: The network uses pretrained 16-layer VGG-Net encoder kernels, He-initialized decoder kernels, and a single-channel depth kernel derived by averaging RGB weights.The implementation is based on Caffe.
  • Evaluation scope: Table I compares single-view semantic segmentation accuracy against state-of-the-art methods on NYUDv2 13-class and 40-class tasks.The supplied table passage gives the task scope but not the accuracy values.
  • Training details: Training uses minibatches of 6 containing two sequences, with one keyframe and two tracking frames per sequence.Samples are randomly shuffled after each epoch across and within sequences.

B. Evaluation Criteria

The paper evaluates semantic segmentation with global pixelwise accuracy, average classwise accuracy, and average IoU, comparing multi-view variants with established methods and baselines.

  • Evaluation criteria: Segmentation performance is measured using global pixelwise accuracy, average classwise accuracy, and average intersection-over-union (IoU).These criteria are calculated from the confusion matrix.
  • Single-frame evaluation: The first experiment evaluates direct semantic segmentation on 654 NYUDv2 test images without fusing nearby-frame predictions.The methods are compared with state-of-the-art approaches using results reported in the original papers where applicable.
  • Baselines: The MVCNet-Mono baseline omits multi-view consistency training, while FuseNet-SF3 and MVCNet-Mono use single-image augmentation.The augmentation includes random scaling, cropping, and mirroring.
  • Results: Multi-view consistency training variants outperform state-of-the-art methods for single-image semantic segmentation.MVCNet-MaxPool has a small advantage over MVCNet-Augment and MVCNet-Bayesian.

D. Multi-View Fused Segmentation

The fused-segmentation experiment aggregates predictions from sampled sequence frames with Bayesian fusion. Multi-view consistency training improves fusion quality, with gains across NYUDv2 class settings and occurrence frequencies.

  • Fusion protocol: Bayesian fusion of 50 uniformly sampled frames improves single-view segmentation by approximately 2% on all evaluation measures.The fusion is applied over test sequences and is used for semantic mapping with RGB-D SLAM.
  • Fusion results: Multi-view consistency training produces a stronger gain over single-view training when segmentations are fused than in single-view segmentation.Qualitative results show more accurate and homogeneous fused segmentations.
  • Classwise analysis: The approach achieves high performance gains across all class-occurrence frequencies in NYUDv2 13-class segmentation.Classwise and average IoU scores are compared in Table III.

VI. CONCLUSION

The paper enforces multi-view consistency in RGB-D segmentation by warping views or feature maps using RGB-D SLAM trajectories and evaluating three training approaches. Multi-view max-pooling gives the best overall gains, while fusion remains vulnerable to mirror reflections.

  • Contribution: The method augments a FuseNet encoder-decoder with multi-scale loss supervision and three approaches for multi-view consistency training.The approaches use an RGB-D SLAM trajectory estimate to warp semantic segmentations or feature maps between viewpoints.
  • Failure cases: Multi-view fusion can degrade performance when mirror reflections are present.In other challenging cases, fusion either fails across all frames or corrects some single-view mistakes.
  • Best variant: Multi-view max-pooling of feature maps provides the best performance gains in both single-view segmentation and multi-view fusion.The method uses feature-map warping to enforce consistency across viewpoints.
  • Benchmark results: All multi-view consistency training approaches outperform single-view-trained baselines on NYUDv2 13-class and 40-class benchmarks.The reported comparisons cover both single-view predictions and fused semantic segmentation.
  • Benchmark results: The end-to-end network reaches state-of-the-art performance for single-view predictions and multi-view fused semantic segmentation without dense-CRF postprocessing.Table III reports the best per-class accuracy and average IoU for the method.
  • Scope and future work: The approach is intended for integration into semantic SLAM, with future work coupling pose tracking and SLAM to semantic predictions.The cited conclusion frames this as future investigation rather than a completed integration.
Loading 1703.08866v2…