PaperlayerView exact revision on arXiv

Source-linked AI summary

Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments

Oier Mees, Andreas Eitel, Wolfram Burgard

arXiv:1707.05733v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Changing conditions can make sensor modalities differently reliable for robot object detection. The paper uses a mixture of CNN experts with learned adaptive fusion, and reports better performance than vision-based and other fusion approaches, including in changing lighting and motion blur.

  • Problem

    Robots need object detection that can handle sensor noise when environmental changes alter the reliability of modalities such as RGB and depth.

    Source: §I · p. 1

  • Method

    A mixture of deep network experts uses a gating network to weight modality-specific classifier outputs from expert feature representations.

    Source: §III · p. 2, A. Fusion via mixture of experts · p. 2

  • Results

    The multimodal method outperforms a vision-based baseline and other fusion techniques, with improved detection in a mobile-robot sequence containing abrupt lighting changes and severe motion blur.

    Source: §V · p. 6

  • Takeaways & Limitations

    The reported comparisons include the publicly available RGB-D People dataset and the InOutDoor People dataset introduced in this work.

    Source: §V · p. 6

Abstract

from arXiv · show

Object detection is an essential task for autonomous robots operating in dynamic and changing environments. A robot should be able to detect objects in the presence of sensor noise that can be induced by changing lighting conditions for cameras and false depth readings for range sensors, especially RGB-D cameras. To tackle these challenges, we propose a novel adaptive fusion approach for object detection that learns weighting the predictions of different sensor modalities in an online manner. Our approach is based on a mixture of convolutional neural network (CNN) experts and incorporates multiple modalities including appearance, depth and motion. We test our method in extensive robot experiments, in which we detect people in a combined indoor and outdoor scenario from RGB-D data, and we demonstrate that our method can adapt to harsh lighting changes and severe camera motion blur. Furthermore, we present a new RGB-D dataset for people detection in mixed in- and outdoor environments, recorded with a mobile robot. Code, pretrained models and dataset are available at http://adaptivefusion.cs.uni-freiburg.de

I. INTRODUCTION

The paper asks how robots can adaptively fuse sensor modalities when changing environmental conditions alter their reliability. It proposes learned fusion of modality-specific experts and evaluates it for human detection in changing RGB-D environments.

I. INTRODUCTION

  • Sensor reliability changes with environmental conditions, making adaptive fusion of raw modalities a central challenge for robust object detection.The introduction contrasts dark indoor settings, where depth may be more reliable, with sunny outdoor settings and distant objects, where depth may be uninformative.Source: §I · p. 1
  • The proposed object-detection scheme uses a mixture of deep network experts and learns their weights with a CNN using high-level expert features, without prior information.Source: §I · p. 1
  • The experiments target two-class human detection in RGB-D data and report greater robustness in changing environments than vision-based and other multimodal fusion approaches.Source: §I · p. 1
  • The mobile-robot sequences move from poorly illuminated indoor environments to very bright outdoor scenes over a short period, challenging both RGB and depth.The introduction also names out-of-range readings and reflective materials as sources of depth-sensor noise.Source: §I · p. 2

II. RELATED WORK

Prior sensor-fusion approaches combine modalities at different stages, use fixed expert weights, hand-designed features, or temporal filtering. This work instead learns per-frame fusion weights from CNN feature representations.

II. RELATED WORK

  • Earlier sensor-fusion methods include appearance-depth feature fusion, fixed-weight experts using appearance, motion, and depth, and late-fusion SVMs with manually designed features.Source: §II · p. 2
  • Other approaches weight detectors using missing-modality information or combine close-range depth and appearance detections through tracking, whereas this method fuses per frame.Source: §II · p. 2
  • Vision-focused pedestrian detectors have used better features, added context or motion, and CNNs, but the cited approaches do not use multiple modalities.Source: §II · p. 2

III. MIXTURE OF DEEP NETWORKS ARCHITECTURE

The proposed mixture of deep network experts (MoDE) combines modality-specific CNN classifiers through a gating network that learns how to weight their outputs from expert features.

III. MIXTURE OF DEEP NETWORKS ARCHITECTURE

  • MoDE fuses modality-specific expert classifiers through an additional gating network that determines their contribution to the final classifier output.Source: §III · p. 2
  • Unlike the original adaptive mixture-of-experts method, the gating network uses higher-level feature representations from each expert rather than raw pixel input.Source: §III · p. 2
  • Training examples pair multiple modality inputs with class labels encoded as C-dimensional one-hot vectors.Source: §III · p. 2

A. Fusion via mixture of experts

The fusion module combines each modality expert’s classifier output using input-dependent gating functions. The gate receives concatenated expert features and its outputs weight the experts’ class predictions.

A. Fusion via mixture of experts

B. Training a mixture of experts

The method trains expert networks first, then trains a gating network while keeping the experts fixed. Experiments use CifarNet and GoogLeNet-xxs architectures with different input modalities.

B. Training a mixture of experts

  • CifarNet and GoogLeNet-xxs serve as expert architectures, with GoogLeNet-xxs using layers through its first softmax classifier.RGB experts are fine-tuned from pretrained models; experts for other domains are trained from scratch.Source: B. Training a mixture of experts · p. 3
  • The model is trained in two stages: first the individual experts, then the gating network with expert weights fixed.During the second stage, the gating network is trained on a validation set using SGD.Source: B. Training a mixture of experts · p. 3
  • The experiments use RGB, depth, and optical flow as input modalities, with dense multiscale sliding windows generating region proposals.Optical flow is computed using the method proposed by Brox et al..Source: B. Training a mixture of experts · pp. 3–4
  • Tables I and II present performance for single and multimodal networks and compare selected models with prior approaches on RGB-D People.The supplied passages identify the tables’ scope but do not provide their cell values.Source: Table I · p. 4, Table II · p. 4

A. RGB-D People Unihall Dataset

The Unihall evaluation uses people detections from three Kinect cameras and compares single networks trained on RGB, depth, and optical flow. Depth CifarNet outperforms HOD-based approaches on the reported metrics.

A. RGB-D People Unihall Dataset

  • The evaluation uses over 3,000 frames from three vertically mounted Kinect cameras, with five random train/test splits and a reported split standard deviation of ±0.8 EER.Training randomly selects 700 frames per camera; evaluation uses the remaining 300 per camera.Source: A. RGB-D People Unihall Dataset · p. 4

1) Single expert performance

  • At IoU 0.4, Depth CifarNet reaches 93.7% EER, a 9.7% relative improvement over Choi et al.’s graph-based segmentation with HOD.At IoU 0.6, the same model reports a 28.2% improvement over previous approaches.Source: 1) Single expert performance · p. 4
  • Depth CifarNet performs better than RGB CifarNet, while the optical-flow CNN underperforms the other experts.The authors hypothesize that illumination changes and low RGB contrast contribute to the depth advantage; optical flow is also limited to moving people.Source: 1) Single expert performance · p. 4

2) Comparison of fusion approaches

Across fusion approaches, MoDE performs best in the reported Unihall scenarios, while the InOutDoor evaluation tests detections through difficult lighting and motion conditions. Its gating weights shift between RGB and depth with the environment.

2) Comparison of fusion approaches

  • 89.3% EER is reported for three-modality MoDE, with a 1.9% relative improvement over HGE’s 87.4% EER.MoDE yields the best performance across the reported fusion scenarios; late fusion of depth and optical flow underperforms, which the authors attribute to their differing feature spaces.Source: 2) Comparison of fusion approaches · p. 4

B. InOutDoor RGB-D People Dataset

  • In dark frame b, the gating weights are 0.38 for RGB and 0.62 for depth, where people are more visible in depth.The reported test includes abrupt lighting changes and severe motion blur that challenge both RGB and depth modalities.Source: B. InOutDoor RGB-D People Dataset · pp. 4–5, Figure 3 · p. 5
  • The InOutDoor test sequence contains 1,066 frames as the robot moves from a dark indoor scene into a relatively dark street scene.The dataset comprises 8,605 RGB-D frames recorded at 30 Hz, with robot transitions from indoors to outdoors in a single take.Source: B. InOutDoor RGB-D People Dataset · p. 5
  • Hard 0/1 expert selection underperforms weighting, while a six-channel end-to-end network slightly underperforms late fusion using the same GoogLeNet architecture.MoDE also outperforms naive averaging of classifier outputs.Source: B. InOutDoor RGB-D People Dataset · p. 5

2) Qualitative results

The gating network adapts expert weights across changing conditions. Its weights favor RGB in bright settings and depth in darkness or blur, with more frequent switches during abrupt camera motion.

2) Qualitative results

  • 8.4% AP over late fusion; the best model combines RGB and depth GoogLeNet-xxs experts through the gating network.The authors identify this combination as GoogLeNet-MoDE.Source: 2) Qualitative results · p. 5
  • Gating weights favor RGB in bright indoor and outdoor scenes, but depth in dark scenes; outdoors, RGB helps distant pedestrians nearly invisible in depth.Fig. 4 visualizes assignment weights for each expert.Source: 2) Qualitative results · p. 5, Figure 4 · p. 6
  • Depth performs better on blurred images, while abrupt camera motion produces more frequent gating switches; Fig. 4 shows weights for exemplary frames.The reported weight changes also correspond to lighting conditions and false depth readings at far ranges.Source: 2) Qualitative results · p. 6, Figure 4 · p. 6

V. CONCLUSION

The conclusion presents adaptive multimodal fusion as a mixture of deep network experts that learns classifier weights from raw input. Experiments report stronger detection than vision-based and other fusion approaches, including in a sequence with abrupt lighting changes and severe motion blur.

V. CONCLUSION

  • The multimodal method outperforms vision-based detection and other fusion techniques, and improves detection in a robot sequence with abrupt lighting changes and severe motion blur.The conclusion also reports better results than previously reported depth-based people-detection approaches on RGB-D People and the InOutDoor People dataset.Source: §V · p. 6
  • A mixture of deep network experts automatically learns from raw input how to weight several domain-based classifiers.Source: §V · p. 6

ACKNOWLEGMENTS

  • The authors thank Jost Tobias Springenberg, Luciano Spinello, and Gian Diego Tipaldi for helpful discussions.Source: ACKNOWLEGMENTS · p. 6

Related papers

Loading 1707.05733v2…