PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis

Daniel Seichter, Mona Köhler, Benjamin Lewandowski, Tim Wengefeld, Horst-Michael Gross

arXiv:2011.06961v3cs.CVcs.LGcs.RO

TL;DR

Mobile robots need semantic scene understanding across multiple tasks despite limited computing and battery resources. The paper proposes ESANet, an efficient RGB-D encoder-decoder optimized for embedded inference. On indoor datasets, it performs on par or better than other state-of-the-art methods while enabling much faster inference, and Cityscapes evaluation supports application beyond indoor scenes.

  • Problem

    Mobile robots require real-time semantic scene analysis for multiple tasks under limited computing and battery capabilities.

  • Method

    ESANet uses enhanced RGB and depth encoders, attention-based fusion, and a learned-upsampling decoder designed for efficient TensorRT-optimized inference.

  • Results

    ESANet performs on par or better than other state-of-the-art methods on NYUv2 and SUNRGB-D while enabling much faster inference; on Cityscapes, it exceeds 30 FPS and outperforms other efficient approaches by at least 2.2 mIoU.

  • Takeaways & Limitations

    ESANet is well suited for embedding in complex mobile-robot scene-analysis systems with limited hardware.

Abstract

from arXiv · show

Analyzing scenes thoroughly is crucial for mobile robots acting in different environments. Semantic segmentation can enhance various subsequent tasks, such as (semantically assisted) person perception, (semantic) free space detection, (semantic) mapping, and (semantic) navigation. In this paper, we propose an efficient and robust RGB-D segmentation approach that can be optimized to a high degree using NVIDIA TensorRT and, thus, is well suited as a common initial processing step in a complex system for scene analysis on mobile robots. We show that RGB-D segmentation is superior to processing RGB images solely and that it can still be performed in real time if the network architecture is carefully designed. We evaluate our proposed Efficient Scene Analysis Network (ESANet) on the common indoor datasets NYUv2 and SUNRGB-D and show that we reach state-of-the-art performance while enabling faster inference. Furthermore, our evaluation on the outdoor dataset Cityscapes shows that our approach is suitable for other areas of application as well. Finally, instead of presenting benchmark results only, we also show qualitative results in one of our indoor application scenarios.

I. INTRODUCTION

The paper targets real-time semantic scene understanding for mobile robots with limited computing and battery resources. It proposes efficient RGB-D segmentation as a shared initial step for downstream perception and mapping tasks.

  • Semantic segmentation provides precise pixel-wise information that supports obstacle avoidance, semantic mapping, navigation, and person perception.
  • The approach provides semantic masks that restrict person perception to person regions, identify floor-based free space, and exclude dynamic classes from mapping.
  • ESANet uses RGB and depth inputs to improve segmentation in cluttered indoor scenes while addressing the additional computational cost of depth processing.The approach uses two shallow encoder branches instead of one deep RGB encoder.
  • The paper evaluates ESANet on NYUv2, SUNRGB-D, and Cityscapes, combining benchmark measurements with qualitative results in an indoor robotic application.
  • ESANet is designed for high NVIDIA TensorRT optimization and fast inference on embedded robotic hardware.

II. RELATED WORK

Encoder-decoder architectures extract semantically rich features while reducing computation, then restore resolution to assign a class to every input pixel.

  • The encoder extracts semantically rich features and downsamples them to reduce computational effort.
  • The decoder restores input resolution and assigns a semantic class to each input pixel.

A. RGB-D Semantic Segmentation

RGB-D segmentation combines complementary appearance and geometric information, but real-time deployment on embedded hardware remains challenging. Existing approaches often incur high complexity or rely on deep encoders with high inference times.

  • Depth images provide complementary geometric information to RGB images and can improve segmentation, but their statistics differ from RGB data.
  • 3D projection-based methods increase computational complexity, while depth-specific convolutions often lack optimized implementations for real-time embedded inference.
  • Most RGB-D methods use separate RGB and depth branches that extract modality-specific features before fusing them later in the network.
  • Fusion strategies include injecting depth into the RGB encoder or recalibrating and returning fused features to both encoders.
  • Deep encoders with 50, 101, or 152 layers produce high inference times and are inappropriate for mobile-robot deployment.

B. Efficient Semantic Segmentation

Efficient semantic segmentation reduces computation through lightweight architectures and blocks. ESANet adapts a shallow ResNet-based encoder-decoder design for RGB-D inputs while maintaining compatibility with TensorRT optimization.

  • Efficient segmentation methods reduce operations and parameters through tailored architectures while retaining good segmentation performance.
  • ESANet replaces ResNet basic blocks with more efficient factorized-convolution blocks to reduce inference time.
  • The ESANet overview combines a shallow pretrained ResNet18 encoder, a context module, a shallow skip-connected decoder, and final upsampling.
  • A separate depth encoder supplies complementary geometric features that are fused into the RGB encoder at multiple stages using attention.
  • The network uses simple components because complex or tailored operations can hinder ONNX and NVIDIA TensorRT conversion and slow inference.

A. Encoder

ESANet uses separate ResNet-based RGB and depth encoders, efficient NBt1D blocks, and attention-based fusion to combine modality-specific features.

  • A. Encoder: The RGB and depth branches use ResNet backbones, with ResNet34 selected as a speed–accuracy trade-off alongside ResNet18 and ResNet50 variants.
  • A. Encoder: Replacing ResNet18 and ResNet34 basic blocks with spatially factorized NBt1D blocks reduces inference time while improving segmentation performance.
  • A. Encoder: At five encoder resolutions, depth features are reweighted with Squeeze-and-Excitation and summed element-wise with RGB features.

C. Context Module

The network adds multiscale context aggregation and a detailed decoder that restores resolution using skip connections, learned upsampling, and auxiliary supervision.

  • C. Context Module: The context module aggregates features at multiple scales with fixed pooling sizes designed for NVIDIA TensorRT compatibility.
  • C. Context Module: The decoder expands SwiftNet’s design with 512 channels initially, progressively narrower convolutions, and three additional NBt1D blocks.
  • C. Context Module: Encoder-to-decoder skip connections project fused RGB-D features with 1×1 convolutions and add them at matching resolutions to recover detail.
  • C. Context Module: Two final learned upsampling modules restore the input resolution after decoder processing stops at features 4× smaller than the input.
  • C. Context Module: Each decoder module receives auxiliary supervision through a 1×1 convolution and a ground-truth segmentation resized to the corresponding scale.

IV. EXPERIMENTS

Experiments evaluate ESANet on indoor RGB-D benchmarks, an outdoor dataset, ablations, and embedded inference, using training and preprocessing procedures described for reproducibility.

  • IV. EXPERIMENTS: ESANet is evaluated on SUNRGB-D and NYUv2, with additional results on Cityscapes to assess suitability beyond indoor scenes.
  • IV. EXPERIMENTS: Training used PyTorch for 500 epochs with batches of 8, SGD or Adam optimization, one-cycle learning-rate scheduling, and geometric image augmentation.
  • IV. EXPERIMENTS: Models were selected by mIoU, and bilinear upsampling resized class mappings before computing the final segmentation mask.
  • IV. EXPERIMENTS: NYUv2 contains 1,449 images with 795 training and 654 testing samples across 40 classes, while SUNRGB-D contains 10,335 images across 37 classes.
  • IV. EXPERIMENTS: Figures compare upsampling methods, RGB-D against single-modality networks, and ResNet18-based configurations on the NYUv2 test set.

B. Results on NYUv2 & SUNRGB-D

On NYUv2 and SUNRGB-D, ESANet combines competitive segmentation with faster embedded inference, while ablations support its architectural choices.

  • B. Results on NYUv2 & SUNRGB-D: RGB-D input outperforms RGB-only and depth-only networks on NYUv2, while a shallow ResNet18 RGB-D model surpasses a deeper ResNet50 RGB model and is faster.
  • B. Results on NYUv2 & SUNRGB-D: Replacing basic blocks with NBt1D blocks further improves both segmentation and inference time.
  • B. Results on NYUv2 & SUNRGB-D: Up to 5× faster inference than PyTorch was achieved with NVIDIA TensorRT, while ESANet performed on par with or better than other approaches.
  • B. Results on NYUv2 & SUNRGB-D: ResNet34 with NBt1D was selected as the best trade-off between inference time and performance for the application.

C. Ablation Study on NYUv2

The ablation study evaluates decoder depth, upsampling, context, skip connections, and modality-feature reweighting, showing how these choices affect segmentation performance and inference time.

  • Study purpose: The ablation study identifies how fundamental architecture components affect segmentation performance and inference time under differing real-time requirements.These findings support adapting the network when deployment constraints change.
  • Decoder design: Three NBt1D blocks in each decoder module outperform alternative block counts and reversed encoder layouts.The study uses these results to justify the selected decoder design.
  • Upsampling: Learned upsampling improves mIoU by 0.9 despite increasing inference time.It also produces finer-grained segmentation and avoids gridding artifacts associated with transposed convolutions.
  • Feature enhancement: A context module, encoder-decoder skip connections, and Squeeze-and-Excitation reweighting each independently improve segmentation performance.Combining all three components produces the best result.

D. Results on Cityscapes

On Cityscapes, ESANet remains competitive across efficient and full-resolution settings while extending the RGB-D approach beyond indoor datasets. The evaluation also examines inference on the NVIDIA Jetson AGX Xavier.

  • Evaluation setup: The Cityscapes evaluation is designed to demonstrate applicability to outdoor environments and uses the 1024×512 resolution commonly adopted for efficient segmentation.The study also reports results at the full 2048×1024 resolution.
  • 1024×512 results: At 1024×512, ESANet-R34-NBt1D exceeds 30 FPS and outperforms other efficient approaches by at least 2.2 mIoU.It also achieves notably higher mIoU than RGB-D LDFNet at similar inference time.
  • RGB-D comparison: Depth increases segmentation performance on Cityscapes, although the gain is smaller than on NYUv2.The authors attribute this difference to less precise Cityscapes disparity images than indoor depth images.
  • Full-resolution results: At 2048×1024, ESANet falls between mobile and non-mobile methods in both mIoU and inference time.This comparison covers SwiftNet, BiSeNet, and non-mobile approaches.
  • Resolution trade-off: With 1024×512 RGB-D input, ESANet-R34-NBt1D matches SwiftNet RGB performance at 2048×1024 while providing slightly faster inference.The comparison uses the smaller input resolution for ESANet.

E. Application on our Robots

The authors deploy ESANet-R34-NBt1D in a robot’s semantic scene-analysis system and present qualitative results from an indoor application. The resulting masks support person perception and semantic mapping, including refined free-space representation.

  • Robot deployment: ESANet-R34-NBt1D is deployed with a Kinect2 sensor in a complex robotic semantic scene-analysis system.The application evaluates qualitative behavior rather than benchmark metrics alone.
  • System benefits: Segmentation masks strengthen person perception and semantic mapping by enriching the robot’s visual perception.The mapping application includes a refined floor representation indicating free space.
  • Conclusion: The paper concludes that ESANet performs on par or better than state-of-the-art methods on NYUv2 and SUNRGB-D while enabling much faster inference.This supports embedding the approach in mobile-robot scene-analysis systems with limited hardware.
  • System overview: Figure 6 provides an overview of the complete robotic scene-analysis system.The figure accompanies the qualitative application results.
Loading 2011.06961v3…