PaperlayerView exact revision on arXiv

Source-linked AI summary

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

Sachin Mehta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, Hannaneh Hajishirzi

arXiv:1803.06815v3cs.CV

TL;DR

Semantic segmentation requires accurate models, but large networks are slow, power hungry, and unsuitable for resource-constrained edge devices. ESPNet addresses this with an efficient spatial pyramid module and achieves fast, efficient segmentation while generalizing across datasets and outperforming efficient baselines.

  • Problem

    Large CNNs are accurate but slow and power hungry, making semantic segmentation difficult to deploy on computationally constrained edge devices.

  • Method

    ESPNet uses ESP, which factorizes convolution into point-wise convolutions and a spatial pyramid of dilated convolutions, and introduces edge-device performance metrics.

  • Results

    ESPNet outperforms efficient segmentation baselines on unseen data and processes high-resolution RGB images at 112 FPS on a high-end GPU and 9 FPS on an edge device.

  • Takeaways & Limitations

    ESPNet provides a fast, efficient semantic segmentation network that preserves accuracy and learns generalizable representations across datasets.

Abstract

from arXiv · show

We introduce a fast and efficient convolutional neural network, ESPNet, for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a new convolutional module, efficient spatial pyramid (ESP), which is efficient in terms of computation, memory, and power. ESPNet is 22 times faster (on a standard GPU) and 180 times smaller than the state-of-the-art semantic segmentation network PSPNet, while its category-wise accuracy is only 8% less. We evaluated ESPNet on a variety of semantic segmentation datasets including Cityscapes, PASCAL VOC, and a breast biopsy whole slide image dataset. Under the same constraints on memory and computation, ESPNet outperforms all the current efficient CNN networks such as MobileNet, ShuffleNet, and ENet on both standard metrics and our newly introduced performance metrics that measure efficiency on edge devices. Our network can process high resolution images at a rate of 112 and 9 frames per second on a standard GPU and edge device, respectively.

1 Introduction

ESPNet targets semantic segmentation on resource-constrained edge devices by combining an efficient spatial pyramid with a fast, compact network design. Its evaluations report strong efficiency, speed, and generalization while retaining segmentation accuracy.

  • Motivation: Large CNNs are slow and power hungry, making semantic segmentation difficult to deploy for online local processing on energy-limited edge devices.PSPNet is cited as having 65.7 million parameters and running at about 1 FPS while discharging a standard laptop at 77 Watts.
  • Approach: ESPNet is designed to be fast, small, low power, and low latency while preserving segmentation accuracy on resource-constrained edge devices.The network is intended for deployment on edge devices with limited resources.
  • Approach: ESP decomposes standard convolution into point-wise convolution and a spatial pyramid of dilated convolutions.Point-wise convolution reduces computation, while dilated convolutions learn representations over a large effective receptive field.
  • Evaluation: ESPNet introduces edge-device metrics including sensitivity to GPU frequency and warp execution efficiency alongside standard evaluation metrics.These metrics are used to analyze CNN performance on edge devices.

2 Related Work

Prior work reduces CNN cost through factorization, compression, low-bit computation, sparsity, and specialized segmentation architectures. ESPNet builds on factorization while addressing the computational and memory demands of dilated spatial-pyramid methods.

  • Efficient CNNs: Convolution factorization decomposes convolution into multiple steps to reduce deep CNN computational complexity.Examples include Inception, factorized networks, and related efficient architectures.
  • Efficient CNNs: Compression-based methods, low-bit networks, and sparse CNN methods are complementary techniques that can also be applied to ESPNet.The paper presents these approaches as compatible with its contribution rather than alternatives that replace it.
  • Dilated Convolutions: Dilated convolutions enlarge effective receptive fields by inserting holes between kernel pixels, but dilation can introduce artifacts.For an n × n kernel with dilation rate r, the effective size is [(n−1)r +1]2.
  • Spatial Pyramids: Existing spatial-pyramid approaches use large receptive fields but can be computationally inefficient because of high memory requirements and many parameters.The paper specifically identifies atrous spatial pyramid modules as examples of this trade-off.
  • Semantic Segmentation: Semantic segmentation research includes recurrent, encoder-decoder, hypercolumn, region-based, and cascaded CNN architectures.These represent several major families of CNN-based segmentation networks discussed in the paper.

3 ESPNet

ESPNet uses ESP modules to factorize convolution into low-dimensional projection and parallel dilated processing, combining large receptive fields with reduced resource use. Hierarchical fusion and residual connections address gridding artifacts and information flow, while comparisons position ESP against other CNN modules.

  • ESP Module: ESP modules apply a 1×1 convolution before parallel n × n dilated convolutions with dilation rates 2^k−1.The point-wise convolution projects high-dimensional feature maps into a low-dimensional space before spatial processing.
  • ESP Module: The ESP pipeline projects, splits, transforms, and merges feature maps to produce an N-dimensional output.K parallel branches process the split feature maps using different dilation rates, then concatenate their outputs.
  • Efficiency: For n = 3, N = M = 128, and K = 4, an ESP module learns approximately 3.6 times fewer parameters while achieving a 17×17 effective receptive field.The comparison is against a standard convolutional kernel with a 3×3 effective receptive field.
  • De-gridding: Hierarchical feature fusion removes gridding artifacts caused by combining dilated-convolution outputs with large effective receptive fields.The module also uses a skip-connection between input and output to improve information flow.
  • Module Comparisons: Compared with MobileNet, ShuffleNet, Inception, and ResNext modules, ESP emphasizes a larger effective receptive field alongside low memory or parameter requirements.The cited comparisons describe different trade-offs among parameters, memory, and receptive field across these modules.
  • Module Comparisons: Figure 3 compares convolutional modules using layer dimensions, dilation rates, grouped-convolution counts, and convolutional-layer memory.Memory values are converted to bytes by multiplying by 4 because one float requires 4 bytes.

4 Experiments

Experiments evaluate ESPNet’s architecture, generalization, segmentation accuracy, and edge-device efficiency across datasets and controlled ablations. Results show strong resource efficiency, competitive accuracy, and benefits from architectural refinements.

  • Experimental set-up: ESPNet uses ESP modules for feature learning and down-sampling, with a lightweight decoder producing masks at the input resolution.ESPNet variants progressively improve information flow through feature sharing, input reinforcement, and decoding.
  • Datasets and evaluation: ESPNet was evaluated on multiple segmentation datasets, including an unseen Mapillary dataset and datasets from different domains.The experiments assess both generalization and segmentation performance beyond the primary Cityscapes evaluation.
  • Generalization: ESPNet outperformed ENet and ERFNet qualitatively and quantitatively on the unseen Mapillary dataset.The compared networks were trained on Cityscapes and tested on Mapillary after mapping annotations to seven Cityscapes categories.
  • Segmentation results: ESPNet achieved 4% higher accuracy than SegNet on PASCAL VOC while learning 81× fewer parameters, and matched a breast-biopsy model with 9.5× fewer parameters.On PASCAL VOC, ESPNet remained 22% less accurate than PSPNet while using 180× fewer parameters.
  • Edge-device performance: 1.12× and 5.45× smaller networks than ENet and ERFNet, respectively, were obtained on the NVIDIA Jetson TX2, while ESPNet had nearly the same frame rate as ENet.ESPNet was more sensitive to GPU frequency; the analysis also examined utilization, efficiency, memory behavior, and power consumption.

5 Conclusion

ESPNet is a semantic segmentation network built around an efficient spatial pyramid module, complemented by system-level metrics for CNN performance analysis. Empirical results indicate that ESPNet is fast, efficient, generalizable, and effective on unseen data.

  • ESPNet is a semantic segmentation network based on an efficient spatial pyramid module.
  • The paper introduces system-level metrics alongside legacy metrics to analyze CNN performance.
  • Empirical analysis suggests that ESPNet is fast and efficient.
  • ESPNet learns generalizable object representations and performs well in the wild.

A Hardware Details

The experiments use three machines with different CPU–GPU memory arrangements, including the NVIDIA Jetson TX2 edge platform. TX2 operating modes alter enabled CPU cores, clock frequencies, and power requirements.

  • Three machines were used in the experiments.Table 3 summarizes their hardware details.
  • Edge devices such as Jetson TX2 share global memory or RAM between CPU and GPU.Laptop and desktop devices instead have dedicated CPU and GPU memory.
  • NVIDIA Jetson TX2 operates in performance mode or normal mode with different active CPU cores and clock frequencies.Max-P enables all CPU cores, whereas Max-Q activates four of six cores.
  • TX2 operating modes produce different application power requirements.

B The path from ESPNet-A to ESPNet

ESPNet evolves from ESPNet-A through feature-map sharing and input reinforcement, then adds a lightweight decoder to produce full-resolution segmentation masks. Its design uses strided ESP modules, depth multipliers, and reduce-upsample-merge aggregation to balance efficiency and information flow.

  • The path from ESPNet-A to ESPNet: ESPNet-A processes RGB images with ESP modules at different spatial levels to produce segmentation masks.ESPNet-B, ESPNet-C, and ESPNet then add feature-map sharing, input reinforcement, and a lightweight decoder, respectively.
  • The path from ESPNet-A to ESPNet: Strided ESP modules replace standard strided convolutions for down-sampling to reduce parameters and enlarge the effective receptive field.The parameter requirement is reduced by a factor of KM M+N, while the effective receptive field increases by approximately [2K−1]2 times.
  • The path from ESPNet-A to ESPNet: ESPNet-B concatenates feature maps from preceding strided ESP and ESP modules to expand network width and improve information flow.
  • The path from ESPNet-A to ESPNet: ESPNet-C reinforces the input image by concatenating its down-sampled representation with feature maps, adding minimal complexity because RGB inputs have three channels.
  • The path from ESPNet-A to ESPNet: Depth multiplier α repeats ESP modules at spatial level l, while avoiding repetition at high-resolution levels l = 0 and l = 1 for memory efficiency.Increasing α2 mainly enlarges network size, whereas increasing α3 increases both network size and parameter count.
  • The path from ESPNet-A to ESPNet: The decoder uses Reduce-Upsample-Merge to aggregate multi-level information and recover output at the input image’s spatial resolution.

C Image Size vs. Inference Speed

ESPNet is faster than ENet and ERFNet at smaller image resolutions, but its speed becomes similar to ENet for high-resolution images on the Jetson TX2. On high-end laptop and desktop devices, ESPNet processes high-resolution images faster than ENet.

  • At 224x224 and 640x360 resolutions, ESPNet is faster than ENet and ERFNet.
  • At high resolutions on the TX2, ESPNet has inference speed similar to ENet.The authors presume that shared, limited TX2 resources create a bottleneck.
  • ESPNet processes high-resolution images faster than ENet on high-end laptop and desktop devices.

D Top-10 Kernels in ESPNet, ENet, and ERFNet

ESPNet’s kernel profile indicates efficient use of edge-device compute resources, with GEMM as its dominant kernel and memory alignment identified as ERFNet’s bottleneck.

  • Kernel contributions: 38% of ESPNet’s total computational time is attributed to its top-1 GEMM kernel.GEMM implements convolution operations in the analyzed networks.
  • Kernel contributions: ESPNet’s top-1 GEMM contribution exceeds ENet’s, while ERFNet’s top kernel is memory alignment.The authors associate ERFNet’s top kernel with a memory-operations bottleneck.
  • Kernel contributions: ESPNet runs 1.27× faster than ENet on NVIDIA TitanX while running at almost the same rate on NVIDIA TX2.The passage links this difference to ESPNet’s greater sensitivity to GPU frequency.
  • Kernel contributions: Figure 10 examines how image size affects inference speed on an edge device.The caption identifies image size and edge-device inference speed as the plotted variables.
  • Kernel contributions: Figure 11 visualizes the top-10 kernels and their contributions to compute-resource utilization, highlighting the top-1 kernel in green.The figure compares kernel profiles across ENet, ERFNet, and ESPNet.

E Resource Utilization Plots for ENet, ERFNet, and ESPNet

Resource-utilization analyses compare ENet, ERFNet, and ESPNet on NVIDIA Jetson TX2, showing high GPU use, low CPU use, and ESPNet’s lower memory footprint than ENet.

  • Resource utilization: Average CPU utilization stays below 25% for these throughput-intensive networks on NVIDIA Jetson TX2.The passage suggests the networks use only one of TX2’s four CPU cores.
  • Resource utilization: Figures 12–14 compare CPU, GPU, and memory utilization over time for ENet, ERFNet, and ESPNet.Measurements were collected using Tegrastats in Max-Q mode on TX2.
  • Resource utilization: GPU utilization rates are high while CPU utilization rates are low for these throughput-intensive networks.This describes the reported utilization pattern across the evaluated networks.
  • Resource utilization: ESPNet has a lower memory footprint than ENet despite learning the same number of parameters.The comparison uses ESPNet with α2 = 2 and α3 = 8.
  • Resource utilization: NVIDIA Jetson TX2 provides 8 GB of maximum available memory shared between CPU and GPU.Figure 14 uses this shared-memory capacity as the device context.

F Results on the Cityscape and the Mapillary Dataset

ESPNet generalizes across Cityscape and Mapillary, showing good category-wise performance and relatively strong behavior on Mapillary’s in-the-wild images despite class-level errors.

  • Dataset results: ERFNet outperformed ENet and ESPNet on every class in Cityscape but performed badly on the Mapillary dataset.On Mapillary, ERFNet struggled with simple classes such as sky, while ENet and ESPNet performed relatively well.
  • Dataset results: ESPNet makes some errors between classes within the same category on the Cityscape validation set.The confusion matrix caption attributes its lower class-wise accuracy partly to these same-category confusions.
  • Dataset results: ESPNet delivers good category-wise accuracy on the Cityscape validation set.The figure caption contrasts this with lower class-wise accuracy.
  • Dataset results: ESPNet outperformed both ENet and ERFNet on the Mapillary validation set’s category-wise comparison.The table caption states that ESPNet learned generalizable object representations and outperformed both networks in the wild.
  • Dataset results: Figures 16 and 17 present qualitative results for the Cityscape and Mapillary validation datasets, respectively.These figures accompany the reported dataset comparisons.
Loading 1803.06815v3…