Source-linked AI summary
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Adam Paszke, Abhishek Chaurasia, Sangpil Kim, Eugenio Culurciello
TL;DR
Real-time pixel-wise semantic segmentation is important for mobile applications, but existing deep networks require substantial computation and have long runtimes. ENet addresses this gap with a low-latency neural architecture, achieving efficient operation while maintaining comparable segmentation accuracy across evaluated settings.
Problem
Existing semantic-segmentation networks require many floating point operations and long runtimes, limiting their usability on mobile and battery-powered devices.
Method
ENet is a neural network architecture designed for fast inference and high accuracy, using a large encoder and small decoder.
Results
ENet provides high frame rates for real-time applications and is efficient on both NVIDIA TX1 embedded hardware and Titan X GPUs.
Takeaways & Limitations
ENet offers an embedded-platform semantic-segmentation solution with gains over baseline models while matching or sometimes exceeding their performance.
Takeaways & Limitations
PReLUs consume more than a quarter of inference time, and kernel fusion could further improve ENet’s speed and efficiency.
Abstract
from arXiv · showhide
The ability to perform pixel-wise semantic segmentation in real-time is of paramount importance in mobile applications. Recent deep neural networks aimed at this task have the disadvantage of requiring a large number of floating point operations and have long run-times that hinder their usability. In this paper, we propose a novel deep neural network architecture named ENet (efficient neural network), created specifically for tasks requiring low latency operation. ENet is up to 18$\times$ faster, requires 75$\times$ less FLOPs, has 79$\times$ less parameters, and provides similar or better accuracy to existing models. We have tested it on CamVid, Cityscapes and SUN datasets and report on comparisons with existing state-of-the-art methods, and the trade-offs between accuracy and processing time of a network. We present performance measurements of the proposed architecture on embedded systems and suggest possible software improvements that could make ENet even faster.
1 Introduction
Real-time pixel-wise segmentation is needed for mobile applications, but existing deep architectures are too large and slow for many battery-powered devices. ENet is proposed as a fast, accurate architecture for this setting and is evaluated across driving and indoor datasets.
- Motivation: Real-time semantic segmentation is needed for applications including augmented reality, home automation, and self-driving vehicles on low-power devices.These algorithms assign each image pixel to an object class.
- Problem: Existing segmentation networks based on VGG16 use many parameters and long inference times, making them unsuitable for applications requiring more than 10 fps.
- Contribution: ENet is a neural network architecture optimized for fast inference and high accuracy.
- Evaluation: The evaluation covers Cityscapes and CamVid road scenes and SUN indoor scenes, using NVIDIA Jetson TX1 and Titan X hardware.
2 Related work
Prior scene-parsing systems commonly combine encoder and decoder networks, but their large architectures remain slow, while post-processing or recurrent components introduce additional burdens.
- Application context: Semantic segmentation supports image understanding and target-object localization in applications such as driving aids and augmented reality.
- Encoder-decoder models: State-of-the-art scene-parsing CNNs combine an encoder with a decoder for semantic segmentation.The encoder classifies the input, while the decoder upsamples its output.
- Limitations: VGG16-based encoder-decoder networks remain too slow for real-time inference despite removing fully connected layers to reduce computation and memory.
- Alternative approaches: CRF post-processing can be onerous and often fails to label classes occupying fewer pixels, while recurrent post-processing degrades speed.
3 Network architecture
ENet uses a staged encoder-decoder architecture built from compact bottleneck modules and varied convolutional operations. Its stages reduce and then restore spatial resolution while limiting computation and memory operations.
- Initial block: The initial block combines non-overlapping 2 × 2 max pooling with a 13-filter convolution, producing 16 feature maps after concatenation.
- Bottleneck modules: Each bottleneck contains a 1 × 1 projection, a main convolution, and a 1 × 1 expansion, with Batch Normalization and PReLU between convolutions.Downsampling bottlenecks add max pooling to the main branch.
- Convolutional operations: The main convolution can be regular, dilated, or full, using 3 × 3 filters, or can replace a 5 × 5 convolution with asymmetric 5 × 1 and 1 × 5 convolutions.
- Stages: ENet divides its architecture into an initial block, encoder stages 1–3, and decoder stages 4–5.Stage 1 has five bottleneck blocks; stages 2 and 3 share a structure, except stage 3 does not initially downsample.
- Architecture specification: The architecture reports output sizes for an example 512 × 512 input image.
- Efficiency choices: ENet omits bias terms in projections to reduce kernel calls and memory operations without affecting accuracy.The decoder uses max unpooling instead of max pooling and spatial convolution without bias.
4 Design choices
ENet’s design choices prioritize real-time semantic segmentation by reducing early and decoder computation while preserving spatial context and accuracy. The architecture combines compact processing, dilated convolutions, factorized bottlenecks, and learned nonlinearities to balance efficiency with segmentation quality.
- Early downsampling: Early downsampling reduces expensive processing on large input frames while retaining a compact feature representation for later layers.ENet heavily reduces input size in its first two blocks and uses few feature maps because visual information is considered spatially redundant.
- Decoder size: A large encoder and small decoder let the encoder process and filter low-resolution features while the decoder mainly fine-tunes upsampled details.This contrasts with symmetric encoder-decoder designs in which the decoder mirrors the encoder.
- Nonlinear operations: PReLU weights learn depth-dependent nonlinearities: main-branch layers resemble ReLUs, while bottleneck weights invert and scale negative values.Initial layers show high variance and slight positive bias, whereas later encoder layers settle into a recurring pattern; decoder weights become more positive and closer to identity.
- Information-preserving dimensionality changes: Using 2 × 2 projections during downsampling considers the full input and improves information flow and accuracy, despite making those layers 4× more expensive.ENet contains few such layers, limiting the overall computational impact.
- Factorizing filters: Asymmetric 5 × 5 convolutions and bottleneck factorization reduce redundancy, parameters, and computation through sequences of smaller operations.The bottleneck sequence acts as a low-rank approximation of a larger convolution and also inserts nonlinear operations between factors.
- Dilated convolutions: Dilated convolutions raise Cityscapes IoU by around 4 percentage points with no additional cost by expanding receptive fields without excessive downsampling.They replace main convolutions in bottlenecks operating at the smallest resolutions and perform best when interleaved with other bottleneck modules.
5 Results
ENet was evaluated against SegNet and other models across road-scene and indoor datasets, using accuracy, inference speed, computational cost, and memory requirements. The results show substantial efficiency gains, competitive segmentation performance, and a software-level limitation associated with fragmented computation.
- Evaluation setup: ENet was evaluated on CamVid, Cityscapes, and SUN RGB-D, with SegNet used as a baseline and tests run on Titan X and TX1 hardware.The evaluation used class average accuracy and intersection over union alongside performance measurements.
- Inference performance: ENet is significantly faster than SegNet across tested input resolutions, providing high frame rates for real-time applications.Inference time and frames per second were compared for single input frames of varying resolution.
- Hardware requirements: ENet’s floating-point-operation and parameter requirements are two orders of magnitude smaller than those of compared models.Its parameter storage requires only 0.7MB in half precision, enabling placement in fast on-chip embedded memory.
- Software limitations: Convolutional-layer factorization reduces FLOPs and parameters but increases kernel calls and memory transactions, with PReLUs consuming more than a quarter of inference time.The authors hypothesize that data movement causes this cost and suggest kernel fusion as a software remedy.
- Benchmarks: On CamVid, ENet outperforms other models in six classes associated with smaller objects.The comparison is reported on the CamVid test set against existing state-of-the-art algorithms.
- Benchmarks: On SUN RGB-D, ENet has comparable class average accuracy to SegNet but lower global average accuracy and IoU, while running nearly 20× faster on embedded platforms.The network uses RGB data without depth information and remains capable of differentiating smaller objects nearly as well as SegNet by class average accuracy.
6 Conclusion
ENet is a semantic-segmentation architecture designed for efficient use of scarce embedded resources while matching or sometimes exceeding larger baseline models. Although targeting mobile devices, it is also reported as efficient on high-end GPUs.
- ENet was designed from the ground up for semantic segmentation and efficient use of scarce embedded-platform resources.
- ENet provides large task gains while matching or sometimes exceeding baseline models with an order of magnitude larger computational and memory requirements.
- ENet's application on NVIDIA TX1 exemplifies a real-time portable embedded solution.
- ENet was also found to be very efficient on the NVIDIA Titan X, despite its primary goal of mobile-device deployment.