Source-linked AI summary
LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation
Abhishek Chaurasia, Eugenio Culurciello
TL;DR
Real-time semantic segmentation requires accuracy with efficient computation, but existing methods can be large and slow. LinkNet introduces direct encoder-to-decoder links that preserve spatial information while reducing wasted computation, and it is evaluated for speed and accuracy across datasets and hardware. The paper reports real-time embedded performance, high-resolution GPU processing, and strong benchmark results.
Problem
Existing semantic-segmentation algorithms can require excessive parameters and operations, limiting processing speed for real-time applications.
Method
LinkNet directly bypasses information from each encoder layer to its corresponding decoder, allowing the decoder to share encoder knowledge and use fewer parameters.
Results
LinkNet provides real-time performance on NVIDIA TX1, processes high-resolution images at 8.5 fps on GPU, and outperforms existing models on Cityscapes.
Takeaways & Limitations
The architecture supports efficient semantic segmentation on embedded platforms while matching or exceeding existing baseline models with larger computational and memory requirements.
Abstract
from arXiv · showhide
Pixel-wise semantic segmentation for visual scene understanding not only needs to be accurate, but also efficient in order to find any use in real-time application. Existing algorithms even though are accurate but they do not focus on utilizing the parameters of neural network efficiently. As a result they are huge in terms of parameters and number of operations; hence slow too. In this paper, we propose a novel deep neural network architecture which allows it to learn without any significant increase in number of parameters. Our network uses only 11.5 million parameters and 21.2 GFLOPs for processing an image of resolution 3x640x360. It gives state-of-the-art performance on CamVid and comparable results on Cityscapes dataset. We also compare our networks processing time on NVIDIA GPU and embedded system device with existing state-of-the-art architectures for different image resolutions.
I. INTRODUCTION
Semantic segmentation is central to scene understanding, but existing encoder-decoder networks often process too slowly for real-time applications. LinkNet targets accurate predictions without compromising processing time by bypassing spatial information directly from encoders to corresponding decoders.
- Semantic segmentation labels every image pixel and supports scene-understanding applications such as augmented reality and self-driving vehicles.
- Encoder-decoder networks encode information into feature space and map it into spatial categories for segmentation.
- Existing semantic-segmentation networks often require excessive processing time despite targeting real-time applications.
- LinkNet bypasses spatial information directly from each encoder to its corresponding decoder, preserving information without additional parameters or operations for relearning.
- The network is evaluated on Cityscapes and CamVid, with processing times recorded on NVIDIA Jetson TX1 and Titan X GPU hardware.
II. RELATED WORK
Semantic segmentation architectures commonly use encoder-decoder designs, with encoders providing discrimination and decoders reconstructing spatial categorization. Prior approaches preserve spatial information through pooling indices, deconvolution, skip connections, or recurrent contextual modeling, while efficiency can reduce accuracy.
- Segmentation networks commonly divide scene parsing into encoder and decoder networks serving discriminative and generative roles.
- Pooling indices can preserve encoder spatial information for decoder upsampling, while fully convolutional and skip architectures avoid storing those indices.
- Recurrent neural networks have been used for contextual information and CRF optimization, but their use makes processing computationally expensive.
- Efficient segmentation designs have optimized forward processing time, sometimes with decreased prediction accuracy.
III. NETWORK ARCHITECTURE
LinkNet uses an encoder-decoder architecture with convolutional downsampling and upsampling, residual encoder blocks, and direct encoder-to-decoder links. These links recover spatial information while allowing a lighter decoder and a more efficient overall network.
- LinkNet’s left half is the encoder and right half is the decoder, with downsampling by strided convolution and upsampling by factor 2.
- The encoder begins with a 7×7 stride-2 convolution and 3 × 3 stride-2 max-pooling, followed by residual encoder blocks.
- LinkNet uses ResNet18 as its encoder instead of larger VGG16 or ResNet101 encoders.The cited passage identifies VGG16 with 138 million parameters and ResNet101 with 45 million parameters.
- The decoder uses full convolution, with convolution operations described by kernel size, input maps, and output maps.
- Each encoder layer input is bypassed to its corresponding decoder output to recover spatial information lost during downsampling.
- Sharing encoder knowledge at every layer lets the decoder use fewer parameters, producing a more efficient network intended for real-time operation.
IV. RESULTS
LinkNet is compared with existing architectures for speed and accuracy, using operation count and forward-pass time alongside dataset performance on Cityscapes and CamVid.
- Performance Analysis: LinkNet’s speed is evaluated by the number of operations required for one forward pass and the time taken for that pass.
- The comparisons cover processing performance and segmentation accuracy as separate evaluation dimensions.
- Benchmarks: LinkNet’s accuracy is compared with existing architectures on the Cityscapes and CamVid datasets.
A. Performance Analysis
LinkNet is evaluated for inference efficiency across image resolutions and hardware platforms. The results emphasize real-time processing on an embedded NVIDIA TX1 while retaining high-resolution GPU throughput and reducing per-frame computation.
- 8.5 fps: LinkNet processes very high-resolution images on an NVIDIA GPU.The comparison uses single input frames at varying resolutions.
- Real-time performance: LinkNet operates on the NVIDIA TX1 embedded system.A dash denotes resolutions the embedded device could not process.
- 640 × 360: LinkNet’s default resolution for reporting operation counts.The authors use operation counts to assess forward-pass efficiency.
- Lower operations per frame and fewer overall parameters demonstrate LinkNet’s efficiency.The paper states that operation reduction is more vital than parameter reduction because operations determine forward-pass time.
B. Benchmarks
LinkNet is evaluated on Cityscapes and CamVid using IoU and iIoU, with comparisons against existing segmentation models. It outperforms existing models on Cityscapes and all compared methods on CamVid.
- Evaluation setup: IoU and iIoU: LinkNet uses these metrics instead of pixel-wise accuracy for evaluation.The metrics follow the Cityscapes evaluation practice.
- Cityscapes: Cityscapes: LinkNet outperforms existing models on the reported results.The dataset includes 5,000 fine-annotated images, with 2,975 for training, 500 for validation, and 1,525 for testing.
- Qualitative results: Cityscapes predictions and CamVid predictions are shown in Figures 4 and 5, respectively.The figures present segmented outputs on test images.
- CamVid: 367 training, 101 validation, and 233 testing images: CamVid provides the dataset split used for evaluation.The dataset contains eleven labeled classes; unlabeled data are ignored during training.
- CamVid: CamVid: LinkNet outperforms all compared algorithms in both IoU and iIoU on the test set.The comparison includes SegNet, ENet, Dilation8, LinkNet without bypass, and LinkNet.
V. CONCLUSION
LinkNet is designed for efficient semantic segmentation on resource-constrained embedded platforms while matching or exceeding existing baseline models. Its deployment on NVIDIA TX1 demonstrates a real-time portable solution, and its efficiency also extends to NVIDIA Titan X.
- Embedded platforms: LinkNet targets scarce computational and memory resources while matching or sometimes exceeding existing baseline models.The baselines are described as having an order of magnitude larger computational and memory requirements.
- NVIDIA TX1: LinkNet exemplifies a real-time portable embedded solution.The conclusion identifies the hardware application as evidence of this deployment scope.
- NVIDIA Titan X: LinkNet is also reported as very efficient on a high-end GPU.The authors suggest this may be useful for processing large numbers of high-resolution images in data centers.