PaperlayerView exact revision on arXiv

Source-linked AI summary

LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation

Abhishek Chaurasia, Eugenio Culurciello

arXiv:1707.03718v1cs.CVcs.LG

TL;DR

Semantic segmentation needs accuracy and speed for real-time use, yet existing methods have given little attention to real-time segmentation. LinkNet bypasses encoder spatial information to corresponding decoder layers, and the authors report that it matches or sometimes exceeds baselines with an order of magnitude greater computational and memory requirements.

  • Problem

    Existing semantic-segmentation methods have given little attention to real-time operation, despite the task’s real-time applications.

    Source: §I · p. 1

  • Method

    LinkNet bypasses each encoder layer’s input to its corresponding decoder output to help recover spatial information lost during downsampling.

    Source: §III · p. 3

  • Results

    LinkNet matches and sometimes exceeds baseline models whose computational and memory requirements are an order of magnitude larger.

    Source: §V · p. 5

  • Takeaways & Limitations

    The NVIDIA TX1 application exemplifies LinkNet as a real-time portable embedded solution for semantic segmentation.

    Source: §V · p. 5

Abstract

from arXiv · show

Pixel-wise semantic segmentation for visual scene understanding not only needs to be accurate, but also efficient in order to find any use in real-time application. Existing algorithms even though are accurate but they do not focus on utilizing the parameters of neural network efficiently. As a result they are huge in terms of parameters and number of operations; hence slow too. In this paper, we propose a novel deep neural network architecture which allows it to learn without any significant increase in number of parameters. Our network uses only 11.5 million parameters and 21.2 GFLOPs for processing an image of resolution 3x640x360. It gives state-of-the-art performance on CamVid and comparable results on Cityscapes dataset. We also compare our networks processing time on NVIDIA GPU and embedded system device with existing state-of-the-art architectures for different image resolutions.

I. INTRODUCTION

LinkNet aims to make semantic segmentation accurate without sacrificing processing speed by passing spatial information directly from encoder stages to corresponding decoder stages, avoiding the need to relearn lost information.

I. INTRODUCTION

  • Semantic segmentation is pixel-level classification and a key step in scene understanding, an area receiving attention amid growth in augmented reality and self-driving-vehicle tasks.The introduction situates this interest alongside CNN successes in classification, localization, and scene understanding,, [5], [6],, [8], [9],,,.Source: §I · p. 1
  • Most semantic-segmentation networks use encoder-decoder architectures, yet many require excessive processing time despite real-time application needs; real-time detection methods include YOLO, Fast RCNN, and SSD.The introduction describes little to no comparable focus on real-time semantic segmentation [19].Source: §I · p. 1
  • Directly forwarding encoder spatial information to corresponding decoder stages is proposed to improve accuracy while reducing processing time without adding parameters or operations to relearn lost information.The authors state that spatial information otherwise lost through pooling or strided convolution can be preserved through these direct connections.Source: §I · p. 1
  • The proposed network was tested on Cityscapes and CamVid, with processing times recorded on an NVIDIA Jetson TX1 embedded system and Titan X GPU.The introduction points to Section III for the architecture and Section IV for the reported results.Source: §I · p. 1

II. RELATED WORK

Related work frames semantic segmentation as a tradeoff between preserving pixel-level spatial information and designing encoder–decoder architectures for accurate, efficient scene parsing. Prior approaches vary in their use of pooling indices, convolutional upsampling, contextual modeling, and speed–accuracy tradeoffs.

II. RELATED WORK

  • Segmentation architectures pair an encoder discriminator with a decoder generator, which may reuse pooling indices or learn convolutional upsampling and may be symmetric or asymmetric.Retaining spatial information is especially important because segmentation labels every image pixel.Source: §II · p. 1
  • Prior segmentation methods included pre-trained VGG as a discriminator with saved pooling indices for decoder upsampling [22], deep deconvolution networks, [24], FCNs with skip architecture, and classification encoders AlexNet, VGG [5], and GoogLeNet.Source: §II · p. 1
  • Dense-prediction networks [27] often failed to run in real time on embedded hardware, while RNN-based context and CRF optimization were computationally expensive.Efficient DCNN designs [30], [19] sped up forward processing at the cost of prediction accuracy.Source: §II · p. 2

III. NETWORK ARCHITECTURE

LinkNet pairs a ResNet18 encoder with a decoder that receives bypassed inputs from corresponding encoder layers, aiming to recover spatial information while using fewer decoder parameters.

III. NETWORK ARCHITECTURE

  • The encoder begins with a 7×7, stride-2 convolution and 3 × 3, stride-2 max-pooling, then uses residual blocks [6].The network’s left half is the encoder and its right half is the decoder.Source: §III · p. 2
  • LinkNet uses ResNet18 as a lighter encoder than VGG16 (138 million parameters) or ResNet101 (45 million parameters), which contemporary segmentation algorithms use.The passage says LinkNet still outperforms those encoders, referring to Section IV.Source: §III · p. 2
  • The decoder uses full-convolution operations as proposed earlier, with batch normalization between convolutional layers followed by ReLU, [32].Figure 1 presents the overall architecture; Figures 2 and 3 show encoder- and decoder-block convolutional modules, respectively.Source: §III · pp. 2–3
  • LinkNet bypasses each encoder layer’s input to its corresponding decoder output to help recover spatial information lost through encoder downsampling.Sharing encoder-learned information at each layer also lets the decoder use fewer parameters.Source: §III · p. 3

IV. RESULTS

The results section compares LinkNet with existing architectures on speed and accuracy. Speed is assessed by forward-pass operations and time, while accuracy is evaluated on Cityscapes and CamVid.

IV. RESULTS

  • The evaluation compares LinkNet with existing architectures across speed and accuracy.Speed is assessed using operations per forward pass and forward-pass time; accuracy is evaluated on Cityscapes and CamVid.Source: §IV · p. 3
  • Speed is measured by the number of operations required for one forward pass.The section also compares time taken for one forward pass.Source: §IV · p. 3
  • Accuracy is evaluated on Cityscapes and CamVid.Source: §IV · p. 3

A. Performance Analysis

LinkNet combines high-resolution GPU throughput with real-time inference on an embedded device, while requiring comparatively few operations and parameters at its default resolution.

A. Performance Analysis

  • 8.5 fps on GPU at high image resolution, with real-time performance also reported on NVIDIA TX1.Inference time was compared across varying input resolutions on NVIDIA TitanX and TX1; a dash indicates a resolution the embedded device could not process.Source: A. Performance Analysis · p. 3
  • At the default 640 × 360 resolution, LinkNet requires a much lower number of operations per frame and fewer overall parameters.The paper argues that reducing operations is more vital than reducing parameters because operations determine forward-pass time.Source: A. Performance Analysis · p. 3

B. Benchmarks

LinkNet outperforms existing models on Cityscapes and CamVid in the reported benchmarks. The evaluation uses IoU and iIoU, including comparisons with and without bypass connections.

Evaluation and training

  • The evaluation uses IoU and instance-level IoU (iIoU), reports results with and without bypass connections, and compares LinkNet with SegNet, ENet, Dilation8/10, and Deep-Lab CRF.Training used Torch7 with RMSProp on four NVIDIA TitanX GPUs; a custom class-weighting scheme performed better than mean average frequency.Source: B. Benchmarks · p. 3

a) Cityscapes

  • Cityscapes has 2,975 training, 500 validation, and 1,525 test images among its 5,000 fine-annotated images.Training used 1024 × 512 inputs, batch size 10, and an initial learning rate of 5e−4.Source: B. Benchmarks · pp. 3–4
  • LinkNet outperforms existing models on the Cityscapes validation results.The network was trained on the 19 classes specified by the official evaluation scripts.Source: Table IV · p. 3, B. Benchmarks · p. 3

b) CamVid

  • LinkNet outperforms SegNet, ENet, and Dilation8 on the CamVid test set in both IoU and iIoU.The comparison also includes LinkNet without bypass connections.Source: Table V · p. 4, B. Benchmarks · p. 4
  • CamVid contains 367 training, 101 validation, and 233 test images, with 11 semantic classes and an unlabeled class excluded during training.Images were downsampled from 960 × 720 by a factor of 1.25, and hardware constraints limited training batch size to 8.Source: B. Benchmarks · p. 4

V. CONCLUSION

The proposed segmentation architecture targets efficient use of scarce embedded resources while matching or sometimes exceeding larger baselines. Its efficiency also extends to high-end GPUs, with potential for data-center processing.

V. CONCLUSION

  • The architecture matches or sometimes exceeds baseline models with an order of magnitude more computational and memory requirements, and its NVIDIA TX1 deployment exemplifies real-time portable embedded solutions.It was designed specifically for semantic segmentation and efficient use of scarce embedded-platform resources.Source: §V · p. 5
  • The network is also efficient on NVIDIA Titan X, potentially benefiting data-center applications that process large numbers of high-resolution images.The authors suggest it could enable faster, more efficient large-scale computations and significant savings.Source: §V · p. 5

Related papers

Loading 1707.03718v1…