Source-linked AI summary
Light-Weight RefineNet for Real-Time Semantic Segmentation
Vladimir Nekrasov, Chunhua Shen, Ian Reid
TL;DR
The paper addresses the computational cost of semantic segmentation models used for real-time, high-resolution processing. It adapts RefineNet through architectural simplifications and achieves substantially higher speed with nearly preserved performance, including lightweight-backbone configurations.
Problem
Real-time semantic segmentation remains difficult because high-resolution applications are hindered by expensive computational requirements.
Method
The method simplifies RefineNet by replacing expensive convolutional blocks, removing redundant residual units, and combining the result with different classification backbones.
Results
55 FPS at 512 × 512 with 81.1% mean iou on PASCAL VOC is achieved by the fastest ResNet model, while the best-performing model reaches 82.7% mean iou at 32 FPS.
Takeaways & Limitations
The simplified RefineNet closely matches the original network’s performance while substantially improving runtime and supporting lightweight classification backbones.
Takeaways & Limitations
The compression methods discussed require an initial powerful, large pretrained model for the task.
Abstract
from arXiv · showhide
We consider an important task of effective and efficient semantic image segmentation. In particular, we adapt a powerful semantic segmentation architecture, called RefineNet, into the more compact one, suitable even for tasks requiring real-time performance on high-resolution inputs. To this end, we identify computationally expensive blocks in the original setup, and propose two modifications aimed to decrease the number of parameters and floating point operations. By doing that, we achieve more than twofold model reduction, while keeping the performance levels almost intact. Our fastest model undergoes a significant speed-up boost from 20 FPS to 55 FPS on a generic GPU card on 512x512 inputs with solid 81.1% mean iou performance on the test set of PASCAL VOC, while our slowest model with 32 FPS (from original 17 FPS) shows 82.7% mean iou on the same dataset. Alternatively, we showcase that our approach is easily mixable with light-weight classification networks: we attain 79.2% mean iou on PASCAL VOC using a model that contains only 3.3M parameters and performs only 9.3B floating point operations.
1 Introduction
The paper targets real-time semantic segmentation, where high computational costs hinder high-resolution applications. It adapts RefineNet to reduce computation while preserving segmentation quality across benchmarks.
- High-resolution semantic segmentation applications often require real-time processing, which is difficult for modern computationally expensive networks.
- The method is designed to work with different backbone classification networks, including ResNet, DenseNet, NASNet, and MobileNet variants.
- The approach reduces RefineNet parameters by more than 50% and removes redundant residual blocks without changing performance.The experiments cover three segmentation benchmarks and five backbone networks.
- 55 FPS at 512 × 512 with 81.1% mean iou on PASCAL VOC demonstrates the fastest ResNet model’s real-time performance.The same model achieves 41.7% mean iou on NYUDv2 and 64.9% mean iou on Person-Part.
- The released models are intended to facilitate the use of efficient semantic segmentation in multiple applications.
2 Related work
Prior semantic segmentation methods often require substantial parameters or computation, limiting real-time use. Existing compression and lightweight-backbone approaches motivate a RefineNet-based alternative that can combine architectural simplification with backbone efficiency.
- Encoder-decoder segmentation architectures progressively downsample and upsample images to produce masks matching the input resolution.
- Real-time segmentation is constrained because many existing methods have large parameter counts, high floating-point operation costs, or both.
- Task-specific methods can achieve real-time speed, but their performance may not generalize across datasets with different resolutions or properties.
- Compression methods such as quantisation, pruning, distillation, and low-rank factorisation can substantially reduce model size but typically require a large pretrained model first.
- The paper combines its approach with lightweight classifiers such as NASNet-Mobile and MobileNet-v2 to obtain competitive results with few parameters and floating-point operations.
3 Light-Weight RefineNet
Light-Weight RefineNet reduces RefineNet’s computational burden by replacing expensive convolutions and removing redundant residual units. The resulting architecture uses lightweight CRP blocks and remains compatible with varied classification backbones.
- 3.1 RefineNet Primer: RefineNet’s decoder uses RCU and CRP abstractions, with CRP combining convolutional and pooling layers in a residual arrangement.
- 3 Light-Weight RefineNet: The proposed simplifications target ubiquitous 3 × 3 convolutions, identified as the most expensive RefineNet components in parameters and floating-point operations.
- 3.2 Replacing 3x3 convolutions: 1 × 1 convolutions preserve performance in RefineNet while avoiding the use of 3 × 3 convolutions except in the final classification layer.
- 3.3 Omitting RCU blocks: Removing RCU blocks does not reduce accuracy in the lightweight architecture, whereas removing them from original RefineNet causes more than a 5% performance drop.
- 3 Light-Weight RefineNet: Replacing 3 × 3 convolutions with 1 × 1 convolutions and removing RCU blocks produces a fast architecture based on CRP blocks with 5 × 5 max-pooling.
- 3.4 Adaptation to different backbones: RefineNet can be combined with any backbone containing several subsampling operations, including efficient NASNet-Mobile and MobileNet-v2.
4 Experiments
Experiments evaluate Light-Weight RefineNet across three segmentation benchmarks, five backbone networks, and efficiency metrics. The models match or outperform relevant baselines while substantially improving runtime and reducing computational cost.
- NYUDv2: Across ResNet backbones, performance is on-par with original RefineNet while using only a slight portion of its parameters and achieving a significant twofold speedup on NYUDv2.
- Person-Part: On PASCAL Person-Part, the proposed models achieve results analogous to the original models.
- PASCAL VOC: On PASCAL VOC, the models match original RefineNet for ResNet networks and outperform MobileNet-v1+DeepLab-v3 and MobileNet-v2+DeepLab-v3 for lightweight backbones.
5 Discussion
The discussion examines why removing expensive components preserves performance, focusing on receptive fields and the relative roles of CRP and RCU blocks. Empirical analyses indicate that CRP and skip summation preserve contextual coverage, while RCU contributes only marginally to accuracy.
- Receptive field: The receptive field remains effective after replacing 3 × 3 convolutions because RefineNet sums low-level and high-level features while retaining CRP blocks for contextual information.
- Receptive field: CRP enlarges the empirical receptive field, while summation with lower-layer features produces significantly larger activation contours.Before CRP, activations concentrate on small object parts and barely cover the objects.
- Representational power: Ablations compare heads placed before RCU, after RCU, and after CRP using fixed network weights and VOC validation data.
- Representational power: CRP is the main driving force behind accurate segmentation and classification, whereas RCU improves results only marginally.
6 Conclusions
The paper rethinks RefineNet for real-time semantic segmentation by simplifying redundant components while largely preserving performance. Its conclusions emphasize faster inference, applicability across backbones and datasets, and the potential redundancy of large decoder kernels.
- Conclusions: 55 FPS on 512×512 inputs is achieved after increasing runtime from an initial 20 FPS while closely matching the original network’s performance.
- Conclusions: The method applies with any classification network and dataset and can benefit from lightweight backbones and other compression approaches.
- Conclusions: The work demonstrates that large-kernel convolutions can be unnecessary in the decoder part of segmentation networks.
7 Experiments
Additional experiments evaluate the approach on PASCAL Context and CityScapes, with quantitative results reported for both datasets and qualitative results shown for several benchmarks.
- PASCAL-Context: Quantitative PASCAL-Context results are reported in Table 5, with multi-scale evaluation identified as msc.The supplied passages do not provide the table's cell values.
- CityScapes: 72.1% mean iou is achieved on the CityScapes test set with a single-scale ResNet-101 model, compared with 73.6% for original RefineNet using multi-scale evaluation.CityScapes contains 5000 high-resolution 1024×2048 images and 19 semantic classes.
8 Receptive field size
The receptive-field analysis explains why removing 3×3 convolutions preserves performance: RefineNet’s skip connections and CRP blocks maintain broad contextual behavior.
- Design explanation: Removing 3×3 convolutions does not significantly reduce receptive-field size because RefineNet sums low-level with high-level features and retains CRP blocks.These two architectural properties are identified as the reasons the expected receptive-field harm is not observed.
- Empirical comparison: Similar empirical receptive-field activation contours are observed for RefineNet-101 and RefineNet-LW-101, although the original produces less jagged boundaries.The comparison uses the last classification layer and models pretrained on PASCAL VOC.
- Visual validation: Visual validation results are shown for PASCAL-Context, CityScapes, PASCAL Person-Part, and NYUDv2 using residual models.The supplied figure captions identify the datasets and residual-model setting but do not report quantitative comparisons.