Source-linked AI summary
RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
Guosheng Lin, Anton Milan, Chunhua Shen, Ian Reid
TL;DR
Deep CNN subsampling reduces image resolution, leaving effective use of multi-level features for high-resolution semantic segmentation as an open question. RefineNet addresses this with multi-path refinement and long-range residual connections, outperforming prior work on seven public benchmarks, including an IoU of 83.4 on PASCAL VOC 2012.
Problem
Repeated subsampling in deep CNNs reduces feature-map resolution, while effectively exploiting features across abstraction levels for high-resolution segmentation remains an open question.
Method
RefineNet uses multiple resolution paths and long-range residual connections to recursively refine coarse semantic features with fine-grained features for high-resolution segmentation.
Results
83.4 IoU on PASCAL VOC 2012, with RefineNet outperforming DeepLab and achieving state-of-the-art performance on seven public datasets.
Takeaways & Limitations
RefineNet combines high-level semantics and low-level detail for high-resolution segmentation while supporting effective end-to-end learning.
Takeaways & Limitations
Deep CNN-based segmentation methods face a trade-off between feature-map resolution and channel capacity caused by sequential subsampling.
Abstract
from arXiv · showhide
Recently, very deep convolutional neural networks (CNNs) have shown outstanding performance in object recognition and have also been the first choice for dense classification problems such as semantic segmentation. However, repeated subsampling operations like pooling or convolution striding in deep CNNs lead to a significant decrease in the initial image resolution. Here, we present RefineNet, a generic multi-path refinement network that explicitly exploits all the information available along the down-sampling process to enable high-resolution prediction using long-range residual connections. In this way, the deeper layers that capture high-level semantic features can be directly refined using fine-grained features from earlier convolutions. The individual components of RefineNet employ residual connections following the identity mapping mindset, which allows for effective end-to-end training. Further, we introduce chained residual pooling, which captures rich background context in an efficient manner. We carry out comprehensive experiments and set new state-of-the-art results on seven public datasets. In particular, we achieve an intersection-over-union score of 83.4 on the challenging PASCAL VOC 2012 dataset, which is the best reported result to date.
1. Introduction
RefineNet addresses the loss of fine image structure in CNN-based semantic segmentation by recursively refining coarse semantic features with multi-level visual features. Its residual connections and chained residual pooling support end-to-end training and large-region context capture, yielding state-of-the-art results across seven public datasets.
- Motivation: CNN pooling and strided convolutions typically reduce predictions by a factor of 32 in each dimension, losing finer image structure.This resolution loss limits high-resolution semantic segmentation.
- RefineNet Architecture: RefineNet exploits features at multiple abstraction levels, recursively refining coarse semantic features with fine-grained low-level features to produce high-resolution maps.High-level features support region-category recognition, while low-level features preserve sharp, detailed boundaries.
- Training: Identity-mapping residual connections propagate gradients through short- and long-range paths, enabling effective end-to-end training of cascaded RefineNets.The architecture can be cascaded and modified in various ways.
- Network Components: Chained residual pooling captures background context from a large image region by pooling with multiple window sizes and fusing features using residual connections and learnable weights.The component is designed to capture context efficiently.
- Results: 83.4 IoU is achieved on PASCAL VOC 2012, while RefineNet reaches state-of-the-art performance on 7 public datasets.The reported datasets include PASCAL VOC 2012, PASCAL-Context, NYUDv2, SUN-RGBD, Cityscapes, ADE20K, and Person-Parts.
2. Background
The background frames semantic segmentation as dense prediction with deep CNNs, whose sequential subsampling reduces feature-map resolution while increasing contextual receptive fields. It contrasts dilated convolutions with the paper’s approach, which aims to retain deresolution’s memory and computational benefits while producing effective high-resolution predictions.
- Deep CNNs: ResNet is adopted as the fundamental building block for semantic segmentation because of its strong object-recognition performance and publicly available ImageNet-pretrained models.The authors note that replacing ResNet with another deep network is straightforward.
- Dense prediction: Semantic segmentation modifies ResNet by replacing single-label prediction with a dense layer that outputs class confidence at every pixel.During the forward pass, feature-map resolution decreases while feature depth increases.
- Resolution loss: Sequential subsampling halves feature-map resolution between blocks while increasing deeper layers’ receptive fields for more global and contextual information.The authors identify this resolution loss as a well-known limitation of deep CNN-based segmentation methods.
- Dilated convolution: Dilated convolution removes subsampling by changing the stride from 2 to 1 and uses dilated convolutions after the first block to retain a large receptive field.The passage describes this method as having state-of-the-art semantic-segmentation performance.
- Motivation: RefineNet seeks both the memory and computational benefits of deresolving and effective, efficient high-resolution segmentation prediction.This objective is presented in contrast to dilated-convolution methods.
3. Proposed Method
RefineNet is a generic multi-path framework that fuses coarse semantic features with fine-grained features through long-range residual connections for high-resolution semantic segmentation. Its residual-based building blocks support gradient propagation and end-to-end training while incorporating multi-resolution fusion and chained residual pooling for background context.
- Multi-path refinement: RefineNet fuses coarse high-level semantic features with finer-grained low-level features through multiple resolution paths to produce high-resolution feature maps.The architecture uses cascaded RefineNet units connected to ResNet outputs and preceding RefineNet blocks.
- Architectural flexibility: The architecture is flexible: RefineNet blocks can accept arbitrary numbers of feature maps with arbitrary resolutions and depths, and multiple cascade variants can be explored.The standard design uses four cascaded units, while two-cascade, single-block, and two-scale seven-path variants are also considered.
- RefineNet block components: Each RefineNet block performs residual convolution, multi-resolution fusion, chained residual pooling, and output convolution operations.Inputs are adapted, up-sampled to the largest resolution, and fused by summation; chained pooling combines multiple window sizes to capture broad background context.
- Residual connections: Long-range residual connections convey low-level visual details during forward propagation and directly propagate gradients to early convolution layers during training.This supports efficient end-to-end training of the full cascaded multi-path network.
- Residual connections: Residual identity-mapping connections are used both locally within RefineNet components and globally between RefineNet modules and ResNet blocks to facilitate effective backward propagation.Short-range connections occur in residual convolution and pooling components, while long-range connections link the modules to ResNet blocks.
4. Experiments
RefineNet is evaluated on seven public datasets using standard segmentation metrics, augmentation, and multi-scale testing. It outperforms prior methods across the reported benchmarks, including an IoU of 83.4 on PASCAL VOC 2012, while ablations show benefits from deeper networks, chained residual pooling, and four-stage cascading.
- Evaluation setup: Training uses random scaling from 0.7 to 1.3, random cropping, and horizontal flipping, while unspecified evaluations average predictions across multiple image scales.Multi-scale evaluation is applied at test time unless otherwise specified.
- Ablation studies: Ablations consistently show that chained residual pooling and deeper networks improve IoU on NYUDv2 and Person-Part.The Person-Part experiments compare RefineNet with state-of-the-art methods, including DeepLab-v2, using the same ResNet initialization.
- Semantic segmentation results: RefineNet outperforms previous methods on all six dense semantic-labeling benchmarks, including NYUDv2, Cityscapes, PASCAL-Context, SUN-RGBD, and ADE20K MIT.On SUN-RGBD, it leads across all evaluation metrics without using depth information; on PASCAL-Context, it also uses no additional training data beyond VOC images.
- Architecture variants: The 4-cascaded RefineNet performs better than 2-cascaded and single-RefineNet variants, while 2-scale input improves over 1-scale input but increases training time.The single-scale 4-cascaded version is therefore selected as the standard architecture.
5. Conclusion
RefineNet is a multi-path refinement network that combines high-level semantics with low-level features for high-resolution segmentation and object parsing. Its identity-mapping-inspired design supports gradient propagation across long-range connections and effective end-to-end learning, outperforming prior work on seven public benchmarks.
- Conclusion: RefineNet combines high-level semantics and low-level features to produce high-resolution segmentation maps for semantic segmentation and object parsing.The model uses a cascaded multi-path refinement architecture.
- Conclusion: Identity-mapping-inspired residual connections facilitate gradient propagation across long-range connections and enable effective end-to-end learning.This design is a central architectural choice of RefineNet.
- Conclusion: RefineNet outperforms all previous works on seven public benchmarks.The conclusion summarizes the network’s comparative performance across seven datasets.
- Conclusion: The RefineNet block handles different numbers of inputs with arbitrary resolutions and dimensions without modification.This capability is illustrated across single, 2-cascaded, and 4-cascaded network variants.