Source-linked AI summary
Residual Conv-Deconv Grid Network for Semantic Segmentation
Damien Fourure, Rémi Emonet, Elisa Fromont, Damien Muselet, Alain Tremeau, Christian Wolf
TL;DR
Semantic segmentation needs full-resolution predictions, but conventional subsampling trades resolution for receptive-field context. GridNet addresses this with interconnected streams at different resolutions in a two-dimensional grid, achieving competitive Cityscapes results while generalizing several established architectures.
Problem
Subsampling reduces feature-map resolution, which is harmful when semantic segmentation must predict a class for every pixel.
Method
GridNet organizes residual horizontal streams and non-residual vertical downsampling or upsampling connections in a two-dimensional multi-resolution grid.
Results
GridNet achieves results comparable with state-of-the-art methods on Cityscapes under matched input-information conditions.
Takeaways & Limitations
The architecture generalizes conv-deconv, U-Net, residual, and FRRN-style models while supporting information flow across resolutions.
Takeaways & Limitations
An alternative concatenation design increased capacity but was much more difficult to train, especially when trained from scratch.
Abstract
from arXiv · showhide
This paper presents GridNet, a new Convolutional Neural Network (CNN) architecture for semantic image segmentation (full scene labelling). Classical neural networks are implemented as one stream from the input to the output with subsampling operators applied in the stream in order to reduce the feature maps size and to increase the receptive field for the final prediction. However, for semantic image segmentation, where the task consists in providing a semantic class to each pixel of an image, feature maps reduction is harmful because it leads to a resolution loss in the output prediction. To tackle this problem, our GridNet follows a grid pattern allowing multiple interconnected streams to work at different resolutions. We show that our network generalizes many well known networks such as conv-deconv, residual or U-Net networks. GridNet is trained from scratch and achieves competitive results on the Cityscapes dataset.
1 Introduction
GridNet is introduced as a multi-stream architecture for full scene labelling, combining high-resolution accuracy with low-resolution contextual information. It addresses resolution loss in semantic segmentation and is designed independently of radically modifying pretrained networks.
- Semantic segmentation assigns meaningful regions to every pixel, unlike image classification, which predicts a single class for an image.
- Fully convolutional networks produce 2D outputs and can process images of varying sizes.
- Conv-deconv networks restore input resolution after convolutional subsampling, while dilated convolutions enlarge receptive fields without subsampling.
- Residual connections ease gradient back-propagation and are often faster to train than classical neural networks.
- Pretrained network structures cannot be changed radically, limiting their suitability for new architectures such as GridNet.
- GridNet uses interconnected streams at different resolutions, combining high-resolution accuracy with low-resolution receptive-field context.
- The paper reviews related architectures, presents GridNet’s components, and evaluates the approach on Cityscapes.
2 Related Work
Prior segmentation architectures manage the trade-off between receptive-field context and resolution through deconvolution, dilation, residual connections, or multiple paths. GridNet instead uses a multi-resolution grid that supports decisions across resolutions and generalizes several existing designs.
- Subsampling enlarges receptive fields but harms semantic-segmentation output resolution by reducing feature-map sizes.
- FCN-style deconvolution and U-Net skip connections restore or reconstruct segmentation resolution after convolutional downsampling.
- GridNet uses downsampling and upsampling as connectors between streams, enabling multi-scale decisions rather than merely correcting resolution loss.
- Dilated convolutions avoid pooling while enlarging the receptive field for semantic labelling.
- GridNet incorporates residual units, drawing on prior analyses of residual-connection strategies.
- FRRN combines conv-deconv and residual networks with two streams, whereas GridNet generalizes this arrangement.
- GridNet’s figure distinguishes residual resolution-preserving units from convolutional downsampling and deconvolutional upsampling units.
- Compared with Convolutional Neural Fabrics, GridNet uses residual units, introduces total dropout, and constrains downsampling to precede upsampling.
3 GridNet
GridNet organizes residual streams at multiple resolutions in a two-dimensional grid, connecting them with subsampling and upsampling layers so information can follow parallel paths. Its topology generalizes established resolution-preserving models, while total dropout addresses the training difficulty of longer low-resolution paths.
- Grid organization: Each feature map Xi,j lies at a grid line and column, with information flowing from the first block of line 0 to the final output through multiple paths.Paths may remain on line 0 or involve other resolution streams.
- Grid organization: Horizontal streams preserve feature-map size and use residual computations, whereas vertical layers change resolution through subsampling or upsampling.Vertical connections are non-residual.
- Resolution changes: Subsampling halves width and height while doubling feature maps, and upsampling returns features toward the original resolution for cross-stream addition.The stated subsampling mapping is (Fi × Wi × Hi) to (2Fi × Wi/2 × Hi/2).
- GridBlock computation: GridBlock feature maps combine horizontal residual computation with vertical computation, using summation to keep capacity low and blend with the residual design.Vertical computation uses the preceding or following grid line depending on whether the column subsamples or upsamples.
- Model generality: GridNet generalizes conv-deconv, U-net, and FRRN models by removing or retaining selected grid connections.Blue connections yield conv-deconv networks, green connections yield U-networks, and yellow connections represent FRRN structure.
- Blockwise dropout: Because high-resolution paths are shorter, low-resolution paths are harder to train; total dropout randomly drops residual streams to encourage using all streams.The corresponding residual mappings are set to zero during stream dropping.
Experimental results
GridNet was evaluated on Cityscapes using a five-stream, six-column configuration and several architectural variants. The experiments examine design choices and compare the resulting network with published methods under matched input conditions.
- Dataset and metrics: Cityscapes evaluation used 1024 × 2048 images, 2,975 training images, 500 test images, and 19 evaluated semantic classes.Performance was reported with class- and category-level IoU and instance-level iIoU measures.
- Model configuration: The tested GridNet used five streams with feature-map dimensions 16, 32, 64, 128, and 256, followed by three subsampling and three upsampling columns.This configuration was selected as a tradeoff between memory consumption, parameter count, and modelling capacity.
- Ablations: Total dropout significantly improved accuracy, whereas full residuality, concatenation-based fusion, and greater network capacity were not advantageous.The experiments also report that total dropout did not resolve learning difficulties and negatively affected training stability.
- Ablations: The study varied the number of columns and streams, training networks to convergence and testing them without data augmentation or multiple scales.Table 2 evaluates how these architectural dimensions affect performance under the stated testing setup.
- Benchmark comparison: GridNet produced results comparable to state-of-the-art Cityscapes networks, particularly FRNN, when compared using the same input information.The comparison excludes methods using coarse annotations or stereo inputs.
- Benchmark comparison: Only RefineNet among nine other reported Cityscapes results performed slightly better than GridNet, while those other results used ImageNet-pretrained classification networks.GridNet itself was evaluated without those classification pretraining conditions.
5 Conclusion
The paper introduces GridNet as a semantic-segmentation architecture that generalizes several existing network families. Its grid connects resolution-preserving residual paths with down- and up-sampling paths, and it achieves promising results when trained from scratch.
- Conclusion: GridNet is a novel architecture specifically designed for semantic segmentation and generalizes conv-deconv, U-Net, and Full Resolution Residual Networks.The paper presents the architecture as a common structure encompassing these existing models.
- Conclusion: A two-dimensional grid lets information flow horizontally through residual resolution-preserving paths and vertically through down- and up-sampling layers.The architecture combines computations across multiple resolutions.
- Conclusion: GridNet shows promising results when trained from scratch without pre-training.The authors suggest that pre-training on ADE20K could further improve weight initialization.