Source-linked AI summary
ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation
Francesco Visin, Marco Ciccone, Adriana Romero, Kyle Kastner, Kyunghyun Cho, Yoshua Bengio, Matteo Matteucci, Aaron Courville
TL;DR
Semantic segmentation requires local pixel detail and long-range context, which conventional CNN and FCN approaches do not jointly preserve. ReSeg extends ReNet with recurrent spatial sweeps over pretrained convolutional features and restores resolution through upsampling. It achieves state-of-the-art performance on Weizmann Horse, Oxford Flowers, and CamVid, although spatially recurrent training remains computationally intensive and one reported enhancement was not tested.
Problem
CNN-based segmentation methods reduce spatial resolution, while FCN and CRF approaches do not fully exploit long-range contextual dependencies.
Method
ReSeg extends ReNet with four-directional recurrent sweeps over pretrained VGG-16 features, followed by upsampling and per-pixel prediction.
Results
ReSeg achieves state-of-the-art performance on the Weizmann Horse, Oxford Flowers, and CamVid segmentation datasets.
Takeaways & Limitations
A few VGG-16 layers are sufficient for ReSeg to handle semantic segmentation while encoding contextual and long-term dependencies.
Takeaways & Limitations
Spatially recurrent neural-network training is computationally intensive, and the benefit of sub-model averaging for ReSeg remains untested.
Abstract
from arXiv · showhide
We propose a structured prediction architecture, which exploits the local generic features extracted by Convolutional Neural Networks and the capacity of Recurrent Neural Networks (RNN) to retrieve distant dependencies. The proposed architecture, called ReSeg, is based on the recently introduced ReNet model for image classification. We modify and extend it to perform the more challenging task of semantic segmentation. Each ReNet layer is composed of four RNN that sweep the image horizontally and vertically in both directions, encoding patches or activations, and providing relevant global information. Moreover, ReNet layers are stacked on top of pre-trained convolutional layers, benefiting from generic local features. Upsampling layers follow ReNet layers to recover the original image resolution in the final predictions. The proposed ReSeg architecture is efficient, flexible and suitable for a variety of semantic segmentation tasks. We evaluate ReSeg on several widely-used semantic segmentation datasets: Weizmann Horse, Oxford Flower, and CamVid; achieving state-of-the-art performance. Results show that ReSeg can act as a suitable architecture for semantic segmentation tasks, and may have further applications in other structured prediction problems. The source code and model hyperparameters are available on https://github.com/fvisin/reseg.
1. Introduction
Semantic segmentation needs both fine-grained pixel resolution and contextual dependencies, but standard CNN and FCN approaches lose resolution and underuse long-range context. ReSeg extends ReNet with an efficient recurrent architecture stacked on convolutional features to address this gap.
- Deep CNNs provide generic visual representations but reduce input resolution through pooling or subsampling, complicating per-pixel semantic segmentation.These operations support invariance but make dense pixel-level prediction difficult.
- FCN-based methods upsample coarse representations but do not jointly preserve local and global contextual dependencies.CRF post-processing locally smooths predictions, while long-range context remains relatively unexploited.
- Spatially recurrent neural networks can retrieve global spatial dependencies for segmentation, but their training is computationally intensive.
- ReSeg extends ReNet for semantic segmentation by combining efficient horizontal and vertical RNN sweeps with convolutional features from VGG-16.The model was evaluated on Weizmann Horse, Oxford Flower, and CamVid, achieving state-of-the-art performance on all three datasets.
2. Related Work
Prior segmentation methods recover resolution through multi-scale processing, interpolation, or learned upsampling, while recurrent models capture long-distance pixel dependencies. ReSeg-related work combines these contextual and resolution-recovery strategies, with ReNet offering parallelizable spatial sequences.
- FCN-based approaches recover segmentation resolution using multi-scale architectures, information from multiple layers, bilinear interpolation, or unpooling.
- RNN and RNN-like segmentation models capture long-distance pixel dependencies through semantic feedback, multidimensional LSTM blocks, and sweeps along image directions.
- Subsequent independent work combined ReSeg with FCN and CRFs and reported state-of-the-art results on Pascal VOC.
- A ReNet layer applies paired vertical and horizontal sweeps, concatenating their feature maps into the layer output.The figure encodes downward, upward, rightward, and leftward processing with distinct dot colors.
- ReNet spatial sequences are readily parallelizable because each RNN depends only along one horizontal or vertical pixel sequence.All rows or columns can therefore be processed simultaneously.
3. Model Description
ReSeg processes convolutional feature maps with recurrent sweeps that encode spatial context, then uses upsampling and softmax to restore pixel-level predictions. Its recurrent and upsampling components support flexible architectural choices while preserving input topology.
- The ReSeg pipeline applies early, non-fine-tuned VGG-16 layers, one or more ReNet layers, upsampling layers, and a per-pixel softmax.The VGG-16 feature maps retain sufficient resolution before recurrent processing, and upsampling restores the input resolution.
- Recurrent layer: Each recurrent layer combines four RNNs that sweep vertically and horizontally in both directions to capture local and global spatial structure.The layer can use vanilla tanh RNN, GRU, or LSTM units; the authors choose GRUs for memory and computational balance.
- Recurrent layer: The input is divided into non-overlapping patches, processed top-down and bottom-up by independent vertical RNNs, then represented as a concatenated contextual feature map.Each column can be processed in parallel, and each location encodes information relative to patches in its input column.
- Recurrent layer: A second pair of RNNs sweeps the rows of the vertically processed feature map, producing horizontally contextualized activations.The horizontal sublayer preserves the first sublayer's granularity when patches are not split further.
- Recurrent layer: Multiple recurrent layers can be concatenated and trained with gradient-descent optimization because the composite model is smooth and continuous.
- Upsampling layer: Non-overlapping patches reduce feature-map resolution, so predictions must be expanded before applying softmax to match the ground-truth resolution.
- Upsampling layer: Transposed convolutions are selected for upsampling because they preserve input topology while providing memory- and computation-efficient resolution recovery.The operation applies the transpose of the direct-convolution transformation, inverting the corresponding input and output shapes.
4. Experiments
The experiments evaluate ReSeg across binary and multiclass segmentation datasets, varying architectural and training choices. Results show benefits from pretrained VGG features and state-of-the-art performance on Oxford Flowers and CamVid, while one proposed comparison remains untested.
- Datasets and evaluation: The evaluation covers Weizmann Horse and Oxford Flowers binary segmentation, followed by the more challenging multiclass CamVid task.CamVid uses 11 categories from 480 × 360 images after downsampling, with 367 training, 101 validation, and 233 test images.
- Architectural settings: The network uses pretrained VGG-16 convolutional preprocessing, stacked ReNet layers, transposed-convolution upsampling, and a final softmax.The illustrated configuration applies recurrent sweeps to image patches, concatenates feature maps, stacks two ReNet layers, and then upsamples.
- Architectural settings: The ReSeg experiments vary the number of ReNet and upsampling layers, their feature counts, and their patch or filter sizes.The model also permits different recurrent implementations and initialization schemes, creating a potentially large hyperparameter space.
- Results: Pretrained VGG-16 features increased average Intersection over Union on Weizmann Horse when the first seven convolutional layers were used.The authors used these layers for low-level generic features while learning task-specific high-level features with ReNet layers.
- Results: Oxford Flowers results clearly outperform the state of the art in both global accuracy and average IoU, while CamVid shows a 17% IoU increase over SegNet.The CamVid comparison includes standard segmentation and neural-network methods; sub-model averaging for ReSeg was not tested.
5. Discussion
The discussion examines preprocessing and class balancing choices across datasets. VGG-16 preprocessing improves Weizmann Horse results, while median frequency balancing targets under-represented CamVid classes.
- Preprocessing: VGG-16 preprocessing improves results on the Weizmann Horse dataset, whereas Local Contrast Normalization provides no apparent advantage.The experiments use some pretrained VGG-16 layers and no other preprocessing.
- Class balancing: Median frequency balancing adds a cross-entropy term that biases predictions toward low-occurrence classes in highly imbalanced CamVid data.Without balancing, the network can maximize scores on frequent classes while effectively ignoring rare ones.
- Evaluation: Global accuracy, average per-class Intersection over Union, and average per-class accuracy provide complementary evaluation measures across the segmentation settings.Per-class accuracy and average per-class accuracy are additionally reported for full semantic segmentation.
6. Conclusion
The conclusion presents ReSeg as an extension of ReNet for semantic segmentation, reporting state-of-the-art performance across the evaluated datasets. It also highlights VGG-16 preprocessing and class balancing as important analyzed components.
- Contribution and results: ReSeg extends ReNet for semantic segmentation and achieves state-of-the-art performance on CamVid, Oxford Flowers, and Weizmann Horses.The conclusion identifies CamVid, Oxford Flowers, and Weizmann Horses as evaluation datasets.
- Analysis: The analysis evaluates VGG-16 input processing and a class-balancing term in the cross-entropy loss for under-represented classes.These are presented as analyzed design choices rather than as separate models.
- Analysis: Using only a few VGG-16 layers is sufficient for ReSeg to handle semantic segmentation while encoding contextual information and long-term dependencies.The conclusion links this observation to the model’s ability to encode contextual information and long-term dependencies.