Source-linked AI summary
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
Tobias Pohlen, Alexander Hermans, Markus Mathias, Bastian Leibe
TL;DR
Semantic segmentation needs both recognition and precise localization, while pooling and reliance on pretrained classification architectures can limit boundary accuracy and architectural flexibility. The paper proposes a two-stream full-resolution residual network that couples pooled multi-scale features with a full-resolution stream. It reports state-of-the-art Cityscapes performance without post-processing or pre-training, including a 71.8% intersection-over-union score.
Problem
Semantic segmentation requires precise pixel boundaries, but pooling-based and pretrained classification architectures can limit localization accuracy and architectural flexibility.
Method
The proposed FRRN combines a pooled stream for large-scale recognition features with a full-resolution residual stream for precise boundary adherence.
Results
71.8% intersection-over-union is achieved on Cityscapes without additional post-processing or pre-training.
Takeaways & Limitations
The architecture reaches state-of-the-art Cityscapes results while being trainable from scratch and preserving boundary adherence without post-processing.
Takeaways & Limitations
The introduction notes that new activation functions often cannot be added into an existing architecture.
Abstract
from arXiv · showhide
Semantic image segmentation is an essential component of modern autonomous driving systems, as an accurate understanding of the surrounding scene is crucial to navigation and action planning. Current state-of-the-art approaches in semantic image segmentation rely on pre-trained networks that were initially developed for classifying images as a whole. While these networks exhibit outstanding recognition performance (i.e., what is visible?), they lack localization accuracy (i.e., where precisely is something located?). Therefore, additional processing steps have to be performed in order to obtain pixel-accurate segmentation masks at the full image resolution. To alleviate this problem we propose a novel ResNet-like architecture that exhibits strong localization and recognition performance. We combine multi-scale context with pixel-level accuracy by using two processing streams within our network: One stream carries information at the full image resolution, enabling precise adherence to segment boundaries. The other stream undergoes a sequence of pooling operations to obtain robust features for recognition. The two streams are coupled at the full image resolution using residuals. Without additional processing steps and without pre-training, our approach achieves an intersection-over-union score of 71.8% on the Cityscapes dataset.
1. Introduction
Semantic segmentation supports scene understanding in autonomous driving, but pooling-based architectures trade localization accuracy for recognition. The paper proposes two coupled processing streams to combine full-resolution boundary adherence with multi-scale recognition.
- Motivation: Semantic segmentation assigns predefined class labels to image pixels and supports modeling street-scene entities such as cars, pedestrians, roads, and sidewalks.Automotive uses include filtering regions unlikely to contain objects of interest and improving object detection.
- Motivation: Many automotive applications require precise region boundaries, motivating segmentation with high-quality boundary adherence.
- Problem: Pooling and strided convolutions enlarge receptive fields and improve translation robustness, but significantly deteriorate localization in semantic segmentation.
- Approach: The proposed architecture combines a full-resolution stream for precise boundaries with a pooled stream for large-scale relationships and robust recognition.The streams are coupled through full-resolution residual units (FRRUs).
- Contributions: The paper targets state-of-the-art street-scene segmentation without additional post-processing or limitations imposed by pre-trained architectures.It also reports releasing code and trained models.
2. Related Work
Related work develops semantic segmentation from CNN-based pixel classification and fully convolutional networks, while addressing reduced output resolution through skip connections, encoder-decoder designs, and smoothing.
- Semantic Segmentation Approaches: CNN-based semantic segmentation evolved from region classification toward pixel-wise predictions using features from multiple scales.
- Semantic Segmentation Approaches: Fully convolutional networks enabled end-to-end semantic segmentation and use of pretrained classification architectures, with skip connections improving information flow to object boundaries.
- Resolution Recovery: Pooling increases receptive-field size and classification performance but produces lower-resolution outputs, motivating skip connections and encoder-decoder strategies.
- Output Smoothing: CRFs, domain transforms, and superpixel-based methods smooth CNN predictions, whereas the proposed approach combines high- and low-resolution information for already smooth outputs.
- Network Architectures: Modern CNN architecture research progressed through deeper VGG and batch-normalized GoogleNet designs before widespread adoption of ResNet.
3. Network Architectures for Segmentation
The paper combines residual-network training behavior with a full-resolution stream and a pooled stream. Full-resolution residual units jointly process both resolutions while passing high-level information and preserving localization.
- Feed-Forward Networks: Traditional feedforward networks compute outputs through a linear sequence of layers parameterized by successive functions.
- Residual Networks: ResNets add learned residuals to inputs through skip connections, improving training characteristics over traditional feedforward networks.
- Full-Resolution Residual Networks: FRRNs unify residual and pooled processing: one stream adds successive residuals while the other applies convolution and pooling operations to the input.
- Design Motivation: Pooling reduces feature-map size, enlarges receptive fields, and improves translation robustness, whereas deep pooling hierarchies make low-level localization difficult.
- FRRUs: Each FRRU has two inputs and two outputs, simultaneously operating on the residual and pooling streams.
- FRRUs: FRRUs can interpolate between residual and feedforward designs: setting G to zero yields a residual unit, while setting H to zero yields a traditional feedforward network.
- FRRU Design: The FRRU design concatenates streams, applies convolutional processing, and uses channel adjustment plus unpooling to form the full-resolution residual.
- FRRU Design: Reusing G's output to compute H gives all unit weights improved gradient-flow properties, while alternative designs remain possible.
4. Training Procedure
The network is trained with a bootstrapped cross-entropy loss that focuses on selected pixels, while memory and overfitting are managed through recomputation and augmentation.
- Loss: The bootstrapped cross-entropy loss is computed over K selected pixels using predicted probabilities for the target classes.The formulation uses target labels and posterior class probabilities for each pixel.
- Loss: The threshold t_K selects pixels with the lowest predicted target-class probabilities, including misclassified pixels and uncertain correct predictions.The threshold is determined by sorting predicted log probabilities and choosing the K + 1-th value.
- Optimization: ADAM is used to minimize the bootstrapped cross-entropy loss.
- Memory management: Full-resolution residual units make training memory intensive because the entire forward pass must be stored for backpropagation.
- Memory management: The computation graph is partitioned into blocks, with derivatives computed separately to control the space-time trade-off.Intermediate feature maps are recomputed on demand rather than all being retained simultaneously.
- Data augmentation: Translation and gamma augmentation are used to reduce overfitting, with reflection padding for images and void-label padding for annotations.
5. Experimental Evaluation
Experiments on Cityscapes compare the proposed FRRNs with ResNet baselines and leading published methods, including performance across resolutions and boundary-focused evaluation. The results show competitive or superior segmentation and boundary adherence without relying on post-processing or pre-trained architectures.
- 5.1. Residual Network Baseline: 65.7% validation mean IoU for FRRN A exceeded the ResNet baseline’s 62.8% after quarter-resolution training.Both networks were trained for 45,000 iterations with batch size 3.
- 5.2. Quantitative Evaluation: FRRNs consistently outperform approaches trained at the same image resolutions throughout the experiments.Figure 5 compares mean IoU scores against the subsampling factors used during training.
- 5.2. Quantitative Evaluation: The proposed architecture matches the published state of the art while training FRRN B on half-resolution images.Full-resolution training was too memory intensive for the current design.
- 5.3. Boundary Adherence: The proposed approach outperforms LRR and Dilation for every trimap radius between 1 and 80 pixels.The trimap evaluation measures mean IoU near ground-truth label boundaries.
- 5.3. Boundary Adherence: A fully connected CRF improved validation IoU by only approximately 0.5%, so the authors rejected post-processing because of its high computation time.The experiment used Gaussian appearance and smoothness kernels with validation-set parameter tuning.
6. Conclusion
The paper concludes that its novel architecture provides accurate semantic segmentation in street scenes without additional post-processing and can be trained from scratch. It achieves state-of-the-art Cityscapes results and is presented as potentially applicable to other per-pixel prediction tasks.
- 6. Conclusion: The proposed architecture combines clean processing, scratch training, strong boundary adherence, and state-of-the-art Cityscapes results without additional post-processing.The authors also state that code and trained models will be provided.
- 6. Conclusion: The architecture may also apply to stereo and optical flow because those tasks make per-pixel predictions.The authors note that the design does not incorporate choices tailored specifically to semantic segmentation.
A. Gamma Augmentation
Gamma augmentation varies image contrast and brightness by transforming intensities with x 7→x^γ, while a novel sampling scheme reduces bias in γ.
- Gamma augmentation transforms unit-interval image intensities as x 7→x^γ to vary contrast and brightness.
- Naively sampling γ uniformly or from a truncated Gaussian centered at 1 produces noticeable bias.
- The method chooses γ so the expected transformed intensity satisfies E_U[U] = 0.5.
- The sampling scheme derives γ from a zero-mean deviation variable Z without explicitly solving for the implicitly defined U.
- Figure 8 compares naive and new γ sampling across 50 samples, with the new scheme reducing bias.
- The FRRN A model outperforms the residual-network baseline by a significant margin, with both trained until convergence.
C. Qualitative Results
Figure 10 presents additional qualitative output labelings from the method on the Cityscapes validation set, alongside ground truth and LRR comparisons.
- Figure 10 compares additional output labelings produced by the method.
- The displayed comparison includes image, ground truth, the method’s output, and LRR labels.
- The qualitative results are from the Cityscapes validation set and omit comparison to Dilation to allow larger images.