Source-linked AI summary
AlignSeg: Feature-Aligned Segmentation Networks
Zilong Huang, Yunchao Wei, Xinggang Wang, Wenyu Liu, Thomas S. Huang, Humphrey Shi
TL;DR
Semantic segmentation networks often overlook misalignment caused by downsampling and indiscriminate contextual fusion. AlignSeg introduces AlignFA for multiresolution feature alignment and AlignCM for adaptive contextual alignment, achieving 82.6% and 45.95% mIoU on Cityscapes and ADE20K, respectively.
Problem
Existing segmentation architectures can overlook feature misalignment caused by step-by-step downsampling and indiscriminate contextual information fusion.
Method
AlignSeg combines AlignFA, which learns alignment for multiresolution feature aggregation, with AlignCM, which lets each pixel select custom contextual information.
Results
82.6% and 45.95% mIoU were achieved on the Cityscapes test set and ADE20K validation set, respectively.
Takeaways & Limitations
AlignSeg provides aligned multiresolution and contextual representations for semantic segmentation and is reported to achieve state-of-the-art performance on two challenging datasets.
Takeaways & Limitations
The reported architecture assumes image height and width can be made divisible by 32 through padding.
Abstract
from arXiv · showhide
Aggregating features in terms of different convolutional blocks or contextual embeddings has been proven to be an effective way to strengthen feature representations for semantic segmentation. However, most of the current popular network architectures tend to ignore the misalignment issues during the feature aggregation process caused by 1) step-by-step downsampling operations, and 2) indiscriminate contextual information fusion. In this paper, we explore the principles in addressing such feature misalignment issues and inventively propose Feature-Aligned Segmentation Networks (AlignSeg). AlignSeg consists of two primary modules, i.e., the Aligned Feature Aggregation (AlignFA) module and the Aligned Context Modeling (AlignCM) module. First, AlignFA adopts a simple learnable interpolation strategy to learn transformation offsets of pixels, which can effectively relieve the feature misalignment issue caused by multiresolution feature aggregation. Second, with the contextual embeddings in hand, AlignCM enables each pixel to choose private custom contextual information in an adaptive manner, making the contextual embeddings aligned better to provide appropriate guidance. We validate the effectiveness of our AlignSeg network with extensive experiments on Cityscapes and ADE20K, achieving new state-of-the-art mIoU scores of 82.6% and 45.95%, respectively. Our source code will be made available.
1 INTRODUCTION
AlignSeg addresses feature misalignment in semantic segmentation caused by multiresolution aggregation and indiscriminate contextual fusion. Its AlignFA and AlignCM modules align features and adapt contextual information, achieving strong results on Cityscapes and ADE20K.
- Motivation: Feature aggregation combines spatially rich early features with semantically rich later features, but existing methods can suffer feature misalignment.The paper identifies this issue as important because segmentation requires both spatial and semantic information.
- Motivation: Step-by-step downsampling creates spatial misalignment when upsampled late features are aggregated with early high-resolution features.Indiscriminate contextual fusion also gives every pixel homogeneous contextual information, potentially causing falsely aligned guidance and misclassification.
- AlignSeg: AlignFA uses learnable alignment to precisely align high- and low-resolution feature maps during multiresolution aggregation.It learns 2D transformation offsets to guide feature alignment instead of relying only on regular-grid interpolation.
- AlignSeg: AlignCM enables each pixel to choose private custom contextual information through adaptive, selective, and rich context pooling.The module integrates learnable alignment with pooling-based context modeling.
- Results: 82.6% and 45.95% mIoU were obtained on the Cityscapes test set and ADE20K validation set, respectively.These results are reported as evidence of the approach’s effectiveness.
2 RELATED WORK
Prior semantic segmentation methods aggregate features across network stages or model contextual information to capture spatial and semantic cues. The paper situates AlignSeg among these approaches while focusing on feature misalignment.
- Semantic segmentation: Semantic segmentation methods commonly use feature aggregation and context modeling to capture rich spatial and semantic information.Feature aggregation combines representations from different network stages, while context modeling introduces contextual features.
- Feature Aggregation: Encoder-decoder and bottom-up architectures are established strategies for aggregating multiresolution features from different convolutional blocks.Examples include SegNet, U-Net, RefineNet, ACNet, DeepLabv3+, FRRN, DLA, and SeENet.
- Feature Alignment: Feature aggregation approaches may suffer feature misalignment, motivating feature-alignment methods such as AlignSeg.The supplied related-work passage introduces this issue but does not provide further details of the compared methods.
3 APPROACH
AlignSeg addresses feature misalignment in multiresolution and contextual aggregation with two learnable alignment modules. Its architecture combines bottom-up feature pathways, aligned feature aggregation, and adaptive contextual modeling.
- Approach: AlignSeg uses AlignFA to align low-resolution high-level and high-resolution low-level features during multiresolution aggregation.The module learns transformation offsets to guide interpolation and spatial alignment.
- Network Architecture: The network uses a bottom-up architecture with two pathways: one follows CNN downsampling, while the other preserves the input feature resolution.The pathways are fused to enrich the spatially preserving pathway with semantic information.
- Network Architecture: The first pathway produces progressively lower-resolution, semantically richer features and adds AlignCM after the res5 block for contextual information.The pathway uses outputs from the final layer of each convolutional block.
- Aligned Feature Aggregation: AlignFA predicts offset maps from concatenated upsampled low-resolution and high-resolution features before applying alignment and aggregation.Both feature inputs are used because alignment cannot be accomplished by a single feature alone.
- Aligned Context Modeling: AlignCM pools contextual features, predicts per-position offsets from local and upsampled contextual features, and concatenates aligned context with local features.This replaces homogeneous contextual fusion with adaptive context selection for each pixel.
4 EXPERIMENTS
The experiments evaluate AlignSeg on Cityscapes and ADE20K, with additional reported results on COCO. The paper states that the approach achieves state-of-the-art performance on Cityscapes and ADE20K.
- Experiments: AlignSeg is evaluated on Cityscapes and ADE20K, with results also reported on COCO.The cited passage identifies these datasets as the experimental evaluation scope.
4.1 Datasets and Evaluation Metrics
The evaluation uses Cityscapes for urban semantic segmentation, ADE20K for densely labeled scene parsing, and COCO for instance segmentation. Tables 1 and 2 report module ablations and feature aggregation comparisons on Cityscapes validation data.
- Datasets: Cityscapes contains 5,000 finely annotated 1024×2048 images across 19 classes, split into 2,975 training, 500 validation, and 1,525 test images.The experiments use only the high-quality pixel-level annotations.
- Datasets: ADE20K contains densely labeled images spanning 150 stuff/object classes, with 20K training, 2K validation, and 3K test images.It is described as a scene parsing dataset.
- Datasets: COCO is described as an instance-segmentation dataset with 115K training, 5K validation, and 20K test images across 80 categories.The passage gives dataset scale and category count but does not describe a specific metric here.
- Evaluation: Table 1 studies the proposed modules on the Cityscapes validation set, while Table 2 compares feature aggregation methods on that set.The captions identify the respective comparison scopes.
4.2 Implementation Details
Implementation uses an ImageNet-pretrained ResNet-101 backbone with synchronized batch normalization and polynomial learning-rate scheduling. Training settings differ between Cityscapes and ADE20K in learning rate and batch size.
- Network Structure: The model uses ImageNet-pretrained ResNet-101, removes its final fully connected layer, and starts feature aggregation from the res2 block.Standard BatchNorm is replaced with synchronized BatchNorm across multiple GPUs.
- Training Settings: Training uses a polynomial learning-rate policy with power = 0.9, momentum 0.9, and weight decay 0.0001.The learning rate is multiplied by 1 − (iter / max iter)^power.
- Training Settings: The initial learning rate is 1e-2 for Cityscapes and 2e-2 for ADE20K, with mini-batches of 16 and 32 respectively.These settings are specified for stochastic gradient descent training.
4.3 Experiments on Cityscapes
Cityscapes experiments show that AlignCM and AlignFA improve the ResNet-101 baseline, while standard testing enhancements further raise validation performance and AlignSeg compares favorably with prior methods.
- Ablation Study: 2.3% mIoU improvement comes from adding AlignCM, while AlignFA adds a further 1.1% mIoU gain.
- Ablation Study: 0.8% mIoU gain results from aligning the high-level feature with the low-level feature in AlignFA.
- Ablation Study: 79.4% mIoU is achieved by AlignCM, comparable to RCCA and higher than PSP and ASPP on Cityscapes val.
- Ablation Study: 81.5% to 82.4% performance improvement follows adding deep supervision, online hard example mining, and multi-scale testing with flipping.
- Compare with State-of-the-Arts: AlignSeg outperforms approaches using the same ResNet-101 backbone and training set, as well as several stronger-backbone or larger-training-set methods.
4.4 Experiments on ADE20K
On ADE20K, AlignSeg achieves higher validation performance than prior ResNet-101 and ResNet-152 approaches.
- 45.95% mIoU surpasses the previous best-performing ACNet on the ADE20K validation set.
4.5 Experiments on COCO
COCO experiments evaluate AlignFA within Mask R-CNN and report improvements over the ResNet-101 baseline across detection and instance-segmentation metrics.
- AlignFA is added to each Mask R-CNN feature aggregation stage, producing the R-101-FPN-AlignFA model.
- AlignFA outperforms the ResNet-101 baseline on all reported box AP and mask AP metrics, particularly mask AP.
4.6 Visualization
Visualization comparisons show progressively improved segmentation with AlignCM and AlignFA, while learned offset maps indicate feature deformation toward a more canonical view.
- AlignCM produces higher-quality segmentation than the baseline and reduces unexpected misclassification in Cityscapes visualizations.
- Different directions and intensities in the ΔF and ΔA offset maps indicate corrections for misalignment between feature F and feature A.
5 CONCLUSION AND FUTURE WORK
AlignSeg addresses feature misalignment in semantic segmentation through Aligned Feature Aggregation and Aligned Context Modeling. It achieves state-of-the-art performance on Cityscapes and ADE20K, while suggesting broader pixel-level applications for future work.
- AlignSeg focuses on feature misalignment in popular feature aggregation architectures for semantic segmentation.
- AlignFA precisely aligns high-resolution and low-resolution feature maps for multi-resolution feature aggregation.
- AlignCM collects adaptive, selective, and rich contextual information by integrating learnable alignment with pooling-based methods.
- 82.6% and 45.95% mIoU scores were achieved on the Cityscapes test set and ADE20K validation set, respectively.
- The authors identify edge and saliency detection as future pixel-level prediction tasks for AlignSeg.