Source-linked AI summary
ACFNet: Attentional Class Feature Network for Semantic Segmentation
Fan Zhang, Yanqin Chen, Zhihang Li, Zhibin Hong, Jingtuo Liu, Feifei Ma, Junyu Han, Errui Ding
TL;DR
Semantic segmentation methods commonly exploit spatial context, while category-specific context remains comparatively underused. ACFNet introduces class centers and an attentional class feature module within a coarse-to-fine architecture, achieving 81.85% mIoU on Cityscapes with only fine-annotated training data.
Problem
Existing context methods mainly use spatial strategies and do not explicitly distinguish pixels from different classes when calculating context.
Method
ACFNet uses class centers for class-level context, adaptively combines them through coarse-segmentation-guided attention, and fuses the result with a base network’s features.
Results
81.85% mIoU was achieved on Cityscapes with only fine-annotated data used for training.
Takeaways & Limitations
Ablation studies and intermediate-result visualizations show the effectiveness of class-level context, while the ACF module further improves performance with an ASPP baseline from 78.42% to 80.08%.
Abstract
from arXiv · showhide
Recent works have made great progress in semantic segmentation by exploiting richer context, most of which are designed from a spatial perspective. In contrast to previous works, we present the concept of class center which extracts the global context from a categorical perspective. This class-level context describes the overall representation of each class in an image. We further propose a novel module, named Attentional Class Feature (ACF) module, to calculate and adaptively combine different class centers according to each pixel. Based on the ACF module, we introduce a coarse-to-fine segmentation network, called Attentional Class Feature Network (ACFNet), which can be composed of an ACF module and any off-the-shell segmentation network (base network). In this paper, we use two types of base networks to evaluate the effectiveness of ACFNet. We achieve new state-of-the-art performance of 81.85% mIoU on Cityscapes dataset with only finely annotated data used for training.
1. Introduction
Semantic segmentation benefits from richer context, but common methods organize context spatially rather than by category. ACFNet introduces class-level context through class centers and selectively combines it in a coarse-to-fine network, achieving 81.85% mean IoU on Cityscapes using only fine-annotated training data.
- Semantic segmentation assigns a class label to every image pixel and supports applications including autonomous driving, scene understanding, and human parsing.
- Spatial context methods aggregate pixels through dilated sampling, pyramid regions, or global average pooling without explicitly distinguishing their classes.
- Class centers represent each category’s overall image representation by aggregating features from pixels belonging to that class.
- The ACF module uses coarse segmentation as an attention map so pixels selectively focus on different class centers.This avoids requiring ground-truth labels at test time and allows class centers to be estimated from coarse predictions and high-level backbone features.
- ACFNet combines a base segmentation network with the ACF module, then fuses attentional class features with base-network features for final segmentation.
- 81.85% mean IoU was achieved on Cityscapes with only fine-annotated data used for training.
2. Related Work
Prior segmentation research exploits context through global, nearby, or multiscale spatial mechanisms and has also explored class-specific context. ACFNet instead harvests categorical context and uses a coarse-to-fine feature-aggregation structure to refine segmentation.
- Semantic Segmentation: Semantic segmentation research includes FCN, encoder-decoder networks, and methods that recover reduced spatial information through progressive upsampling.
- Context: Context methods use global average pooling, atrous spatial pyramid pooling, or pyramid-scale pooling to capture image-level, nearby, or multiscale information.
- Context: ACFNet harvests contextual information from a categorical perspective rather than relying only on conventional spatial context.
- Context: EncNet uses whole-scene information to enhance or weaken channel-wise class-level features, whereas ACFNet selectively uses class-specific context at the pixel level.
- Attention: Attention-based segmentation methods explore context through mechanisms including spatial-temporal dependencies and self-attention.
- Coarse-to-fine Methods: Existing coarse-to-fine segmentation methods combine coarse results with higher-resolution information or use coarse locations before fine smoothing.
- Coarse-to-fine Methods: ACFNet uses coarse-to-fine feature-level aggregation to improve the final segmentation result.
3. Methodology
ACFNet introduces class centers as categorical global context and uses an attentional class feature module within a coarse-to-fine segmentation framework. The module estimates class centers from coarse predictions and features, then adaptively combines them per pixel.
- ACF Module: The ACF module contains a Class Center Block for calculating class centers and a Class Attention Block for producing attentional class features.Its input is the coarse segmentation result and the base network feature map.
- Class Center: A class center represents the average feature of all pixels belonging to a category, providing categorical global context for semantic segmentation.The ideal formulation uses ground-truth pixel indicators to aggregate features for each class.
- Class Center: During testing, coarse class probabilities weight pixel features to approximate a robust class center without ground-truth labels.Pixels with higher probability for a class contribute more to that class center.
- Class Center: Class centers help pixels compare their features with image-specific class representations, refining class distributions and correcting some misclassifications.The paper illustrates correction when a pixel initially confused between classes A and B is evaluated using both class centers.
- Class Attention: The Class Attention Block uses coarse predictions to selectively combine class centers for each pixel, emphasizing likely or confusing categories rather than absent classes.The resulting attentional class feature is refined with a 1 × 1 convolution.
- Network: ACFNet combines a complete base segmentation network with the ACF module, concatenates the attentional class feature with base features, and predicts the final segmentation.Auxiliary, coarse, and fine segmentation losses are balanced by λa, λc, and λf.
4. Experiments
The experiments evaluate ACFNet on Cityscapes, an urban-scene semantic segmentation dataset, using mean class-wise Intersection over Union as the metric.
- Dataset: Cityscapes contains 5,000 high-resolution images across 19 semantic classes for urban scene understanding and segmentation evaluation.The dataset provides 2,975 training images, 500 validation images, and 1,525 test images.
- Metric: The evaluation metric is mean Intersection over Union averaged across classes.The paper denotes this metric as mIoU.
4.1. Network Architecture
The network-architecture experiments test ACFNet with ResNet-101 and ResNet-101 plus ASPP base networks to assess effectiveness and generality.
- Base Networks: ACFNet uses ResNet-101 as a baseline base network and ResNet-101 with ASPP as a second base network.The latter tests whether ACF improves a network already containing a state-of-the-art context module.
- Baseline Network: The baseline ResNet-101 removes its classification and final pooling layers, uses dilation rates 2 and 4 in its last two blocks, and has output stride 8.The backbone is initialized from ImageNet pretraining.
- ASPP Network: The ASPP base network contains one 1 × 1 convolution branch and three 3 × 3 branches with dilation rates 12, 24, and 36.This configuration is used to assess ACF module generalization with ASPP.
- ACF Configuration: The ACF module reduces its input and final output feature channels to 512 to lower computation and memory usage.This channel setting applies to the module’s intermediate input and final output.
4.2. Implementation Details
Training uses SGD with a polynomial learning-rate schedule, synchronized in-place batch normalization, and standard geometric data augmentation on Cityscapes.
- Optimization: Cityscapes models use SGD with learning rate 0.01, weight decay 0.0005, momentum 0.9, and 40k training iterations.The learning rate follows a ‘poly’ policy with exponent 0.9.
- Loss: The auxiliary, coarse, and fine loss weights are set to λa = 0.4, λc = 0.6, and λf = 0.7.These weights balance the three losses in the training objective.
- Normalization: All BatchNorm layers are replaced by InPlaceABN-Sync during training.The supplied implementation details identify this as the normalization configuration.
- Augmentation: Data augmentation includes random horizontal flipping, random scaling from 0.5 to 2.0, and random cropping of 769 × 769 patches.These augmentations follow prior segmentation work cited by the paper.
4.3. Ablation Study
The ablation study evaluates the ACF module and its components on Cityscapes using an atrous ResNet-101 baseline, showing gains from class centers and adaptive class-feature aggregation. Visualizations further associate ACFNet with more discriminative class features, improved coarse-to-fine consistency, and corrections in challenging regions.
- Experimental Setup: The experiments use an atrous ResNet-101 baseline with final results obtained by directly upsampling its output and auxiliary supervision applied throughout.The baseline results are reported on the Cityscapes validation set in Table 1.
- Class Center Ablation: Adding class centers improves coarse segmentation to 76.42% and fine segmentation to 77.94%, corresponding to gains of 0.57% and 2.09%.The class center is upsampled and concatenated with the base-network feature map after removing the Class Attention Block.
- ACF Aggregation: ACF(sum) reaches 79.32% fine-segmentation performance, slightly exceeding ACF(concat) at 79.17%.Both strategies use pixel-wise coarse-segmentation probabilities to weight class centers; ACF(sum) is selected as the default strategy.
- Feature Similarity: ACFNet learns more discriminative class features than the baseline after adding class-level context, as illustrated by cosine-similarity maps for terrain and car pixels.The comparison uses baseline and ACFNet feature maps, with ACFNet evaluated before fine segmentation.
- Coarse-to-Fine Refinement: Feature visualizations compare maps before coarse and fine segmentation to examine whether class-level context helps pixels check consistency with each class and refine predictions.Improved regions are marked with yellow squares in Figure 6.
- Qualitative Comparison: Qualitative comparisons show ACFNet correcting challenging baseline errors, including truck-versus-car confusion and building-versus-wall mislabeling.The corrected regions are highlighted with yellow squares in Figure 7.
4.4. Experiments on Baseline Network with ASPP
The ACF module is appended to a ResNet-101 with ASPP and evaluated alongside online bootstrapping and multi-scale/flipping test strategies on Cityscapes validation data.
- Performance with ACF Module: 1.7% performance improvement, from 78.42% to 80.08%, after adding ACF to ResNet-101 with ASPP.This result supports compatibility between ACF and another state-of-the-art context module.
- Online Bootstrapping: 0.91% validation-set improvement from online bootstrapping during training.The method focuses training on hard pixels whose correct-class probabilities are below θ=0.7 while retaining at least K=100,000 pixels per batch.
- MS/Flip: 1.38% validation-set improvement from multi-scale and left-right flipping during testing.The multi-scale evaluation uses scales [0.75, 1.0, 1.25, 1.5, 1.75, 2.0].
4.5. Comparing with the State-of-the-Art
On the Cityscapes test set, ACFNet outperforms prior methods under both train-fine-only and train-fine-plus-validation-fine training settings.
- Cityscapes Test Set: 81.85% mIoU is achieved by ACFNet when trained with both train-fine and validation-fine data, establishing a new state of the art.The reported test-set submission uses ResNet-101 with ASPP and ACF, online bootstrapping, and multi-scale plus flipping test strategies.
- Cityscapes Test Set: About 2.2% improvement over PSANet is reported when ACFNet uses only train-fine data.Under this setting, ACFNet also performs better than most methods that use the validation set for training.
5. Conclusion
ACFNet represents class-level context with class centers and selectively combines it per pixel through a coarse-to-fine architecture, achieving 81.85% mIoU on Cityscapes.
- 81.85% mIoU on Cityscapes establishes ACFNet’s reported new state-of-the-art performance.
- Class centers represent class-level context, while ACFNet calculates and selectively combines this context according to each pixel’s features.
- Ablation studies and intermediate-result visualizations support the effectiveness of class-level context for segmentation.