Source-linked AI summary
Adaptive Affinity Fields for Semantic Segmentation
Tsung-Wei Ke, Jyh-Jing Hwang, Ziwei Liu, Stella X. Yu
TL;DR
Pixel-wise segmentation can miss spatial structure and fine details, while existing structural methods can impose computational or training burdens. AAF matches neighbouring-pixel label relations and adaptively selects category-specific affinity ranges through adversarial minimax learning during training only. The method achieves competitive benchmark performance and generalizes to unseen data, while avoiding inference overhead.
Problem
Pixel-wise approaches lack spatial discrimination for weak foreground evidence and small structures, while structural methods can require substantial computational cost or tuning.
Method
AAF matches ground-truth-based same-or-different relations between neighbouring pixels and uses adversarial minimax learning to adapt affinity ranges by semantic category.
Results
AAF achieves competitive performance on standard benchmarks and generalizes well on unseen data; on VOC, AAF improves mean IoU by 2.27% over the referenced baseline.
Takeaways & Limitations
AAF provides rich structural relations, requires no inference overhead, and is robust to visual domain changes because it operates on label relations rather than pixel values.
Takeaways & Limitations
Compared with other region-wise approaches, AAF-related alternatives can require substantial hyperparameter tuning and be prone to overfitting, producing small gains over strong baselines.
Abstract
from arXiv · showhide
Semantic segmentation has made much progress with increasingly powerful pixel-wise classifiers and incorporating structural priors via Conditional Random Fields (CRF) or Generative Adversarial Networks (GAN). We propose a simpler alternative that learns to verify the spatial structure of segmentation during training only. Unlike existing approaches that enforce semantic labels on individual pixels and match labels between neighbouring pixels, we propose the concept of Adaptive Affinity Fields (AAF) to capture and match the semantic relations between neighbouring pixels in the label space. We use adversarial learning to select the optimal affinity field size for each semantic category. It is formulated as a minimax problem, optimizing our segmentation neural network in a best worst-case learning scenario. AAF is versatile for representing structures as a collection of pixel-centric relations, easier to train than GAN and more efficient than CRF without run-time inference. Our extensive evaluations on PASCAL VOC 2012, Cityscapes, and GTA5 datasets demonstrate its above-par segmentation performance and robust generalization across domains.
1 Introduction
Pixel-wise segmentation can miss spatial structure and fine details, motivating AAF, which matches neighbouring-pixel label relations and adaptively selects category-specific neighbourhood sizes during training.
- Pixel-wise classifiers can fail when foreground and background are visually mixed, especially for weak evidence or small structures such as motorcycle shields and wheel spokes.
- CRF and GAN methods incorporate structural reasoning, but CRF relies on costly iterative inference and visual appearance, while GAN introduces a more complex alternative.
- AAF matches same-versus-different label relations between neighbouring pixels rather than relying on image appearance or independent pixel labels.
- Fixed affinity-field sizes cannot represent structures at different ranges, while naive size search can select trivial neighbourhoods that add little learning value.
- Adversarial learning formulates size adaptation as a minimax problem, selecting fields that are difficult enough to matter but still support improvement toward ground truth.
- AAF encodes distributed pixel-centric structure, adds no testing-time inference process, and is evaluated on Cityscapes, PASCAL VOC 2012, and GTA5.
2 Related Works
AAF offers a training-only alternative for structure modeling by matching neighbouring-pixel relations in label space, enabling deployment without run-time inference.
- AAF models structure by matching relations between neighbouring pixels in the label space during segmentation-network training.
- After training, the segmentation network can be deployed without additional run-time inference.
3 Our Approach: Adaptive Affinity Fields
The approach extends pixel-wise segmentation supervision to neighborhood-level affinity matching, then adaptively selects kernel sizes through adversarial weighting. This provides structural supervision based on label relations while avoiding extra inference computation.
- From Pixel-wise Supervision to Region-wise Supervision: Region-wise supervision evaluates prediction and ground-truth neighborhoods jointly rather than treating pixels independently.The neighborhood is centered at each pixel and represented by N(·).
- Affinity Field Loss Function: Pairwise affinity uses ground-truth label relations: same-category pixels receive a grouping force, while different-category pixels receive a separating force.The affinity loss applies these forces to predicted categorical probabilities.
- Affinity Field Loss Function: Affinity-field loss uses KL divergence between binary classification probabilities to compare neighboring predictions.The divergence is defined for Bernoulli distributions and is consistent with the unary cross-entropy term.
- Adaptive Kernel Sizes from Adversarial Learning: A fixed neighborhood size is inadequate because the ideal kernel size varies with category-specific object size and shape complexity.Different neighborhood sizes encode structural relations at different spatial ranges.
- Adaptive Kernel Sizes from Adversarial Learning: The size-adaptive loss optimizes weights over multiple affinity-field sizes for each category.Direct minimization can produce trivial weighting that favors easy local or distant relationships, reducing the effect of pairwise supervision.
- Adaptive Kernel Sizes from Adversarial Learning: Adversarial learning maximizes affinity errors across kernel sizes while segmentation minimizes the resulting maximal loss in a best worst-case training scenario.This pushes same-segment predictions together and different-segment predictions apart as far as possible.
4 Experimental Setup
The experiments compare AAF-related methods with established structural and feature-based alternatives across PASCAL VOC 2012 and Cityscapes, using pixel-, instance-, and boundary-focused evaluation. Training uses standard segmentation backbones and augmentation, while dense-CRF adds inference time that the proposed methods avoid.
- Datasets: PASCAL VOC 2012 and Cityscapes provide the evaluation datasets, with 20 object categories plus background and 19 urban-scene categories, respectively.The augmented VOC split contains 10,582 training, 1,449 validation, and 1,456 testing images; Cityscapes contains 2,975 training, 500 validation, and 1,525 testing images.
- Evaluation Metrics: Pixel-wise mIoU is complemented by instance-wise mIoU and boundary precision, recall, and f-measure to assess smaller objects and fine structures.Instance-wise mIoU reduces the bias toward large objects, while boundary metrics evaluate semantic contours.
- Methods of Comparison: The comparison includes GAN adversarial learning, contrastive loss, and CRF-based processing alongside the proposed affinity-field methods.These alternatives represent adversarial region-structure supervision, feature-map region supervision, and dense-CRF post-processing.
- Methods of Comparison: Dense-CRF requires an additional 40 seconds on Cityscapes for final-result generation, whereas the proposed methods introduce no inference overhead.The comparison uses dense-CRF as a post-processing baseline.
- Implementation: Experiments use PSPNet or FCN base architectures, a poly learning-rate policy, random mirroring and resizing, and output-stride-8 training labels.The base learning rate is 0.001, with 30K training iterations on VOC and 90K on Cityscapes.
- Implementation: AAF and affinity-field losses use λ = 1.0 and margin m = 3.0 with an ImageNet-pretrained ResNet101 backbone.These settings are used for the reported experiments.
5 Experimental Results
AAF improves pixel-, instance-, and boundary-level segmentation on VOC and Cityscapes, with especially strong gains for fine structures and complicated boundaries. Adaptive kernel sizes vary by category, and AAF generalizes across domains without fine-tuning.
- Pixel-level Evaluation: 82.17% and 79.07% mIoU: AAF outperforms PSPNet by 1.54% and 2.77% on VOC and Cityscapes testing sets.
- Boundary-level Evaluation: 7.9% on VOC and 8.0% on Cityscapes: AAF improves overall boundary recall, with gains for every category.
- Adaptive Affinity Field Analysis: Person and dog favor smaller kernels, whereas cow and plant favor larger kernels, indicating category-dependent affinity ranges.
- Generalizability: Without fine-tuning, AAF outperforms PSPNet on GTA5 by 9.5% in mean pixel accuracy and 1.46% in mIoU.
6 Summary
AAF incorporates geometric regularities through adversarially learned local relations with adaptive ranges. It is effective, efficient without inference overhead, robust to domain changes, and competitive on standard benchmarks.
- AAF encodes rich structural relations while introducing no additional inference overhead.
- AAF is robust to domain changes and generalizes well to unseen data, including GTA5 evaluations.
- Visual comparisons cover VOC 2012, Cityscapes, and GTA5 using images, ground truth, PSPNet, affinity fields, and AAF.