Source-linked AI summary
Semantic Segmentation using Adversarial Networks
Pauline Luc, Camille Couprie, Soumith Chintala, Jakob Verbeek
TL;DR
Semantic segmentation models typically factorize pixel-label predictions, motivating methods that capture higher-order consistency. The paper trains a segmentation network with an adversarial network that distinguishes predicted from ground-truth label maps, and reports improved accuracy on Stanford Background and PASCAL VOC 2012.
Problem
CNN segmentation models predict label variables independently during training, so they do not explicitly capture higher-order spatial consistency.
Method
The paper combines multi-class cross-entropy with an adversarial loss that discriminates predicted label maps from ground-truth maps and penalizes higher-order mismatches.
Results
The approach improves semantic segmentation accuracy on both the Stanford Background and PASCAL VOC 2012 datasets.
Takeaways & Limitations
Adversarial training regularizes factorized segmentation predictions by enforcing higher-order consistency without adding complexity to the test-time model.
Abstract
from arXiv · showhide
Adversarial training has been shown to produce state of the art results for generative image modeling. In this paper we propose an adversarial training approach to train semantic segmentation models. We train a convolutional semantic segmentation network along with an adversarial network that discriminates segmentation maps coming either from the ground truth or from the segmentation network. The motivation for our approach is that it can detect and correct higher-order inconsistencies between ground truth segmentation maps and the ones produced by the segmentation net. Our experiments show that our adversarial training approach leads to improved accuracy on the Stanford Background and PASCAL VOC 2012 datasets.
1 Introduction
Semantic segmentation predicts a class label for every image pixel, but conventional CNN training predicts labels independently and does not explicitly model spatial consistency. The paper proposes adversarial training to enforce higher-order consistency, reporting improved accuracy on two datasets.
- Semantic segmentation is a dense labeling task that assigns a category to each pixel in an input image.
- CNN segmentation methods commonly predict label variables independently during training, leaving spatial contiguity implicit.Post-processing methods such as CRFs have been explored to reinforce spatial contiguity.
- Higher-order potentials can improve label consistency, but prior approaches are restricted to specific potential classes or few learnable parameters.
- The proposed approach combines conventional multi-class cross-entropy with an adversarial term that encourages generated label maps to resemble ground-truth maps.
- The work presents, to the authors’ knowledge, the first application of adversarial training to semantic segmentation.
- The approach enforces long-range spatial label contiguity without adding complexity to the model used at test time.
- The experiments show improved labeling accuracy on the Stanford Background and PASCAL VOC 2012 datasets.
2 Related work
Prior work combines CNN-based pixel predictions with CRFs, recurrent networks, and adversarial or generative modeling approaches. This paper distinguishes its approach by using a high-capacity adversarial model to capture broad higher-order label interactions without manually specifying them or retaining higher-order machinery at test time.
- GANs use an adversarial model alongside a generative network to learn a data distribution, with later work addressing architectural stability and image detail.
- Conditional generative models produce deterministic outputs from conditioning variables, including object attributes or input images.
- Modern semantic segmentation commonly uses fully convolutional architectures, which avoid redundant computation from overlapping image patches.
- CNN predictions have been combined with CRFs using pairwise or higher-order terms, while recurrent networks provide another mechanism for long-range label interactions.
- Unlike approaches with manually defined higher-order terms, the proposed adversarial model can detect mismatches across a wide range of higher-order statistics.
- After training, the model avoids higher-order terms and recurrence in its own inference architecture.
3 Adversarial training for semantic segmentation networks
The framework combines per-pixel cross-entropy with adversarial training so a segmentation model produces maps that match ground-truth maps in higher-order label statistics. Experiments use Stanford Background and PASCAL VOC 2012 architectures with multiple adversarial inputs and receptive fields.
- 3.1 Adversarial training: The hybrid objective combines multi-class cross-entropy with an adversarial term.Cross-entropy encourages correct labels independently at each pixel, while the adversarial component evaluates generated maps against ground-truth maps.
- 3.1 Adversarial training: The adversarial network penalizes higher-order mismatches, including region shapes, class proportions, and other spatial label statistics.Its large field of view allows it to assess patterns that the standard per-pixel factorized loss cannot access.
- 3.1 Adversarial training: Training alternates roles: the adversarial model discriminates ground-truth from generated maps, while the segmentation model minimizes cross-entropy and makes generated maps harder to distinguish.The segmentation update uses a modified adversarial target to provide a stronger gradient when discrimination is accurate, which preliminary experiments found important for faster training.
- 3.2 Network architectures: Experiments cover Stanford Background with a multi-scale patch-trained network and PASCAL VOC 2012 with a fine-tuned Dilated-8 architecture.The VOC architecture preserves higher output resolution by removing two max-pooling layers and uses dilated convolutions plus a context module.
- 3.2 Network architectures: The adversarial input variants are Basic probability maps, Product RGB-probability maps with 3C channels, and Scaling label distributions constrained by a minimum correct-label mass.Scaling uses τ=0.9 in the experiments; unlabeled pixels are zeroed in both inputs and corresponding gradients.
- 3.2 Network architectures: LargeFOV and SmallFOV use 34 × 34 and 18 × 18 label-map fields of view, targeting broader co-occurrence patterns versus local boundaries and spurious labels.Because segmentation outputs are eight times down-sampled, each field of view corresponds to a larger region in the input image.
4 Experimental evaluation results
Experiments on Stanford Background and PASCAL VOC 2012 evaluate adversarial segmentation against standard training using visual, per-class, IoU, and boundary metrics. Adversarial training improves spatial consistency and yields small, consistent accuracy gains, with more stable training obtained through slow alternation.
- Datasets and evaluation: The experiments use Stanford Background and PASCAL VOC 2012, with validation and test settings described for both datasets.Stanford Background contains 715 images; PASCAL VOC 2012 uses 10,582 augmented training images, 1,449 validation images, and 1,456 test images.
- Datasets and evaluation: The evaluation includes standard IoU and the BF measure, which measures accuracy along object contours.Mean BF uses the 1,103 fully annotated validation images because void border labels are ignored in the original annotations.
- Stanford Background: Adversarial training smooths class probabilities over large areas, sharpens class boundaries, and removes spurious labels across small areas on Stanford Background.Figure 3 compares class probabilities without and with adversarial training, followed by superimposed class labels.
- Stanford Background: Adversarial training produces less overfitting and improved validation accuracy on Stanford Background.Figure 4 shows per-class accuracy across training epochs for train and validation data, with and without adversarial training.
- Training procedure: Slow alternation between segmentation and adversarial-network updates was more stable than alternating at every SGD iteration.The reported results use 500 iterations of each network update before switching.
- PASCAL VOC 2012: Against a cross-entropy-only Dilated8 baseline, adversarial setups show small but consistent gains, with LargeFOV most effective overall.The baseline has mean IoU 71.8 and mean BF 47.4; Table 2 compares adversarial architectures and input encodings.
- PASCAL VOC 2012: On the PASCAL VOC 2012 test set, the baseline scores 73.1, while LargeFOV-Product and LargeFOV-Scaling score 73.3 and 73.2, respectively.These results confirm the small but consistent validation gains.
5 Discussion
The paper frames adversarial training as a variational loss that regularizes factorized segmentation predictions by enforcing higher-order consistency. Experiments show accuracy improvements on both datasets, with larger gains on the smaller Stanford Background dataset.
- Discussion: The adversarial network acts as a variational loss that regularizes segmentation by enforcing higher-order consistency without adding higher-order terms to the predictive model.The approach retains tractable cross-entropy computation while using adjustable adversarial parameters.
- Discussion: Adversarial training improves semantic segmentation accuracy on both Stanford Background and PASCAL VOC 2012.The gains are more pronounced on Stanford Background than on PASCAL VOC 2012.
- Discussion: The larger Stanford Background gains are attributed most likely to greater overfitting risk in the smaller dataset and more powerful architectures on PASCAL VOC 2012.This explanation is presented as a likelihood, not as a demonstrated causal result.
— Supplementary Material —
The supplied passages identify the paper’s authors and affiliations.
- Authors and affiliations: The authors are Camille Couprie, Pauline Luc, Soumith Chintala, and Jakob Verbeek.Couprie, Luc, and Chintala are affiliated with Facebook AI Research; Verbeek is affiliated with INRIA, Laboratoire Jean Kuntzmann.
1 Network architectures
The experiments use adversarial architectures that process segmentation maps and RGB images through separate branches. For PASCAL VOC 2012, the study varies field of view, model capacity, and input encoding.
- Stanford Background: The Stanford Background experiments use local contrast normalization on RGB images before either network receives them.
- Stanford Background: The Stanford Background adversarial architecture processes class segmentations and RGB images in separate branches.The segmentation branch receives class segmentations, while the other branch processes the corresponding RGB image.
- PASCAL VOC 2012: PASCAL VOC 2012 experiments compare LargeFOV and SmallFOV adversarial architectures with lighter-capacity variants.The variants are named LargeFOV-light and SmallFOV-light.
- PASCAL VOC 2012: The adversarial architectures are summarized with layers ordered top-to-bottom and approximate parameter counts.Parameter counts depend on the input encoding: C channels for Basic and Scaling, versus 3C for Product.
2 Additional results
Additional experiments report baseline and adversarial configurations, selected per-class test accuracy, and results summarized for the PASCAL VOC test set.
- Baselines and settings: The non-adversarial baseline fine-tunes Dilated8 with learning rates 10^-5, 10^-6, and 10^-7, achieving its best result at 10^-6.
- Baselines and settings: Adversarial experiments vary adversarial learning rates from 0.01 to 0.5, segmentation learning rates from 10^-6 to 10^-4, and λ over 0.1 and 1.BF results also include the across-image standard deviation σ.
- Test results: Per-class accuracy on the test set is reported for selected models in Table 3.
- Test results: Table 2 reports performance and hyper-parameters for the baseline and adversarial architectures with different input encodings.