Source-linked AI summary
Feedforward semantic segmentation with zoom-out features
Mohammadreza Mostajabi, Payman Yadollahpour, Gregory Shakhnarovich
TL;DR
Semantic segmentation is challenging because it involves complex interactions among image elements and traditionally relies on expensive structured prediction and inference. The paper casts segmentation as feedforward superpixel classification using multilevel zoom-out features and asymmetric-loss multilayer networks. It reports 64.4% average accuracy on the PASCAL VOC 2012 test set and explores the limits of this non-structured formulation.
Problem
Semantic segmentation is challenging because it depends on neighboring and distant image elements, global context, and complex structured prediction models with difficult inference and learning.
Method
The method classifies superpixels with a feedforward multilayer network using features from nested spatial contexts, convolutional representations, and asymmetric loss.
Results
64.4% average accuracy on the PASCAL VOC 2012 test set surpassed previously published segmentation results and advanced the state of the art from about 52%.
Takeaways & Limitations
Feedforward semantic labeling with multilevel zoom-out features can substantially surpass existing state-of-the-art segmentation results without explicitly representing segmentation structure.
Takeaways & Limitations
The paper does not dismiss structured prediction and notes that such tools may be complementary to its architecture.
Abstract
from arXiv · showhide
We introduce a purely feed-forward architecture for semantic segmentation. We map small image elements (superpixels) to rich feature representations extracted from a sequence of nested regions of increasing extent. These regions are obtained by "zooming out" from the superpixel all the way to scene-level resolution. This approach exploits statistical structure in the image and in the label space without setting up explicit structured prediction mechanisms, and thus avoids complex and expensive inference. Instead superpixels are classified by a feedforward multilayer network. Our architecture achieves new state of the art performance in semantic segmentation, obtaining 64.4% average accuracy on the PASCAL VOC 2012 test set.
1. Introduction
Semantic segmentation is difficult because it depends on local, distant, and global image relationships, yet conventional structured models make inference and learning complex. This paper instead labels superpixels with a feedforward classifier using features gathered by zooming out across spatial contexts, achieving 64.4% accuracy on PASCAL VOC 2012.
- Semantic segmentation remains challenging because neighboring and distant image elements, global context, and instance-level detection interact.
- Structured prediction approaches commonly use random fields or structured support vector machines, but many such models make inference and learning intractable.
- The proposed approach treats segmentation as single-stage classification, labeling each superpixel with a feedforward model from image-derived evidence.
- 64.4% average accuracy on PASCAL VOC 2012 advanced the reported state of the art from about 52%.
- Zoom-out features combine representations from the superpixel, proximal and distant neighborhoods, and the entire image before classification.
- The architecture uses convolutional features, multilayer neural networks, and asymmetric loss to make structured information implicit in feedforward classification.
2. Zoom-out feature fusion
The zoom-out architecture labels superpixels using concatenated features from nested spatial scopes, combining local appearance with proximal, distant, and global context. It also uses inverse-frequency loss weighting to address imbalanced labels.
- Architecture: The architecture classifies each nearly uniform superpixel using features extracted at multiple spatial extents around it.The same classification machine is applied to superpixels, while features from different context levels contribute to each labeling decision.
- Local zoom: Local features capture color, texture, intensity, and gradient evidence within the superpixel, which may differ sharply across object boundaries.These features provide local evidence for or against a category label.
- Proximal zoom: Proximal regions enlarge the receptive area, smoothing statistics through overlap while capturing boundary appearance and class distributions unavailable locally.They remain generally too limited for confident object-presence reasoning.
- Distant zoom: Distant regions include sizeable object fractions and encode shape, complex patterns, spatial layout, appearance, and context.Their overlap across neighboring superpixels can induce gradual feature changes and implicit smoothness.
- Global zoom: Global features summarize the entire scene, supporting local labels through image-level category evidence and shared soft constraints across superpixels.All superpixels in an image receive the same global features, creating an implicit high-order interaction.
- Learning: The resulting zoom-out features are concatenated into one superpixel representation, while inverse-frequency weighting gives rarer classes greater loss importance.The weighted loss retains convexity and uses all training data rather than discarding examples from common classes.
3. Related work
Prior segmentation work commonly uses structured models, proposal-based multi-stage pipelines, or contextual and multi-level features. The paper differs by combining nested-region representations with a feedforward classifier rather than explicit structured inference.
- Structured prediction: Many segmentation methods use CRFs with unary evidence and pairwise or higher-order interactions over pixel or superpixel nodes.The paper instead lets zoom-out features capture higher-order structure in unary-like representations.
- Multi-stage methods: Another major line of work generates category-independent or category-aware proposal regions before scoring, ranking, or combining them.Some methods re-rank multiple candidate segmentations with discriminatively trained models.
- Context: Context-focused studies compute neighborhood or global information for segmentation, but one reported approach remained significantly inferior to state of the art.This literature motivates using non-local evidence without requiring explicit non-unary terms.
- Multi-level features: Related approaches use multi-resolution convnet features, recurrent processing, hand-crafted multi-level features, or tree-structured smoothness over superpixels.The paper distinguishes itself by using different feature extractors across levels in a feedforward architecture.
- Concurrent work: A concurrent method achieved 62.2% mean accuracy on VOC 2012 test by fusing predictions from multiple convnet layers over a coarse image grid.Its fusion of spatial levels resembles the paper's zoom-out representations.
- Concurrent work: Hypercolumns also pool evidence from nested regions, but their system takes a hypothesized detection bounding box rather than an entire image as input.Its pooled regions include local, proximal-like, and box-global information.
4. Experiments
The experiments evaluate category-level semantic segmentation on the PASCAL VOC benchmark, using test-set accuracy as the primary measure. VOC 2012 test evaluation requires submitting predictions to the benchmark server.
- Dataset: The main experiments use PASCAL VOC category-level segmentation with 21 categories, including background.The training data comprise train and validation portions of about 1,500 images each, supplemented by labels for 9,118 additional images.
- Evaluation: The primary evaluation is accuracy on the 1,456-image VOC 2012 test set, whose ground truth is unavailable locally.Test accuracy is obtained by uploading predicted segmentations to the evaluation server.
4.1. Superpixels and neighborhoods
The segmentation pipeline uses roughly uniform SLIC superpixels and two increasingly broad neighborhoods. Proximal regions follow graph neighborhoods, while distant regions use a rectangular bounding box around third-degree neighbors.
- Superpixels: SLIC produces roughly 500 superpixels per image, averaging 21×21 pixels and generally adhering to local color boundaries.The parameter m controlling spatial-versus-color proximity is set to 15.
- Neighborhoods: A proximal region contains superpixels within radius 2, averaging 100x100 pixels and allowing arbitrary shape.It includes immediate neighbors and their immediate neighbors.
- Neighborhoods: A distant region contains neighbors through degree 3 and their bounding box, making it rectangular with an average size of 170×170 pixels.Unlike proximal regions, its shape is always rectangular.
- Benchmark scope: COCO was identified as a prospective benchmark but lacked a fully established test set and evaluation procedure at the time.The paper therefore focuses its main experiments on VOC.
4.2. Zoom-out feature computation
The architecture combines handcrafted and learned features from superpixels and increasingly broader image regions. Local, distant, and global representations are extracted with color, texture, SIFT, location, and convolutional-network features.
- Local features: Superpixels are represented with handcrafted features and learned features from a local convolutional network.The local network processes a padded 35 × 35 L*a*b bounding-box input and produces a 21-dimensional softmax feature vector plus two foreground-background features.
- Local features: Color features include fixed-bin and adaptive-bin L*a*b histograms, with entropy values added for the fixed-bin histograms.The color representation contributes 120 fixed-bin dimensions, 3 entropy dimensions, and 120 adaptive-bin dimensions.
- Local features: Texture, SIFT, and location features capture texton statistics, appearance, and normalized position within the image.The texture representation has 65 channels; SIFT contributes 1500 histogram values plus 6 entropy values; location contributes four values.
- Local features: The shared handcrafted representation for local regions has 1818 feature dimensions.This feature set is reused for the corresponding local-region representations.
- Distant and global features: Distant and global regions are encoded by deep convolutional networks pretrained for 1000-category ImageNet classification without fine-tuning.CNN-S and VGG-16 provide 4096-dimensional last fully connected-layer activations after resizing each region or image to 224×224 pixels.
4.3. Learning setup
The learning setup uses millions of superpixel examples, evaluates the accuracy ceiling imposed by superpixel labeling, and trains classifiers with asymmetric loss on a GPU.
- Training data: More than 10,000 images and roughly 500 superpixels per image provide more than 5 million training examples.
- Superpixel labeling: Majority-labeling superpixels achieves 94.4% validation accuracy, indicating limited achievable-accuracy loss from imperfect boundary adherence.The paper compares this assignment accuracy with contemporary methods that perform well below it.
- Optimization: The classifiers are trained with asymmetric loss using Caffe on a single Tesla K40 GPU.Training uses a fixed learning rate of 0.0001 and weight decay factor of 0.001.
4.4. Results on PASCAL VOC 2012
Experiments on VOC 2012 show that zoom-out features at multiple spatial levels improve superpixel segmentation, while multilayer classifiers and VGG-16 features produce the strongest validation models. The selected architecture substantially outperforms prior published methods on VOC test data, though predictions can remain under-smoothed.
- Feature ablations: Each zoom-out level contributes to mean class accuracy, with distant and global convnet features producing the most striking improvement.Local and proximal features remain important for segment localization.
- Feature ablations: Asymmetric loss yields dramatically better performance in the linear-model ablation.
- Classifier comparison: A two-layer classifier with 512 hidden units performs best with CNN-S, while a 1024-unit hidden layer performs best with VGG-16 on VOC 2012 val.Deeper networks generalize less well, even with dropout.
- Test evaluation: 58.4 mean IU is the best reported test result for a two-layer classifier using CNN-S features.
- Test evaluation: The selected zoom-out architecture achieves higher accuracies than previously published methods on VOC test sets, leading on 15 of 20 object categories.The largest category-level margins include dogs, cats, and trains.
- Qualitative behavior: Predictions tend to be under-smoothed and contain small irrelevant category islands, although a region-flipping post-process improves validation results by about 0.5%.The authors do not pursue this ad-hoc correction and defer more principled remedies.
4.5. Results on Stanford Background Dataset
On the Stanford Background Dataset, the zoom-out architecture was evaluated under five-fold cross-validation using multiple performance measures and outperformed two compared methods on both reported accuracies.
- SBD contains 715 outdoor-scene images with dense labels for eight categories.
- The established evaluation protocol uses five-fold cross-validation because SBD has no standard train/test partition.Per-pixel accuracy and average class accuracy are both commonly reported, with the latter differing from the VOC measure by not directly penalizing false positives.
- The zoom-out architecture obtains better class accuracy and pixel accuracy than and.
5. Conclusions
The paper concludes that multilevel zoom-out features and nonlinear feedforward classifiers can substantially advance semantic segmentation without explicitly representing its structured nature. It also identifies several directions for integrating learned features, end-to-end training, and structured inference.
- Conclusions: The paper explores feedforward semantic labeling of superpixels using multilevel zoom-out features, multilayer neural networks, and asymmetric loss.
- Conclusions: The method far surpasses existing state of the art despite its apparent simplicity and lack of explicit representation of segmentation structure.
- Conclusions: Segmentation can benefit from deep convolutional networks, alongside image classification, detection, and other recognition tasks.
- Future work: The authors plan to replace handcrafted local and proximal features with features learned from data and fine-tune the off-the-shelf networks used for distant and global extraction.
- Future work: The authors also plan to add zoom-out levels and eventually replace separate feature extractors with a single zoom-out network aligned with end-to-end learning.
- Future work: Structured inference is not dismissed; the architecture may benefit from inference that cleans up predictions, potentially unrolled into additional feedforward-network layers.