Source-linked AI summary
Scene Parsing with Multiscale Feature Learning, Purity Trees, and Optimal Covers
Clément Farabet, Camille Couprie, Laurent Najman, Yann LeCun
TL;DR
Scene parsing requires labeling every pixel while integrating object detection, segmentation, recognition, and context over varying spatial ranges. The paper combines learned multiscale raw-pixel features with hierarchical segments and an entropy-minimizing cover, achieving state-of-the-art accuracy with subsecond inference and no post-training parameter tuning.
Problem
Full scene labeling must simultaneously solve detection, segmentation, recognition, and contextual integration, with pixel categories depending on both short- and long-range information.
Method
The method combines dense multiscale raw-pixel convolutional features, a segmentation tree, region-wise aggregation, class-histogram estimation, and an entropy-based optimal purity cover.
Results
The system achieves state-of-the-art accuracy across the Stanford Background, SIFT Flow, and Barcelona datasets while dramatically outperforming competing models in inference time.
Takeaways & Limitations
The trained system provides parameter-free full-image parsing with linear or almost-linear processing and labels a 320 × 240 image in less than 1 second on a conventional CPU.
Takeaways & Limitations
The system relies on a single segmentation tree constructed from image gradients and assumes the correct segmentation is contained in that tree.
Abstract
from arXiv · showhide
Scene parsing, or semantic segmentation, consists in labeling each pixel in an image with the category of the object it belongs to. It is a challenging task that involves the simultaneous detection, segmentation and recognition of all the objects in the image. The scene parsing method proposed here starts by computing a tree of segments from a graph of pixel dissimilarities. Simultaneously, a set of dense feature vectors is computed which encodes regions of multiple sizes centered on each pixel. The feature extractor is a multiscale convolutional network trained from raw pixels. The feature vectors associated with the segments covered by each node in the tree are aggregated and fed to a classifier which produces an estimate of the distribution of object categories contained in the segment. A subset of tree nodes that cover the image are then selected so as to maximize the average "purity" of the class distributions, hence maximizing the overall likelihood that each segment will contain a single object. The convolutional network feature extractor is trained end-to-end from raw pixels, alleviating the need for engineered features. After training, the system is parameter free. The system yields record accuracies on the Stanford Background Dataset (8 classes), the Sift Flow Dataset (33 classes) and the Barcelona Dataset (170 classes) while being an order of magnitude faster than competing approaches, producing a 320 \times 240 image labeling in less than 1 second.
1. Overview
The paper addresses scene parsing by combining multiscale learned features with a segmentation tree and an entropy-based optimal cover. The resulting system achieves accurate, fast, parameter-free full-image labeling.
- Full scene labeling must simultaneously detect, segment, recognize, and contextually integrate objects, including dependencies spanning short and long distances.
- The method combines dense multiscale feature extraction, a minimum-spanning-tree segmentation hierarchy, region-wise feature aggregation, class-histogram estimation, and optimal purity cover.
- The feature extractor learns low-level and mid-level representations end to end from raw pixels, reducing reliance on hand-engineered features.
- The optimal cover selects tree nodes whose regions cover the image while minimizing average class-distribution entropy.
- Less than 1 second is required to produce a full parse of a 320 × 240 image on a conventional CPU.
- The trained system is parameter free, and its operations are linear or almost linear in the number of pixels.
2. Related work
Prior scene-parsing systems commonly combine candidate segments and graphical-model inference to obtain consistent image-wide labelings. This paper instead emphasizes dense multiscale ConvNet features, boundary-aligned segment representations, and an efficient tree-node cover optimized for class purity.
- Many prior methods use MRFs, CRFs, or other graphical models to enforce labeling consistency and incorporate context.
- Existing approaches often begin with super-pixels or other segment candidates, then extract features from individual and neighboring segments before inference.
- Socher et al. aggregate segments greedily with a trainable scoring function, but their deep feature extractor operates on hand-engineered features.
- Multiscale dense extraction addresses the challenge of using wide context for local decisions by aligning coarse feature maps with the finest scale.
- Trees provide candidate segments and support fast inference, while this paper selects covering tree nodes by minimizing a criterion.
- ConvNets compute dense features efficiently over large images and can learn low-level and mid-level features directly from raw pixels.
- Unlike earlier raw-pixel ConvNet scene parsing, this system uses boundary-based over-segmentation and feature pooling for size-independent segment representations.
3. An end-to-end trainable model for scene parsing
The model combines multiscale convolutional features with a hierarchy of image segments to select a purity-optimized cover for scene parsing. Dense features encode broad context, while component-level descriptors and tree optimization support spatially coherent labeling.
- 3.2. Parameter-free hierarchical parsing: The hierarchy of segmentations supplies multiple spatial observation levels, allowing the system to choose components that best explain each pixel.A connected component containing each pixel is selected by minimizing its associated cost; restricting candidates to a tree makes the search efficient.
- 3.1. Scale-invariant, scene-level feature extraction: Multiscale convolutional networks extract dense features from image pyramids, combining local patch and scene-level descriptors at every pixel.Networks operate across scales with shared parameters; outputs are upsampled and concatenated into a common feature map.
- 3.2.1 Optimal purity cover: The optimal cover is formed by taking the minimum-cost component along each leaf-to-root path and combining the selected components into an image cover.The selected components may overlap, while the resulting labeling is expressed over disjoint image sets.
- 3.2.2 Producing the confidence costs: Component costs represent class purity: ground-truth costs use class-distribution entropy, while test-time costs are predicted from component descriptors.The classifier estimates the distribution of categories present in each component, enabling purity-based selection without ground-truth labels at test time.
4. Training procedure
Training proceeds in two stages: first learning discriminative pixel features, then training a component classifier to predict label distributions used for purity costs.
- 4.1. Learning discriminative scale-invariant features: The feature extractor is first trained for pixelwise classification so its dense representations are maximally discriminative.This stage uses normalized linear-classifier predictions and multiclass cross entropy with hard pixel targets.
- 4.2. Teaching a classifier to find its best observation level: After feature learning, training hierarchies and component descriptors are constructed over the training data for classifier learning.The resulting descriptors Ok form a new training set for the component classifier.
- 4.2. Teaching a classifier to find its best observation level: The component classifier minimizes KL-divergence between true label histograms and predicted distributions, producing purity costs for later parsing.Component targets are normalized histograms rather than hard pixel labels; the predictions are used in the purity-cost equation.
5. Experiments
Experiments compare the complete scene-parsing system with a multiscale-network baseline across three datasets, while examining hierarchy construction, feature training, sampling, accuracy, and computation.
- Segmentation hierarchy: The hierarchy is built from raw image gradients using a volume criterion, with non-informative components smaller than 100 pixels removed.Graph-cut, Kruskal, and Power Watersheds partitioning methods performed systematically worse than the optimal-cover method.
- Experimental setup: The Stanford Background experiments compare a multiscale-network baseline with the complete model, while the other datasets use the complete model only.Results are reported in Tables 1–3, with SIFT Flow parses illustrated in Figure 4.
- Stanford Background: The baseline achieves good pixelwise accuracy but produces poor spatial consistency and object delineation.It trains the multiscale network as a per-location class predictor using Lcat.
- Stanford Background: The complete network-and-hierarchy system performs significantly better than the baseline, particularly in object delineation.The model uses 8, 33, or 170 output units for Stanford Background, SIFT Flow, and Barcelona, respectively.
- Training choices: Balanced class frequencies improve small-object discrimination on SIFT Flow but reduce overall pixelwise accuracy, whereas they perform poorly on Barcelona.The Barcelona result is attributed to many classes with few training examples, causing faster overfitting.
- Computation: Convolutional networks offer a computational advantage over competing algorithms, with Stanford Background results typically obtained in 24 hours of training on a regular server.The passage reports training time but does not provide the competing algorithms’ exact compute values.
6. Discussion
The paper presents a raw-pixel, multiscale convolutional framework that combines learned region features with purity-based optimal covers, achieving state-of-the-art accuracy and fast inference while retaining a tree-coverage assumption.
- Contributions: The framework learns low- and mid-level features directly from raw pixels using a supervised multiscale convolutional network.This avoids engineered features and trains the feature extractor end to end.
- Evaluation: The paper reports performance using natural and balanced class-frequency sampling on both the SIFT Flow and Barcelona datasets.Table 2 covers SIFT Flow and Table 3 covers Barcelona, each reporting per-pixel and average per-class accuracy.
- Contributions: Class purity guides whether segments represent single objects, and an efficient optimal-cover procedure selects a cover maximizing overall segmentation purity.The paper identifies purity criteria and optimal cover as two key contributions.
- Results: State-of-the-art accuracy is reported on Stanford Background, SIFT Flow, and Barcelona, measured per pixel and averaged per class.The system also dramatically outperforms competing models in inference time.
- Limitations: The current system relies on one segmentation tree built from image gradients and assumes the correct segmentation is contained in that tree.Proposed future directions include multiple trees, alternative graphs, and structured learning for better low-level features.