Source-linked AI summary
CRF Learning with CNN Features for Image Segmentation
Fayao Liu, Guosheng Lin, Chunhua Shen
TL;DR
CRF segmentation has largely relied on hand-crafted features, while contextual co-occurrence relationships also need to reflect spatial layout. The paper transfers ImageNet CNN features into SSVM-based CRF learning and adds spatially related co-occurrence pairwise potentials, achieving strong benchmark performance and new baselines. Its co-occurrence-potential construction assumes sufficiently large training data.
Problem
Most CRF segmentation studies rely on hand-crafted features, despite the importance of feature representation and contextual relationships between co-occurring objects.
Method
The method transfers ImageNet-trained CNN features to construct superpixel potentials, learns CRF parameters with SSVM, and incorporates spatially related co-occurrence pairwise potentials during inference.
Results
The method outperforms compared methods on the Weizmann horse and Graz-02 datasets, provides new baselines across five benchmarks, and matches Carreira et al.'s accuracy on PASCAL VOC 2011 under the same settings.
Takeaways & Limitations
ImageNet CNN features combined with SSVM-based CRF learning are reported as effective for image segmentation, while spatial co-occurrence potentials provide further performance gains.
Takeaways & Limitations
Constructing the co-occurrence pairwise potentials assumes that the training data is sufficiently large.
Abstract
from arXiv · showhide
Conditional Random Rields (CRF) have been widely applied in image segmentations. While most studies rely on hand-crafted features, we here propose to exploit a pre-trained large convolutional neural network (CNN) to generate deep features for CRF learning. The deep CNN is trained on the ImageNet dataset and transferred to image segmentations here for constructing potentials of superpixels. Then the CRF parameters are learnt using a structured support vector machine (SSVM). To fully exploit context information in inference, we construct spatially related co-occurrence pairwise potentials and incorporate them into the energy function. This prefers labelling of object pairs that frequently co-occur in a certain spatial layout and at the same time avoids implausible labellings during the inference. Extensive experiments on binary and multi-class segmentation benchmarks demonstrate the promise of the proposed method. We thus provide new baselines for the segmentation performance on the Weizmann horse, Graz-02, MSRC-21, Stanford Background and PASCAL VOC 2011 datasets.
1. Introduction
The paper addresses limitations of hand-crafted features in CRF-based segmentation by transferring ImageNet CNN features and adding spatial co-occurrence reasoning. It reports new segmentation baselines and further gains from co-occurrence pairwise potentials.
- Motivation: Hand-crafted color, HOG, SIFT, and bag-of-words features have dominated CRF-based segmentation, motivating learned representations.The paper highlights feature representation as highly influential for segmentation accuracy.
- Motivation: Spatial co-occurrence statistics capture likely neighboring object pairs and mutual exclusions, such as grass with cows rather than monitors.The paper uses these contextual relationships to constrain plausible labels.
- Contributions: The method transfers CNN features learned from labelled ImageNet data to segmentation and evaluates features across depths against traditional and unsupervised alternatives.This comparison is used to assess the value of cross-domain CNN representations for segmentation.
- Contributions: SSVM-based CRF learning with CNN features provides new segmentation baselines on Weizmann horse, Graz-02, MSRC-21, Stanford Background, and PASCAL VOC 2011.The contribution is stated across binary and multi-class benchmark datasets.
- Contributions: Spatially related co-occurrence pairwise potentials are incorporated during inference and yield a further performance boost.The potentials favor frequently co-occurring object pairs in particular spatial arrangements.
2. Related work
Related work combines CNN-based representations, CRF learning, and co-occurrence reasoning for segmentation. This paper differs by transferring ImageNet CNN features and modeling spatial relations in co-occurring label pairs.
- CNN-based segmentation: Earlier convolutional-network scene parsing used supervised pixel-input CNNs and achieved only marginal improvement over CRF-based segmentation.The paper contrasts this earlier approach with transferred deep CNN features combined with SSVM-based CRF learning.
- CNN-based segmentation: ImageNet-trained CNN features were also transferred successfully to object detection and scene labelling, motivating their use for segmentation.The paper identifies this prior transfer-learning evidence as inspiration for its approach.
- Co-occurrence modeling: Prior co-occurrence methods used post-processing or global potentials, whereas this work incorporates spatial relations directly into pairwise potentials.The proposed formulation distinguishes co-occurrence pairs by their spatial layout.
3. Method
The method learns CRF segmentation models with feature mappings built from superpixel observations and combines standard potentials with spatial co-occurrence context during inference.
- Segmentation using CRF models: CRF models label superpixels by minimizing an energy with unary and pairwise potentials that depend on observations and learned parameters.Inference seeks a maximum a posteriori labelling over the image's superpixel nodes and edges.
- Learning CRF in the large-margin framework: SSVM learning encourages the ground-truth labelling to have lower energy than incorrect labellings by at least a loss-dependent margin.The optimization iteratively finds the most violated constraint for each training example.
- Learning CRF in the large-margin framework: The energy is linear in w, with unary and pairwise potentials represented as inner products between parameters and corresponding feature mappings.The combined parameter vector stacks the unary and pairwise parameter vectors.
- Implementation details: Unary mappings encode the active class indicator together with each superpixel's feature vector, while multiclass models may substitute one-vs-all SVM confidence scores.This reduces dimensionality when the learned superpixel features are high-dimensional.
- Implementation details: Standard pairwise mappings penalize different neighbouring labels using a relation weight based on shared boundary length or inverse colour difference.These features enforce spatial smoothness between neighbouring superpixels.
- Inference with co-occurrence pairwise potentials: Spatial co-occurrence potentials favour frequently co-occurring label pairs in specified layouts and assign infinite cost to unseen pairs, preventing implausible labellings.The layouts are above, below, left, and right; a trade-off parameter α is tuned from 0.5 to 2 on validation data.
4. Experiments
Experiments compare CNN representations, CRF learning variants, and spatial pairwise potentials across binary and multi-class segmentation benchmarks. Transferred CNN features generally outperform traditional and unsupervised features, while co-occurrence potentials further improve several multi-class results.
- 4.2. Baseline Comparison: CNN features outperform traditional bag-of-words and unsupervised learned features on the Weizmann horse and Graz-02 benchmarks.On Weizmann horse, CNN features perform better in both SVM and SSVM; on Graz-02, CNN features are the strongest feature family.
- 4.2. Baseline Comparison: SSVM-based CRF learning produces substantially better segmentations than simple SVM models on the binary benchmarks.This pattern is reported for both the Weizmann horse and Graz-02 datasets.
- 4.2. Baseline Comparison: The 6th CNN layer performs best on the binary datasets, whereas the 7th layer performs best on MSRC-21.The authors select 6th-layer features for binary datasets and 7th-layer features for multi-class datasets in later comparisons.
- 4.3. State-of-the-art comparison: The method outperforms all compared methods on Weizmann horse and Graz-02, except for one MSRC-21 comparison method.On MSRC-21, the cited exception has slightly higher global accuracy but does not report average per-category accuracy.
- 4.3. State-of-the-art comparison: Co-occurrence pairwise potentials improve multi-class results and outperform mutex potentials on MSRC-21, Stanford Background, and PASCAL VOC 2011.Mutex potentials provide no boost on Stanford Background, while co-occurrence potentials yield further improvement; PASCAL VOC shows constant improvements on most categories.
- 4.3. State-of-the-art comparison: On PASCAL VOC 2011, the method matches Carreira et al. under the same train/validation settings while using 4096-dimensional descriptors instead of 33589.The comparison uses the VOC train/validation set and does not use an extra dataset.
5. Conclusion
The paper combines ImageNet-pretrained CNN features with SSVM-based CRF learning for image segmentation, incorporating spatial co-occurrence pairwise potentials during inference. Experiments across five segmentation datasets demonstrate advantages and establish new performance baselines.
- The method learns CRFs with SSVM using features from a pretrained deep CNN transferred from ImageNet.The CNN features support CRF learning for image segmentation.
- Inference incorporates co-occurrence pairwise potentials to exploit contextual relationships between object labels.These potentials are part of the proposed CRF inference framework.
- Experiments on five segmentation datasets demonstrate the method’s advantages and provide new baselines for further research.The evaluated datasets are Weizmann horse, Graz-02, MSRC-21, Stanford Background, and PASCAL VOC 2011.