Source-linked AI summary
Superpixel Convolutional Networks using Bilateral Inceptions
Raghudeep Gadde, Varun Jampani, Martin Kiefel, Daniel Kappler, Peter V. Gehler
TL;DR
Semantic segmentation must model relationships among pixel labels, but per-pixel classifiers ignore those dependencies. This paper inserts bilateral inception modules into CNNs to propagate information between superpixels with learned, multi-scale bilateral filtering. The authors report improved results over baseline and DenseCRF approaches across three datasets, with faster inference than the compared DenseCRF techniques.
Problem
Per-pixel classifiers ignore relationships among labels, although pixels in a common image are not independent and segmentation methods therefore model output structure.
Source: §1 · pp. 1–2Method
Bilateral inception modules apply multi-scale bilateral filtering between superpixels in CNN representations, learning filter feature spaces end-to-end and producing full-resolution output.
Source: §1 · p. 3, §3.2 · p. 6Results
BI improves results on all three datasets over DenseCRF interpolation/inference techniques while being faster, and compares favorably with some dense-pixel prediction methods.
Source: §1 · p. 3Takeaways & Limitations
Image-adaptive convolutions retain the appealing effect of sharp-edged segmentation masks, while superpixel representation and full-resolution interpolation are orthogonal choices.
Source: §5 · p. 14Superpixels are used for computational reasons, and their quantization error enters only when they are used for interpolation; other sampling strategies remain to be explored.
Source: §3.3 · p. 8, Hierarchical Clustering Analysis · p. 12
Abstract
from arXiv · showhide
In this paper we propose a CNN architecture for semantic image segmentation. We introduce a new 'bilateral inception' module that can be inserted in existing CNN architectures and performs bilateral filtering, at multiple feature-scales, between superpixels in an image. The feature spaces for bilateral filtering and other parameters of the module are learned end-to-end using standard backpropagation techniques. The bilateral inception module addresses two issues that arise with general CNN segmentation architectures. First, this module propagates information between (super) pixels while respecting image edges, thus using the structured information of the problem for improved results. Second, the layer recovers a full resolution segmentation result from the lower resolution solution of a CNN. In the experiments, we modify several existing CNN architectures by inserting our inception module between the last CNN (1x1 convolution) layers. Empirical results on three different datasets show reliable improvements not only in comparison to the baseline networks, but also in comparison to several dense-pixel prediction techniques such as CRFs, while being competitive in time.
1 Introduction
Semantic segmentation must account for relationships among pixel labels, and the paper’s bilateral inception (BI) module propagates information between superpixels using learned, edge-aware filtering. The authors report improved results over DenseCRF techniques on all three datasets while being faster.
1 Introduction
- Semantic segmentation assigns a label to every image pixel and can be formulated as structured prediction, evaluated against human labels using losses such as IoU or pixel-wise Hamming loss.A patch classifier evaluated across image locations is one direct approach, but it ignores relationships among output labels.Source: §1 · p. 1
- Because pixel labels are related rather than independent, segmentation methods commonly model output structure; CRFs combine pixel predictions with shared structure through unary, pairwise, and higher-order factors.Pixels that are spatially and photometrically similar are hypothesized to be more likely to share a label; DenseCRF encodes both similarities.Source: §1 · p. 2
- The BI module performs multi-scale bilateral filtering between superpixels, learning feature spaces and other parameters end-to-end while providing full-resolution output without post-processing.It propagates information across different spatial CNN units using superpixel layouts rather than the CNN’s spatial grid.Source: §1 · p. 3
- BI modules improve results on all three datasets over DenseCRF interpolation/inference techniques while being faster, and compare favorably with some dense-pixel prediction methods.The modules are inserted into existing CNN segmentation models.Source: §1 · p. 3
2 Related Work
Prior CNN segmentation work incorporates output structure through CRFs or structural layers, with approaches limited by expensive inference, small receptive fields, or costly long-range connections. This work instead propagates information across superpixels using bilateral filters with learned feature spaces.
2 Related Work
- CNN–CRF approaches combine CNN unary predictions with pairwise or higher-order factors, while this work filters intermediate CNN representations instead of label unaries and reports better results than DenseCRF while being faster.Prior approaches include learned mean-field inference and faster domain-transform filtering.Source: §2 · p. 3
- Structural CNN layers include local deconvolution, spatial pyramid pooling, normalized-cut layers, and dilated convolutions, while long-range pixel connections can raise complexity.The paper says its method models long-range superpixel dependencies without compromising computational efficiency.Source: §2 · pp. 3–4
- Unlike independent superpixel classifiers, this method propagates information across superpixels using bilateral filters with learned feature spaces.The approach defines convolutions over superpixels through connectivity without requiring superpixel rearrangement.Source: §2 · p. 4
- Unlike bilateral-filtering approaches that use predefined feature spaces, this method backpropagates through explicitly computed kernels to learn task-specific spaces within end-to-end trainable CNNs.Prior approaches backpropagate through permutohedral-lattice approximations to learn filter parameters or optimize in bilateral space.Source: §2 · p. 4
3 Superpixel Convolutional Networks
The section develops superpixel-based CNN representations and bilateral inception modules that filter intermediate activations across feature spaces. Superpixels support information propagation between image regions and can also map lower-resolution representations back to full image resolution.
3.1 Superpixels
- Superpixels reduce the prediction units from pixels to regions, but introduce quantization error when a region contains pixels with different ground-truth labels.The paper reports that this effect is small compared with current best segmentation performance on Pascal VOC and MINC.Source: §3.1 · p. 5, Figure 2 · p. 5
3.2 Bilateral Inceptions
- Bilateral inception combines multiple bilateral-filter outputs linearly, with filters operating at different scales in a shared transformed feature space.The module can be inserted between layers of existing CNN architectures.Source: §3.2 · pp. 5–7
- The bilateral-filtering motivation is to propagate information between intermediate CNN activations, rather than only between the condensed label beliefs used by a DenseCRF.Both approaches encode the prior that nearby pixels with similar color are likely to share a label.Source: §3.2 · pp. 7–8
3.3 Superpixel Convolutions
- Superpixel representations use each region’s mean color and position as features, so spatially close, similarly colored regions influence one another more strongly.The module can switch between grid and superpixel representations.Source: §3.3 · p. 8
- Superpixels enable propagation to full image resolution, while their quantization error enters when they are used for interpolation.The paper also notes that random image points are possible intermediate representations and that dense CNN evaluation performed poorly relative to interpolation methods.Source: §3.3 · p. 8
3.2 Bilateral Inceptions
- The module’s filter scales, feature transformation, and combination weights are learned through backpropagation, including gradients with respect to its inputs.The implementation decomposes filtering into five sub-layers and initializes filter scales at separated values.Source: §3.2 · p. 6, Figure 3 · p. 6, Backpropagation and Training · p. 8
4 Experiments
The experiments insert bilateral inception modules into existing CNN architectures and evaluate semantic, material, and street-scene segmentation across three datasets.
4 Experiments
- The study evaluates modified CNN architectures on Pascal VOC12, MINC, and Cityscapes, using DeepLab, CRFasRNN, and AlexNet-based models.For Cityscapes, the base DeepLab model was trained on half-resolution images and bilinearly interpolated to full resolution.Source: §4 · p. 9, §4.1 · p. 9, §4.2 · p. 12
4.1 Semantic Segmentation
- On Pascal VOC12 test data, full-network fine-tuning (FULL) raises DeepLab IoU by 5.2 points over the baseline and 1.4 over DenseCRF; reported figures are test numbers, not validation numbers.Source: DeepLab · p. 10
- On Pascal VOC12, the CRFasRNN-based BI model reaches 74.9 IoU, 0.2 above CRFasRNN, while removing deconvolution and DenseCRF and running much faster.The comparison reports 72.0 IoU for the deconvolution-only model and 74.7 after ten mean-field steps.Source: CRFasRNN · p. 11, Table 2 · p. 11
- With trained parameters reused across hierarchical superpixel layouts, validation IoU decreases slowly as the number of superpixels falls, then drops below 200.The network was trained with 1000 gSLIC superpixels per image; the authors identify allocating resolution selectively as future work.Source: Hierarchical Clustering Analysis · pp. 11–12, Figure 6 · p. 12
4.2 Material Segmentation
- On MINC, bilateral inception improves total and class-averaged pixel accuracy by 12% over CNN predictions and 2–4% over CNN+DenseCRF.The model uses single-scale, no-sliding CNN predictions with BI modules after FC7 and FC8; no learned setup is provided because training data are very limited.Source: §4.2 · pp. 12–13, Table 3 · p. 12
4.3 Street Scene Segmentation
- On Cityscapes, BI modules yield modest improvements over the base model and outperform DenseCRF while running faster.The experiments used either BI6(2) alone or BI6(2)-BI7(6), with 6000 superpixels.Source: §4.3 · p. 13, Table 4 · p. 13
5 Conclusion
Bilateral Inception brings CRF-style bilateral filtering into intermediate CNN representations, using image-adaptive layouts and task-specific learned feature spaces. The same representation can interpolate coarser features to full-image resolution.
5 Conclusion
- BI filters intermediate CNN representations rather than label beliefs, while jointly learning task-specific feature spaces and using image-adaptive layouts.The 1 × 1 convolution computations scale with the number of superpixels, and the representation can also interpolate coarser features to full image resolution.Source: §5 · p. 14
- Sharp segmentation edges do not depend on superpixels; their use in FC layers is orthogonal to interpolation choices such as bilinear interpolation, up-convolutions, or DenseCRFs.The authors plan to investigate higher-layer sampling strategies and image-adaptive ideas for video.Source: §5 · p. 14
- The authors present BI as an easy-to-implement way to incorporate CRF-factor structure into CNN forward architectures, applicable to CNNs for structured output prediction.Source: §5 · p. 14