Source-linked AI summary
Superpixel Segmentation with Fully Convolutional Networks
Fengting Yang, Qian Sun, Hailin Jin, Zihan Zhou
TL;DR
The paper addresses limited use of superpixels in deep networks and the loss of boundaries and details caused by low-resolution stereo processing and bilinear upsampling. It predicts superpixels on a regular grid with a fully convolutional network and uses them for trainable downsampling and upsampling. The method achieves competitive or state-of-the-art superpixel segmentation, runs at about 50fps, and improves disparity estimation on public datasets.
Problem
Superpixels are useful for reducing image primitives, but their integration into deep networks is limited because standard convolution operates on regular grids; low-resolution stereo processing also loses boundaries and details.
Method
A fully convolutional network predicts pixel-superpixel associations on a regular grid, which are used in a joint, end-to-end trainable downsampling/upsampling framework for stereo matching.
Results
The method achieves competitive or state-of-the-art superpixel segmentation at about 50fps and improves disparity estimation on public datasets.
Takeaways & Limitations
Predicted superpixels can support detail-preserving downsampling and upsampling in dense prediction networks while benefiting both superpixel segmentation and disparity estimation.
Takeaways & Limitations
The method constrains each pixel to association with one of nine nearby grid cells, favoring local spatial coherence and compactness over directly forming long or large superpixels.
Abstract
from arXiv · showhide
In computer vision, superpixels have been widely used as an effective way to reduce the number of image primitives for subsequent processing. But only a few attempts have been made to incorporate them into deep neural networks. One main reason is that the standard convolution operation is defined on regular grids and becomes inefficient when applied to superpixels. Inspired by an initialization strategy commonly adopted by traditional superpixel algorithms, we present a novel method that employs a simple fully convolutional network to predict superpixels on a regular image grid. Experimental results on benchmark datasets show that our method achieves state-of-the-art superpixel segmentation performance while running at about 50fps. Based on the predicted superpixels, we further develop a downsampling/upsampling scheme for deep networks with the goal of generating high-resolution outputs for dense prediction tasks. Specifically, we modify a popular network architecture for stereo matching to simultaneously predict superpixels and disparities. We show that improved disparity estimation accuracy can be obtained on public datasets.
1. Introduction
The paper uses a simple fully convolutional network to predict superpixels on a regular grid, then incorporates them into a trainable downsampling/upsampling framework for dense prediction. Experiments report competitive superpixel segmentation and improved stereo disparity estimation while preserving boundaries and details.
- Motivation: 3D convolution-based stereo networks often operate at reduced resolution, and bilinear upsampling can blur object boundaries and lose fine details.The cited stereo methods compute disparity volumes below the input resolution before producing full-resolution disparity maps.
- Motivation: Superpixels reduce image primitives by grouping perceptually similar pixels, but standard CNN convolutions are defined on regular grids rather than superpixels.This mismatch has limited the adoption of superpixels for dimension reduction in deep neural networks.
- Method: The proposed FCN predicts association scores between image pixels and regular grid cells, following a grid-based initialization strategy used by traditional superpixel algorithms.This formulation learns superpixel segmentation directly on the regular image grid.
- Method: The superpixel framework replaces stride-2 convolutions and bilinear upsampling in the task network to preserve object boundaries and fine details.The resulting joint framework is end-to-end trainable, allowing superpixel segmentation and the downstream task to influence one another.
- Results: 50fps is the reported runtime, while benchmark experiments show competitive or better superpixel segmentation and improved disparity estimation over the original PSMNet.The superpixel method is evaluated on BSDS500 and NYUv2, and disparity estimation is evaluated on SceneFlow, HR-VS, and Middlebury-v3.
2. Related Work
Related work spans traditional grid-initialized superpixel algorithms, learned superpixel segmentation, superpixel-assisted deep networks, and stereo methods using segmentation or low-resolution disparity volumes. The paper differs by directly predicting pixel-superpixel associations and integrating predicted superpixels into a dense prediction pipeline.
- Superpixel segmentation: Traditional methods initialize superpixels on regular grids or at regular seed intervals, then refine regions through growth, clustering, or pixel exchanges.Examples include Turbopixels, SEEDS, and geodesic-distance clustering methods.
- Superpixel segmentation: SLIC and its variants use hand-crafted positional, color, or manifold features with K-means, weighted K-means, or region growing.These methods differ in feature space and optimization procedure while retaining hand-crafted or engineered representations.
- Learned superpixels: Recent learned approaches estimate pixel affinity or features and may use differentiable K-means clustering, whereas this paper directly predicts the pixel-superpixel association map.The cited comparison identifies SSN as an end-to-end trainable learned superpixel method.
- Superpixels in DNNs: Superpixels have also been integrated into deep-learning pipelines to preserve image properties such as boundaries or to represent patterns compactly for downstream processing.Prior examples use pre-computed superpixels to manipulate learned features or convert image patterns into sequential representations.
- Superpixels in DNNs: Deformable convolution can provide adaptive receptive fields, but unlike superpixels it does not require every pixel to contribute to an output feature.The comparison distinguishes adaptive sampling from the superpixel representation constraint.
- Stereo matching: Segmentation-based stereo methods first divide images into regions and fit disparity models, while other work jointly segments reference images and estimates disparity.These approaches differ from the paper’s learned pixel-superpixel association framework.
3. Superpixel Segmentation Method
The method learns superpixels directly on a regular grid by predicting pixel–superpixel associations with an FCN, avoiding a separate clustering stage. It uses local assignments and task-specific losses, achieving competitive segmentation performance, better generalization in reported comparisons, and substantially faster runtime.
- Learning Superpixels on a Regular Grid: Each pixel searches only its 9 surrounding grid cells, producing a compact association tensor instead of evaluating all pixel–superpixel pairs.This local restriction reduces computational cost and favors spatial coherence, despite limiting the size of individual superpixels.
- Learning Superpixels on a Regular Grid: The method predicts a soft pixel–superpixel association map directly with a fully convolutional network on a regular grid.Each pixel is associated with nearby grid cells, and the highest-probability assignment determines its superpixel.
- Learning Superpixels on a Regular Grid: Unlike SSN, the network combines feature extraction and superpixel segmentation in one step rather than applying K-means clustering after CNN feature extraction.The single-step design is intended to make the method faster and easier to integrate into CNN frameworks.
- Network Architecture: The encoder-decoder predicts the association map using skip connections, with softmax applied at the prediction layer.The decoder progressively upsamples features while incorporating corresponding encoder features.
- Loss Functions: The loss jointly encourages similarity in a chosen pixel property and spatial compactness, supporting color or semantic-label properties.The formulation uses a task-specific distance metric and a weight balancing the two terms.
- Experiments: 3 to 8 times faster than SSN and more than 50 times faster than SEAL, the method achieves competitive or better segmentation results across reported BSDS500 and NYUv2 comparisons.On BSDS500 it is competitive or better on ASA and BR-BP and significantly higher on CO; on NYUv2 it outperforms SEAL and SSN on BR-BP and CO and is among the best on ASA.
4. Application to Stereo Matching
The paper integrates predicted superpixels into stereo matching to replace conventional resolution changes and preserve boundaries and fine details in high-resolution disparity outputs. Experiments compare fixed and jointly trained variants with PSMNet across synthetic and real-world datasets.
- Motivation: 3D convolution and low-resolution disparity processing limit high-resolution stereo matching, while bilinear upsampling can blur object boundaries and lose fine details.
- Network design: The method modifies PSMNet by replacing stride-2 convolutions and bilinear upsampling with a superpixel-based downsampling/upsampling scheme.The superpixel network predicts association maps, which are used to compute superpixel center maps as downsampled images.
- Training: Joint training combines disparity and superpixel objectives, whereas the fixed variant pretrains the superpixel network and keeps its parameters fixed during disparity training.
- SceneFlow: Ours joint achieves the lowest EPE on SceneFlow, while both proposed variants preserve fine details better than the original PSMNet.
- HR-VS: On HR-VS, both proposed models outperform PSMNet, with significantly lower EPE from joint training; the larger gain coincides with 16× rather than 4× upsampling.
- Middlebury-v3: On Middlebury-v3, the model outperforms PSMNet across all reported metrics, although the experiment targets verification of the scheme rather than the highest leaderboard rank.
5. Conclusion
The paper concludes that a simple FCN can efficiently produce generalizable superpixels, and that integrating them into stereo matching improves disparity accuracy by preserving boundaries and fine details.
- The proposed FCN is computationally efficient and consistently achieves state-of-the-art superpixel performance with good generalizability.
- Using superpixels in a popular stereo matching network yields higher disparity estimation accuracy while preserving object boundaries and fine details.
- Future work will apply superpixel-based downsampling/upsampling to object segmentation and optical flow estimation.
A. Supplementary Materials
The supplementary materials provide additional architectural specifications and qualitative results for superpixel segmentation and disparity estimation across the evaluated datasets.
- The supplement details the superpixel and stereo matching network architectures and reports additional qualitative results across BSDS500, NYUv2, SceneFlow, HR-VS, and Middlebury-v3.
A.1. Superpixel Segmentation Network
The superpixel segmentation network uses a standard encoder-decoder FCN with skip connections to predict a superpixel association map.
- The network uses an encoder-decoder design with skip connections to predict the superpixel association map Q.
- Batch normalization and leaky ReLU with negative slope 0.1 are used throughout the convolutional layers except the association prediction layer.
- The association prediction layer applies softmax to produce association outputs.
A.2. Stereo Matching Network
The stereo network is adapted to use superpixel-based downsampling and upsampling, with different configurations for standard- and high-resolution inputs.
- Architecture: SPPSMNet modifies PSMNet to perform superpixel-based downsampling and upsampling.The modified layers are highlighted in the architecture design.
- Standard-resolution configuration: A 4 × 4 superpixel grid performs 4× downsampling and upsampling for 256 × 512 inputs with maximum disparity D = 192.
- High-resolution configuration: High-resolution stereo matching uses 1024 × 2048 inputs, maximum disparity D = 768, and a 16 × 16 grid for 16× downsampling and upsampling.
- High-resolution configuration: The high-resolution configuration reduces GPU memory by narrowing selected channels and removing batch normalization from the superpixel segmentation network.The “cnv4a” and “cnv4b” channel counts are reduced from 256 to 128.
A.3.1 Superpixel Segmentation
On BSDS500 and NYUv2, the learning-based methods recover finer boundaries than SLIC, while the proposed method generally produces more compact superpixels than SEAL and SSN.
- Qualitative results: Learning-based methods recover more detailed boundaries than SLIC on BSDS500 and NYUv2.Examples include a windmill hub and a pillow on the right bed.
- Qualitative results: The proposed method usually generates more compact superpixels than SEAL and SSN.
A.3.2 Application to Stereo Matching
The superpixel-based stereo system better preserves fine details and handles some textureless regions than PSMNet across the shown datasets. Joint training also makes superpixels more compact and attentive to disparity boundaries.
- Disparity prediction: The method better preserves fine details than PSMNet on SceneFlow, HR-VS, and Middlebury-v3.Examples include a headset wire, street lamp post, and leaves.
- Disparity prediction: The method better handles some textureless areas, such as a car back on HR-VS.The passage attributes this to a larger input receptive field that leverages surrounding context.
- Joint training: Ours joint produces more compact superpixels and attends more to disparity boundaries than Ours fixed on HR-VS.Color boundaries misaligned with disparity boundaries can be ignored by Ours joint.
- Visual comparison: The disparity visualizations compare predicted disparity maps above corresponding error maps, where darker colors indicate lower EPE or error.
- Superpixel comparison: The HR-VS superpixel comparison does not enforce superpixel connectivity.