Source-linked AI summary
W-Net: A Deep Model for Fully Unsupervised Image Segmentation
Xide Xia, Brian Kulis
TL;DR
Unsupervised image segmentation remains important where pixel-level labels are difficult to obtain. W-Net combines two fully convolutional networks with reconstruction and soft normalized-cut losses, achieving near-human performance on Berkeley benchmarks, including 0.86 PRI versus 0.87 by humans.
Problem
Existing semantic segmentation methods require substantial pixelwise labeled data, motivating unsupervised segmentation for domains where such supervision is difficult to collect.
Method
W-Net concatenates two U-Net-like fully convolutional networks into an encoder-decoder trained with reconstruction error and soft normalized-cut losses.
Results
0.86 PRI versus 0.87 by humans, with W-Net favorably comparing against classical and recent segmentation methods on the Berkeley Segmentation Data Set.
Takeaways & Limitations
W-Net provides a fully unsupervised deep approach that achieves performance near human level by some metrics on the Berkeley Segmentation Data Set.
Takeaways & Limitations
The authors identify further refinement and an architecture that avoids additional postprocessing as future work.
Abstract
from arXiv · showhide
While significant attention has been recently focused on designing supervised deep semantic segmentation algorithms for vision tasks, there are many domains in which sufficient supervised pixel-level labels are difficult to obtain. In this paper, we revisit the problem of purely unsupervised image segmentation and propose a novel deep architecture for this problem. We borrow recent ideas from supervised semantic segmentation methods, in particular by concatenating two fully convolutional networks together into an autoencoder--one for encoding and one for decoding. The encoding layer produces a k-way pixelwise prediction, and both the reconstruction error of the autoencoder as well as the normalized cut produced by the encoder are jointly minimized during training. When combined with suitable postprocessing involving conditional random field smoothing and hierarchical segmentation, our resulting algorithm achieves impressive results on the benchmark Berkeley Segmentation Data Set, outperforming a number of competing methods.
1. Introduction
The paper addresses unsupervised image segmentation as an alternative to supervised semantic segmentation, whose pixelwise labeling requirements can be difficult to satisfy. It introduces W-Net, which combines an encoder-decoder architecture with reconstruction and normalized-cut objectives, followed by CRF smoothing and hierarchical merging.
- Motivation: Unsupervised segmentation produces pixelwise clusters corresponding to coherent image objects, extending a longstanding clustering-based vision problem.Classical approaches include normalized cuts, Markov random fields, and mean-based methods.
- Motivation: Supervised semantic segmentation has dominated recent deep-learning work but requires substantial pixelwise labeled training data that can be difficult to collect.These methods typically use fully convolutional networks to predict pixels and learn segments for novel images.
- W-Net: W-Net connects two fully convolutional networks, with an encoder producing a segmentation and a decoder reconstructing the image.Each network is similar to the U-Net architecture.
- W-Net: Training jointly minimizes autoencoder reconstruction error and a soft normalized-cut loss on the encoding layer.The normalized-cut objective encourages the encoded segmentation while reconstruction trains the autoencoder.
- Evaluation: The initial segmentation is postprocessed with fully connected CRF smoothing and hierarchical merging before evaluation on the Berkeley Segmentation Data Set benchmark.The study evaluates segmentation covering, probabilistic Rand index, and variation of information, reporting favorable comparisons with classical and recent methods.
2. Related Work
Related work spans traditional unsupervised segmentation, supervised deep semantic segmentation, fully convolutional networks, and encoder-decoder models. The paper adapts encoder-decoder ideas by mapping images to dense pixelwise segmentation layers rather than compact representations.
- Unsupervised segmentation: Traditional unsupervised segmentation commonly extracts color, brightness, or texture features from local patches before clustering pixels.Widely used methods include graph-based segmentation, Normalized Cuts, and Mean Shift; edge-detection methods have also been proposed.
- Deep segmentation: Deep neural networks are widely used for supervised semantic segmentation through pixel-wise annotations and classification.Several cited approaches use supervised deep networks to produce segmentation labels.
- Fully convolutional networks: Fully convolutional networks transform fully connected layers into convolutional layers, enabling arbitrary-size inputs and correspondingly sized output maps.This architecture supports pixelwise prediction for semantic segmentation.
- Encoder-decoders: Encoder-decoders learn compact representations with an encoder and reconstruct inputs with a decoder, while this paper instead uses a dense pixelwise segmentation layer.The proposed encoder preserves the input image’s spatial size rather than mapping to a low-dimensional space.
3. Network Architecture
W-Net extends a U-shaped encoder-decoder into a 46-layer W-shaped network that jointly reconstructs input images and predicts pixelwise segmentations without labels. Its encoder produces dense K-class predictions and is trained with differentiable soft normalized-cut and reconstruction objectives.
- Overall architecture: W-Net uses separate UEnc and UDec components in a W-shaped architecture for simultaneous image reconstruction and unlabeled segmentation prediction.The network has 46 convolutional layers organized into 18 modules; the first nine form the dense prediction base and the second nine form the reconstruction decoder.
- Encoder-decoder paths: The UEnc combines a contracting path for context with an expansive path for precise localization, using pooling for downsampling and transposed convolutions for upsampling.Feature channels double at each downsampling step and are halved along the expansive path.
- Convolutional operations: Most modules use depthwise separable convolutions, which independently model spatial and cross-channel correlations.Modules 1, 9, 10, and 18 are exceptions to this modification.
- Segmentation output and loss: The UEnc outputs a normalized 224 × 224 × K dense prediction, whose argmax gives a K-class label for every pixel.The corresponding hard normalized-cut objective is replaced during training by a differentiable soft version because argmax is nondifferentiable.
- Joint training objectives: Training alternates soft normalized-cut updates for UEnc with reconstruction-loss updates for the full W-Net, balancing segment consistency against reconstruction accuracy.The soft normalized-cut loss increases within-segment association and decreases between-segment disassociation, while reconstruction encourages encoded representations to retain input information.
4. Postprocessing
Postprocessing refines the encoder’s initial segmentation in two stages: fully connected CRF smoothing sharpens boundaries, followed by hierarchical merging that combines over-segmented regions into final segments.
- CRF smoothing: A fully connected CRF combines final UEnc responses with unary label probabilities and pairwise Gaussian-kernel penalties to improve fine-grained boundary localization.The unary potential is Φ(u) = −log p(u), while pairwise potentials penalize differing labels across pixels using two Gaussian kernels.
- CRF smoothing: After CRF processing, boundaries become sharper and small spurious regions are smoothed or removed.Before CRF refinement, the softmax output predicts object locations with coarse boundaries.
- Hierarchical merging: The argmax of the CRF output typically remains over-segmented, so the method converts boundaries into weighted maps using multi-scale local cues and global spectral-clustering measurements.The cues include brightness, color, and texture channels; mPb measures all image edges, while sPb captures salient curves.
- Hierarchical merging: Hierarchical segmentation uses contour2ucm, applying Oriented Watershed Transform to create an initial partition and Ultrametric Contour Map for greedy graph-based region merging.This hierarchy converts the over-segmented partition into the final image segments.
5. Experiments
Experiments train W-Net without ground-truth labels and evaluate it on BSDS300 and BSDS500 using segmentation-quality metrics. The soft normalized-cut loss improves hidden-representation consistency and smooths pixelwise predictions, while trading off reconstruction quality and convergence stability.
- Experimental setup: W-Net is trained on PASCAL VOC2012 without ground-truth labels, which are used only to evaluate segmentations on BSDS300 and BSDS500.PASCAL VOC2012 contains 11,530 images and 6,929 segmentations; BSDS300 and BSDS500 contain 300 and 500 images, respectively.
- Loss analysis: Including Jsoft−Ncut makes Jreconstr decrease more slowly and less stably, although excluding it yields faster reconstruction-loss convergence.At reconstruction-loss convergence, training with Jsoft−Ncut remains higher than training without it.
- Loss analysis: Adding Jsoft−Ncut smooths pixelwise predictions and produces a more consistent hidden representation, while reducing reconstruction quality relative to a classical encoder-decoder.Without the loss, the model reconstructs well but produces noisier, more discrete softmax outputs.
- Postprocessing: Hierarchical segmentation is applied to W-Net outputs after CRF smoothing, producing BSDS500 segmentations evaluated at the optimal dataset scale and optimal image scale.A W-Net+ucm variant combines CRF-smoothed W-Net output with UCM as initial boundaries.
- Benchmark comparison: The method is compared with ten existing unsupervised segmentation methods using Variation of Information, Probabilistic Rand Index, and Segmentation Covering.The comparison includes DC-Seg-full, gPb-owt-ucm, Taylor, Felz-Hutt, Mean Shift, Canny-owt-ucm, SWA, Chan Vese, Multiscale Normalized Cuts, and Quad-Tree.
- Benchmark comparison: On BSDS300 and BSDS500, W-Net has competitive performance relative to the computationally demanding gPb-owt-ucm method.Its initial boundaries come from the UEnc encoder followed by a fully connected CRF, with multi-scale local cues computed only on detected edges.
6. Conclusion
The paper introduces a fully unsupervised image-segmentation method built from concatenated U-Net variants, trained with reconstruction and soft normalized-cut losses. It targets settings where pixelwise labels are difficult to obtain and suggests future refinement through alternative losses or postprocessing.
- 6. Conclusion: The proposed approach concatenates two fully convolutional U-Net variants into an encoder-decoder for fully unsupervised image segmentation.Training iteratively minimizes the decoder’s reconstruction error and the encoder’s soft normalized cut.
- 6. Conclusion: The method is intended for domains where labeled pixelwise supervision is difficult to obtain, including biomedical image analysis.The authors also propose exploring different loss functions and postprocessing steps, ideally eliminating the need for additional postprocessing.
7. Appendix
The appendix presents additional BSDS500 image-segmentation results for W-Net and W-Net+ucm.
- Additional results: Additional results are shown for W-Net on BSDS500 images and for W-Net+ucm on BSDS500 images.W-Net results appear in Figures 9 and 10, while W-Net+ucm results appear in Figures 11 and 12.