Source-linked AI summary
Learning random-walk label propagation for weakly-supervised semantic segmentation
Paul Vernaza, Manmohan Chandraker
TL;DR
Semantic segmentation lacks large training datasets because dense annotation is expensive. The paper introduces RAWKS, which uses differentiable random-walk propagation from sparse labels to jointly train a propagator and segmentation predictor. The method produces uncertainty-aware dense labels and achieves encouraging results on challenging benchmarks, while relying on assumptions about label coverage and facing resolution and computational constraints.
Problem
Semantic-segmentation datasets are much smaller than those for classification and detection because obtaining annotations is substantially more expensive.
Method
RAWKS jointly trains a CNN segmentation predictor and a differentiable random-walk label propagator using sparse clicks or scribbles, with uncertainty estimates incorporated into the loss.
Results
RAWKS produces encouraging results on challenging benchmarks, and its propagated labels are nearly as good as or better than the best superpixel-based results on VOC and CONTEXT.
Takeaways & Limitations
The framework can obtain competitive segmentation labels at a fraction of the cost or collect labeled data at larger scale for a given budget.
Takeaways & Limitations
The probabilistic model assumes at least one label for each connected component of the true label image, and violations may yield poor results.
Abstract
from arXiv · showhide
Large-scale training for semantic segmentation is challenging due to the expense of obtaining training data for this task relative to other vision tasks. We propose a novel training approach to address this difficulty. Given cheaply-obtained sparse image labelings, we propagate the sparse labels to produce guessed dense labelings. A standard CNN-based segmentation network is trained to mimic these labelings. The label-propagation process is defined via random-walk hitting probabilities, which leads to a differentiable parameterization with uncertainty estimates that are incorporated into our loss. We show that by learning the label-propagator jointly with the segmentation predictor, we are able to effectively learn semantic edges given no direct edge supervision. Experiments also show that training a segmentation network in this way outperforms the naive approach.
1. Introduction
Semantic-segmentation datasets are much smaller than those for classification and detection because annotation is substantially more expensive. RAWKS addresses this gap by jointly learning sparse-label propagation and image-based segmentation, with differentiable random-walk propagation enabling semantic-edge learning without direct edge supervision.
- Semantic-segmentation datasets remain orders of magnitude smaller than classification and detection datasets because annotation is substantially more expensive.
- The proposed method trains CNN-based segmentation networks from sparse annotations such as image scribbles.
- RAWKS jointly learns networks that propagate sparse labels to unlabeled points and predict true labels from the image alone.
- Differentiable probabilistic label propagation lets gradient-based optimization learn semantic boundaries without direct edge observations.
- Random-walk hitting probabilities provide the propagation mechanism and enable efficient inference and gradient-based learning.
2. Method
RAWKS jointly learns semantic boundaries and image-only labels from sparse annotations by propagating labels with differentiable random-walk hitting probabilities. The resulting uncertainty-aware objective enables end-to-end training through efficient linear-system inference and backpropagation.
- Random-walk backpropagation: Differentiability permits pure backpropagation through label propagation, allowing boundary scores to be learned without alternating optimization.Derivatives with respect to boundary scores are obtained through a sparse adjoint system after expressing the inference constraints in matrix form.
- Random-walk label propagation: Random-walk hitting probabilities assign each pixel a label distribution according to paths that reach sparse labels while crossing predicted boundaries.Path probabilities decay exponentially with accumulated boundary scores, and the probability of a label is the probability of hitting a point carrying that label.
- Overview: RAWKS trains a segmentation predictor and semantic edge predictor jointly from sparse labels using mutually consistent propagated and predicted label distributions.The image, sparse labels, and predicted boundaries produce propagated probabilities that are compared with the segmentation predictor through a cross-entropy objective.
- Probabilistic justification: The probabilistic justification assumes conditional independence of dense labels given sparse labels and boundaries, with boundaries treated as a deterministic function of the image.These assumptions imply that image information beyond sparse labels and predicted boundaries does not affect dense labels, and poorly labeled connected components can yield poor results.
- Random-walk inference: The propagated probabilities are computed efficiently by forming partition-function recursions and solving a sparse linear system with labeled and out-of-image boundary conditions.Partition functions sum path weights for each starting pixel and terminal label; the resulting system has a unique solution from which label probabilities are derived.
- Uncertainty-weighting the loss: The probabilistic formulation produces uncertainty estimates that downweight loss where propagated labels may be incorrect, while entropy regularization prevents uniformly high-entropy solutions.The loss uses weights w(x) = exp(−αH(P_y(x)|ŷ,B_φ,I)); uncertainty is especially relevant near incorrectly predicted boundary gaps.
3. Related work
RAWKS differs from prior weak-supervision methods by learning semantic boundaries and label propagation jointly with differentiable, gradient-based training rather than relying on fixed heuristics or alternating optimization.
- Comparison with prior weak supervision: RAWKS learns an inductive rule for predicting boundaries and propagating labels, unlike Lin et al.'s non-adaptive superpixels and CRF potentials.This allows the propagation model to adapt to larger datasets in a data-driven way.
- Sparse-label supervision: Earlier sparse-label methods used objectness to mitigate overfitting, whereas RAWKS directly propagates sparse labels.
- Constraint-based approaches: RAWKS imposes spatial label-propagation constraints, unlike methods that impose only aggregate regional statistics.Its constraints provide for learning boundaries.
- Learning semantic edges: RAWKS learns semantic edges without direct edge supervision or bounding-box supervision, unlike related CNN-based edge-learning methods.
- Random-walk foundations: Random walks had previously been used for interactive segmentation, but not for learning boundary scores or general semantic image segmentation.
4. Experiments
Experiments evaluate RAWKS on VOC 2012 and PASCAL CONTEXT using jointly trained boundary and segmentation networks, showing competitive performance and high-quality propagated labels, while exposing resolution limits.
- Implementation: RAWKS uses fully convolutional ResNet-101 networks for semantic boundaries and segmentation, producing outputs at four-times downsampled resolution.
- Datasets and protocol: Training uses publicly available scribble annotations and the established VOC 2012 and PASCAL CONTEXT splits.The VOC setup contains 10582 training images and 1449 validation images.
- Quantitative evaluation: RAWKS slightly underperforms Lin et al. on VOC 2012 validation but slightly outperforms it on PASCAL CONTEXT.The study evaluates both image-only predictions Q and propagated labels P.
- Superpixel comparison: Propagated labels are nearly as good as, or better than, the best possible superpixel labelings, and training on them is competitive with optimal superpixel supervision.Superpixel baselines cannot improve with more training data, whereas RAWKS propagation is refined as larger datasets are used.
- Dataset differences: RAWKS performs relatively better on CONTEXT, where adaptive boundaries may better suit its 60 classes than VOC's 21 classes.The authors hypothesize that joint propagator training may also regularize against overfitting on CONTEXT.
- Qualitative results: Qualitative results show high-quality semantic boundaries and propagated labels close to dense ground truth, with boundary predictions generalizing to validation images.
- Limitations: Fine boundaries such as a boat mast and airplane wing are frequent error regions because limited resolution can miss boundaries.
- Limitations: Quarter-resolution propagation reduces computational cost but limits accuracy; solving random-walk systems consumes about 800 ms of a 1.1-second pass.The authors anticipate GPU implementation could increase resolution and prediction accuracy.
5. Conclusions
The framework uses sparse clicks or scribbles to reduce semantic-segmentation labeling costs while supporting larger-scale labeled-data collection. Its random-walk propagator is differentiable, produces uncertainty estimates, and is not artificially bounded by superpixel baselines.
- RAWKS trains semantic-segmentation networks using only sparse clicks or scribbles, mitigating the expense of procuring dense labels.
- The random-walk label-propagation mechanism is differentiable and usable with deep neural network architectures for semantic segmentation.
- The predictor-propagator paradigm produces uncertainty estimates for inferred dense labels from sparse labels.
- For a fixed dataset, the framework targets competitive labels at lower cost; for a fixed budget, it targets labeled data at larger scale.
- Unlike prior superpixel-based approaches, RAWKS label propagation can keep improving with larger-scale training data.