Source-linked AI summary
Normalized Cut Loss for Weakly-supervised CNN Segmentation
Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, Christopher Schroers
TL;DR
Weakly supervised segmentation often relies on generated masks, but errors in those masks can mislead CNN training. The paper combines partial cross entropy on labeled scribbles with normalized cut regularization over network outputs, achieving state-of-the-art scribble-based segmentation. Its scope is limited to normalized cut, with broader regularizer comparisons left for future work.
Problem
Fully annotated masks are costly, while generated full-mask proposals can contain errors that mislead weakly supervised CNN training.
Method
The paper combines partial cross entropy on labeled pixels with normalized cut regularization over CNN outputs.
Results
Experiments show that normalized cut loss achieves state-of-the-art weakly supervised semantic segmentation with scribbles.
Takeaways & Limitations
Partial cross entropy on scribbles works surprisingly well even without normalized cut, while the joint loss provides a direct alternative to iterative proposal generation.
Takeaways & Limitations
The paper focuses on normalized cut, leaving empirical comparison with other regularizers for future work.
Abstract
from arXiv · showhide
Most recent semantic segmentation methods train deep convolutional neural networks with fully annotated masks requiring pixel-accuracy for good quality training. Common weakly-supervised approaches generate full masks from partial input (e.g. scribbles or seeds) using standard interactive segmentation methods as preprocessing. But, errors in such masks result in poorer training since standard loss functions (e.g. cross-entropy) do not distinguish seeds from potentially mislabeled other pixels. Inspired by the general ideas in semi-supervised learning, we address these problems via a new principled loss function evaluating network output with criteria standard in "shallow" segmentation, e.g. normalized cut. Unlike prior work, the cross entropy part of our loss evaluates only seeds where labels are known while normalized cut softly evaluates consistency of all pixels. We focus on normalized cut loss where dense Gaussian kernel is efficiently implemented in linear time by fast Bilateral filtering. Our normalized cut loss approach to segmentation brings the quality of weakly-supervised training significantly closer to fully supervised methods.
1. Introduction
The paper targets weakly supervised CNN segmentation, where generated full-mask proposals can contain errors that mislead cross-entropy training. It proposes combining partial cross entropy on labeled scribbles with normalized cut regularization, achieving state-of-the-art scribble-based training.
- Fully supervised semantic segmentation typically trains fully convolutional networks with pixel-wise cross entropy on ground-truth masks.
- Fully annotated masks are costly, motivating weak annotations such as scribbles, bounding boxes, clicks, and image-level tags.
- Generated full-mask proposals can contain errors that cross entropy treats as labels, causing networks to over-fit mistakes.
- The proposed loss directly integrates normalized cut with partial cross entropy for weakly supervised CNN training.The approach is motivated by semi-supervised learning and shallow segmentation criteria.
- Partial cross entropy on scribbles works surprisingly well compared with cross entropy over generated full masks, even without normalized cut.
- Experiments show that normalized cut loss achieves state-of-the-art semantic segmentation training with scribbles.
2. Background and Motivation
The paper frames weakly supervised CNN segmentation as regularized semi-supervised learning: labeled pixels receive empirical loss, while shallow segmentation criteria regularize network outputs over all pixels. It focuses on normalized cut because its normalization encourages balanced segments and supports continuous relaxation.
- Regularized semi-supervised CNN losses combine fidelity to labeled pixels with regularization of the entire network output.
- The regularization term propagates empirical losses from partially labeled inputs to unlabeled points.
- The proposed framework combines empirical risk over labeled pixels with a shallow-segmentation regularizer over all pixels.
- A relaxed formulation allows standard segmentation energies to operate on real-valued CNN outputs, with hard seed constraints becoming partial cross entropy.
- Normalized cut differs from Potts and dense CRF through normalization that encourages balanced clustering and addresses shrinking bias.
- The paper focuses on normalized cut, while empirical comparisons with other regularizers are deferred to future work.
3. Related Work
Prior work uses graph-based semi-supervision, weak annotations, and segmentation proposals, but proposal-generation pipelines can be expensive and propagate erroneous labels. The paper instead regularizes CNN outputs directly with normalized cut, a balanced clustering criterion.
- Graph-based semi-supervised learning assumes that nodes with larger affinity are more likely to share labels.
- Weakly supervised segmentation has used scribbles, bounding boxes, clicks, and image-level tags.
- Proposal-based methods generate fake masks and minimize cross entropy against them, potentially over-fitting proposal errors.
- The joint loss avoids iterative training and heuristic segmentation proposals by training in one pass.
- Proposal mistakes can reinforce themselves in self-learning, which lacks a convergence guarantee.
- The normalized cut loss regularizes network outputs for unlabeled data and is coupled with partial fidelity loss for implicit label propagation.
- Normalized cut is a graph clustering algorithm whose extra normalization encourages balanced clustering and reduces shrinking bias.
4. Our Method
The method combines partial cross entropy on labeled scribble pixels with normalized cut regularization over unlabeled pixels. A dense Gaussian-kernel implementation uses fast bilateral filtering, while the resulting gradients encourage color-consistent segmentation.
- Joint Loss: The joint loss combines partial cross entropy and normalized cut for weakly-supervised CNN segmentation.Partial cross entropy handles labeled pixels, while normalized cut regularizes the segmentation.
- Partial Cross Entropy: Partial cross entropy evaluates only labeled pixels, ignoring regions without known labels.For scribbles, the sampling variable is one on labeled pixels and zero elsewhere.
- Partial Cross Entropy: More than 85% of full-labeling accuracy is achieved when training only with partial cross entropy.This loss also outperforms training from GrabCut proposals in the reported experiments.
- Normalized Cut Loss: Normalized cut softly evaluates unlabeled pixels using a relaxed segmentation and a Gaussian affinity kernel over RGBXY space.Its regularization encourages balanced nonlinear partitioning, although simple RGBXY clustering may not always be semantically meaningful.
- Normalized Cut Loss: Scribble labels are clamped as seeds, allowing normalized cut gradients to propagate their labels toward unknown pixels during training.The scribbles remain fixed to their ground-truth labeling in the normalized cut implementation.
- Efficient Implementation: Dense Gaussian normalized cut has a naive O(|Ω|^2) implementation, but bilateral filtering with a permutohedral lattice reduces computation to linear time.The same efficient layer supports forward evaluation and back-propagation.
- Gradient Computation: Normalized cut gradients are high-order and non-intuitive, yet visualizations show they encourage better color clustering, especially near object boundaries.The gradients correspond to slopes of a linear upper bound for normalized cut and are observed to drive improved color clustering.
5. Experiments
Experiments show that networks can optimize normalized cut directly, while the joint loss improves weakly supervised segmentation substantially and approaches fully supervised performance.
- 5.1. Normalized Cut and K-means Network: The experiments used binary MSRA10K color-clustering tasks with DeepLab-VGG-16, pretrained saliency networks, and no supervision during clustering-loss training.Normalized cut used σrgb = 15 and σxy = 40; K-means used RGB features only.
- 5.1. Normalized Cut and K-means Network: Networks minimized normalized cut and K-means losses by a large margin, producing segmentations with improved color clustering.Normalized cut gradients particularly improve clustering along object boundaries.
- 5.2. Weakly-supervised Semantic Segmentation: Partial cross entropy on scribbles achieved 55.8% mIOU, outperforming training from GrabCut proposals at 54.7%.The authors interpret partial cross entropy as sampling full cross entropy using scribbles.
- 5.2.1 Results Using Our Loss: 65.1% mIOU with the joint loss was close to full supervision at 68.7% on the PASCAL VOC 2012 validation set.The joint loss combined partial cross entropy with extra normalized cut loss; a non-exist label penalty slightly improved results.
- Implementation Details: Joint-loss training took 0.15 sec/image versus 0.05 sec/image for cross entropy alone in the reported CPU-based filtering implementation.The authors note that GPU-based bilateral filtering could further accelerate training.
- 5.2.2 Comparison to Segmentation Proposal Approach: The framework enables one-pass training with scribbles instead of iterative segmentation-proposal generation.Comparisons included GrabCut, KernelCut, and a seeded normalized-cut variant.
- 5.2.2 Comparison to Segmentation Proposal Approach: The joint loss reduced ScribbleSup’s 5.6% full-supervision gap to 3.6%, while partial cross entropy alone outperformed GrabCut proposals.The authors attribute proposal-based errors to network overfitting and report that normalized cut further boosts accuracy.
- 5.2.3 General Applicability: DeepLab-ResNet-101 trained with the joint loss reached 74.5% mIOU, compared with 76.8% under full supervision.This result supports applicability beyond the DeepLab-largeFOV architecture.
6. Conclusion and Future Work
The paper proposes combining partial cross entropy with normalized cut for weakly supervised scribble segmentation. It reports state-of-the-art performance and identifies broader supervision types and regularizers as future directions.
- Conclusion: The proposed loss combines partial cross entropy with normalized cut for weakly supervised semantic segmentation using scribbles.The framework is motivated by semi-supervised learning and shallow segmentation criteria.
- Conclusion: Proposal mistakes can mislead training, whereas partial cross entropy on scribbles works well without generated full masks.The paper describes partial cross entropy as loss sampling and reports state-of-the-art scribble-supervised segmentation.
- Future Work: Future work includes adapting the framework to tags, boxes, domain adaptation, MRF/CRF losses, and intermediate-representation regularization.These directions are presented as extensions of the current framework.