Source-linked AI summary
On Regularized Losses for Weakly-supervised CNN Segmentation
Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, Yuri Boykov
TL;DR
Weakly supervised semantic segmentation often relies on proposal masks generated through explicit regularization inference, while direct regularized-loss training has been largely overlooked. The paper integrates MRF/CRF-inspired regularizers into losses over partial masks, evaluates several formulations, and reports state-of-the-art performance near full-supervision quality with better quality and efficiency than proposal-generation alternatives.
Problem
Weakly supervised CNN segmentation commonly uses computationally expensive proposal generation to mimic full supervision, while direct regularized-loss training is largely overlooked.
Method
The paper integrates relaxed MRF/CRF regularization terms directly into losses over partial labels and evaluates Potts, dense CRF, and kernel cut formulations.
Results
The regularized losses achieve state-of-the-art weakly supervised CNN segmentation near full-supervision accuracy, with better quality and efficiency than proposal-generating methods or normalized cut loss.
Takeaways & Limitations
Direct regularized-loss training avoids explicit full-mask inference steps while providing a principled framework for weak supervision with partial masks.
Takeaways & Limitations
The paper uses a basic quadratic relaxation, leaving evaluation of alternative CRF relaxations and efficient gradient computation for future work.
Abstract
from arXiv · showhide
Minimization of regularized losses is a principled approach to weak supervision well-established in deep learning, in general. However, it is largely overlooked in semantic segmentation currently dominated by methods mimicking full supervision via "fake" fully-labeled training masks (proposals) generated from available partial input. To obtain such full masks the typical methods explicitly use standard regularization techniques for "shallow" segmentation, e.g. graph cuts or dense CRFs. In contrast, we integrate such standard regularizers directly into the loss functions over partial input. This approach simplifies weakly-supervised training by avoiding extra MRF/CRF inference steps or layers explicitly generating full masks, while improving both the quality and efficiency of training. This paper proposes and experimentally compares different losses integrating MRF/CRF regularization terms. We juxtapose our regularized losses with earlier proposal-generation methods using explicit regularization steps or layers. Our approach achieves state-of-the-art accuracy in semantic segmentation with near full-supervision quality.
1 Introduction
The paper advocates integrating MRF/CRF-inspired regularizers directly into losses for weakly supervised CNN segmentation, replacing proposal-generation pipelines that infer fake full masks. It evaluates several such losses and reports state-of-the-art performance near full-supervision accuracy with improved quality and efficiency.
- Motivation: Regularized losses apply unsupervised structure-aware terms directly to network outputs over partially labeled data.The approach avoids explicitly guessing full training masks or adding separate MRF/CRF inference layers.
- Motivation: Weakly supervised CNN segmentation commonly synthesizes fully labeled proposal masks using graph cuts or dense CRF inference.These methods alternate CNN training with regularization-based proposal generation.
- Contributions: The paper proposes and evaluates losses based on Potts, dense CRF, and kernel cut regularizers.These losses combine standard regularization ideas with weak supervision from partial masks.
- Contributions: The proposed losses are presented as direct alternatives to proposal-generation methods, avoiding their explicit inference steps.The paper also connects iterative CNN training and proposal generation to approximate alternating optimization of regularized losses.
- Results: The experiments report state-of-the-art weakly supervised segmentation with near full-supervision accuracy and better quality and efficiency than proposal-generating methods or normalized cut loss.Alternating proposal-generation schemes give higher loss at convergence.
2 Our Regularized Semi-supervised Losses
The paper formulates weakly supervised CNN segmentation as joint optimization of partial-label loss and differentiable regularizers, adapting MRF/CRF objectives directly to network outputs. It studies CRF, normalized cut, and kernel cut losses, with kernel cut achieving the best weakly supervised segmentation in experiments.
- 2 Our Regularized Semi-supervised Losses: The joint loss combines a ground-truth loss on partial labels with a regularization term applied to the network’s softmax segmentation.The segmentation output is S = fθ(I), and cross entropy is applied over labeled pixels.
- 2 Our Regularized Semi-supervised Losses: The framework studies relaxed shallow-segmentation regularizers as differentiable loss functions for CNN training.The paper focuses on relaxations of MRF/CRF terms and normalized cut objectives.
- 2.1 Potts/CRF Losses: Quadratic Potts relaxations penalize pairwise disagreement between soft segment indicators using an affinity matrix W.The affinity matrix may be sparse for boundary alignment or dense with Gaussian kernels for DenseCRF relaxation.
- 2.2 Kernel Cut Loss: Normalized cut losses measure each segment’s cut cost relative to its weighted size, while kernel cut combines normalized cut and CRF terms.The combination targets balanced partitioning together with object-boundary regularization or edge alignment.
- 2.2 Kernel Cut Loss: Kernel cut loss achieved the best weakly supervised segmentation among the CRF, normalized cut, and kernel cut losses evaluated.The combined objective’s gradient can be implemented efficiently despite being a high-order objective.
- 2 Our Regularized Semi-supervised Losses: Gradient descent during CNN training permits flexible inclusion of regularization terms when they have reasonable relaxations.This differs from the specialized optimization techniques commonly used for shallow normalized cut and CRF objectives.
3 Connecting proposals generation and loss optimization
The paper interprets proposal-generation methods as approximate alternating optimization schemes that separate network training from regularized label inference. This perspective motivates direct back-propagation through regularized losses, avoiding dependence on iteratively generated proposals that can contain reinforcing errors.
- 3 Connecting proposals generation and loss optimization: Weakly supervised methods commonly generate full segmentation proposals from partial labels using graph cuts, random walkers, or related shallow techniques.These proposals are then used as pseudo-ground truth for network training.
- 3 Connecting proposals generation and loss optimization: Training is vulnerable to proposal mistakes because errors can reinforce themselves in alternating self-taught learning schemes.The paper characterizes direct regularized semi-supervised optimization as a more direct alternative to fitting potential proposal errors.
- 3 Connecting proposals generation and loss optimization: Proposal-based training alternates between fitting network parameters to fixed proposals and generating new proposals by optimizing shallow regularization functionals.The proposals equal known labels on labeled pixels and provide latent labels on unlabeled pixels.
- 3 Connecting proposals generation and loss optimization: For dense CRF regularization, proposal generation can use mean-field inference with parallel updates and high-dimensional filtering.These techniques reduce computational time, and parallel convergence is guaranteed for concave CRF models such as Potts.
- 3 Connecting proposals generation and loss optimization: Proposal methods can be viewed as approximate alternating direction methods for optimizing the regularized loss, but their splitting does not directly account for network variables θ.The decomposition separates network outputs from latent proposal distributions and minimizes a divergence between them.
4 Experiments
Experiments show that directly optimizing regularized losses improves weakly supervised segmentation quality and efficiency across scribble and image-level supervision, while naive regularization can harm fully supervised training.
- 4.1 Comparison of regularized losses: Combining CRF and normalized cut losses as KernelCut yielded the best performance in scribble-supervised segmentation.The experiments compare CRF, normalized cut, and KernelCut losses.
- 4.1 Comparison of regularized losses: 66.7% mIOU with KernelCut surpassed the previous 65.1% normalized-cut result and reached 97.6% of full-supervision quality using 3% scribbled pixels.This result used DeepLab-MSc-largeFOV with CRF post-processing on PASCAL VOC12.
- 4.1 Comparison of regularized losses: KernelCut improved edge alignment over normalized cut by combining regional color clustering with pairwise DenseCRF regularization.The paper attributes the edge-alignment improvement to the additional pairwise CRF loss.
- 4.2 Direct loss vs proposal generation: Direct loss optimization outperformed proposal generation in segmentation accuracy across networks and produced higher training and validation mIOU with lower regularized loss.The comparison used matched Gaussian kernels for the direct and proposal-generation methods.
- 4.2 Direct loss vs proposal generation: For image-level labels, direct CRF loss produced similar segmentations to SEC while training faster because iterative CRF inference was unnecessary.CRF loss improved training accuracy from 38.4% to 43.9%, although segmentation mIOU improved only slightly.
- 4.2 Direct loss vs proposal generation: With shortened scribbles, the direct-loss method outperformed ScribbleSup across length ratios, with larger gains for shorter scribbles and clicks.Zero-length scribbles correspond to one click or spot per object.
- 4.3 Fully and semi supervised segmentation: Adding regularization loss naively to fully supervised training worsened cross-entropy loss and reduced mIOU when the regularizer disagreed with ground truth.The authors describe this as a negative result for fully supervised segmentation.
5 Conclusion and Future Work
The paper applies regularized losses to weakly supervised CNN segmentation and reports state-of-the-art performance near full-supervision accuracy. It also identifies broader extensions while leaving alternative CRF relaxations and non-segmentation applications for future work.
- Regularized losses based on Potts/CRF, normalized cut, and KernelCut extend the paper’s study of shallow-segmentation objectives to weakly supervised CNN segmentation.
- The proposed losses integrate regularization directly into training rather than relying on proposal masks or separate inference procedures.
- The method achieves state-of-the-art weakly supervised segmentation near full-supervision accuracy, with better quality and efficiency than proposal-generation methods and normalized cut loss.
- The framework is presented as applicable to semi-supervised learning with unlabeled images, including reported improvement from applying RCRF(S) to unlabeled images.
- Evaluating alternative CRF relaxations and applying the framework to weakly supervised vision problems beyond segmentation are left for future work.
A Mean-field inference for DenseCRF
The appendix connects DenseCRF mean-field inference to optimization of a relaxed CRF objective. It explains this connection through iterative optimization, entropy, and probability-simplex constraints.
- Mean-field inference minimizes an objective combining a DenseCRF regularizer with unary potentials supplied, for example, by the network.
- For positive semidefinite affinity matrices, the objective’s concave-convex structure supports applying the concave-convex procedure.
- The concave-convex procedure linearizes the concave part at the current estimate and iteratively solves an approximation of the energy.
- The KKT approach minimizes the relevant objective subject to probability-simplex constraints.
- The resulting normalized updates are exactly the mean-field updates for DenseCRF and can also be interpreted as minimizing CRF potential plus negative entropy.