PaperlayerView exact revision on arXiv

Source-linked AI summary

Constrained Convolutional Neural Networks for Weakly Supervised Segmentation

Deepak Pathak, Philipp Krähenbühl, Trevor Darrell

arXiv:1506.03648v2cs.CVcs.LG

TL;DR

The paper tackles the expense of pixel-level labels for semantic segmentation by learning from image-level tags expressed as linear output constraints. CCNN introduces a latent-distribution constrained loss that integrates with SGD, achieving state-of-the-art weakly supervised segmentation and improved performance with additional supervision.

  • Problem

    Pixel-level supervision for semantic segmentation is expensive and time consuming, motivating methods that can learn dense labels from weaker annotations.

  • Method

    CCNN models a latent distribution over labels, enforces arbitrary linear constraints on it, and alternates convex and gradient-based optimization within SGD.

  • Results

    CCNN achieves state-of-the-art performance on Pascal VOC 2012 weakly supervised semantic segmentation without pixel-level object labels during training.

  • Takeaways & Limitations

    Constraints provide a general way to describe desired output labelings and reduce the strong supervision required by CNNs; added weak supervision can further improve performance.

  • Takeaways & Limitations

    The framework must handle potentially unsatisfiable constraints by adding a slack variable, and comparisons using cropped weak labels may include limited spatial information.

Abstract

from arXiv · show

We present an approach to learn a dense pixel-wise labeling from image-level tags. Each image-level tag imposes constraints on the output labeling of a Convolutional Neural Network (CNN) classifier. We propose Constrained CNN (CCNN), a method which uses a novel loss function to optimize for any set of linear constraints on the output space (i.e. predicted label distribution) of a CNN. Our loss formulation is easy to optimize and can be incorporated directly into standard stochastic gradient descent optimization. The key idea is to phrase the training objective as a biconvex optimization for linear models, which we then relax to nonlinear deep networks. Extensive experiments demonstrate the generality of our new learning framework. The constrained loss yields state-of-the-art results on weakly supervised semantic image segmentation. We further demonstrate that adding slightly more supervision can greatly improve the performance of the learning algorithm.

1. Introduction

The paper addresses the cost and scalability limits of pixel-level supervision by expressing weak annotations as linear constraints on CNN output labelings. CCNN uses a constrained loss and latent label distribution to train segmentation models, achieving strong weak-supervision results and supporting added supervision.

  • Motivation: Pixel-level labels for semantic segmentation are expensive and time consuming, limiting the scalability of fully supervised CNN training.Image-level tags and bounding boxes are less costly forms of weak supervision.
  • Approach: The framework incorporates weak supervision through linear constraints on the output space of a CNN.In segmentation, image-level tags can specify label existence and expected pixel distributions.
  • Approach: CCNN optimizes arbitrary linear constraints on structured pixel-label outputs with a novel loss function.The method is designed for convolutional networks rather than only linear models.
  • Optimization: A latent probability distribution lets the method enforce constraints indirectly while alternating convex and gradient-based optimization integrates with SGD.The network output is pulled toward the closest plausible point on the constrained manifold.
  • Results: CCNN achieves state-of-the-art performance on Pascal VOC 2012 weakly supervised semantic segmentation without pixel-level labels for objects during training.The framework also incorporates additional weak supervision such as rough object-size estimates.

2. Related Work

Prior weak-segmentation methods commonly use graphical models or multiple-instance learning, but CNN-based approaches can be initialization-sensitive and dependent on pretrained classifiers. CCNN instead optimizes constrained objectives through an alternating procedure and supports broader constraint formulations.

  • Multiple-instance learning: Multiple-instance learning has been extended from detection to semantic segmentation with CNNs, but these methods iteratively reinforce outputs consistent with image-level tags.Their optimization is non-convex and can depend strongly on initialization.
  • Existing approaches: Weak segmentation has traditionally used graphical models and parametric structured models, often connecting regions with similar low-level appearance.Some work also uses top-down segmentation priors for object discovery.
  • Comparison: CCNN is described as less sensitive than comparable approaches and able to recover a good solution from random classification-layer initialization.The cited comparison contrasts it with methods requiring carefully pretrained convolutional classifiers.
  • Constrained formulations: Adaptive bias methods can be interpreted as constrained optimization, but the constraints they model are limited and cannot exploit the full power of weak labels.CCNN is introduced to apply more general constraints.
  • Optimization: The proposed optimization alternates between convex and gradient-based steps, addressing the difficulty of directly minimizing highly non-convex constrained neural-network objectives.The related formulation connects to generalized expectation and posterior regularization while avoiding direct constraint penalties that are hard to evaluate for CNNs.

3. Preliminaries

The paper represents an image labeling as discrete pixel variables and models their distribution with CNN probabilities. Weak learning then replaces unavailable full labels with high-level constraints on the output-labeling distribution.

  • Label representation: A pixel-wise labeling is represented by random variables X={x_0,...,x_n}, with each pixel variable taking one of m discrete labels.The label set is L={1,...,m}.
  • CNN distribution: The CNN models a probability distribution Q(X|θ,I) over the pixel-label variables.The supplied formulation states that this distribution is commonly represented through independent marginal probabilities.
  • Weak-learning pipeline: The fully convolutional network in the weak-learning pipeline receives an input image and produces an output labeling.Training imposes linear constraints derived from image-level tags on that output.
  • Probability model: The network score function assigns higher likelihood to labels with higher real-valued scores.The partition function normalizes the pixel-level probabilities.
  • Weak supervision: Because full ground-truth labeling is unavailable under weak supervision, training uses high-level constraints on the output-labeling distribution.The paper introduces examples of useful constraints in the subsequent formulation.

4. Constrained Optimization

CCNN separates constraint satisfaction from network fitting by introducing a latent label distribution, then alternates constrained optimization with SGD. The procedure supports slack variables for conflicting constraints and is designed for efficient optimization.

  • Latent distribution: CCNN introduces a latent probability distribution P constrained to the feasible region while encouraging it to match the network output Q.This decouples the linear constraints from the nonlinear network output and uses KL-divergence to align P and Q.
  • Alternating optimization: For fixed network parameters, optimizing P is convex; for fixed P, optimizing network parameters becomes a standard cross-entropy loss trained with SGD.Deep networks use back-propagation and stochastic gradient descent with momentum for the network-update step.
  • Optimization guarantee: Each optimization step is guaranteed to decrease the overall energy and converge to a good local optimum under the stated procedure.
  • Latent distribution optimization: The P-update maximizes a concave dual function with projected gradient ascent, increasing dual variables for unsatisfied constraints.The projected dual gradient method usually converges in fewer than 50 iterations.
  • SGD optimization: At each SGD iteration, CCNN infers P, uses it to define the loss and gradient, and updates the network output from Q(t) to Q(t+1).Although strict decrease would require holding P fixed for several SGD iterations, recomputing P every step was faster in practice without hurting performance.
  • Inconsistent constraints: Slack variables and hinge regularization make the constrained objective satisfiable for arbitrary linear constraints and allow difficult constraints to be discounted.The hinge-loss weight bounds the dual variables and enables trade-offs among competing constraints.

5. Constraints for Weak Semantic Segmentation

CCNN encodes image-level labels through complementary linear constraints on pixel-label distributions. These constraints suppress absent classes, require present classes, control foreground/background area, and can use rough object-size information.

  • Constraint framework: For each image, CCNN applies complementary constraints to the output space according to the set of image-level labels.The constraints exploit the image-level labels in different ways rather than relying on a single constraint.
  • Label-presence constraints: Suppression constraints prevent labels absent from an image from receiving predicted pixels.
  • Label-presence constraints: Foreground lower bounds require present labels to occupy pixels, addressing the all-background solution that satisfies suppression alone.The foreground constraint can encourage multiple pixels to take a specified label, unlike the usual MIL requirement of at least one positive pixel.
  • Area constraints: Background lower and upper bounds indirectly control the combined minimum and maximum area assigned to foreground labels.The reported practical bounds are a0 = 0.3n and b0 = 0.7n.
  • Size constraints: Rough size supervision can improve training by boosting classes covering more than 10% of an image and bounding classes known to be small.A threshold bl < 0.01n worked slightly better than a tight threshold in practice.

6. Implementation Details

The implementation uses a fully convolutional, ImageNet-pretrained VGG-derived network adapted to Pascal VOC classes. Constraints are applied to coarse heatmaps during SGD training, with optional CRF refinement at inference.

  • Network architecture: The experiments use a VGG 16-layer-derived CNN pretrained on the ImageNet 1K dataset and adapted to 21 Pascal VOC outputs.The outputs represent 20 object classes plus background.
  • Network architecture: A denser 8s network stride performs better in weakly supervised training than the original 32s stride architecture.
  • Training: The fully convolutional network produces coarse class heatmaps, and CCNN applies constrained optimization to those heatmaps to reduce computational cost.Training uses SGD with momentum.
  • Training: Training uses batch size 1, momentum 0.99, initial learning rate 1e-6, and 60000 iterations.The 60000 iterations correspond to roughly 5 epochs.
  • Inference: At inference, a fully connected conditional random field may optionally refine the final segmentation.

7. Experiments

Experiments evaluate CCNN for dense semantic segmentation under image-level tags and added spatial or size supervision. The results show strong weakly supervised performance, improved accuracy with additional constraints, robustness to constraint bounds, and remaining gaps versus methods using richer box information.

  • Experimental setup: CCNN learns dense multi-class segmentation from image-level tags using simple spatial constraints, without pixel-level labels for objects during training.The experiments target pixel-wise labeling and avoid additional low-level graph-cut methods during training.
  • Image-level supervision: CCNN outperforms prior weakly labeled semantic segmentation methods by a significant margin on the PASCAL VOC 2012 comparison.Table 1 compares contemporary weak segmentation methods.
  • Additional supervision: Limited spatial information lets CCNN slightly outperform EM-Adapt, although the difference is smaller than in the pure weak-label setting.The comparison uses random crops and attributes the result mainly to stronger background constraints.
  • Additional supervision: A single size bit indicating whether a class exceeds 10% of the image dramatically increases accuracy, while EM-Adapt cannot directly incorporate this constraint.The size constraint is one example of the broader linear-constraint flexibility of CCNN.
  • Robustness and qualitative results: Accuracy varies little across a wide range of constraint bounds, with an average standard deviation of 0.73%.Slack variables contribute to this insensitivity, and the authors also report qualitative comparisons across supervision levels.
  • Additional supervision: Bounding-box constraints reach 54% IoU, compared with 52.3% for training on all pixels inside boxes and roughly 58.5−62.0% for richer box-supervision systems.The authors attribute the remaining gap to stronger use of pixel-level segmentation information.
  • Optimization: CCNN naturally integrates arbitrary linear constraints into standard stochastic gradient descent and can be used in frameworks such as Caffe.The supplementary optimization uses alternating minimization over the latent distribution and network parameters.

I. Latent Distribution

The latent label distribution is transformed into a biased CNN distribution under linear constraints, allowing the constrained objective to be expressed through KL divergence and dual optimization. Its optimum factorizes across variables.

  • Latent Distribution: The constrained cross-entropy is rephrased as KL divergence between the latent distribution P and a biased CNN distribution ˜Q.The biased distribution incorporates constraint effects through dual variables and local partition functions.
  • Latent Distribution: The primal constrained objective and its dual are equivalent because strong duality holds for the convex problem with linear constraints.The derivation uses minimization over P followed by maximization over dual variables.
  • Latent Distribution: For fixed network parameters, the minimizing latent distribution fully factorizes over variables without loss of generality.The factorized form arises because the KL divergence is minimized when P matches the corresponding biased distribution.
  • Latent Distribution: The derivation uses independent local CNN marginals, with log partition functions treated as constants during optimization.Each marginal is normalized by a local partition function, and its contribution simplifies through linearity of expectation.

II. Optimizing Constraints with Slack Variable

The framework relaxes potentially infeasible linear constraints with slack variables. For positive slack weights, the resulting dual variables are bounded, recovering the non-slack formulation when substituted back.

  • Optimizing Constraints with Slack Variable: Slack variables make the optimization robust to arbitrary linear constraints that may not be simultaneously satisfiable.The relaxation permits constraint violations while preserving an optimizable objective.
  • Optimizing Constraints with Slack Variable: The slack-relaxed Lagrangian adds a weighted penalty for constraint violations alongside the entropy and cross-entropy terms.The Lagrangian includes β⊤ξ and dual terms coupling slack ξ to the constraint residual.
  • Optimizing Constraints with Slack Variable: For β > 0, stationarity with respect to ξ and nonnegative dual variables imply an upper bound on the constraint multipliers.The slack weight yields λ ≤ β.

III. Ablation Study for Parameter Selection

The sensitivity study varies constraint upper and lower bounds while holding other parameters fixed. CCNN remains robust in the image-tag plus 1-bit size-supervision setting.

  • Ablation Study for Parameter Selection: 0.73% standard deviation in accuracy was observed across line searches over constraint parameters.The study varied each upper and lower bound independently while keeping the remaining parameters fixed.
  • Ablation Study for Parameter Selection: The sensitivity experiments used image-level tags together with 1-bit size supervision during training.The reported robustness therefore applies to this supervision setting.
  • Ablation Study for Parameter Selection: Table 4 reports sensitivity to the chosen CCNN optimization parameters and marks the values used in other experiments in bold.The parameters are defined in Equations (8) and (9) of the main paper.

IV. Ablation Study in Semi-Supervised Setting

The semi-supervised ablation adds fully supervised images to the weak objective and evaluates the resulting accuracy curve. The study reports that the model makes good use of this additional supervision.

  • Ablation Study in Semi-Supervised Setting: Fully supervised images were incorporated in addition to the weak objective, with accuracy measured as their amount varied.The resulting accuracy curve is shown in Figure 5.
  • Ablation Study in Semi-Supervised Setting: The model makes good use of additional supervision in the semi-supervised ablation.Figure 5 summarizes the ablation across varying amounts of fully supervised images.
Loading 1506.03648v2…