Source-linked AI summary
Puzzle-CAM: Improved localization via matching partial and full features
Sanghyun Jo, In-Jae Yu
TL;DR
Image-level supervision makes semantic segmentation training less dependent on costly pixel-level labels, but CAM-based pseudo-labels often cover only discriminative object parts. Puzzle-CAM matches partial and full-image CAMs using a puzzle module and reconstructing regularization. The paper reports state-of-the-art performance on PASCAL VOC 2012.
Problem
Pixel-level annotations are costly, while CAM-based WSSS often focuses on only the most discriminative object parts.
Source: §1 · p. 1Method
Puzzle-CAM uses a puzzle module to merge CAMs from tiled image patches and reconstructing regularization to match them with CAMs from the original image.
Source: §1 · p. 1, §3.3 · p. 3, §3.4 · p. 3Results
Takeaways & Limitations
Puzzle-CAM consistently generates features from local tiled patches and better fits the shapes of ground-truth masks.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Weakly-supervised semantic segmentation (WSSS) is introduced to narrow the gap for semantic segmentation performance from pixel-level supervision to image-level supervision. Most advanced approaches are based on class activation maps (CAMs) to generate pseudo-labels to train the segmentation network. The main limitation of WSSS is that the process of generating pseudo-labels from CAMs that use an image classifier is mainly focused on the most discriminative parts of the objects. To address this issue, we propose Puzzle-CAM, a process that minimizes differences between the features from separate patches and the whole image. Our method consists of a puzzle module and two regularization terms to discover the most integrated region in an object. Puzzle-CAM can activate the overall region of an object using image-level supervision without requiring extra parameters. % In experiments, Puzzle-CAM outperformed previous state-of-the-art methods using the same labels for supervision on the PASCAL VOC 2012 test dataset. In experiments, Puzzle-CAM outperformed previous state-of-the-art methods using the same labels for supervision on the PASCAL VOC 2012 dataset. Code associated with our experiments is available at https://github.com/OFRIN/PuzzleCAM.
1. INTRODUCTION
WSSS uses image-level annotations to avoid the cost of pixel-level labels, but CAM-based methods often activate only small, discriminative object regions. Puzzle-CAM matches CAMs from tiled patches with those from the original image to improve localization, and is reported to outperform existing methods under the same supervision on PASCAL VOC 2012.
1. INTRODUCTION
- Image-level supervision is less costly to produce than pixel-level annotations, while CAM-based WSSS often focuses on only small, discriminative object parts.Source: §1 · p. 1
- The method uses reconstructing regularization to align CAMs from tiled and original images, while the puzzle module splits images and merges tiled-image CAMs.Source: §1 · pp. 1–2
- Puzzle-CAM combines reconstructing regularization with a puzzle module to enhance CAM quality without adding layers.Source: §1 · p. 2
- Puzzle-CAM outperformed existing state-of-the-art methods with the same level of supervision on the PASCAL VOC 2012 dataset.Source: §1 · p. 2
2. RELATED WORK
The related work reviews CNN attention mechanisms and CAM-based weakly supervised segmentation methods. It describes Puzzle-CAM’s chosen attention approach and places it among methods that refine CAMs to approximate segmentation masks.
2.1. Attention Mechanisms Using CNNs
- The paper uses the combined attention model with global average pooling because it is more manageable than other attention mechanisms.Source: §2.1 · p. 2
2.2. Weakly Supervised Semantic Segmentation
- Prior WSSS methods refine classifier-generated CAMs toward segmentation masks, using learned pixel similarities, boundary activations, or pixel-correlation attention.Source: §2.2 · p. 2
3. METHODOLOGY
Puzzle-CAM matches CAMs from tiled image patches to those from the full image. Its puzzle module and classification and reconstruction losses refine CAMs using image-level supervision.
3.1. Motivation
- Puzzle-CAM trains a classifier with reconstruction loss to minimize differences between the full-image CAM and the merged CAM from image patches.The authors motivate this matching by noting that patch-derived CAMs can highlight object areas more accurately than a single-image CAM.Source: §3.1 · p. 2
3.2. The Employed CAM Method
- Class CAMs are formed by applying class-specific classifier weights to input-image feature maps, then normalizing each map by its maximum and concatenating the class maps.Source: §3.2 · pp. 2–3
3.3. The Puzzle Module
- Puzzle-CAM tiles an image into four non-overlapping half-size patches, generates CAMs for each patch, then merges them into a CAM matching the full-image CAM shape.This module is designed to narrow the gap between partial and full features.Source: §3.3 · p. 3
3.4. The Loss Design for Puzzle-CAM
- GAP produces original and reconstructed classification predictions; their classification losses support image classification, while reconstruction regularization aligns their CAMs and narrows the pixel- versus image-supervision gap.α balances the different losses.Source: §3.4 · p. 3
4. EXPERIMENTAL RESULTS
Puzzle-CAM’s experiments evaluate its loss components and compare the resulting pseudo-labels and segmentation performance with prior methods on PASCAL VOC 2012. The reported comparison shows improved validation and test performance over baseline methods under the same training settings.
4.1. Implementation Details
- Experiments use PASCAL VOC 2012 with an augmented 10,582-image training set, and inference uses classifiers without the puzzle module.The augmentation adds Semantic Boundary Dataset annotations; inference uses multi-scale and horizontal-flip processing.Source: §4.1 · p. 3
4.2. Ablation Studies
- mIoU rose from 47.82% baseline to 49.21% with tiled-patch Lre, while Lp-cls alone was similar to baseline and combining losses improved localization without classification loss.The ablation evaluates the components using mIoU.Source: §4.2 · p. 3
- Fig. 3(d) shows final CAMs suppressing over-activation while expanding activation to complete object coverage.The figure visualizes prediction tags and CAMs for combinations of loss functions.Source: §4.2 · p. 3, Figure 3 · p. 4
4.3. Comparison with Existing State-of-the-Art Methods
- The pseudo-label comparison evaluates PASCAL VOC 2012 training-set mIoU, including random walk with AffinityNet and dense conditional random field refinements.The comparison reports original CAMs used by baseline AffinityNet and Puzzle-CAM.Source: Table 2 · p. 4, §4.3 · p. 4
- Puzzle-CAM improved performance over baseline methods on both PASCAL VOC 2012 validation and test sets under the same training settings.Table 3 compares methods on both splits; Fig. 4 presents qualitative validation predictions for large and small objects.Source: §4.3 · p. 4, Table 3 · p. 4, Figure 4 · p. 4
5. CONCLUSIONS
Puzzle-CAM matches partial and full features through a puzzle module and reconstructing regularization to generate consistent CAMs from image-level labels. The authors report state-of-the-art segmentation performance on PASCAL VOC 2012 and suggest the training-module concept may extend to other weakly and semi-supervised tasks.
5. CONCLUSIONS
- Puzzle-CAM combines a puzzle module with reconstructing regularization to match partial and full features and generate consistent CAMs using image-level labels.The method generates features from local tiled patches and fits ground-truth mask shapes better.Source: §5 · p. 4
- The segmentation network trained with synthesized pixel-level pseudo-labels achieved state-of-the-art performance on PASCAL VOC 2012.The authors present this result as evidence of the approach’s effectiveness.Source: §5 · p. 4
- The authors propose that Puzzle-CAM’s training-module concepts may benefit other weakly and semi-supervised tasks, including semantic and instance segmentation.This is stated as a prospective generalization.Source: §5 · p. 4