Source-linked AI summary
CIAN: Cross-Image Affinity Net for Weakly Supervised Semantic Segmentation
Junsong Fan, Zhaoxiang Zhang, Tieniu Tan, Chunfeng Song, Jun Xiao
TL;DR
Weakly supervised semantic segmentation seeks complete pixel predictions from inexpensive image-level labels, but existing approaches often treat images independently and recover incomplete regions. CIAN introduces an end-to-end cross-image affinity module that propagates supplementary information between related images, reaching 65.3% mIoU on the VOC 2012 test set and reported state-of-the-art performance.
Problem
Weakly supervised semantic segmentation must recover precise spatial information from image-level labels, while existing approaches often treat images independently and CAM produces sparse, incomplete regions.
Method
CIAN is an end-to-end module that explicitly models pixel-level affinities across related images to retrieve supplementary information and refine representations.
Results
65.3% mIoU on the Pascal VOC 2012 test set is reported as a new state-of-the-art result using only image-level labels.
Takeaways & Limitations
Cross-image relationships support more integral object-region estimations and rectify ambiguity in weakly supervised segmentation.
Takeaways & Limitations
CIAN training requires query-reference pairs with at least one common class; unconstrained random pairing would not learn reliable relationships.
Abstract
from arXiv · showhide
Weakly supervised semantic segmentation with only image-level labels saves large human effort to annotate pixel-level labels. Cutting-edge approaches rely on various innovative constraints and heuristic rules to generate the masks for every single image. Although great progress has been achieved by these methods, they treat each image independently and do not take account of the relationships across different images. In this paper, however, we argue that the cross-image relationship is vital for weakly supervised segmentation. Because it connects related regions across images, where supplementary representations can be propagated to obtain more consistent and integral regions. To leverage this information, we propose an end-to-end cross-image affinity module, which exploits pixel-level cross-image relationships with only image-level labels. By means of this, our approach achieves 64.3% and 65.3% mIoU on Pascal VOC 2012 validation and test set respectively, which is a new state-of-the-art result by only using image-level labels for weakly supervised semantic segmentation, demonstrating the superiority of our approach.
Introduction
Weakly supervised segmentation reduces annotation costs but must recover complete spatial regions from image-level labels. CIAN addresses this by propagating supplementary information across related images through pixel-level affinities, achieving state-of-the-art VOC results.
- Motivation: Pixel-level annotations are expensive, motivating weakly supervised segmentation from weaker signals such as image-level class labels.Semantic segmentation requires per-pixel predictions, while fully supervised methods need large datasets with precise pixel annotations.
- Motivation: CAM-based attention maps focus on discriminative regions rather than target integrity, producing sparse and incomplete seeds.Existing methods use strategies such as iterative erasing, dilation, affinity learning, region growing, and additional constraints to expand these seeds.
- Approach: CIAN models pixel-level relationships between different images and retrieves complementary information to obtain more integral regions.The proposed end-to-end cross-image affinity module can be plugged into existing segmentation networks.
- Approach: Cross-image relationships provide supplementary information, refine ambiguous or false predictions, and promote more consistent representations across the dataset.They also allow labels to be shared across image groups, making greater use of weak supervision.
- Results: CIAN remains useful across seed qualities and can potentially be combined with future methods that generate better seeds.The paper states that extensive experiments demonstrate the usefulness of modeling cross-image relationships.
- Results: 65.3% mIoU on the VOC 2012 test set is reported as a new state-of-the-art result using only image-level labels.The paper also reports 64.3% mIoU on the VOC 2012 validation set.
Related Work
Prior weakly supervised segmentation methods generate pseudo-masks from image-level labels using CAM and seed-expansion strategies. CIAN differs by explicitly exploiting relationships across images, while remaining related to group-based co-segmentation.
- Weakly Supervised Semantic Segmentation: Image-level weakly supervised segmentation commonly generates pseudo-masks before training segmentation networks, with CAM serving as a cornerstone for seed generation.CAM activates only the most discriminative regions, leaving seeds incomplete for segmentation.
- Weakly Supervised Semantic Segmentation: Prior methods expand sparse seeds through iterative erasing, multiple dilation rates, region growing, alternating networks, heuristic constraints, or randomly dropped connections.These approaches address incomplete activations using varied seed-generation and regularization strategies.
- Cross-Image Relationships: Existing methods generally ignore relationships across different images, whereas CIAN uses them to obtain consistent and integral regions.This distinguishes CIAN from the preceding image-centered approaches.
- Co-segmentation: Co-segmentation predicts common-object masks for a group of images and is related to CIAN because both operate across image groups.The supplied passage establishes a task-level relationship without equating the two tasks.
Our Approach
CIAN augments segmentation representations by modeling pixel-level relationships between query and reference images, retrieving supplementary information to complete regions. It combines cross-image and self-affinity during training so the model can operate on single images at testing.
- Initial seeds and training: CIAN first generates initial seeds from image labels using CAM, then trains a segmentation network equipped with the proposed module.Pixels without seed assignments are ignored during cross-entropy training.
- Cross-image affinity: The cross-image affinity module computes pixel-wise affinities between query and reference feature maps and uses them to retrieve supplementary information.Learnable functions produce affinities, compress reference features, and weight their contributions before aggregation.
- Cross-image affinity: Aggregated messages from reference pixels are normalized, merged into each query representation, and passed to the final classification layer for segmentation.The resulting representation is called the cross-affinity augmented representation.
- Multiple pairs: Multiple reference images can be supported by merging their messages before adding the result to the query representation.Maximum and average functions are given as possible message-merging strategies.
- Completion loss: The completion loss uses online pseudo-labels from augmented predictions to compensate for the sparsity of the initial seeds.Online pseudo-labels retain predictions that match both the image-level labels and the network output.
- Overall loss and testing: Training samples image pairs with at least one common class, while self-affinity reduces unmatched-class effects and supports single-image testing.Self-affinity is also used during testing to avoid the representation-distribution change caused by removing the affinity residual.
Experiments and Analysis
Experiments on Pascal VOC 2012 evaluate CIAN’s cross-image affinity module through implementation analysis, ablations, pair-sampling tests, seed-quality tests, and qualitative visualizations. The results show improvements from valid cross-image relationships, completion and retraining strategies, stronger seeds, and low training overhead, while revealing a failure case involving similar interweaving objects.
- Computation Complexity: CIAN adds only marginal computational overhead because its affinity module operates on top-layer features with relatively small spatial dimensions.Its computation complexity matches the self-affinity baseline during testing, with overhead occurring only during training.
- Ablation Study: 0.8% improvement comes from adding cross-image affinity with cross-entropy, followed by another 3.4% from completion loss and 1.8% from retraining with CRF refinement.The completion loss is described as necessary for using complementary information from otherwise sparse initial seeds.
- Pair Sampling: Random-class pairs perform near the baseline, whereas valid common-class pairs outperform it by 4.2%, showing the importance of reliable cross-image relationships.Random-class pairs can still yield a 1.4% gain with completion loss, but their completion remains limited and inferior.
- Seed Quality: 4.4% and 5.1% gains over the baseline occur when 5% and 10% of training seeds are replaced by stronger seeds, respectively.The reported gains support combining CIAN with methods that generate better seeds.
- Overall Results: 64.3% and 65.3% mIoU are achieved on the VOC12 validation and test sets with ResNet101, while visualizations show completed regions, fewer false positives, reduced clutter, and confusion in one failure case.The failure case involves interweaving objects with similar appearance and small spatial scale.
Conclusion
CIAN leverages cross-image relationships to refine incomplete regions and produce more integral object-region estimates for weakly supervised semantic segmentation using only image-level labels. Visualizations show typical improvements over the baseline, alongside failures when multiple classes have similar appearance.
- CIAN builds pixel-level affinities across images and can be plugged into existing segmentation networks.
- CIAN uses supplementary information from related images to refine incomplete regions and rectify ambiguity in object-region estimates.
- The method achieves state-of-the-art performance on the VOC 2012 semantic segmentation task using only image-level labels.
- Figure 4 shows typical cases where CIAN outperforms the baseline and a failure case involving multiple classes with similar appearance.