Source-linked AI summary
Token Contrast for Weakly-Supervised Semantic Segmentation
Lixiang Ru, Heliang Zheng, Yibing Zhan, Bo Du
TL;DR
The paper addresses incomplete CAM regions and ViT patch-token over-smoothing in weakly supervised semantic segmentation. It introduces Token Contrast through Patch Token Contrast and Class Token Contrast, achieving stronger single-stage results and performance comparable to multi-stage methods on PASCAL VOC and MS COCO.
Problem
CAM-based pseudo labels often miss integral object regions, while ViT-based WSSS introduces over-smoothing in which final patch tokens become nearly uniform.
Method
Token Contrast combines Patch Token Contrast, which supervises final patch-token relations with intermediate-layer knowledge, and Class Token Contrast, which aligns local uncertain regions with global object representations.
Results
ToCo significantly outperforms state-of-the-art single-stage WSSS methods and achieves comparable performance with multi-stage competitors on PASCAL VOC and MS COCO.
Takeaways & Limitations
Token-level contrast improves CAM quality and object activation completeness within a single-stage WSSS framework.
Takeaways & Limitations
CTC is not a perfect objective when global images contain multiple semantics but local images cover only partial semantics.
Abstract
from arXiv · showhide
Weakly-Supervised Semantic Segmentation (WSSS) using image-level labels typically utilizes Class Activation Map (CAM) to generate the pseudo labels. Limited by the local structure perception of CNN, CAM usually cannot identify the integral object regions. Though the recent Vision Transformer (ViT) can remedy this flaw, we observe it also brings the over-smoothing issue, \ie, the final patch tokens incline to be uniform. In this work, we propose Token Contrast (ToCo) to address this issue and further explore the virtue of ViT for WSSS. Firstly, motivated by the observation that intermediate layers in ViT can still retain semantic diversity, we designed a Patch Token Contrast module (PTC). PTC supervises the final patch tokens with the pseudo token relations derived from intermediate layers, allowing them to align the semantic regions and thus yield more accurate CAM. Secondly, to further differentiate the low-confidence regions in CAM, we devised a Class Token Contrast module (CTC) inspired by the fact that class tokens in ViT can capture high-level semantics. CTC facilitates the representation consistency between uncertain local regions and global objects by contrasting their class tokens. Experiments on the PASCAL VOC and MS COCO datasets show the proposed ToCo can remarkably surpass other single-stage competitors and achieve comparable performance with state-of-the-art multi-stage methods. Code is available at https://github.com/rulixiang/ToCo.
1 School of Computer Science, Wuhan University, China. 2 JD Explore Academy, China
The paper lists affiliations for Wuhan University and JD Explore Academy, China.
- The authors are affiliated with Wuhan University and JD Explore Academy in China.
1. Introduction
Image-level WSSS commonly uses CAM pseudo labels, but CNN locality leaves object regions incomplete and ViT introduces over-smoothing. ToCo addresses these issues with patch- and class-token contrast, improving CAM quality and segmentation performance.
- Image-level labels are cheap but provide limited information for WSSS.
- CNN-based CAM often activates only the most discriminative regions because convolution primarily perceives local features.
- ViT can produce nearly uniform patch tokens through over-smoothing, causing CAM to assign monotonous labels across image regions.
- PTC supervises final patch-token relations with semantic knowledge from intermediate layers, while CTC aligns uncertain local regions with global objects through class-token contrast.
- ToCo significantly outperforms state-of-the-art single-stage WSSS methods and achieves comparable performance with multi-stage competitors on PASCAL VOC and MS COCO.
2. Related Work
Prior WSSS methods address CAM’s incomplete activation through training schemes and auxiliary tasks that improve pseudo-label generation.
- WSSS research has used erasing, online attention accumulation, and cross-image semantic mining to address CAM’s limited object activation.
- Other approaches regularize training with auxiliary tasks such as visual-word learning, sub-category exploration, and scale-invariant feature learning.
3. Preliminaries
CAM generates normalized class-specific activation maps from classification features, while ViT’s global self-attention can progressively make patch tokens overly similar. Early Transformer layers retain more semantic diversity, motivating intermediate-layer supervision.
- Class Activation Map: CAM weights classification feature maps by class-specific classifier weights, removes negative activations with ReLU, and max-normalizes the result to [0, 1].
- Class Activation Map: A background threshold β separates background and foreground regions in CAM-based pseudo labels.
- Vision Transformer: ViT forms patch tokens, appends a learnable class token, and processes them through Transformer blocks with global multihead self-attention.
- Over-smoothing: Patch-token similarity increases in late Transformer blocks, whereas early blocks preserve semantic diversity.
- Over-smoothing: This layerwise pattern motivates supervising final-layer tokens with intermediate-layer knowledge to counter over-smoothing.
4. Methodology
ToCo addresses ViT over-smoothing in weakly-supervised segmentation with Patch Token Contrast (PTC) and uses Class Token Contrast (CTC) to improve activation of uncertain object regions. Its auxiliary and final CAMs support token-relation supervision, local-global contrast, and pseudo-label generation.
- Overall framework: ToCo adds an auxiliary classification layer whose CAM guides PTC and CTC, while a final classification layer produces the final CAM and pseudo labels.The auxiliary CAM supplies intermediate supervision and local crop proposals for the two contrastive modules.
- Patch Token Contrast: PTC uses semantic relations from an intermediate layer, whose patch tokens retain greater diversity, to supervise the final patch-token similarities.The intermediate layer is selected because early layers may lack high-level semantics and late layers may already be smoothed.
- Patch Token Contrast: PTC derives positive and negative token pairs from auxiliary pseudo labels and encourages similar final-token representations for positive pairs but discriminative representations for negative pairs.The loss uses absolute cosine similarity in practice to avoid negative cosine values remaining highly correlated.
- Class Token Contrast: CTC crops local images from uncertain auxiliary-CAM regions and background regions, then contrasts their projected class tokens with the global class token.Positive local crops are aligned with the global representation, whereas negative background crops are separated from it.
- Class Token Contrast: CTC stops gradients through the global projection head and updates it with an exponential moving average of the local projection head.The InfoNCE objective uses q+ and q− for uncertain and background crops, respectively, with temperature τ and stability term ϵ.
- Training objective and WSSS: ToCo combines token-contrast losses with classification and auxiliary classification losses, while single-stage WSSS additionally uses refined pseudo labels to supervise a segmentation decoder.The pseudo labels are refined with pixel-adaptive refinement before segmentation training.
5. Experiments
Experiments on VOC and COCO show that ToCo produces more complete CAMs and strong single-stage segmentation results, while PTC and CTC address complementary ViT limitations.
- Pseudo Labels: ToCo produces more integral and accurate CAMs than CNN, AFA, and the ViT baseline, while alleviating ViT over-smoothing.The comparison includes ViT with PTC and ViT with PTC plus CTC.
- Semantic Segmentation Results: ToCo achieves 71.1% mIoU on VOC val, 72.2% on VOC test, and 42.3% on COCO val.These results substantially outperform previous single-stage methods.
- Semantic Segmentation Results: ToCo’s single-stage results exceed multi-stage WSSS methods using image-level labels only, while methods using additional saliency maps slightly outperform it.The comparison uses semantic segmentation results on VOC and COCO.
- Semantic Segmentation Results: ToCo produces more accurate segmentation masks than AFA, with results described as close to ground truth.The qualitative comparison is shown on VOC and COCO examples.
- Ablation: PTC improves final CAM mIoU from 27.9% to 62.5%, raises intermediate CAM mIoU to 57.8%, and CTC adds 4.7% mIoU to pseudo-label quality.The ablation evaluates final CAM, auxiliary CAM, and semantic segmentation on VOC val.
- Ablation and Analysis: PTC lowers late-layer patch-token cosine similarity, while CTC activates less discriminative regions through global-local class-token comparisons.The analyses connect token diversity and local-to-global semantic consistency with improved CAM quality.
- Ablation and Analysis: The CTC objective is limited when global images contain multiple semantics but local crops contain only partial semantics, although class tokens can capture multi-class information.The authors describe this as an imperfect objective and report that CTC works well on multi-semantic images.
- Hyper-parameter Analysis: A local crop size of 962 achieves the best performance, while EMA momentum ρ = 0.9 is the best tested choice but other values remain favorable.These hyper-parameter studies are evaluated on the VOC validation set.
6. Conclusion
ToCo addresses ViT over-smoothing in WSSS through complementary patch-token and class-token contrast modules. Experiments on VOC and COCO show strong performance against competing methods.
- PTC supervises final patch representations with intermediate-layer knowledge to counter over-smoothing and improve WSSS pseudo labels.CTC contrasts global and local class tokens to improve activation completeness in CAM.
- ToCo is evaluated within a single-stage WSSS framework on the VOC and COCO datasets.
- ToCo significantly outperforms other competitors in the reported experiments.
7. Additional Results
Additional experiments examine backbone generality, hyperparameters, cosine-similarity choices, and qualitative segmentation behavior. Across these analyses, ToCo addresses over-smoothing and improves pseudo-label and segmentation quality.
- ViT-S and ViT-L† also exhibit over-smoothing, while ToCo addresses it across these backbone configurations.ViT-S uses 12 Transformer blocks and ViT-L uses 24.
- Absolute cosine similarity performs best for PTC, whereas directly minimizing cosine similarity is unsatisfactory.The reported explanation is that negatively similar patch tokens can remain correlated.
- ToCo produces better pseudo labels than AFA and segmentation results close to ground truth in the qualitative comparisons.The reported visualizations attribute improved CAM behavior to PTC and CTC addressing over-smoothing and uncertain regions, respectively.
VOC COCO
The paper reports semantic segmentation results on VOC and COCO and visualizes class-token attention for global and local views. These figures support comparison across datasets and views.
- Figure 11 presents semantic segmentation results on the VOC and COCO datasets.
- Figure 12 visualizes class-token attention to patch tokens for global images and randomly cropped local views.Brighter regions indicate larger attention values.