Source-linked AI summary
Few-Shot Segmentation via Cycle-Consistent Transformer
Gengwei Zhang, Guoliang Kang, Yi Yang, Yunchao Wei
TL;DR
Few-shot segmentation must adapt to novel categories from very few annotated examples, while prior methods underuse pixel-wise support information. The paper introduces CyCTR, which aggregates relevant support pixels into query features using cycle-consistent attention, achieving state-of-the-art results on Pascal-5i and COCO-20i.
Problem
Few-shot segmentation seeks rapid adaptation to novel categories with very few annotated samples, but prior methods mainly use semantic-level support prototypes instead of all pixel-wise information.
Method
CyCTR aggregates pixel-wise support features into query features through self-alignment and cross-alignment, using cycle-consistent attention to suppress confusing support pixels.
Results
CyCTR achieves state-of-the-art results on the Pascal-5i and COCO-20i few-shot segmentation benchmarks.
Takeaways & Limitations
The results support utilizing pixel-level support features while filtering harmful information for few-shot segmentation.
Takeaways & Limitations
Cycle-consistent attention may over-suppress positive support regions when support and query images differ extremely.
Abstract
from arXiv · showhide
Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Cycle-Consistent TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-$5^i$ and COCO-$20^i$ datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art methods by 5.6% and 7.1% respectively.
1 Introduction
Few-shot segmentation addresses rapid adaptation to novel categories when dense annotation is scarce. CyCTR uses pixel-wise support-query alignment and cycle-consistent attention to aggregate useful support information while filtering confusing features.
- Motivation: Few-shot segmentation trains models to adapt quickly to novel categories using very few exemplars, addressing the limits of dense annotation and semi-supervised generalization.Dense segmentation masks are time-consuming to label, while semi-supervised approaches may fail to generalize to novel classes with few exemplars.
- Motivation: Previous few-shot segmentation methods primarily use semantic-level support prototypes, limiting their access to pixel-wise support information.These methods follow a metric-learning pipeline derived from PrototypicalNet.
- Method: CyCTR aggregates pixel-wise support features into query features through self-alignment and cross-alignment transformer blocks.Self-alignment encodes query context, while cross-alignment uses query features as Query and support features as Key and Value.
- Method: Scale, color, and scene differences make only a small proportion of support pixels beneficial, so foreground and background pixels can confuse attention.A foreground support pixel may have its most similar query counterpart in a different semantic category and should therefore be ignored.
- Method: Cycle-consistent attention filters confusing support features and enables beneficial foreground and background pixel-level information to be considered.The framework is designed to provide each query pixel with relevant support information through pixel-wise alignment.
2 Related Work
Related work frames few-shot segmentation largely through metric learning and situates transformer and cycle-consistency techniques within broader computer-vision research. These lines of work motivate the paper’s pixel-wise alignment approach.
- Few-Shot Segmentation: Few-shot segmentation methods commonly formulate novel-category segmentation through metric learning, including prototype-based approaches such as PrototypicalNet, PANet, SG-One, and CANet.These methods use support-derived prototypes or similarity and alignment modules to guide query prediction.
- Transformer: Transformer research introduced attention-based architectures for sequence modeling and later extended them to computer vision through self-attention and encoder-decoder frameworks.Examples include Nonlocal, CCNet, ViT, and DETR.
- Cycle-Consistency: Cycle-consistency learning has been applied to align image pairs, learn 3D correspondence, maintain video-frame consistency, and associate different domains.The paper distinguishes its attention operation from cycle-consistency losses used in these prior works.
3 Methodology
CyCTR encodes query features through self-alignment and aggregates relevant pixel-wise support information through cross-alignment. Cycle-consistent attention suppresses inconsistent support pixels, while mask-guided sampling improves scalability for multiple shots.
- Problem Setting: Few-shot segmentation uses support images and masks to condition query-image predictions for novel categories with few exemplars.
- CyCTR Framework: CyCTR stacks encoders containing self-alignment for query context and cross-alignment for aggregating relevant support features into query representations.The cross-alignment block attends between flattened query features and sampled support features.
- Cycle-Consistent Attention: Cycle-consistent attention compares query-support correspondences in both directions and retains support pixels whose labels remain consistent through the cycle.The method computes an affinity map, finds the most similar query pixel for each support pixel, then finds the corresponding support pixel again.
- Cycle-Consistent Attention: Cycle-inconsistent support features receive zero attention weight, preventing irrelevant support information from entering query-feature aggregation.The additive bias is applied element-wise to the affinity values before attention aggregation.
- Self-Alignment: In self-alignment, learnable offsets and attention weights select consistent query pixels when query masks are unavailable.The block uses query features with positional encodings to predict sampled locations and their attention weights.
- K-shot Setting: For K-shot inputs, mask-guided sampling selects foreground and background support tokens to reduce computation and improve scalability as support resolution grows.Sampling also balances foreground and background tokens and accounts for different object-region sizes.
4 Experiments
Experiments on Pascal-5i and COCO-20i show that CyCTR achieves state-of-the-art few-shot segmentation performance. Ablations and qualitative results support the contribution of cycle-consistent attention, while also revealing a remaining failure mode under large support-query differences.
- Comparisons with State-of-the-Art Methods: CyCTR achieves state-of-the-art performance on Pascal-5i and COCO-20i for few-shot segmentation.
- Comparisons with State-of-the-Art Methods: 64.0% mIoU is achieved with ResNet-50 for 1-shot Pascal-5i segmentation, exceeding previous state-of-the-art results by 3.2%.
- Comparisons with State-of-the-Art Methods: 5.6% and 6.0% mIoU gains are obtained over prior state-of-the-art methods for 5-shot Pascal-5i segmentation with ResNet-50 and ResNet-101, respectively.
- Component-Wise Ablations: 0.4% mIoU is gained by vanilla cross-attention over the baseline, while cycle-consistent attention adds a further 0.6% mIoU.
- Component-Wise Ablations: CyCTR’s cycle-consistent attention outperforms predicted aggregation by 0.9% because it globally considers varying conditional information from support and query images.
- Model Capacity: Increasing encoder number or hidden dimension improves results within a certain range, with L = 2 selected as the default accuracy-efficiency trade-off.
- Qualitative Results: Qualitative examples show cycle-consistency correcting several false-positive regions, although it can slightly worsen a mask boundary when support and query images differ extremely.
5 Conclusion
CyCTR addresses few-shot segmentation by using all pixel-level support features while filtering confusing or harmful features through cycle-consistent attention.
- CyCTR uses all pixel-level support features rather than only semantic-level prototypes or foreground features to encode query features.
- Cycle-consistency attention reduces the aggregation of confusing and harmful support features during pixel-level feature transfer.
- CyCTR outperforms previous state-of-the-art methods by a significant margin on two popular few-shot segmentation benchmarks.
A.1 Implementation
The experimental network extracts query and support features with a shared backbone, prepares them for pixel-wise comparison, and uses CyCTR to produce encoded query features for prediction.
- The backbone first extracts features for both query and support images before transformer processing.
- Averaged foreground support features are expanded and concatenated with query and support features to enable pixel-wise comparison.
- CyCTR receives flattened query features, support features, and the flattened support mask, then produces encoded query features for prediction.
- The backbone is an ImageNet-pretrained dilated ResNet that remains fixed during few-shot segmentation training.
- The baseline merges query features with residual blocks and combines concatenated support global features with a prior map for support information.
A.2 Dataset Settings
The experiments use established four-fold cross-validation splits for PASCAL-5i and COCO-20i, reserving class subsets for testing and the remainder for training.
- The dataset split settings follow the settings proposed in prior work.
- PASCAL-5i uses four-fold cross-validation, with 5 test classes and 15 training classes in each row.
- COCO-20i uses four-fold cross-validation, with 20 test classes and the remaining classes used for training in each row.
B More Visualizations
Additional Pascal-5i visualizations show cycle-consistency relationships by distinguishing valid foreground and background correspondences from inconsistent pixels.
- In one example, only a small part of the foreground region is activated, whereas most foreground regions are valid in another.
- Gray pixels on a person indicate a possible negative impact when segmenting a cat.
- The visualizations mark cycle-consistent foreground pixels in red, background pixels in blue, and cycle-inconsistent pixels in gray.