Source-linked AI summary
Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation
Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, Yefeng Zheng
TL;DR
Few-shot semantic segmentation must use scarce support images effectively because prototype compression and partial pixel-level support use can lose information. DCAMA performs dense query-support attention and aggregates all support-mask pixels, enabling one-pass n-shot inference. It achieves state-of-the-art performance across three standard benchmarks, with reported 1-shot mIoU gains of 3.1%, 9.7%, and 3.6%.
Problem
Few-shot semantic segmentation needs to exploit scarce support-image information, but prototype compression and foreground-only pixel methods lose support information.
Method
DCAMA predicts query masks by attention-weighted aggregation of all foreground and background support-mask pixels, extending to n-shot inference by pooling support pixels in one pass.
Results
DCAMA sets the state of the art across PASCAL-5i, COCO-20i, and FSS-1000 in both 1-shot and 5-shot settings, including 3.1%, 9.7%, and 3.6% 1-shot mIoU gains.
Takeaways & Limitations
Dense pixel-wise support utilization and one-pass aggregation provide a general DCAMA framework for few-shot segmentation across varying shot settings.
Takeaways & Limitations
The 1/4-scale features are not cross-attended because of hardware constraints, and training n-specific models for n > 1 remains computationally prohibitive.
Abstract
from arXiv · showhide
Research into Few-shot Semantic Segmentation (FSS) has attracted great attention, with the goal to segment target objects in a query image given only a few annotated support images of the target class. A key to this challenging task is to fully utilize the information in the support images by exploiting fine-grained correlations between the query and support images. However, most existing approaches either compressed the support information into a few class-wise prototypes, or used partial support information (e.g., only foreground) at the pixel level, causing non-negligible information loss. In this paper, we propose Dense pixel-wise Cross-query-and-support Attention weighted Mask Aggregation (DCAMA), where both foreground and background support information are fully exploited via multi-level pixel-wise correlations between paired query and support features. Implemented with the scaled dot-product attention in the Transformer architecture, DCAMA treats every query pixel as a token, computes its similarities with all support pixels, and predicts its segmentation label as an additive aggregation of all the support pixels' labels -- weighted by the similarities. Based on the unique formulation of DCAMA, we further propose efficient and effective one-pass inference for n-shot segmentation, where pixels of all support images are collected for the mask aggregation at once. Experiments show that our DCAMA significantly advances the state of the art on standard FSS benchmarks of PASCAL-5i, COCO-20i, and FSS-1000, e.g., with 3.1%, 9.7%, and 3.6% absolute improvements in 1-shot mIoU over previous best records. Ablative studies also verify the design DCAMA.
1 Introduction
Few-shot semantic segmentation must extract more information from scarce support images than prototype-based or foreground-only methods preserve. DCAMA addresses this with dense pixel-wise query-support attention, support-mask aggregation, and one-pass n-shot inference, achieving state-of-the-art results across standard benchmarks.
- Motivation: Few-shot semantic segmentation adapts to novel classes using only a few annotated support images, making effective support-information use central.The task evaluates segmentation on query images containing the same novel classes as the support set.
- Motivation: Prototype-based methods compress support images into class-wise representations, while pixel-level methods may use only foreground features, discarding scarce support information.The introduction identifies this information loss as a key challenge in FSS.
- DCAMA: DCAMA predicts each query-pixel mask value by aggregating all support-mask values according to dense similarities between query and support pixels.The aggregation includes both foreground and background support pixels.
- DCAMA: DCAMA implements this pixel-wise aggregation with Transformer dot-product attention, treating query pixels as tokens and computing the mask through attention-weighted support values.Flattened query features form Q, support features form K, and support labels form V in the attention computation.
- N-shot inference: DCAMA extends to n-shot inference by collecting pixels from all support images at once, avoiding separate 1-shot inferences and n-specific models.The same 1-shot-trained model can aggregate additional support pixels during inference.
- Results: DCAMA sets the state of the art on PASCAL-5i, COCO-20i, and FSS-1000 in both 1-shot and 5-shot settings.The reported comparison covers all three benchmarks and both evaluated few-shot settings.
2 Related Work
Related FSS work progresses from prototype-based metric learning toward pixel-level query-support correlations, while Transformer-based approaches have also largely retained prototyping.
- Semantic segmentation commonly uses encoder-decoder networks with skip connections and multi-scale processing for dense pixel prediction.
- Few-shot segmentation methods typically compress support information into class-wise prototypes and classify query images by distances in metric space.
- Multiple-prototype methods address the limited representation power of a single prototype, but still compress support information into concentrated representations.
- Pixel-level approaches model dense query-support correspondences using graph attention or 4D correlation tensors.
- Transformer-based FSS methods explored global enhancement and query-specific classifier adaptation, but retained the prototyping pipeline.
- The proposed framework is illustrated as a 1-shot pipeline in Fig. 2.
3 Methodology
DCAMA performs dense pixel-wise attention between query and support features, then aggregates support-mask labels to predict query segmentation. Its formulation also enables one-pass n-shot inference by pooling pixels from all support images as attention tokens.
- Framework overview: DCAMA processes multi-scale query and support features together with downsampled support masks for cross-attention-based mask aggregation.The framework uses pretrained feature extraction, multi-scale features, and multi-layer feature maps before combining aggregated query masks with image features through skip connections.
- Cross-attention weighted aggregation: Scaled dot-product attention computes dense pixel-wise similarities between query and support features and weights aggregation of support-mask values.For each query pixel, QK^T measures similarity to all support pixels, while multiplication with V aggregates their mask values.
- Implementation limitation: The 1/4-scale features are not cross-attended because of a hardware constraint.
- Cross-attention weighted aggregation: DCAMA uses both foreground and background support pixels, allowing similarity-weighted voting toward the corresponding segmentation label.Pixels more similar to foreground than background support pixels vote for foreground, and vice versa.
- Multi-scale mask fusion: The method combines independently aggregated masks across layers and scales, then fuses them with image features through convolution, upsampling, additions, and skip connections.The resulting mask-feature mixer restores the output resolution and produces foreground and background channels for 1-way segmentation.
- n-shot inference: One-pass n-shot inference treats pixels from additional support images as extra key and value tokens while keeping the DCAMA process unchanged.This avoids performing separate 1-shot inference for every support image and follows directly from DCAMA's attention formulation.
4 Experiments and Results
DCAMA is evaluated on three standard FSS benchmarks under multiple backbones and shot settings, with comparisons, qualitative analyses, convergence studies, and intermediate-mask evaluations. It achieves strong performance while maintaining comparable inference speed and benefiting from multi-scale aggregation.
- Comparison with state of the art: DCAMA achieves the best mIoU and FB-IoU for almost all backbone and shot-setting combinations, with comparable performance to HSNet in the ResNet-101 PASCAL-5i exception.With Swin-B, mIoU gains over HSNet are 3.1% and 4.5% on PASCAL-5i, 9.7% and 8.8% on COCO-20i, and 3.6% and 1.9% on FSS-1000 for 1-shot and 5-shot settings.
- Comparison with state of the art: DCAMA uses dense pixel-level similarities and both foreground and background support information, contrasting with methods that prototype or omit background features.The comparison discussion distinguishes DCAMA’s mask inference from HSNet and CyCTR while linking pixel-wise correlations to stronger performance than prototyping methods.
- Efficiency: DCAMA takes substantially fewer training epochs to converge than HSNet while achieving comparable inference speed, reaching about 8 FPS with Swin-B and 20 FPS with ResNet-101 for 1-shot segmentation.Both methods have O(N^2) pixel-wise correlation complexity and comparable per-epoch training time.
- Qualitative and intermediate-mask analysis: Multi-scale intermediate masks are valid but less accurate individually, while the final prediction integrates them to achieve optimal performance.The 1/16-scale masks are relatively strong, whereas 1/8 and 1/32 masks are weaker; qualitative visualizations show valid intermediate segmentations.
5 Conclusion
The conclusion presents DCAMA as a metric-learning paradigm for FSS that aggregates support-mask values using pixel-wise query-support similarities. It emphasizes full support-pixel utilization and one-pass n-shot inference, with state-of-the-art results across three benchmarks.
- Conclusion: DCAMA predicts each query-pixel mask value by similarity-weighted additive aggregation of all support-pixel mask values.The framework uses scaled dot-product attention in a Transformer structure for simplicity and efficiency.
- Conclusion: DCAMA fully uses foreground and background support pixels and performs one-pass n-shot inference with pixels from all support images considered simultaneously.These design choices distinguish the framework from approaches that compress support information or process shots independently.
- Conclusion: DCAMA sets the state of the art on all three commonly used FSS benchmarks.The paper identifies adapting the paradigm to other few-shot dense-prediction tasks such as detection as future research.
A. More Results
Additional experiments examine stability, region-wise segmentation quality, computational efficiency, and n-shot scaling. They report stable repeated trials, reduced training cost, and approximately linear increases in memory and latency with the number of shots.
- Stability and robustness: Repeated trials on PASCAL-5i Fold-0 produce stable IoUs of 72.1±0.25, while tested batch-size and learning-rate pairs yield IoUs from 71.9 to 72.2.The results support stability across repeated trials and the examined hyperparameter settings.
- Region-wise evaluation: ROM and RUM provide threshold-free region-wise measures of over-segmentation and under-segmentation, with smaller values preferred.These metrics are intended to improve interpretability of semantic segmentation performance in real-world applications.
- Computational efficiency: DCAMA has slightly larger FLOPS but comparable epoch-training and inference times to HSNet, while converging in substantially fewer epochs.The reported computational profile implies lower total training time despite the larger FLOPS.
- n-shot scaling: Memory and latency for n-shot inference increase approximately linearly as n rises from 1 to 5.The analysis covers n=1 through n=5.
B. Further Ablation Studies
Further ablations identify effective skip connections, multi-scale attention, and support-feature usage as important components of DCAMA. The experiments also show that intermediate multi-scale masks remain useful in challenging 5-shot cases.
- Skip-connection configuration: Jointly concatenating 1/4- and 1/8-scale features improves mIoU by +3.1% and FB-IoU by +1.1% over no skip connection.Adding 1/16-scale features instead causes obvious performance deterioration, so the framework uses the 1/4 and 1/8 scales.
- Multi-scale attention: Removing 1/8-scale attention decreases mIoU and FB-IoU by 2.1% and 1.4%, while additionally removing 1/16-scale attention causes decreases of 8.5% and 6.6%.These ablations indicate that multi-scale attention is important to the proposed framework.
- Support features and pixel counts: Excluding support features from skip connections or normalizing foreground and background support-region sizes impairs performance.Skip-connected high-level feature maps retain relatively large receptive fields and remain helpful even when not aligned.
- Ablation setting: The ablations use Swin-B on PASCAL-5i, while the supplementary qualitative examples include 5-shot segmentation under intra-class variation, scale differences, complex backgrounds, and occlusions.The studies therefore combine controlled component tests with challenging multi-shot visual examples.
C. More Visual Analysis
The visual analyses examine DCAMA’s qualitative failures, attention behavior, and learned feature space. Additional informative support images address several 1-shot failure cases, while attention and t-SNE visualizations illustrate the model’s pixel correspondences and feature separability.
- Failure analysis: Extra support images accurately resolve the representative 1-shot failure cases in the 5-shot setting.The additional images and masks provide information that is absent or insufficient in the original support example.
- Failure analysis: DCAMA’s 1-shot failures arise from limited support representativeness, intra-class variation, and inter-class similarity.These limitations include occlusion, small objects, viewpoint changes, appearance variation, and confusion between different classes.
- Support-image analysis: An informative extra support image fixes the 1-shot failures in 2-shot inference, whereas a non-informative image helps little.The informative image supplies completed class information, within-class variation, and inter-class differentiation.
- Attention visualization: Foreground and background query pixels assign strongest attention to semantically similar regions in the support image.The visualized maps show weaker responses around similar regions belonging to other instances.
- Feature-space visualization: DCAMA’s penultimate-layer features are more separable than frozen Swin-B features in the t-SNE visualization.The comparison qualitatively indicates more effective learning of the metric space.