PaperlayerView exact revision on arXiv

Source-linked AI summary

Hypercorrelation Squeeze for Few-Shot Segmentation

Juhong Min, Dahyun Kang, Minsu Cho

arXiv:2104.01538v3cs.CV

TL;DR

Few-shot semantic segmentation must predict query masks from a few annotated support examples while modeling fine-grained correspondences across diverse visual cues. HSNet constructs multi-level hypercorrelations and processes them with efficient center-pivot 4D convolutions in a pyramidal coarse-to-fine architecture. The method reports significant improvements on PASCAL-5i, COCO-20i, and FSS-1000.

  • Problem

    Few-shot segmentation must learn query-object masks from only a few annotated support examples while finding reliable, fine-grained correspondences.

  • Method

    HSNet constructs hypercorrelations from multi-level intermediate features and progressively processes them with efficient 4D convolutions to predict query masks.

  • Results

    HSNet reports significant performance improvements on the PASCAL-5i, COCO-20i, and FSS-1000 few-shot segmentation benchmarks.

  • Takeaways & Limitations

    Learning feature relations from multiple visual aspects is effective for fine-grained segmentation under limited supervision.

  • Takeaways & Limitations

    4D convolutions have quadratic complexity with respect to input-feature size and can be over-parameterized.

Abstract

from arXiv · show

Few-shot semantic segmentation aims at learning to segment a target object from a query image using only a few annotated support images of the target class. This challenging task requires to understand diverse levels of visual cues and analyze fine-grained correspondence relations between the query and the support images. To address the problem, we propose Hypercorrelation Squeeze Networks (HSNet) that leverages multi-level feature correlation and efficient 4D convolutions. It extracts diverse features from different levels of intermediate convolutional layers and constructs a collection of 4D correlation tensors, i.e., hypercorrelations. Using efficient center-pivot 4D convolutions in a pyramidal architecture, the method gradually squeezes high-level semantic and low-level geometric cues of the hypercorrelation into precise segmentation masks in coarse-to-fine manner. The significant performance improvements on standard few-shot segmentation benchmarks of PASCAL-5i, COCO-20i, and FSS-1000 verify the efficacy of the proposed method.

1. Introduction

Few-shot segmentation must generalize from very limited annotation by matching diverse visual cues across support and query images. HSNet addresses this with multi-level hypercorrelations, efficient 4D convolutions, and coarse-to-fine mask prediction.

  • Few-shot segmentation is limited by the need for densely annotated examples, whose collection requires substantial human effort.
  • Reliable correspondence across support and query instances is central to generalizing object appearances from limited supervision.
  • HSNet combines multi-level intermediate features and 4D convolutions to analyze diverse geometric and semantic correspondences.
  • Hypercorrelations collect 4D correlation tensors from multiple intermediate CNN layers, representing correspondences across several visual aspects.
  • A pyramidal architecture progressively squeezes hypercorrelations into precise masks by combining high-level semantic and low-level geometric cues.
  • The efficient 4D kernel reduces computational burden through weight sparsification while supporting real-time inference.

2. Related Work

Related work frames few-shot segmentation as correspondence learning between support and query images. HSNet extends this direction with multi-level feature correlations and efficient 4D convolutional processing in an encoder-decoder architecture.

  • Semantic segmentation: Semantic segmentation commonly uses encoder-decoder architectures that aggregate deep features before decoding them into segmentation masks.
  • Semantic segmentation: Prototypical methods can lose spatial structure through masked average pooling, motivating pair-wise feature correlations for fine-grained mask prediction.
  • Learning visual correspondences: Semantic correspondence methods show that exploiting convolutional features from multiple levels benefits matching accuracy.
  • Learning visual correspondences: HSNet adapts multi-level features and 4D convolutions to construct and process high-dimensional relational features for few-shot segmentation.
  • HSNet: The proposed Hypercorrelation Squeeze Networks analyze dense feature matches with stacked 4D convolutions and introduce a faster center-pivot kernel.
  • HSNet: The network architecture comprises hypercorrelation construction, a 4D-convolutional pyramid encoder, and a 2D-convolutional context decoder.

3. Problem Setup

Few-shot semantic segmentation predicts a query-image mask from only a few annotated support examples. Episodic training learns this mapping on disjoint training classes and evaluates it on novel test classes without further optimization.

  • Few-shot semantic segmentation performs segmentation using only a few annotated examples.
  • Each episode contains a support image and mask plus a query image and mask, with training learning a mapping from support-query inputs to the query mask.
  • Episodic training samples episodes from disjoint training and test class sets to reduce overfitting from insufficient training data.
  • After training, the model predicts masks for test episodes without further optimization.

4. Proposed Approach

HSNet predicts fine-grained query masks by constructing multi-level hypercorrelations and progressively squeezing them with a pyramidal encoder using efficient center-pivot 4D convolutions.

  • Hypercorrelation construction: HSNet constructs hypercorrelations by applying cosine similarity between query and masked support features at multiple intermediate CNN layers.Correlations with matching spatial sizes are concatenated along the channel dimension into pyramidal hypercorrelation tensors.
  • Pyramidal encoder: The encoder squeezes each hypercorrelation pyramid through 4D convolution blocks, merges adjacent pyramid levels, and propagates information top-down.Squeezing blocks reduce support dimensions while preserving query dimensions; mixing blocks process merged features before producing a condensed representation.
  • Mask prediction: The decoder applies 2D convolutions, ReLU, upsampling, and softmax to predict foreground and background probabilities for every query pixel.During testing, the maximum-probability channel at each pixel forms the final query mask.
  • Center-pivot 4D convolution: Center-pivot 4D convolution sparsifies each local 4D kernel by retaining neighbors adjacent to either of the two 2D center positions.It is formulated as the sum of convolutions over the two center-pivot neighbor sets and is designed to reduce the resource burden of dense 4D convolution.
  • Center-pivot 4D convolution: The center-pivot operation performs two convolutions on separate 2D subspaces, giving it linear complexity and improving efficiency in memory and time.The paper reports experimental superiority over existing kernels in accuracy, memory, and time.

5. Experiment

HSNet is evaluated across three few-shot segmentation benchmarks using mIoU and FB-IoU, with experiments examining performance, robustness, and architectural components. Results show strong benchmark performance, benefits from multi-level hypercorrelations and pyramidal cues, and efficiency from center-pivot 4D convolutions.

  • Results and analysis: HSNet sets new state-of-the-art results on PASCAL-5i, COCO-20i, and FSS-1000 with the smallest number of learnable parameters among its tested backbones.With ResNet101, it improves PASCAL-5i mIoU by 6.1%p in 1-shot and 4.8%p in 5-shot over cited prior methods, and COCO-20i by 2.7%p and 6.8%p.
  • Results and analysis: HSNet improves mIoU over prior methods by 6.1%p and 4.8%p on PASCAL-5i, and by 2.7%p and 6.8%p on COCO-20i, for 1-shot and 5-shot settings respectively.These comparisons use the ResNet101 backbone and the cited baselines [70] and on PASCAL-5i, and [70] on COCO-20i.
  • Ablation study: Diverse correlations from dense intermediate CNN layers are crucial: single-channel hypercorrelations show large performance gaps relative to the full hypercorrelation representation.Performance degradation with the deepest-layer correlation also indicates that reliable feature representations typically appear at deeper CNN layers.
  • Ablation study: Removing geometric or semantic levels from the hypercorrelation pyramid damages boundary refinement or target localization, showing that both cue types support fine-grained segmentation.The full pyramid outperforms single-channel alternatives, while shallow and deep intermediate correlations provide complementary information.
  • Ablation study: The center-pivot 4D kernel achieves the fastest inference with the smallest memory and FLOPs requirements while remaining comparably effective to alternative kernels.Compared with separable 4D convolution, it uses 2.6M versus 4.4M parameters and runs at 25.51ms versus 28.48ms per episode.
  • Ablation study: Increasing the number of 4D layers improves accuracy through three layers, after which performance saturates; HSNet therefore uses three layers per building block.The depth study reports this pattern for both 1-shot and 5-shot mIoU on PASCAL-5i.

6. Conclusion

The paper presents HSNet, a fully convolutional framework for analyzing complex feature correlations with lightweight 4D convolutions. Results across three standard benchmarks support learning feature-relation patterns from multiple visual aspects for fine-grained segmentation under limited supervision.

  • Conclusion: HSNet analyzes complex feature correlations fully convolutionally using lightweight 4D convolutions.The framework is presented as a novel approach for few-shot segmentation.
  • Conclusion: Performance improvements on three standard benchmarks support learning feature-relation patterns from multiple visual aspects for fine-grained segmentation under limited supervision.The conclusion identifies this relation-focused learning as effective within the evaluated few-shot setting.
  • Conclusion: Discarding insignificant weights decomposes a 4D kernel into two 2D kernels, enabling extensive 4D convolution use at significantly lower cost.The paper suggests this design may facilitate 4D convolutions in other domains requiring high-dimensional correlation analysis.

Appendix A. Complete derivation of the center-pivot 4D convolution

The appendix derives center-pivot 4D convolution by sparsifying a local 4D neighborhood around two 2D centers. This reduces the operation to two separate convolutions on 2D subspaces with linear complexity.

  • Derivation: Typical 4D convolution aggregates activations over a local 4D neighborhood around each position.The neighborhood is represented as P(x, x′) = P(x) × P(x′).
  • Center-pivot neighborhood: Center-pivot convolution retains neighbors adjacent to either 2D center x or x′ within the corresponding subspace.The selected neighbors form two subsets whose union is the center-pivot neighborhood.
  • Center-pivot convolution: The center-pivot kernel is formulated as the sum of convolutions using kernels kc and kc′ over their respective neighbor sets.The formulation is given as (c ∗kCP)(x, x′) = (c ∗kc)(x, x′) + (c ∗kc′)(x, x′).
  • Computational form: Each component is equivalent to a 2D convolution on a slice of the 4D correlation tensor.The appendix identifies the corresponding 2D kernel operation on c(x, :).
  • Computational form: The resulting operation performs two separate convolutions on 2D subspaces with linear complexity.This provides the stated efficiency benefit over a dense 4D convolution.

Appendix B. Implementation details

The implementation uses ImageNet-pretrained VGG or ResNet backbones and extracts intermediate features to form pyramidal representations. The appendix also specifies evaluation tables for PASCAL-5i and K-shot experiments.

  • Backbone and features: HSNet uses ImageNet-pretrained VGG16, ResNet50, or ResNet101 backbone networks.Feature extraction uses late convolutional or bottleneck layers, producing three pyramidal layers.
  • Evaluation settings: The PASCAL-5i evaluation table reports mIoU and FB-IoU without using ignore_label.The compared results are drawn from prior methods.
  • Evaluation settings: The K-shot evaluation compares ResNet50 results for K ∈ {1, 5, 10}.The appendix compares these results with methods from prior work.

Appendix C. Additional results and analyses

Additional analyses report improvements in higher-shot evaluation and ablations of hypercorrelations and pyramidal layers. The appendix also clarifies boundary-label handling and comparison fairness.

  • Additional K-shot results: 2.5%p and 4.6%p mIoU improvements over the previous best method are reported on PASCAL-5i and COCO-20i, respectively, in the 10-shot setting.These gains establish new state-of-the-art results for the reported 10-shot comparisons.
  • Ablation analyses: 4.5%p mIoU improvement over Cdeep is achieved by using diverse feature correlations from multiple CNN layers.The ablation is summarized numerically in Table A3.
  • Ablation analyses: The second pyramidal layer is especially effective, with C(2:3) reaching 63.9 versus 55.5 for C(3).The comparison concerns robust mask prediction in the pyramidal-layer ablation.
  • Evaluation considerations: PASCAL-5i uses ignore_label regions to exclude ambiguous boundary pixels during training and evaluation.The appendix discusses this convention when comparing methods without ignore_label.
  • Evaluation considerations: Methods that do not use ignore_label are excluded from one comparison because boundary prediction is especially challenging.The exclusion is presented as a fairness measure.

Appendix D. Qualitative results

Qualitative analyses test HSNet across scale changes, occlusions, illumination changes, clutter, tiny objects, and multiple query instances. The results show broad robustness but also identify failures under severe occlusion, variation, and extreme small-object conditions.

  • Robustness analyses: Without support feature masking, HSNet still segments target instances in query images on PASCAL-5i.The appendix relates this behavior to learning feature-correlation patterns across multiple visual aspects.
  • Challenging conditions: Qualitative examples cover large scale differences, truncations, occlusions, illumination changes, exceptionally small objects, intra-class variation, and noisy background clutter.These conditions are illustrated across PASCAL-5i and COCO-20i examples.
  • Multiple instances: A single support image-annotation pair can support segmentation of multiple query instances.The appendix presents one-to-many and many-to-many examples across three benchmarks.
  • Failure cases: Representative failures occur with severe occlusions, intra-class variations, and extremely tiny support or query objects.These failure cases define concrete boundaries for the qualitative robustness claims.
  • Qualitative observations: The appendix reports occasional predictions that are more accurate than human-annotated ground truths.This claim is illustrated qualitatively rather than quantified.
  • Effect of shot count: Predictions become more reliable when multiple support images and mask annotations are available in many challenging cases.The comparison is made between 1-shot and 5-shot results on PASCAL-5i.
Loading 2104.01538v3…