PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Object Detection with Fully Cross-Transformer

Guangxing Han, Jiawei Ma, Shiyuan Huang, Long Chen, Shih-Fu Chang

arXiv:2203.15021v2cs.CVcs.AIcs.MM

TL;DR

FSOD seeks to detect novel objects from very few examples, while prior two-branch methods limit query-support interaction to the detection head. FCT uses cross-transformers in both the feature backbone and detection head, with asymmetric-batched cross-attention for differently sized branches. Experiments on PASCAL VOC and MSCOCO demonstrate the effectiveness of these multi-level interactions.

  • Problem

    Prior two-branch FSOD methods restrict query-support interaction to high-level detection-head features, despite visual differences and domain gaps between the branches.

  • Method

    FCT jointly extracts and aligns query-support features through fully cross-transformer modules in the backbone and detection head, using asymmetric-batched cross-attention.

  • Results

    Experiments on the PASCAL VOC and MSCOCO FSOD benchmarks demonstrate the effectiveness of FCT's multi-level query-support interactions.

  • Takeaways & Limitations

    Multi-level interactions provide an effective basis for two-branch FSOD similarity learning.

Abstract

from arXiv · show

Few-shot object detection (FSOD), with the aim to detect novel objects using very few training examples, has recently attracted great research interest in the community. Metric-learning based methods have been demonstrated to be effective for this task using a two-branch based siamese network, and calculate the similarity between image regions and few-shot examples for detection. However, in previous works, the interaction between the two branches is only restricted in the detection head, while leaving the remaining hundreds of layers for separate feature extraction. Inspired by the recent work on vision transformers and vision-language transformers, we propose a novel Fully Cross-Transformer based model (FCT) for FSOD by incorporating cross-transformer into both the feature backbone and detection head. The asymmetric-batched cross-attention is proposed to aggregate the key information from the two branches with different batch sizes. Our model can improve the few-shot similarity learning between the two branches by introducing the multi-level interactions. Comprehensive experiments on both PASCAL VOC and MSCOCO FSOD benchmarks demonstrate the effectiveness of our model.

1. Introduction

FSOD seeks to detect novel objects from very few examples, but conventional approaches struggle with data scarcity and limited query-support interaction. FCT addresses this by jointly processing both branches with cross-transformers throughout the backbone and detection head.

  • Motivation: FSOD aims to detect objects in query images from only a few training examples, requiring strong generalization to unseen images.Deep-learning methods commonly need thousands of examples and can overfit in data-scarce settings.
  • Prior approaches: Existing FSOD architectures include single-branch detectors and two-branch siamese models that compare query regions with few-shot examples.Single-branch methods are prone to overfitting, while two-branch methods use metric learning for similarity-based detection.
  • Motivation: Prior two-branch methods restrict query-support interaction to high-level detection-head features, leaving hundreds of earlier layers to extract features separately.This may be suboptimal when the branches differ in pose, scale, illumination, occlusion, or background.
  • Proposed approach: FCT incorporates cross-transformers into both the feature backbone and detection head to jointly extract and align query-support features at multiple levels.The model uses a multi-layer cross-transformer rather than separately extracting deep visual features for each branch.
  • Proposed approach: Asymmetric-batched cross-attention aggregates key information from branches with different batch sizes, while transformer attention captures within-branch and cross-branch context.The method uses multi-scale visual tokens and spatial-reduction operations to reduce computational complexity.

2. Related Works

Related work spans general object detection, few-shot learning, FSOD architectures, and transformer-based vision models. FSOD methods mainly use single-branch detectors or two-branch siamese networks for comparing query regions with support examples.

  • Object Detection: Object detection methods are commonly grouped into proposal-based and proposal-free approaches, with deep convolutional networks as a dominant approach.Proposal-based methods divide detection into stages involving object proposals.
  • Few-Shot Learning: Few-shot learning transfers knowledge from data-abundant base classes to recognize novel classes using only a few examples.Existing methods include optimization-based approaches such as MAML.
  • Few-Shot Object Detection: Few-shot object detection must both recognize novel objects from few examples and localize them in images.This distinguishes FSOD from few-shot recognition alone.
  • Few-Shot Object Detection: FSOD architectures mainly comprise single-branch methods and two-branch methods based on siamese networks.Two-branch methods process query and support inputs in parallel and compare image regions with few-shot examples.
  • Transformer and Its Application in Computer Vision: Transformers model long-range dependencies with self-attention and have been extended to vision-language pre-training and image classification.ViLT uses a convolution-free vision-language transformer focused on modality interactions, motivating FCT.

3. Our Approach

FCT is a pure cross-transformer FSOD model that jointly extracts query and support features through multi-level interactions in both the backbone and detection head. It uses asymmetric-batched cross-attention to handle the branches’ different batch sizes and follows a staged Faster R-CNN-based training procedure.

  • Model Overview: FCT replaces separate deep feature encoders with a pure cross-transformer model based on Faster R-CNN.The model uses cross-transformer modules for joint feature extraction instead of deep convolutional networks.
  • Cross-Transformer Feature Backbone: Spatial reduction subsamples feature maps for keys and values to reduce attention complexity and summarize key information.The operation can use strided convolution or spatial pooling, with shared projection weights across branches.
  • Asymmetric-Batched Cross-Attention: Asymmetric-batched cross-attention aggregates key-value information across query and support branches despite their different batch sizes.Support features are average-pooled to match the query batch, while query features are repeated for support aggregation before concatenation.
  • Cross-Transformer Feature Backbone: Three cross-transformer backbone stages progressively interact through low-level, mid-level, and high-level visual features.The stages produce decreasing sequence lengths and increasing channel dimensions.
  • Cross-Transformer Detection Head: The detection head jointly extracts RoI features for query proposals and support images before pairwise matching and final detection.The head averages support images, applies asymmetric-batched cross-attention, and then uses a pairwise matching network with binary cross-entropy and bounding-box regression losses.
  • Model Training: Training proceeds by pretraining a single-branch base-class model, then training the fully cross-transformer two-branch model initialized from it.The initialization reuses most parameters and is intended to ease FCT training.

4. Experimental Results

Experiments on PASCAL VOC and MSCOCO evaluate FCT across backbones, architectural components, training strategies, and state-of-the-art comparisons. Results support multi-level cross-branch interaction, concatenation-based aggregation, staged training, and strong benchmark performance.

  • Benchmark setup: FCT is evaluated on PASCAL VOC and MSCOCO FSOD benchmarks under multiple few-shot settings.PASCAL VOC reports AP50 for 1, 2, 3, 5, and 10 shots; MSCOCO reports AP for 1, 2, 3, 5, 10, and 30 shots.
  • Baseline comparison: The two-branch baseline outperforms the single-branch baseline with both ResNet-101 and PVTv2-B2-Li, especially in extremely few-shot settings.The reported explanation is that two-branch learning compares query regions with few-shot classes, while single-branch classification is prone to overfitting.
  • Architecture ablation: Adding cross-transformer stages improves performance through mutual alignment and feature fusion, with the detection-head stage producing the largest improvement.Using the first three stages forms the cross-transformer feature backbone, while combining those stages with the detection-head stage yields the best FCT results.
  • Backbone comparison: PVTv2-B2-Li achieves performance very similar to PVTv2-B2 while providing faster training and testing, motivating its default use.PVTv2-B2 outperforms the smaller B0 and B1 variants because of its larger model capacity.
  • Information aggregation: Concatenating key-value pairs with branch embeddings outperforms element-wise addition or multiplication, which can lose information.The branch embedding identifies the source branch and slightly improves performance.
  • State-of-the-art comparison: FCT achieves state-of-the-art results across most shots and evaluation settings on both benchmarks, including the most challenging MSCOCO 1-shot multiple-run setting.Compared with single-branch methods, FCT achieves the second-best results under the multiple-run setting.

5. Conclusion

The paper concludes that FCT integrates cross-transformers into both the feature backbone and detection head, using asymmetric-batched cross-attention for query-support interaction. Quantitative results and visualizations support the effectiveness of these multi-level interactions.

  • FCT incorporates cross-transformers into both the feature backbone and detection head for few-shot object detection.
  • Asymmetric-batched cross-attention aggregates key-value pairs from query and support branches with different batch sizes.
  • The evaluation reports quantitative results on two widely used FSOD benchmarks and qualitative visualizations of learned multi-level cross-attention.
  • The reported evidence supports the effectiveness of multi-level interactions between query and support branches.

Appendix

The appendix provides implementation details and additional visualizations of the proposed multi-level cross-attention.

  • The supplementary materials first describe the training framework's implementation details.
  • The appendix then presents additional visualization examples of the proposed multi-level cross-attention.

A. Implementation Details for Model Training

Model training uses three stages: base-class pre-training, base-class training of the fully cross-transformer two-branch model, and novel-class fine-tuning before evaluation.

  • Fine-tuning on novel classes: The final stage fine-tunes the two-branch model on a sub-sampled K-shot dataset containing both base and novel classes, then evaluates it.
  • Pre-training the single-branch model: Training begins by pre-training a single-branch Faster R-CNN with a PVTv2 vision-transformer backbone on abundant base-class data.The model is initialized from an ImageNet-pretrained model.
  • Pre-training the single-branch model: The single-branch model uses classification and bounding-box regression losses for region proposals and detection.The classification losses include binary cross-entropy for foreground/background and multi-class cross-entropy over base classes plus background; localization uses smooth L1 loss.
  • Training the two-branch model: The second stage trains the FCT two-branch model over base classes using initialization from the first-stage pretrained model.FCT reuses most first-stage parameters and learns branch embeddings and a pairwise matching network.
  • Training the two-branch model: The two-branch model's attention RPN and matching network use binary cross-entropy and bounding-box regression losses.
  • Training settings: MSCOCO base training uses AdamW with learning rate 0.0001, weight decay 0.0002, batch size 4, and 20,000 iterations.The learning rate is divided by 10 after 15,000 and 20,000 iterations.
  • Training settings: PASCAL VOC base training uses 10,000 iterations with learning rate 0.0002, divided by 10 after 7,500 and 10,000 iterations.
  • Training settings: Novel-class fine-tuning uses AdamW with learning rate 0.0002, weight decay 0.0001, batch size 4, and shot-dependent iteration schedules.Thirty-shot fine-tuning runs for 5,000 iterations; ten-shot or fewer runs for 3,000 iterations.

B. Visualization of Multi-level Cross-attention

The appendix includes additional visualizations of the multi-level cross-attention learned by the proposed model.

  • Figure 5 provides a visualization of the multi-level cross-attention, corresponding to Figure 4 in the main paper.
  • Figure 6 provides another visualization of the multi-level cross-attention, also corresponding to Figure 4 in the main paper.
Loading 2203.15021v2…