PaperlayerView exact revision on arXiv

Source-linked AI summary

Few-Shot Object Detection via Variational Feature Aggregation

Jiaming Han, Yuqiang Ren, Jian Ding, Ke Yan, Gui-Song Xia

arXiv:2301.13411v1cs.CV

TL;DR

Few-shot object detectors can become biased toward abundant base classes and sensitive to the variance of scarce novel examples. The paper addresses this with Class-Agnostic Aggregation and Variational Feature Aggregation, using cross-class interactions and VAE-based class distributions. On benchmark evaluations, the method outperforms strong baselines and previous methods, with reported gains of 9%∼16% and 3%∼7% on PASCAL VOC Novel Set 1.

  • Problem

    Few-shot object detectors are biased toward base classes and sensitive to novel-example variance when trained with abundant base samples and few-shot novel examples.

  • Method

    The paper proposes CAA for cross-class feature aggregation and VFA for VAE-based class-level support features, with classification and regression decoupled.

  • Results

    9%∼16%: the method outperforms the strong Meta R-CNN++ baseline and 3%∼7%: previous best results on PASCAL VOC Novel Set 1.

  • Takeaways & Limitations

    CAA reduces class bias through cross-class interactions, while VFA improves robustness by transforming instance-wise support features into class-level distributions.

Abstract

from arXiv · show

As few-shot object detectors are often trained with abundant base samples and fine-tuned on few-shot novel examples,the learned models are usually biased to base classes and sensitive to the variance of novel examples. To address this issue, we propose a meta-learning framework with two novel feature aggregation schemes. More precisely, we first present a Class-Agnostic Aggregation (CAA) method, where the query and support features can be aggregated regardless of their categories. The interactions between different classes encourage class-agnostic representations and reduce confusion between base and novel classes. Based on the CAA, we then propose a Variational Feature Aggregation (VFA) method, which encodes support examples into class-level support features for robust feature aggregation. We use a variational autoencoder to estimate class distributions and sample variational features from distributions that are more robust to the variance of support examples. Besides, we decouple classification and regression tasks so that VFA is performed on the classification branch without affecting object localization. Extensive experiments on PASCAL VOC and COCO demonstrate that our method significantly outperforms a strong baseline (up to 16\%) and previous state-of-the-art methods (4\% in average). Code will be available at: \url{https://github.com/csuhan/VFA}

Introduction

Few-shot object detection must transfer knowledge from abundant base classes to novel classes despite base-class bias and support-set variance. The paper introduces CAA and VFA to improve cross-class representation learning and robustness.

  • The approach targets base-class bias and sensitivity to novel-example variance caused by few-shot fine-tuning and prototype averaging.
  • CAA aggregates query and support features across classes, encouraging class-agnostic representations and reducing confusion between base and novel classes.
  • VFA encodes support examples into class-level features by modeling class distributions with VAEs, producing prototypes robust to few-shot example variance.
  • The framework decouples classification and regression so feature aggregation can improve classification without affecting object localization.
  • 9%∼16%: VFA outperforms the strong Meta R-CNN++ baseline on PASCAL VOC Novel Set 1, while exceeding previous best results by 3%∼7%.

Related Work

Related work spans generic one- and two-stage object detectors, meta-learning approaches for few-shot detection, and variational feature learning for modeling intra-class variance.

  • Generic Object Detection: Generic object detectors are broadly organized into one-stage and two-stage architectures.
  • Few-Shot Object Detection: Few-shot detection research explores meta-learning architectures, feature aggregation schemes, feature-space augmentation, and fine-tuning-based alternatives.
  • Variational Feature Learning: Variational feature learning transforms images or features into distributions and samples from them to model intra-class variance across several learning tasks.

Background and Meta R-CNN++

The paper formalizes FSOD as detecting novel classes from K annotated shots after base-class training, then develops Meta R-CNN++ as a stronger meta-learning baseline.

  • Problem Definition: FSOD splits classes into abundant base classes and K-shot novel classes, transferring representations learned on base classes to novel-object detection.
  • Meta R-CNN++: Stronger Meta-Learning Baseline: Under 1-shot PASCAL VOC Novel Set 1 evaluation, the baseline analysis compares Meta R-CNN and TFA across training-stage and architectural choices.
  • Meta-Learning Based FSOD: Meta R-CNN uses query and support encoders, a feature aggregator, and a detection head within episodic training.
  • Meta R-CNN++: Stronger Meta-Learning Baseline: Meta R-CNN++ aligns key hyperparameters with TFA, including parameter freezing and classifier-weight transfer, to strengthen the meta-learning baseline.
  • Meta R-CNN++: Stronger Meta-Learning Baseline: Meta R-CNN++ with parameter freezing outperforms Meta R-CNN and achieves higher novel AP than TFA, motivating its use as the subsequent baseline.

The Proposed Approach

The proposed Class-Agnostic Aggregation revises class-specific feature interaction by allowing query features to aggregate with support features from different classes.

  • Class-Specific Aggregation: Class-specific aggregation combines each query feature with support features from the same class during training and limits interactions with query-image classes during testing.
  • Class-Agnostic Aggregation: CAA permits cross-class query-support aggregation, encouraging class-agnostic representations and reducing bias toward base classes.
  • Class-Agnostic Aggregation: CAA additionally models relations between different classes, helping prevent confusion between novel classes and similar base classes.

Variational Feature Aggregation𝒜𝒜

VFA extends class-agnostic feature aggregation by transforming support examples into class-level distributions and sampling robust variational features. It applies aggregation to classification while preserving original query features for regression.

  • Framework Overview: The framework contains query and support feature extractors, variational encoder and decoder, feature aggregation, and a detection head, while omitting RPN and regression visualization.The overview labels FQ, FS, Fenc, Fdec, FD, and A as the main components.
  • Class-Agnostic Aggregation: CAA aggregates query and support features across classes, encouraging class-agnostic representations and reducing bias toward base classes.During training, a randomly selected support class is aggregated with each query feature, while testing follows class-specific aggregation.
  • Variational Feature Learning: The variational autoencoder uses reconstruction and KL-divergence objectives to encode support features as distributions from which variational features are sampled.The encoder produces distribution parameters, the decoder reconstructs support features, and the KL term regularizes the latent distribution toward a normal prior.
  • Variational Feature Aggregation: VFA transforms support features into class distributions, samples variational features, and aggregates them with query features for robust feature representation.A variational encoder and decoder model support distributions; sampled features retain generic class information while reducing sensitivity to support-example variance.
  • Optimization: The framework combines detection, regression, classification, consistency, reconstruction, and KL losses in an end-to-end objective.The total loss includes RPN, regression, classification, consistency, reconstruction, and weighted KL-divergence terms.
  • Classification-Regression Decoupling: VFA aggregates features only for classification, while regression uses the original query feature to preserve localization-sensitive information.The design addresses the mismatch between translation-invariant classification features and translation-covariant regression features.

Experiments and Analysis

Experiments on PASCAL VOC and COCO evaluate VFA against prior methods and analyze how CAA, VFA, and support-feature choices affect robustness and class discrimination.

  • PASCAL VOC: VFA achieves the best or second-best results across PASCAL VOC settings and reports a 56.1% average score, 4.2% above the second-best result.In Novel Set 1, its 2-shot result reaches 64.6%, exceeding the previous best 10-shot result of 63.4%.
  • COCO: On COCO, VFA achieves the best nAP among meta-learning methods and the second-best result overall.DeFRCN outperforms VFA in nAP, while the reported VFA configuration omits advanced techniques used by DeFRCN.
  • Visual analysis: CAA reduces class bias and confusion by learning class-agnostic representations, while VFA further improves intra-class similarity and reduces inter-class similarity.The visual analysis reports larger intra-class/inter-class similarity gaps after applying CAA and VFA than with CSA.
  • Robust prototypes: VFA produces prototypes whose distances to real class centers remain more stable as the shot decreases, unlike the baseline’s shot-sensitive prototypes.The estimated prototypes leverage base-class distributions to model novel-class distributions.
  • Support-feature analysis: Latent encodings µ, σ, and z outperform original and reconstructed support features, while z = µ + σ is the default satisfactory choice.The stochastic feature e_z performs worst because its inference process is indeterminate.
  • Ablation studies: The consistency loss L_cons is important because it makes shared-VAE outputs class-conditional and preserves class-specific information in z.The ablation applies L_cons to support features and evaluates its effect in Table 6.

Conclusion

The paper revisits feature aggregation in meta-learning FSOD with CAA and VFA, and experiments on PASCAL VOC and COCO demonstrate their effectiveness.

  • Conclusion: CAA reduces class bias and confusion, while VFA transforms instance-wise support features into class distributions for robust feature aggregation.The proposed aggregation schemes target bias between base and novel classes and variance among few-shot examples.
  • Conclusion: Extensive experiments on PASCAL VOC and COCO demonstrate the effectiveness of the proposed method.

Additional Main Results.

Generalized FSOD experiments show that the method remains competitive across PASCAL VOC and COCO, especially in low-shot settings.

  • PASCAL VOC: On generalized PASCAL VOC FSOD, the method achieves the best results in 12 of 15 settings and second-best results in the remaining three.Its 1-shot gains are 7.2%, 4.2%, and 8.8% on Novel Sets 1, 2, and 3, respectively.
  • COCO: On generalized COCO FSOD, the method achieves the second-best nAP, narrowing its 10-shot gap to DeFRCN to 0.9% versus 2.3% in the traditional setting.

Additional Ablation Studies

Additional studies examine generalized FSOD performance, hyperparameter sensitivity, fine-tuning choices, and feature-space discrimination.

  • VFA design: VFA is sensitive to encoder and decoder hyperparameters at low shot counts, with differences reaching 4% in 1-shot settings but shrinking to 1% in 3- and 5-shot settings.
  • Generalized FSOD: In generalized FSOD, the method performs best or second-best across PASCAL VOC settings and achieves the second-best COCO nAP.
  • Fine-tuning: Table 9 compares freezing or fine-tuning the VFA encoder and decoder to assess how these choices affect performance.
  • Feature discrimination: CAA and VFA enlarge the gap between intra-class and inter-class similarities, reaching 0.12–0.54 with VFA compared with 0.05–0.3 for CSA.

Visualization

The paper visualizes detection results after base training and few-shot fine-tuning on 1, 3, and 5 shots of PASCAL VOC Novel Set 1. As the support set grows, detected novel objects receive higher confidence scores.

  • The model is pretrained on PASCAL VOC base classes, fine-tuned with 1, 3, or 5 shots, and evaluated on the VOC07 test set.
  • Increasing the support set from 1-shot to 5-shot produces more confident detections of novel objects.

More Training Details

Training follows a two-stage base-training and few-shot fine-tuning strategy using query and support datasets. Additional visual analyses report cosine similarities for feature aggregation and detection results across shot counts.

  • More Training Details: Training uses base-class pretraining followed by few-shot fine-tuning, with query and support datasets constructed during base training.
  • More Training Details: During base training, the query dataset contains all base-class data, while the support dataset is balanced-sampled from the query dataset.
  • More Training Details: Figure 8 reports averaged cosine similarities across five-shot examples under the same setting as Figure 5 of the main paper.
  • More Training Details: Figure 9 visualizes detection results for models trained with 1, 3, and 5 shots of PASCAL VOC Novel Set 1 on the VOC07 test set.
Loading 2301.13411v1…