PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Relation Reasoning for Shot-Stable Few-Shot Object Detection

Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, Marios Savvides

arXiv:2103.01903v2cs.CV

TL;DR

Few-shot object detection is limited by scarce novel-class data and sensitivity to both explicit and implicit shots. SRR-FSD combines visual representations with word-embedding semantic spaces and dynamic relation reasoning, achieving stable performance across shot variations, particularly in lower-shot and reduced-pretraining settings. The proposed removal of implicit shots offers a more realistic benchmark protocol.

  • Problem

    Few-shot object detection remains sensitive to scarce novel-class data and to implicit novel-class knowledge encoded in pretrained backbones.

  • Method

    SRR-FSD projects visual object representations into a word-embedding semantic space and augments semantic embeddings with a dynamic relation graph for explicit relation reasoning.

  • Results

    SRR-FSD achieves competitive higher-shot results and significantly better lower-shot performance while maintaining performance more steadily when implicit novel-class shots are removed from pretraining.

  • Takeaways & Limitations

    Semantic relation reasoning supports robust few-shot detection across explicit and implicit shot variation, and the reduced-pretraining protocol provides a more realistic evaluation setting.

  • Takeaways & Limitations

    Raw semantic embeddings can be misleading when more images are available because of the domain gap between vision and language, motivating embedding augmentation.

Abstract

from arXiv · show

Few-shot object detection is an imperative and long-lasting problem due to the inherent long-tail distribution of real-world data. Its performance is largely affected by the data scarcity of novel classes. But the semantic relation between the novel classes and the base classes is constant regardless of the data availability. In this work, we investigate utilizing this semantic relation together with the visual information and introduce explicit relation reasoning into the learning of novel object detection. Specifically, we represent each class concept by a semantic embedding learned from a large corpus of text. The detector is trained to project the image representations of objects into this embedding space. We also identify the problems of trivially using the raw embeddings with a heuristic knowledge graph and propose to augment the embeddings with a dynamic relation graph. As a result, our few-shot detector, termed SRR-FSD, is robust and stable to the variation of shots of novel objects. Experiments show that SRR-FSD can achieve competitive results at higher shots, and more importantly, a significantly better performance given both lower explicit and implicit shots. The benchmark protocol with implicit shots removed from the pretrained classification dataset can serve as a more realistic setting for future research.

1. Introduction

Few-shot object detection remains sensitive to scarce novel-class data and to the implicit shots supplied by pretraining. SRR-FSD combines visual information with semantic relation reasoning to achieve more stable performance, especially under limited-shot settings.

  • Motivation: Few-shot detection is difficult because real-world long-tail distributions leave some object classes with very few annotated examples.Simply increasing annotation effort cannot eliminate these scarce cases.
  • Motivation: Performance is sensitive to both explicit novel-class shots and implicit shots encoded by pretrained classification models.Removing novel classes from pretraining can substantially reduce detection performance.
  • Key Insight: Semantic relations between base and novel classes remain available regardless of novel-class visual data, complementing image-based learning.The paper motivates relations such as similarity, interaction, and object-carrying relations between classes.
  • Approach: SRR-FSD projects visual object representations into a word-embedding semantic space and introduces explicit relation reasoning for few-shot detection.The method addresses the vision-language domain gap and heuristic graph construction through learned relation augmentation.
  • Results: SRR-FSD achieves competitive higher-shot performance and significantly better lower-shot performance than prior few-shot detectors.The authors report stable performance across shot variation, with strong gains when novel-class data is extremely limited.
  • Results: Removing implicit novel-class shots from pretraining provides a more realistic benchmark, where SRR-FSD maintains performance more steadily than previous methods.The proposed protocol is intended as a benchmark setting for future research.

2. Related Work

Related work spans conventional and few-shot object detection, while semantic embeddings and knowledge graphs have mainly been explored in zero-shot or recognition settings. SRR-FSD extends these semantic-reasoning ideas to few-shot object detection with explicit relation reasoning.

  • Object Detection: Object detectors are commonly categorized as single-stage or multi-stage systems based on how they predict classes and bounding boxes.Examples include YOLO, SSD, RetinaNet, and FreeAnchor for single-stage detection, and Faster R-CNN for multi-stage detection.
  • Few-Shot Detection: Few-shot detection methods address limited data using knowledge transfer, sample selection, metric learning, or class-conditioned feature reweighting.Representative methods include LSTD, RepMet, FSRW, and Meta R-CNN.
  • Semantic Reasoning in Vision Tasks: Semantic embeddings map visual features toward semantic spaces in zero-shot recognition and object detection, and can guide feature augmentation or synthesis.These approaches use embeddings to transfer information toward unseen classes.
  • Semantic Reasoning in Vision Tasks: Knowledge graphs are often combined with word embeddings for relation reasoning, but are typically constructed heuristically from common-sense or co-occurrence information.The paper identifies this prior use as related to, but not equivalent to, its dynamic relation reasoning for FSOD.
  • Semantic Reasoning in Vision Tasks: Semantic embeddings and knowledge graphs remain rarely explored in few-shot object detection, with ASD identified as the only cited prior FSOD method using word embeddings.The paper distinguishes its approach by considering explicit relation reasoning.

3. Semantic Relation Reasoning Few-Shot Detector

SRR-FSD integrates semantic embeddings, visual features, and learned relation reasoning into a two-phase few-shot detector. It addresses domain-gap and heuristic-graph issues by augmenting semantic embeddings with a data-driven dynamic relation graph.

  • FSOD Preliminaries: SRR-FSD builds on Faster R-CNN and uses two training phases: base training followed by balanced fine-tuning on base and novel data.Only selected late layers are fine-tuned in the second phase, including the relation reasoning module and projection matrix for classification.
  • Semantic Space Projection: Class concepts are represented by fixed word embeddings, while a learnable projection maps visual features into the semantic space.Novel classes can be added by expanding the embedding matrix, without introducing new parameters in the projection matrix.
  • Semantic Space Projection: Raw semantic projection can suffer from a vision-language domain gap, because embeddings may guide learning with few images but become misleading when more visual data are available.The method therefore augments semantic embeddings rather than relying on them directly.
  • Relation Reasoning: The dynamic relation graph uses self-attention over transformed word embeddings and propagates knowledge among classes through graph convolution.The learned graph is driven by image inputs and also helps reduce the domain gap between visual and language representations.
  • Relation Reasoning: A predefined knowledge graph is problematic for FSOD because its classes are not strongly hierarchical, and object co-occurrence does not necessarily represent semantic relation.These limitations motivate learning the relation structure from data instead of defining it heuristically.

4. Experiments

Experiments evaluate SRR-FSD across standard VOC and COCO settings, cross-domain transfer, implicit-shot removal, and component ablations. The results show shot-stable performance, strong low-shot gains, preserved base-class accuracy, and benefits from dynamic relation reasoning.

  • Existing Settings: SRR-FSD is evaluated on VOC and COCO using established FSOD protocols and fixed data splits, with VOC testing 1-, 2-, 3-, 5-, and 10-shot settings.VOC uses three base/novel splits; COCO evaluates 20 novel classes with 10 or 30 shots per class.
  • Existing Settings: Up to 10+ mAP separates SRR-FSD from the second-best method below 5 shots on VOC, while performance remains competitive at 5- and 10-shot.Against ASD on VOC Novel Set 1, SRR-FSD is 24.2 mAP better at 3-shot and 6.0 mAP better at 5-shot.
  • Existing Settings: Base AP does not drop after novel-class adaptation, while novel AP and base AP compare favorably with prior methods using the same Faster R-CNN and ResNet-101.The authors report that base AP even increases slightly, possibly because semantic relation reasoning transfers information from novel to base objects.
  • Cross-Domain Evaluation: 44.5 mAP is achieved by SRR-FSD in 10-shot COCO-to-VOC cross-domain detection, indicating strong cross-domain generalization.The cross-domain setting uses 60 COCO base classes and 20 VOC novel classes.
  • Realistic Setting: After novel classes are removed from the pretrained classification dataset, SRR-FSD remains more stable and outperforms previous methods especially in lower-shot conditions.This protocol enforces C0 ∩ Cn = ∅ to eliminate implicit shots from pretraining.
  • Ablation Study: Semantic space projection improves 1-shot and 2-shot performance by 7.9 mAP and 2.4 mAP, respectively, but raw embeddings fall below baseline at 5 and 10 shots.The ablation attributes low-shot gains to language guidance and higher-shot degradation to the vision-language domain gap.
  • Ablation Study: Dynamic relation reasoning improves novel-object detection across shot conditions and outperforms trainable transformation and heuristic knowledge-graph alternatives.Its data-driven relation graph updates with image inputs, adapting raw word embeddings toward the vision domain.
  • Ablation Study: Decoupled fine-tuning reduces novel-class false positives by allowing classification and localization subnets to learn independently.Figure 6 reports that most false positives arise from confusion with similar categories.

5. Conclusion

The paper introduces semantic relation reasoning for few-shot object detection by combining semantic relations with visual information. SRR-FSD achieves robust performance across shot levels and under a realistic protocol that removes novel concepts from pretraining, while its core components can transfer to other few-shot detectors.

  • Conclusion: SRR-FSD explicitly integrates semantic relations between base and novel classes with visual information to improve learning when novel-class data is extremely limited.The method is applied to a standard two-stage Faster R-CNN detector.
  • Conclusion: SRR-FSD demonstrates robust few-shot performance against variation in shot numbers and achieves state-of-the-art results on several few-shot detection settings.The conclusion also reports performance under a realistic setting where novel concepts are eliminated from the pretrained backbone.
  • Conclusion: Semantic space projection and relation reasoning can be applied directly to the classification subnet of other few-shot detectors.

A. Removing Novel Classes from ImageNet

The paper proposes removing novel classes from ImageNet pretraining to eliminate implicit shots and better reflect real-world scarcity. For COCO, it recommends selecting tail-frequency classes instead because its novel classes are common.

  • A. Removing Novel Classes from ImageNet: Novel classes are removed from the classification dataset used to initialize the detector backbone, preventing their concepts from being encoded before few-shot training.The motivation is that pretraining on data-scarce novel classes is unrealistic.
  • A. Removing Novel Classes from ImageNet: ImageNet classes are removed hierarchically by synset and all corresponding hyponyms, so one novel class may cover multiple ImageNet classes.The procedure uses the ImageNet API to identify each synset's full subtree.
  • A. Removing Novel Classes from ImageNet: For VOC, the protocol specifies WordNet IDs for each novel class, including aeroplane, bird, boat, bottle, bus, cat, cow, horse, motorbike, and sheep.These IDs define which ImageNet categories are removed for the VOC benchmark.
  • A. Removing Novel Classes from ImageNet: For COCO, the paper recommends choosing data-scarce tail classes as novel classes because its candidate novel classes are common in the real world.Removing common classes from ImageNet is considered less meaningful than removing scarce classes.

B. Visualization of Relation Reasoning

The relation reasoning module slightly increases correlations between novel and base-class semantic embeddings. The visualization highlights stronger associations for some novel classes, including sofa with bottle and table.

  • B. Visualization of Relation Reasoning: Nearly all novel–base semantic correlations increase slightly after relation reasoning, indicating improved knowledge propagation between the two class groups.Figure 7 compares correlation maps before reasoning, after reasoning, and their difference.
  • B. Visualization of Relation Reasoning: Some novel classes become more correlated than others, including sofa with bottle and sofa with table.The paper attributes these stronger associations as probably reflecting frequent co-occurrence contexts.

C. Using Other Word Embeddings

The semantic space projection can use different language-model word embeddings. In the reported comparison, Word2Vec provides better representations than GloVe.

  • C. Using Other Word Embeddings: Word2Vec and GloVe are evaluated as alternative word-embedding representations for the semantic space projection.The embeddings provide vector representations for class concepts.
  • C. Using Other Word Embeddings: Word2Vec provides better representations than GloVe in VOC Novel Set 1 FSOD experiments.Table 7 reports mAP50 comparisons using ResNet-50 with 300d and 200d embeddings.

D. Reduced Dimension in Relation Reasoning

The relation reasoning module reduces word-embedding dimensions before computing attention to save computation. Experiments select dimension 32, while broader finetuning does not guarantee better few-shot performance.

  • D. Reduced Dimension in Relation Reasoning: Word-embedding dimensions are reduced by linear layers before attention-map computation, saving computational time.The module uses dimensionality reduction as part of relation reasoning.
  • D. Reduced Dimension in Relation Reasoning: Table 7 compares Word2Vec and GloVe embeddings using mAP50 on VOC Novel Set 1 with ResNet-50.The table varies embedding dimensionality between 300d and 200d.
  • D. Reduced Dimension in Relation Reasoning: Dimension 32 achieves the best reported performance, while other tested dimensions are only slightly worse.Table 8 evaluates reduced feature dimensions on VOC under the main experimental setting.
  • D. Reduced Dimension in Relation Reasoning: Finetuning more parameters does not guarantee better few-shot detection performance compared with TFA's last-layer finetuning baseline.The comparison progressively unfreezes fully connected, RPN, and backbone layers.

E. Finetuning More Parameters

The detector includes a finetuning stage for generalization to novel classes and updates selected classification-subnet components while freezing earlier layers. The passage also raises, but does not complete, a comparison about whether this parameter choice explains the improvement.

  • Finetuning is used to make the detector generalized to novel classes.
  • In the classification subnet, the relation reasoning module and projection matrix are finetuned while parameters in previous layers remain frozen.
  • The authors address whether finetuning more parameters than Faster R-CNN explains SRR-FSD’s improvement over the baseline.The supplied passage ends before reporting the result of this analysis.

F. Complete Results on VOC

The VOC results compare SRR-FSD with prior few-shot detectors, including MPSR, whose visual-only direction is complementary to semantic relation reasoning. The reported comparison emphasizes performance under extremely low-shot conditions.

  • MPSR uses an auxiliary branch to generate multi-scale positive samples as object pyramids and refine predictions at various scales.
  • Table 10 reports AP50 for each novel class on the few-shot VOC dataset.The table compares SRR-FSD with results presented for FSRW, Meta R-CNN, and MPSR.
  • SRR-FSD combines visual information with semantic relation reasoning, a direction described as orthogonal and complementary to MPSR’s visual-only approach.
  • SRR-FSD can achieve superior performance under extremely low-shot conditions such as 1- and 2-shot settings.

G. Interpretation of the Dynamic Relation Graph

The relation reasoning module learns a data-driven dynamic relation graph using self-attention rather than a predefined fixed knowledge graph. Its computation propagates knowledge among word embeddings while also applying independent learnable transformations.

  • The module learns a dynamic relation graph driven by data, unlike predefined fixed knowledge graphs.The graph is implemented with a self-attention architecture and can be interpreted as a knowledge-graph-related computation.
  • The augmented word embeddings are used as weights for computing classification scores, with δ applying softmax along the input matrix’s last dimension.
  • The term δ(WeTfTTg) is interpreted as an N × N dynamic knowledge graph with learnable parameters Tf and Tg.
  • The graph-convolution operation connects the N word embeddings in We to enable knowledge propagation among them.
  • The ThTl component applies a learnable transformation independently to each embedding.
Loading 2103.01903v2…