Source-linked AI summary
One-Shot Object Detection with Co-Attention and Co-Excitation
Ting-I Hsieh, Yi-Chen Lo, Hwann-Tzong Chen, Tyng-Luh Liu
TL;DR
One-shot object detection asks a model to find every instance of an unseen class in a target image using one query patch. CoAE addresses this with query-conditioned proposals, adaptive channel reweighting, and margin-based proposal ranking, achieving state-of-the-art performance on VOC and MS-COCO. The setup excludes COCO- and VOC-related classes from ImageNet pretraining.
Problem
One-shot object detection must localize instances of a class absent from training using a query image patch, while learned metrics still require suitable candidate regions.
Method
CoAE uses non-local query-target co-attention for region proposals, squeeze-and-co-excitation for channel reweighting, and margin-based ranking in a two-stage detector.
Results
The model achieves better performance than baseline methods on seen and unseen VOC classes and than Siamese Mask-RCNN on seen and unseen COCO classes.
Takeaways & Limitations
CoAE's query-target evidence supports non-local proposals and shared-feature emphasis, yielding state-of-the-art performance on two popular datasets.
Takeaways & Limitations
The unseen-class protocol depends on reduced ImageNet pretraining that removes COCO-related classes and thereby excludes VOC classes as well.
Abstract
from arXiv · showhide
This paper aims to tackle the challenging problem of one-shot object detection. Given a query image patch whose class label is not included in the training data, the goal of the task is to detect all instances of the same class in a target image. To this end, we develop a novel {\em co-attention and co-excitation} (CoAE) framework that makes contributions in three key technical aspects. First, we propose to use the non-local operation to explore the co-attention embodied in each query-target pair and yield region proposals accounting for the one-shot situation. Second, we formulate a squeeze-and-co-excitation scheme that can adaptively emphasize correlated feature channels to help uncover relevant proposals and eventually the target objects. Third, we design a margin-based ranking loss for implicitly learning a metric to predict the similarity of a region proposal to the underlying query, no matter its class label is seen or unseen in training. The resulting model is therefore a two-stage detector that yields a strong baseline on both VOC and MS-COCO under one-shot setting of detecting objects from both seen and never-seen classes. Codes are available at https://github.com/timy90022/One-Shot-Object-Detection.
1 Introduction
One-shot object detection seeks to localize instances of an unseen object class in a target image from a single query exemplar. The paper addresses the proposal-selection and similarity-learning challenges by using query-target co-attention, co-excitation, and metric learning.
- Human-like performance requires grouping object pixels, extracting distinctive comparison cues, and focusing attention for localization.
- One-shot detection localizes all likely occurrences of an unseen query object in a new target image despite appearance and viewing variations.
- A learned metric alone is insufficient because the detector must first identify target regions suitable for comparison with the query.
- Non-local feature maps provide query-conditioned region proposals, while squeeze-and-co-excitation emphasizes correlated features in query and target images.
2 Related work
Prior work covers two-stage proposal-based detection and metric learning for unseen-class classification, with few-shot object detection remaining relatively underexplored. This paper formulates unseen-class detection as class-agnostic and training-free after initial learning.
- Object detection: Two-stage detectors first generate region proposals and then classify and refine their locations, unlike one-stage proposal-free pipelines.
- Few-shot classification via metric learning: Metric-learning approaches support inference on unseen classes from labeled examples, including the one-shot case with K = 1.
- Few-shot object detection: Few-shot object detection is newer and less explored, with preliminary approaches based on transfer learning, meta learning, and metric learning.
- Positioning of this work: The proposed formulation detects unseen novel classes without knowing their classes beforehand or fine-tuning after training.
3 Our method
The method is a two-stage Faster R-CNN-based detector for one-shot detection, designed around query-conditioned proposals, adaptive feature reweighting, and proposal ranking. Its training combines standard detection losses with a margin-based ranking objective.
- Architecture and problem formulation: The detector partitions classes into training classes C0 and inference-only classes C1, then uses a two-stage Faster R-CNN-based architecture.
- Architecture and problem formulation: Given a query patch from C1, the task is to uncover all corresponding instances in a target image assumed to contain at least one such instance.
- Non-local object proposals: Non-local operation combines target and query feature maps so the RPN can generate proposals incorporating query-target co-attention.
- Squeeze and co-excitation: Squeeze-and-co-excitation adaptively reweights the shared N feature channels of query and target maps to match the query with candidate proposals.
- Proposal ranking: A two-layer MLP ranks K = 128 proposals using concatenated proposal and query features, with foreground labels assigned when IoU exceeds 0.5.
- Proposal ranking: The margin-based loss separates foreground and background scores using m+ = 0.7 and m− = 0.3, alongside Faster R-CNN cross-entropy and regression losses.
4 Experiments
Experiments evaluate the model on VOC and COCO under seen/unseen-class settings, using controlled splits, baselines, and ablations. The reported results show stronger performance than comparison methods on both seen and unseen classes, while component studies assess the proposed modules.
- Datasets and setup: The experiments use VOC and COCO benchmarks, with COCO alternating one of four class splits as unseen and the other three as seen during training.VOC uses established train/test splits, while COCO evaluates on val 2017 after alternating seen and unseen class partitions.
- Baselines: The study compares against SiamFC, SiamRPN, CompNet, and Siamese Mask-RCNN across the evaluated datasets.The baselines include tracking-oriented and metric-based few-shot detection methods.
- Ablation and analysis: The ablation study evaluates co-attention through non-local proposals, co-excitation through SCE, and the margin-based ranking loss.These experiments isolate the proposed components rather than evaluating only the complete detector.
- Overall performance: Our model outperforms baseline methods on both seen and unseen VOC classes and exceeds Siamese Mask-RCNN on both seen and unseen COCO classes.On VOC, performance further improves when using the all-1,000-class ImageNet backbone; artifact classes are reported as hardest on COCO.
5 Ablation studies
The ablations show that co-attention and co-excitation are both important, while co-excitation weights adapt to query-target similarities in color, texture, and shape.
- Ablation results: Adding either non-local RPN or SCE significantly improves performance, while combining both modules provides additional gains on VOC and COCO.Either module increases mAP by 4.4/6.3 and AP50 by 8.2/9.8; both together add 1.8/0.9 mAP and 1.9/0.3 AP50.
- Ablation results: The margin-based ranking loss moderately enhances performance, supporting its usefulness for learning the desired similarity metric.
- Non-local proposals: Non-local features enable the RPN to focus proposals on regions jointly relevant to the query and target, producing a co-attention effect.
- Co-excitation analysis: Co-excitation learns class-specific weight distributions, with animal-related classes closer together and the person class distant from other classes.The distances reflect shared attributes in texture or shape across classes.
- Co-excitation analysis: For different queries of one target, co-excitation emphasizes color or shape channels according to query-target similarity.Queries p1 and p2 share target color, whereas p3 and p4 differ in color and therefore emphasize shape-related features.
- Co-excitation analysis: For one query across different targets, co-excitation distances indicate texture emphasis for I1 and I2 and shape emphasis for I3 and I4.
6 Conclusion
The CoAE detector uses query-target evidence rather than relying solely on training labels, enabling non-local proposals and feature emphasis for one-shot detection. It achieves state-of-the-art performance on VOC and MS-COCO, with future work extending the method to k-shot detection.
- Conclusion: CoAE uses correlated evidence from query-target pairs instead of relying solely on training labels.
- Conclusion: The method generates non-local object proposals and emphasizes features shared by query and target images.
- Conclusion: The resulting one-shot detector achieves state-of-the-art performance on two popular datasets.
- Conclusion: Future work will generalize the method to k-shot object detection for k ≥ 0.