Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning

Xiaopeng Yan; Ziliang Chen; Anni Xu; Xiaoxi Wang; Xiaodan Liang; Liang Lin

Source-linked AI summary

Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning

Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, Liang Lin

arXiv:1909.13032v2Sep 28, 2019cs.CVcs.LG

TL;DR

Few-shot detection and segmentation are difficult because novel objects are entangled with multiple objects and diverse backgrounds. Meta R-CNN addresses this by remodeling Faster/Mask R-CNN predictor heads with PRN-generated attention over RoI features, achieving state-of-the-art detection and improved segmentation, with reported gains across backbones but lower improvement on challenging MS COCO.

Problem
Few-shot detection and segmentation must predict novel-object classes and structure labels despite scarce examples and information entangled across multiple objects and backgrounds.
Method
Meta R-CNN extends Faster/Mask R-CNN with a Predictor-head Remodeling Network that generates class-attentive vectors and applies channel-wise attention to RoI features.
Results
Meta R-CNN achieves state-of-the-art few-shot novel-class detection and segmentation, while retaining competitive base-class detection across benchmarks and backbones.
Takeaways & Limitations
RoI-based meta-learning provides a flexible framework for adapting Faster/Mask R-CNN to few-shot object detection and segmentation.
Takeaways & Limitations
On MS COCO, improvement is lower than on PASCAL VOC because the dataset contains more occlusion, ambiguities, and small objects.

Abstract

from arXiv · show

Resembling the rapid learning capability of human, few-shot learning empowers vision systems to understand new concepts by training with few samples. Leading approaches derived from meta-learning on images with a single visual object. Obfuscated by a complex background and multiple objects in one image, they are hard to promote the research of few-shot object detection/segmentation. In this work, we present a flexible and general methodology to achieve these tasks. Our work extends Faster /Mask R-CNN by proposing meta-learning over RoI (Region-of-Interest) features instead of a full image feature. This simple spirit disentangles multi-object information merged with the background, without bells and whistles, enabling Faster /Mask R-CNN turn into a meta-learner to achieve the tasks. Specifically, we introduce a Predictor-head Remodeling Network (PRN) that shares its main backbone with Faster /Mask R-CNN. PRN receives images containing few-shot objects with their bounding boxes or masks to infer their class attentive vectors. The vectors take channel-wise soft-attention on RoI features, remodeling those R-CNN predictor heads to detect or segment the objects that are consistent with the classes these vectors represent. In our experiments, Meta R-CNN yields the state of the art in few-shot object detection and improves few-shot object segmentation by Mask R-CNN.

1. Introduction

Few-shot learning addresses scarce novel-class data, but instance-level detection and segmentation remain difficult because objects are entangled with other objects and backgrounds. Meta R-CNN combines Faster/Mask R-CNN with RoI-based meta-learning and reports strong performance across tasks and backbones.

Few-shot learners use 1–10 labeled examples from novel classes, supported by base classes with sufficient annotations.
Instance-level detection and segmentation require predicting novel-class categories alongside bounding boxes or masks in complex scenes.These structure labels require more annotation labor than image-level labels.
Meta R-CNN performs meta-learning over Faster/Mask R-CNN RoI features, where each RoI corresponds to a single object or background and helps disentangle scene information.
The Predictor-head Remodeling Network infers class-attentive vectors from few-shot objects and applies channel-wise attention to RoI features for detection or segmentation.
Meta R-CNN achieves state-of-the-art few-shot novel-class detection and segmentation while retaining competitive base-class detection across three benchmarks and three backbones.The authors also describe it as general, simple, effective, and fast at inference.

2. Related Work

Related work spans few-shot recognition, object detection, and object segmentation. Recognition methods use Bayesian, similarity-learning, or meta-learning approaches, while detection and segmentation follow distinct architectural paradigms.

Few-shot object recognition research is organized around Bayesian approaches, similarity learning, and meta-learning.Bayesian methods model latent organization, whereas similarity learning compares within- and between-class feature similarity.
Object detection methods divide mainly into one-stage and two-stage detectors.R-CNN-family methods use region proposals to classify objects and regress their locations.
Object segmentation methods are categorized as image-based or proposal-based, with proposal-based methods predicting masks from generated regions.The passage states that related few-shot segmentation research remained absent.

3. Tasks and Motivation

Few-shot detection and segmentation must identify novel objects within multi-object images and predict their classes and structure labels. The paper motivates RoI-level meta-learning because RoI features can separate object and background information before prediction.

Few-shot learning separates sufficiently sampled base classes from sparsely sampled novel classes, whose imbalance causes overfitting or inadequate training.Meta-learning addresses this setting by promoting rapid adaptation to novel tasks.
Unlike recognition, instance-level tasks process images containing multiple objects with diverse classes, positions, and shapes.
Few-shot detection and segmentation must identify novel-class objects among other objects and backgrounds, then predict classes, bounding boxes, or masks.
Motivation: The target meta-learning function is object-level rather than image-level because image-level meta-predictors contain complex multi-object information merged with backgrounds.
Motivation: Faster/Mask R-CNN produces RoI features through RPN proposals and RoIAlign, disentangling objects and background for second-stage prediction.
Motivation: Meta R-CNN remodels the predictor head by combining each RoI feature with PRN-derived class-attentive vectors to detect or segment novel-class objects.

4. Meta R-CNN

Meta R-CNN adapts Faster/Mask R-CNN for few-shot detection and segmentation by applying meta-learning to RoI features. Its PRN derives class-attentive vectors from few-shot objects and uses them to remodel the R-CNN predictor heads.

Review the R-CNN family: Faster/Mask R-CNN produces RoI features from region proposals, while its predictor heads classify, localize, and optionally segment the corresponding objects.Mask R-CNN adds a parallel mask branch to the predictor head.
Predictor-head Remodeling Network (PRN): PRN remodels the standard predictor head into a meta-predictor conditioned on a few-shot meta-set Dmeta.The meta-set contains few-shot objects from base and novel classes with bounding boxes or masks.
Predictor-head Remodeling Network (PRN): The class-attentive vectors apply channel-wise soft attention to every RoI feature, selecting features relevant to the represented classes.The attended RoI features are fed into the original predictor heads for detection or segmentation.
Predictor-head Remodeling Network (PRN): The method supports end-to-end joint training and can be applied flexibly across Faster/Mask R-CNN backbones.The channel-wise multiplication combines RoI features with the vectors inferred by PRN.
Predictor-head Remodeling Network (PRN): PRN shares most backbone parameters with Faster/Mask R-CNN and infers class-attentive vectors from few-shot objects.Each meta-set object is represented by a four-channel RGB image and foreground structure label.
Predictor-head Remodeling Network (PRN): Meta R-CNN generates class-specific detection outcomes for each RoI, selects the highest-confidence class, and discards proposals below the objectness threshold.The selected attended feature branch is used to locate or segment the object.

5. Implementation

Meta R-CNN is trained with class-consistent meta-learning batches and uses PRN alongside the Faster/Mask R-CNN losses. Its inference design preserves nearly the original detector speed while adapting to base and novel classes.

Mini-batch construction: Each training mini-batch combines m classes, a K-shot meta-set, and a matching m-class training set.The meta-set and training image use consistent class identities, with resized structure-labeled inputs for the meta-set.
Channel-wise soft-attention layer: The channel-wise soft-attention layer pools backbone features and applies a sigmoid to produce 2048×1 attentive vectors.These vectors maintain the spatially aligned feature size used by the RoI representations.
Meta-loss: The auxiliary meta-loss encourages attentive vectors from different classes to produce diverse feature-selection effects.It uses cross-entropy to associate each object attentive vector with its object class.
RoI meta-learning: Meta-training constructs meta-sets and training sets from base classes, while meta-testing considers both base and novel classes.Novel objects are ignored during meta-training when an image contains both base and novel objects.
Inference: During testing, Meta R-CNN sets Cmeta to Cbase ∪ Cnovel and uses PRN together with the Faster/Mask R-CNN module.The design is intended to preserve the fast inference of Faster/Mask R-CNN.
Inference: 83.0 ms/im is the Faster R-CNN inference speed at 3 shots, compared with 84.2 ms/im for Meta R-CNN; at 10 shots, Meta R-CNN runs at 85.4 ms/im.These measurements use a single GTX TITAN XP.

6. Experiments

Experiments across detection and segmentation benchmarks evaluate Meta R-CNN against Faster R-CNN variants and other few-shot baselines. The results show improved novel-class performance, competitive base-class detection, backbone flexibility, and benefits from RoI-level meta-learning.

Few-shot object detection: Meta R-CNN consistently outperforms three Faster R-CNN baselines across novel-class splits and shot counts on PASCAL VOC.The evaluation uses K = 1, 2, 3, 5, and 10 shots across three novel/base-class splits.
Few-shot object detection: Meta R-CNN exceeds YOLO-Few-shot in most cases, with gains of +11.8% in 5-shot split 1 and +6.8 in 10-shot split 3.The paper notes that 5/10-shot settings are more robust than extremely few-shot comparisons.
Few-shot object detection: 72.7 mAP in base classes versus 4.3 in novel classes exposes FRCN+joint’s severe novel-class degradation at K=3.The paper attributes this contrast to extreme object-quantity imbalance in the few-shot setup.
Backbone ablation: Meta R-CNN outperforms FRCN+ft-full on novel classes with both ResNet-34 and ResNet-101 backbones, reaching 35.0% versus 32.8% and 51.5% versus 45.6%.These results support deployment across different Faster R-CNN backbones.
RoI meta-learning: RoI meta-learning outperforms full-image meta-learning, while meta-learning improves Meta R-CNN performance over the non-meta variant.The full-image variant suffers a heavy performance drop and performs worse than Faster R-CNN without the meta-strategy.
Few-shot object segmentation: Meta R-CNN improves Mask R-CNN for few-shot instance segmentation across 5-, 10-, and 20-shot settings on MS COCO.The reported comparison uses MRCN+ft+full as the baseline and finds significant margins.

7. Discussion and Future Work

Meta R-CNN provides a flexible, lightweight framework for few-shot detection and segmentation, with potential extension to broader R-CNN-based visual structure tasks.

7. Discussion and Future Work: Meta R-CNN generalizes Faster/Mask R-CNN to few-shot detection and segmentation while remaining lightweight and plug-and-play.It remodels predictor branches into a meta-learner and is available across diverse Faster/Mask R-CNN backbones.
7. Discussion and Future Work: The framework may extend to few-shot relationship detection and other challenging visual structure prediction tasks.The paper identifies these as potential extensions beyond detection and segmentation.
7. Discussion and Future Work: Few-shot instance-level learning could support generalization to newly emerged concepts without heavily consuming labor-intensive annotations.This motivation applies to visual tasks requiring object structures such as bounding boxes or segmentation masks.

8. Appendix

The appendix analyzes Meta R-CNN’s optimization behavior, attentive-vector representations, benchmark settings, segmentation resolution, and predictor-head design choices. These analyses connect faster adaptation and vector diversity with performance across detection and segmentation experiments.

8.1. Optimization analysis: Meta R-CNN upper-bounds FRCN+ft-full across training iterations, indicating faster performance improvement during adaptation.The comparison is framed through stochastic optimization and task adaptation acceleration.
8.1. Optimization analysis: Normalized mAP is computed from class-specific AP normalized by each class’s converged AP over training iterations.The figure reports mean and variance of this normalized metric.
8.2. Attentive vector analysis: With meta-loss, object attentive vectors cluster by class and separate from other classes, producing more deterministic class-attentive vectors.The resulting lower inter-class variance supports more stable class-attentive vectors when different objects induce them.
8.2. Attentive vector analysis: Without meta-loss, object attentive vectors become more diverse with very large inter-class variance, causing unstable vectors and potentially ambiguous class-specific predictions.Changing the input objects can substantially change the inferred class-attentive vectors, while nearby vectors can make detections ambiguous.
8.2. Attentive vector analysis: Classes with similar semantics, such as Car, Bus, and Train, appear closer in attentive-vector space, suggesting base classes can aid related novel classes.The appendix links this semantic proximity to novel-class object detection through shared information.
8.3. Experimental settings: The appendix evaluates detection and segmentation across specified VOC and COCO novel/base-class splits, with VOC tables reporting AP and mAP for varying novel-class shots.The cited captions identify VOC2007 evaluation under ResNet-101 and COCO evaluation under Mask R-CNN with ResNet-50 for 5/10/20-shot settings.
8.4. Few-shot object segmentation: Increasing reference-image resolution can improve segmentation on data-starved categories, but all baselines use 224x224 images for fair comparison.The resolution observation concerns the reference set, while training and evaluation images remain standardized across baselines.
8.5. Predictor-head ablation: Channel-wise attention performs better on novel detection, whereas concatenation performs better on base detection among the tested PRN variants.The appendix compares concatenate, plus, unshared, and limited-meta-set designs.