Source-linked AI summary
Few-shot Adaptive Faster R-CNN
Tao Wang, Xiaopeng Zhang, Li Yuan, Jiashi Feng
TL;DR
The paper addresses object-detector degradation under domain shift when only a few loosely annotated target images are available. FAFRCNN combines feature pairing, bi-level adaptation, and source-model feature regularization, and reports strong results across FDA and UDA settings. Its practical advantages include fast adaptation, reduced data-collection cost, and more stable training.
Problem
Object detection adapts poorly to new domains when target data is scarce, while detection requires both localization and classification and limited-data adaptation risks over-adaptation and instability.
Method
FAFRCNN pairs source and target features, aligns image-level patches and instance-level object features, and applies source model feature regularization during adaptation.
Results
FAFRCNN surpasses compared methods across few-shot scenarios, outperforms a state-of-the-art method using full target data, and achieves new state-of-the-art UDA results.
Takeaways & Limitations
A source-trained Faster R-CNN can be robustly adapted with very few loosely annotated target samples while retaining strong performance across evaluated domain-shift scenarios.
Abstract
from arXiv · showhide
To mitigate the detection performance drop caused by domain shift, we aim to develop a novel few-shot adaptation approach that requires only a few target domain images with limited bounding box annotations. To this end, we first observe several significant challenges. First, the target domain data is highly insufficient, making most existing domain adaptation methods ineffective. Second, object detection involves simultaneous localization and classification, further complicating the model adaptation process. Third, the model suffers from over-adaptation (similar to overfitting when training with a few data example) and instability risk that may lead to degraded detection performance in the target domain. To address these challenges, we first introduce a pairing mechanism over source and target features to alleviate the issue of insufficient target domain samples. We then propose a bi-level module to adapt the source trained detector to the target domain: 1) the split pooling based image level adaptation module uniformly extracts and aligns paired local patch features over locations, with different scale and aspect ratio; 2) the instance level adaptation module semantically aligns paired object features while avoids inter-class confusion. Meanwhile, a source model feature regularization (SMFR) is applied to stabilize the adaptation process of the two modules. Combining these contributions gives a novel few-shot adaptive Faster-RCNN framework, termed FAFRCNN, which effectively adapts to target domain with a few labeled samples. Experiments with multiple datasets show that our model achieves new state-of-the-art performance under both the interested few-shot domain adaptation(FDA) and unsupervised domain adaptation(UDA) setting.
1. Introduction
The paper targets object detection under domain shift when only a few loosely annotated target images are available. FAFRCNN combines feature pairing, two-level adaptation, and source-model regularization, achieving strong FDA and UDA results with fast, low-cost, stable adaptation.
- Motivation: Domain shift causes substantial detection degradation in unseen environments, while existing adaptation methods often require abundant target data and are mainly designed for classification.Object detection additionally requires localizing and classifying individual objects over high-resolution inputs.
- Proposed framework: The framework adapts source-trained detectors using a feature-pairing mechanism, image-level patch alignment, instance-level object alignment, and source model feature regularization.Pairing augments limited target data; the two adaptation modules address global and instance-level shifts, while SMFR stabilizes training and limits over-adaptation.
- Practical benefits: Only hundreds of adaptation updates are empirically needed to reach desirable performance, compared with tens of thousands for prior UDA methods.The paper presents this as a fast-adaptation benefit of the proposed framework.
- Practical benefits: FAFRCNN reduces target-data collection and annotation costs by using only a few representative, loosely annotated samples.The paper states that this substantially reduces human annotation time.
- Training stability: SMFR reduces run-to-run variance and provides more stable adaptation when limited-data fine-tuning and adversarial objectives risk overfitting or instability.The regularizer does not significantly improve the two adversarial modules’ performance individually, but it substantially reduces variance across runs.
- Empirical efficacy: FAFRCNN significantly surpasses compared methods across few-shot scenarios and outperforms a state-of-the-art method using full target-domain data, while also achieving new state-of-the-art UDA results.Experiments use Cityscapes, SIM10K, Udacity self-driving, and Foggy Cityscapes.
2. Related Work
Related work spans deep-CNN object detection, cross-domain adaptation, and few-shot learning. The paper distinguishes its FDA setting from few-shot learning because the data distribution changes while the detection task remains the same.
- Object Detection: Modern object detectors include two- or multi-stage architectures such as Faster R-CNN and single-stage architectures such as YOLO, SSD, and RetinaNet, but require substantial carefully annotated training data.
- Cross-domain Object Detection: Prior cross-domain detection work includes DPM mitigation, R-CNN subspace alignment, and iterative domain transfer with pseudolabeling, while much domain-adaptation research targets classification.
- Few-shot Learning: Few-shot learning studies learning new categories or tasks from few examples, whereas this paper’s FDA setting changes the data distribution while keeping the task unchanged.
3. Method
FAFRCNN adapts a source-trained Faster R-CNN to a novel domain using paired source–target features, image- and instance-level adversarial alignment, and source-model feature regularization. The design addresses sparse target data, object-level domain shift, and instability or over-adaptation during few-shot adaptation.
- 3. Method: FAFRCNN combines feature pairing, two-level adversarial adaptation, and source model feature regularization to adapt Faster R-CNN with few target examples.The image-level module aligns local patches, the instance-level module aligns object features, and regularization stabilizes adaptation.
- 3.2. Image-level Adaptation: Split pooling samples local feature patches across locations, scales, and aspect ratios for multi-scale image-level domain alignment.Random grid offsets reduce biased sampling and redundancy, while ROI pooling converts variable-sized regions into fixed-sized features.
- 3. Method: Feature pairing augments limited target data by forming source–source and source–target pairs for local patches and object features.The paired features support adaptation when only a very small target set is available.
- 3.3. Instance-level Adaptation: The instance-level module keeps high-overlap foreground ROIs, pairs source and target object features by class, and uses a multi-way discriminator to avoid inter-class confusion.Foreground ROIs use an IoU threshold of 0.7 in the implementation, and the discriminator has 2 × C outputs.
- 3.4. Source Model Feature Regularization: SMFR constrains adapted features toward source-model responses using an ℓ2 penalty focused on estimated foreground regions rather than noisy background areas.The foreground mask uses anchor locations whose IoU with ground-truth boxes exceeds 0.5 in the implementation.
- 3.5. Training: Training alternates detector optimization using detection, image-level, instance-level, and regularization losses with discriminator optimization using image- and instance-level losses.The full detector minimizes Lg, while domain discriminators minimize Ld.
4. Experiments
Experiments evaluate FAFRCNN across multiple cross-domain detection scenarios under few-shot and unsupervised adaptation settings. The results show strong adaptation gains, with complementary image- and instance-level modules, feature pairing, and SMFR improving effectiveness or stability.
- Quantitative Results: 5.2 AP improvement over the raw source model is obtained in Scenario-1 FDA by combining all proposed components.The combined system also outperforms the state-of-the-art UDA method in this scenario.
- Quantitative Results: 6.4 AP improvement over the source model is observed for the proposed split-pooling adaptation in Scenario-1 UDA.The gain is reported for settings with either few or sufficient target-domain images.
- Quantitative Results: FAFRCNN significantly surpasses compared methods across few-shot scenarios and outperforms a state-of-the-art method using full target-domain data, while also achieving state-of-the-art UDA results.Experiments use multiple datasets and cross-domain scenarios, including SIM10K, Cityscapes, Udacity, and Foggy Cityscapes.
- Ablation Analysis: Removing feature pairing causes a significant performance drop, supporting its role in augmenting discriminator inputs under limited target data.The pairing effect is evaluated for both split-pooling and object-instance adaptation modules.
- Ablation Analysis: Using more target images and annotated boxes improves adaptation, while the roughly linear gain with exponentially increasing image count suggests saturation.The analysis covers S→U, U→C, and C→F settings with varying image and annotation counts.
- Ablation Analysis: SMFR drastically reduces adaptation variance and prevents the severe overfitting observed when fine-tuning directly on limited target data.Across ten random initializations, fine-tuning can underperform the source model, whereas SMFR enables benefit from limited target samples.
5. Conclusion
FAFRCNN adapts a source-trained Faster R-CNN detector to a target domain using very few loosely annotated images, while outperforming state-of-the-art methods using the full unlabeled target set.
- FAFRCNN robustly adapts a source-trained Faster R-CNN model with very few loosely annotated target images.