Source-linked AI summary
Cross-domain Object Detection through Coarse-to-Fine Feature Adaptation
Yangtao Zheng, Di Huang, Songtao Liu, Yunhong Wang
TL;DR
Cross-domain object detection degrades under domain shift, while collecting diverse labeled data is costly. The paper introduces coarse-to-fine adaptation using attention-based foreground alignment and category prototypes, achieving state-of-the-art results across varied scenarios.
Problem
Domain shift reduces detector performance on unseen domains, and addressing it by collecting sufficiently diverse labeled data is expensive and time-consuming.
Method
The framework combines ART, which aligns attention-extracted foreground distributions, with PSA, which aligns global prototypes for matching categories across domains.
Results
The approach achieves state-of-the-art results across various cross-domain detection scenarios and three major benchmarks.
Takeaways & Limitations
Coarse-to-fine feature adaptation transfers domain knowledge in foreground regions for cross-domain object detection.
Takeaways & Limitations
PSA must address false target-domain pseudo-labels and class mismatches, which can impair semantic alignment.
Abstract
from arXiv · showhide
Recent years have witnessed great progress in deep learning based object detection. However, due to the domain shift problem, applying off-the-shelf detectors to an unseen domain leads to significant performance drop. To address such an issue, this paper proposes a novel coarse-to-fine feature adaptation approach to cross-domain object detection. At the coarse-grained stage, different from the rough image-level or instance-level feature alignment used in the literature, foreground regions are extracted by adopting the attention mechanism, and aligned according to their marginal distributions via multi-layer adversarial learning in the common feature space. At the fine-grained stage, we conduct conditional distribution alignment of foregrounds by minimizing the distance of global prototypes with the same category but from different domains. Thanks to this coarse-to-fine feature adaptation, domain knowledge in foreground regions can be effectively transferred. Extensive experiments are carried out in various cross-domain detection scenarios. The results are state-of-the-art, which demonstrate the broad applicability and effectiveness of the proposed approach.
1. Introduction
Cross-domain object detection suffers when detectors trained on one distribution are applied to unseen domains. The paper proposes coarse-to-fine adaptation that first aligns foreground regions and then aligns category-specific foreground prototypes.
- Domain shift causes CNN detectors to lose performance on novel scenes, while collecting sufficiently diverse labeled data is expensive and time-consuming.
- Existing image-level and instance-level adaptation methods align diverse object regions collectively and overlook category-specific distributions.Their fixed region-grouping alternatives can also depend on predefined cluster numbers and region sizes.
- The framework uses Attention-based Region Transfer (ART) to extract foregrounds and align their feature distributions across domains in a class-agnostic coarse stage.ART uses attention and multi-layer adversarial learning to emphasize foreground regions during alignment.
- Prototype-based Semantic Alignment (PSA) performs fine-grained adaptation by building global prototypes for each category across source and target domains.The prototypes are updated adaptively to reduce the effects of false pseudo-labels and class mismatches.
- The approach is presented as a progressive adaptation method for cross-domain two-stage object detection that aligns deep features more accurately.
- Experiments across three major benchmarks and typical adaptation scenarios report state-of-the-art results and evaluate ART and PSA through ablations.
2. Related Work
Cross-domain detection research addresses domain shift through domain-invariant representations, but feature-level methods remain limited by inaccurate alignment. The proposed approach targets this limitation with coarse-to-fine alignment of foreground importance and class distributions.
- Domain adaptation methods reduce cross-domain discrepancy through statistical-distance minimization or adversarial domain classifiers.
- Existing cross-domain detection methods include semi-supervised, pixel-level, and feature-level adaptation approaches.
- The proposed coarse-to-fine scheme improves feature alignment by learning foreground importance with ART and class-wise distribution properties with PSA.
3. Method
The framework adapts a Faster R-CNN detector from a labeled source domain to an unlabeled target domain through coarse foreground-aware alignment and fine category-aware prototype alignment.
- Framework Overview: The framework combines a detection network with Attention-based Region Transfer and Prototype-based Semantic Alignment modules.The adaptive detector is trained with labeled source and unlabeled target data.
- Coarse-grained Adaptation: ART uses RPN-derived attention to emphasize foreground regions during multi-layer adversarial feature alignment.Multiple domain classifiers distinguish source from target features while the backbone learns domain-confusing representations through gradient reversal.
- Coarse-grained Adaptation: The attention map averages RPN activations across channels, filters values below the mean threshold, and is up-sampled for different convolution blocks.A skip connection mitigates errors when foreground regions are mistakenly assigned zero attention.
- Fine-grained Adaptation: PSA aligns source and target foregrounds by minimizing distances between prototypes of the same category across domains.Source prototypes use ground-truth regions, whereas target prototypes use RoI-head pseudo-labels.
- Fine-grained Adaptation: Global prototypes are initialized from the pretrained source detector and updated adaptively from local mini-batch prototypes.This avoids relying on category matches that may be absent from a small mini-batch.
- Network Optimization: The total objective combines detection, ART, and PSA losses using separate trade-off factors for the two adaptation modules.The optimization procedure alternates coarse-grained and fine-grained adaptation before updating the detection model.
4. Experiments
Experiments evaluate the framework across weather, synthetic-to-real, and cross-camera domain shifts, with analyses of foreground feature discrepancy, detection errors, and qualitative attention.
- Experimental Settings: Four datasets support three adaptation scenarios: Normal-to-Foggy, Synthetic-to-Real, and Cross-Camera.The scenarios use Cityscapes, FoggyCityscapes, SIM10k, and KITTI with the stated source-target configurations.
- Implementation Details: The experiments use Faster R-CNN with a VGG16 backbone, two images per batch, and staged SGD training followed by Adam-trained domain classifiers.Images are resized to 600 pixels on the shorter side; training runs for 50k plus 20k iterations.
- Results: 38.6% mAP is achieved on Normal-to-Foggy, the best result among compared methods, with a +17.8% gain over the source-only model.The paper also evaluates the gain relative to each method’s pre-adaptation performance.
- Results: 43.8% performance on Car is reported for Synthetic-to-Real, a +0.8% gain over SCDA.The passage attributes the smaller transfer difficulty to the single shared category and notes PSA remains complementary for foreground focus.
- Results: 41.0% mAP is reached on Cross-Camera, with a +7.6% gain over the non-adaptive model.The method reduces negative transfer for Car detection and outperforms the baseline on the remaining categories.
- Further Analysis: Compared with SWDA, correct detections increase from 39.3% to 43.0% while other highest-confidence error types decrease.The analysis categorizes detections as correct, mislocalized, or background, and qualitative results show more true positives and fewer false positives.
5. Conclusion
The paper concludes that ART and PSA form a coarse-to-fine framework that aligns foreground features across domains at category-agnostic and semantic levels. Experiments across multiple adaptation scenarios report state-of-the-art results.
- Conclusion: ART highlights foreground regions through attention and aligns their feature distributions across domains in a category-agnostic manner.PSA then performs fine-grained semantic adaptation using category-level prototypes.
- Conclusion: The framework reaches state-of-the-art results across various adaptation scenarios, demonstrating its effectiveness for cross-domain object detection.