Source-linked AI summary
Exploring Categorical Regularization for Domain Adaptive Object Detection
Chang-Dong Xu, Xing-Ran Zhao, Xin Jin, Xiu-Shen Wei
TL;DR
Domain adaptive object detection must bridge substantial source–target gaps, while existing alignment can overlook crucial regions and important instances. The paper introduces categorical regularization with image-level classification and image–instance consistency to target those elements. Across varied domain-shift scenarios, it reports significant gains over existing Domain Adaptive Faster R-CNN detectors and state-of-the-art benchmark results.
Problem
Domain adaptive detectors face substantial source–target gaps, while existing alignment overlooks crucial image regions and important instances relevant to detection.
Method
The framework combines image-level categorical regularization from a multi-label classifier with categorical consistency regularization between image-level and instance-level predictions.
Results
The framework significantly boosts existing Domain Adaptive Faster R-CNN detectors across varied domain shifts and produces state-of-the-art results on benchmark datasets.
Takeaways & Limitations
The plug-and-play framework focuses alignment on object-related regions and hard aligned instances without requiring further annotations or detector-specific alignment algorithms.
Takeaways & Limitations
CCR weights only foreground target instances using detection-head predictions, while source and target-background instance weights remain unchanged.
Abstract
from arXiv · showhide
In this paper, we tackle the domain adaptive object detection problem, where the main challenge lies in significant domain gaps between source and target domains. Previous work seeks to plainly align image-level and instance-level shifts to eventually minimize the domain discrepancy. However, they still overlook to match crucial image regions and important instances across domains, which will strongly affect domain shift mitigation. In this work, we propose a simple but effective categorical regularization framework for alleviating this issue. It can be applied as a plug-and-play component on a series of Domain Adaptive Faster R-CNN methods which are prominent for dealing with domain adaptive detection. Specifically, by integrating an image-level multi-label classifier upon the detection backbone, we can obtain the sparse but crucial image regions corresponding to categorical information, thanks to the weakly localization ability of the classification manner. Meanwhile, at the instance level, we leverage the categorical consistency between image-level predictions (by the classifier) and instance-level predictions (by the detection head) as a regularization factor to automatically hunt for the hard aligned instances of target domains. Extensive experiments of various domain shift scenarios show that our method obtains a significant performance gain over original Domain Adaptive Faster R-CNN detectors. Furthermore, qualitative visualization and analyses can demonstrate the ability of our method for attending on the key regions/instances targeting on domain adaptation. Our code is open-source and available at \url{https://github.com/Megvii-Nanjing/CR-DA-DET}.
1. Introduction
Domain adaptive detection must handle large source–target gaps while avoiding alignment of irrelevant backgrounds and low-value proposals. The proposed categorical regularization framework addresses this with image-level and instance-level regularization that focuses alignment on object-related regions and hard instances.
- Domain gaps in object appearance, background, and weather limit detector generalization, while obtaining new bounding-box annotations is costly.
- DA Faster R-CNN aligns image and instance distributions, but plain image-level alignment can include non-transferable backgrounds and instance alignment can miss hard instances among low-value proposals.
- The framework adds image-level categorical regularization and categorical consistency regularization to the Domain Adaptive Faster R-CNN series.
- Image-level categorical regularization attaches a multi-label classifier to the detection backbone to identify crucial regions using classification CNNs’ weak localization ability.
- Categorical consistency regularization compares image-level and instance-level predictions to automatically identify hard aligned target-domain instances.
- The framework is plug-and-play, requires no further annotations or hyperparameters for vanilla detectors, and significantly boosts existing detectors across varied domain shifts.
2. Preliminaries and Related Work
Domain adaptive Faster R-CNN methods extend two-stage detection with adversarial alignment of image- and instance-level distributions, but can align backgrounds and miss crucial regions or informative instances. Classification CNNs provide weak localization of category-related objects, motivating region-aware adaptation.
- CNN-based Object Detection: Faster R-CNN is a two-stage detector whose Region Proposal Network shares the feature-extraction backbone with the detection head.The two-stage pipeline first generates region proposals and then classifies regions of interest.
- Domain Adaptive Faster R-CNN Series: DA Faster R-CNN aligns image and instance distributions using adversarial domain classifiers, while enforcing image–instance domain consistency for RPN robustness.Image-level classifiers operate on feature-map activations; instance-level classifiers operate on RoI features.
- Domain Adaptive Faster R-CNN Series: The DA Faster R-CNN objective combines detection, image-alignment, instance-alignment, and consistency losses, with λ balancing detection and adaptation components.Adversarial adaptation uses a gradient reversal layer that flips gradients entering the base convolutional layers.
- Domain Adaptive Faster R-CNN Series: Plain image-level alignment can fit non-transferable source backgrounds and fail to align crucial regions or important instances, motivating weak or region-level alignment variants.Recent methods replace plain image-level alignment with weak or region-level alignment models.
- Weakly Localization by Classification CNNs: Multi-label classification CNNs weakly localize objects associated with image-level categories, as illustrated by VGG-16 trained on Cityscapes.Cityscapes heatmaps localize category-related objects such as cars, persons, and riders.
3. Approach
The proposed categorical regularization framework adds image-level and instance-level categorical signals to Domain Adaptive Faster R-CNN detectors. ICR focuses alignment on category-related regions, while CCR weights target foreground instances according to categorical disagreement.
- Overview: The framework uses image-level categorical regularization (ICR) and categorical consistency regularization (CCR) to improve alignment of crucial regions and important instances across domains.The two modules are designed as flexible additions to Domain Adaptive Faster R-CNN series detectors.
- Image-Level Categorical Regularization: ICR attaches a source-supervised multi-label classifier to the detection backbone, using image-level category vectors collected from instance labels without additional annotation cost.The classifier uses global average pooling followed by a 1×1 convolution.
- Image-Level Categorical Regularization: ICR encourages category-specific object features and reduces background fitting because its training provides no background supervision.These features support alignment of crucial regions in both domains through an image-level alignment model.
- Categorical Consistency Regularization: CCR measures categorical consistency between image-level and instance-level predictions to identify hard target instances for instance-level alignment.A disagreement between image-level absence and instance-level presence of a category indicates a hard, informative target sample.
- Categorical Consistency Regularization: CCR weights only target foreground instances according to detection-head predictions; source and target-background instance weights remain unchanged at d_j = 1.The weighted instance-level adversarial loss is implemented by weighting gradients passed through the gradient reversal layer.
- Integration with DA Faster R-CNN Series: The framework integrates with DA-Faster and SW-Faster, adds no additional hyperparameters to DA-Faster, and remains independent of the specific alignment algorithms.For SW-Faster, the framework adds a DA-Faster-style instance-level alignment model so CCR can be applied.
4. Experiments
Experiments across weather, scene, and dissimilar domain shifts show that categorical regularization consistently improves domain-adaptive detectors. Visualization and distance analyses further support better alignment of image-level features, instance features, crucial regions, and important instances.
- Experimental settings: The experiments cover weather adaptation from Cityscapes to Foggy Cityscapes, scene adaptation to daytime BDD100k, and dissimilar adaptation from PASCAL VOC to Clipart1k.Five public datasets are used, with DA-Faster and SW-Faster as baseline methods.
- Weather adaptation: 1.4% and 2.6% mAP improvements are obtained over DA-Faster and SW-Faster, respectively, for weather adaptation.The framework also reduces the gap to the oracle detector to about 5% mAP and improves difficult categories such as train through CCR.
- Scene adaptation: 1.3% and 1.6% improvements are obtained over DA-Faster and SW-Faster, respectively, for scene adaptation.Scene layout shift remains challenging, and CCR improves difficult objects such as truck.
- Dissimilar domain adaptation: 2.0% and 1.5% mAP improvements are obtained over DA-Faster and SW-Faster, respectively, for dissimilar domain adaptation.The method also outperforms a recent one-stage adaptive detector using self-training.
- Qualitative analysis: Detection examples show more accurate SW-Faster results on Foggy Cityscapes, BDD100k, and Clipart1k under complex environments and large domain shifts.The qualitative comparison covers three target datasets.
- Visualization and analyses: For PASCAL VOC-to-Clipart1k adaptation, EMD decreases from 8.84 for SW-Faster to 8.59 with ICR and 8.15 with ICR-CCR.t-SNE visualizations also show closer same-category cross-domain instances and better image- and instance-level alignment than SW-Faster.
5. Conclusions
The proposed categorical regularization framework improves Domain Adaptive Faster R-CNN detectors by focusing alignment on object-related regions and hard instances, achieving state-of-the-art benchmark results.
- The framework uses multi-label classification and categorical consistency to align object-related local regions and hard target-domain instances.These mechanisms exploit weak localization and agreement between image-level and instance-level predictions.
- The framework significantly improves existing Domain Adaptive Faster R-CNN detectors and produces state-of-the-art results on public benchmark datasets.
- Visualization and analyses validate the effectiveness of the proposed method.