Source-linked AI summary
Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection
Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, Changick Kim
TL;DR
Object detectors degrade beyond benchmark domains, while existing adaptations face source-biased discriminativity and imperfect translation. The paper combines Domain Diversification and Multi-domain-invariant Representation Learning in a structured framework, outperforming state-of-the-art methods by 3% ∼12% mAP.
Problem
Object detectors degrade beyond benchmark domains, and manual annotation for new application-specific datasets is costly; feature-level and pixel-level adaptations also face distinct limitations.
Method
The approach combines Domain Diversification, which generates distinctive shifted source domains, with Multi-domain-invariant Representation Learning, which encourages domain-invariant features in a structured adaptation framework.
Results
3% ∼12% mAP improvement over state-of-the-art methods is reported across various cases.
Takeaways & Limitations
DD addresses source-biased discriminativity, while MRL addresses imperfect image translation in unsupervised object-detection domain adaptation.
Takeaways & Limitations
The practical DD realization assumes shifted domains can preserve source semantic information, while adversarial loss alone cannot guarantee a given translation.
Abstract
from arXiv · showhide
We introduce a novel unsupervised domain adaptation approach for object detection. We aim to alleviate the imperfect translation problem of pixel-level adaptations, and the source-biased discriminativity problem of feature-level adaptations simultaneously. Our approach is composed of two stages, i.e., Domain Diversification (DD) and Multi-domain-invariant Representation Learning (MRL). At the DD stage, we diversify the distribution of the labeled data by generating various distinctive shifted domains from the source domain. At the MRL stage, we apply adversarial learning with a multi-domain discriminator to encourage feature to be indistinguishable among the domains. DD addresses the source-biased discriminativity, while MRL mitigates the imperfect image translation. We construct a structured domain adaptation framework for our learning paradigm and introduce a practical way of DD for implementation. Our method outperforms the state-of-the-art methods by a large margin of 3%~11% in terms of mean average precision (mAP) on various datasets.
1. Introduction
Object detectors degrade beyond benchmark domains, while manual annotation is costly and existing adaptation methods face distinct feature- and pixel-level limitations. The paper proposes Domain Diversification and Multi-domain-invariant Representation Learning to address these issues in a unified framework.
- Motivation: Object detectors suffer degeneration when applied beyond benchmark domains, and manually annotating application-specific datasets is costly.The paper also notes that extending an application to different environments creates further generalization issues.
- Motivation: Feature-level adaptation risks source-biased discriminativity, while pixel-level adaptation is limited by imperfect image translation.The detection data combines instances of interest with relatively unimportant background, complicating discriminative target-instance feature extraction.
- Proposed approach: Domain Diversification generates distinctive shifted source domains, and Multi-domain-invariant Representation Learning unifies dispersed domains in a common feature space.Together, they train domain-invariant detection layers using diversified annotated data.
- Framework and evaluation: The framework integrates DD and MRL and is evaluated with Faster R-CNN on real-world, artistic-media, and urban-scene datasets.The experiments are designed to demonstrate effectiveness across various datasets.
2. Related work
Related work spans feature- and pixel-level domain adaptation, but object-detection adaptation has been explored by only a few pioneering methods. Existing detection approaches address domain shift through pixel translation or separate image- and instance-level alignment.
- General domain adaptation: Deep domain adaptation methods mainly reduce domain discrepancy through feature-level adaptation or pixel-level adaptation.Feature-level methods align source and target distributions in feature space, whereas pixel-level methods translate visual appearance between domains.
- Feature-level adaptation: Feature-level approaches include Maximum Mean Discrepancy minimization and adversarial feature confusion.These methods target domain discrepancy through feature-space strategies.
- Pixel-level adaptation: Pixel-level approaches translate source images toward the target style using image-to-image translation and Generative Adversarial Networks.This line of work developed as image-to-image translation produced promising results.
- Object detection adaptation: Unsupervised domain adaptation for object detection had few pioneers compared with its broader use in computer vision.The cited detection methods include a weakly supervised framework using pixel-level adaptation and a method aligning image- and instance-level distributions.
3. Methods
The method combines Domain Diversification, which creates distinctive shifted source domains, with Multi-domain-invariant Representation Learning, which uses adversarial learning to unify them. A structured framework applies these domains to jointly learn domain-invariant representations and adapt the detector.
- Domain Diversification: Domain Diversification generates distinctive shifted domains from labeled source images to reduce source-biased discriminativity.The approach uses domain shifters that preserve source semantics while varying image appearance.
- Domain Diversification: Varying constraint losses diversifies domain-shifter appearance, while the objective balances adversarial learning with differentiated constraint losses.The constraints include an additional module when needed, and β balances the two losses.
- Multi-domain-invariant Representation Learning: Multi-domain-invariant Representation Learning uses adversarial learning across diversified, source, and target domains to encourage domain-invariant features.The framework treats translated source domains as separate domains because pairwise shifts can arise from both adaptation and imperfect translation.
- Multi-domain-invariant Representation Learning: The multi-domain discriminator receives feature maps and predicts domain probabilities for each spatial location, with a gradient reversal layer driving feature confusion.The domain label is defined for each feature vector location, and the discriminator assigns probabilities across domains.
- Structured Domain Adaptation framework for Object Detection: The structured framework uses all n + 2 domains for MRL, while source and shifted domains train the detector’s localization and classification layers.The framework is built on the object detection network and combines representation learning with detector losses.
4. Experiments
Experiments evaluate the method across real-world-to-artistic and urban-scene adaptation settings, comparing domain-shift configurations, shifted-domain counts, detection performance, localization, classification, and errors. The method achieves strong cross-domain detection results and improves discriminative classification while reducing background errors.
- Experimental settings: Experiments cover Real-world→Clipart1k, Real-world→Watercolor2k, Real-world→Comic2k, and Cityscapes→Foggy Cityscapes.Artistic-media experiments use whole target-domain images during training and test sets for evaluation; the urban experiment uses Cityscapes training images and Foggy Cityscapes validation images.
- Domain-shifter configurations: Three shifted domains are generated for each adaptation task using color-preservation, reconstruction, or combined constraints.The domain-shifter study examines how these constraint configurations produce distinctive shifted domains.
- Performance comparison: The method exceeds state-of-the-art methods by 3%–12% mAP across adaptation tasks, including around 9%–12% mAP on Real-world→AMD tasks.It achieves the highest class-wise AP in nearly all compared classes, except table in Clipart1k, car in Watercolor2k, and bus in Cityscapes.
- Ablation study: Overall results improve as the number of shifted domains increases, while combining DD with MRL significantly boosts performance.The performance improvement from MRL is amplified when more shifted domains are used.
- Localization and discriminativity: All adaptation methods improve RPN localization over baseline, while methods using DD achieve higher classification accuracy than methods without DD.The comparison uses RPN mean IoU for localization and target-domain instance classification accuracy for the Real-world→Clipart1k case.
- Error analysis: DD with and without MRL reduce background detection errors, while adding MRL to DD significantly increases correct detections.Errors are categorized as correct detections, mislocalization errors, or background errors using the top 1,000 detections per method.
5. Conclusion
The proposed learning paradigm combines Domain Diversification (DD) and Multi-domain-invariant Representation Learning (MRL) to address limitations in domain adaptation for object detection. The method reports positive impacts from both schemes and outperforms state-of-the-art methods in various cases.
- DD diversifies the distribution of labeled data to mitigate source-biased discriminativity in feature-level adaptation.
- The framework incorporates DD and MRL into a structured domain adaptation framework for object detection networks.
- MRL encourages unbiased semantic representations among multiple domains to address imperfect image translation.
- In-depth analysis confirmed the positive impact of DD and MRL and verified the effectiveness of both schemes.
- The method outperforms state-of-the-art methods in various cases.