PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptive Faster R-CNN for Object Detection in the Wild

Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, Luc Van Gool

arXiv:1803.03243v1cs.CV

TL;DR

Domain shift between training and test data can substantially reduce object-detection performance, especially when target-domain annotations are unavailable. The paper introduces Domain Adaptive Faster R-CNN, aligning image- and instance-level representations with adversarial H-divergence components and consistency regularization. Evaluations across multiple domain-shift scenarios demonstrate the approach’s effectiveness, including an 8.8% improvement over Faster R-CNN on SIM10K-to-Cityscapes adaptation.

  • Problem

    Object detection must handle differing source and target distributions, but target-domain bounding-box annotations are unavailable in the unsupervised adaptation setting.

  • Method

    Domain Adaptive Faster R-CNN adds image- and instance-level H-divergence adaptation components plus consistency regularization to learn a domain-invariant RPN.

  • Results

    The method is effective across multiple domain-shift scenarios, improving Faster R-CNN by +8.8% and achieving 38.97% AP for SIM10K-to-Cityscapes adaptation.

  • Takeaways & Limitations

    The approach obtains a robust detector for a new domain without additional labeled target data and can be trained end-to-end.

Abstract

from arXiv · show

Object detection typically assumes that training and test data are drawn from an identical distribution, which, however, does not always hold in practice. Such a distribution mismatch will lead to a significant performance drop. In this work, we aim to improve the cross-domain robustness of object detection. We tackle the domain shift on two levels: 1) the image-level shift, such as image style, illumination, etc, and 2) the instance-level shift, such as object appearance, size, etc. We build our approach based on the recent state-of-the-art Faster R-CNN model, and design two domain adaptation components, on image level and instance level, to reduce the domain discrepancy. The two domain adaptation components are based on H-divergence theory, and are implemented by learning a domain classifier in adversarial training manner. The domain classifiers on different levels are further reinforced with a consistency regularization to learn a domain-invariant region proposal network (RPN) in the Faster R-CNN model. We evaluate our newly proposed approach using multiple datasets including Cityscapes, KITTI, SIM10K, etc. The results demonstrate the effectiveness of our proposed approach for robust object detection in various domain shift scenarios.

1. Introduction

Object detection can lose substantial performance when real-world training and test domains differ in appearance, conditions, or data collection. The paper addresses this unsupervised cross-domain setting with image- and instance-level adaptation integrated into Faster R-CNN.

  • Motivation: Real-world detection faces domain shifts from viewpoints, object appearance, backgrounds, illumination, image quality, and camera or weather differences.Autonomous-driving data may also differ across cities and between synthetic and real imagery.
  • Motivation: Domain shifts can cause significant performance drops, while collecting additional bounding-box annotations is expensive and time-consuming.This motivates adaptation to visually different target domains without requiring more labeled data.
  • Problem setting: The paper studies unsupervised domain adaptation with full source-domain supervision and no target-domain supervision.The intended target-domain improvement therefore incurs no additional annotation cost.
  • Approach: Domain Adaptive Faster R-CNN minimizes domain discrepancy at image and instance levels using two adaptation components based on H-divergence.The method is built on Faster R-CNN and targets differences such as image style, illumination, object appearance, and size.
  • Approach and evaluation: A consistency regularization reinforces the domain classifiers and encourages a domain-invariant region proposal network within an end-to-end Faster R-CNN system.The paper evaluates the approach across multiple datasets, including Cityscapes, KITTI, and SIM10K, reporting effectiveness across domain-discrepancy scenarios.

2. Related Work

Prior work established strong object detectors and broad domain-adaptation methods, but detection under domain shift received less attention than classification. Existing approaches included task-specific adaptation and image translation, with realistic high-resolution translation remaining challenging.

  • Object Detection: Faster R-CNN introduced a region proposal network and achieved state-of-the-art detection results, but conventional follow-up work generally did not address domain adaptation.This paper uses Faster R-CNN as its base detector and seeks to improve generalization to a new target domain.
  • Domain Adaptation: Domain adaptation has been extensively studied for image classification using kernel, metric, subspace, covariance-alignment, and deep-network methods.The related work surveys several conventional and contemporary adaptation strategies.
  • Domain Adaptation: Unpaired image translation can provide pixel-level domain adaptation, but producing realistic high-resolution images remains challenging for autonomous-driving applications.The limitation is especially relevant when downstream systems require high-resolution visual inputs.
  • Domain Adaptation Beyond Classification: Compared with classification, domain adaptation for other vision tasks received less attention, including detection, semantic segmentation, and fine-grained recognition.Earlier detection work addressed domain shift for deformable part-based models through an adaptive SVM.

3. Preliminaries

Faster R-CNN combines shared convolutional features, an RPN, and an ROI classifier, while H-divergence frames domain alignment as making source and target features difficult for a domain classifier to distinguish. Adversarial training provides a way to optimize this alignment.

  • Faster R-CNN: Faster R-CNN is a two-stage detector comprising shared convolutional layers, a region proposal network, and an ROI-based classifier.Its architecture is illustrated in Figure 2.
  • Faster R-CNN: The shared layers produce a feature map, the RPN generates candidate proposals, and ROI pooling supplies features for category prediction.This separates proposal generation from ROI-wise classification.
  • Faster R-CNN: Both RPN and ROI-classifier losses include classification and bounding-box regression terms.Classification measures predicted-probability accuracy, while regression improves box localization.
  • H-divergence: H-divergence measures the distance between source and target sample distributions through the errors of a domain classifier.A high best-classifier error indicates that the domains are difficult to distinguish and therefore close under this measure.
  • H-divergence: Feature-producing networks can be trained to minimize domain distance by making source and target representations difficult to classify by domain.The paper describes adversarial optimization using a gradient reversal layer integrated into a CNN.

4. Domain Adaptation for Object Detection

The paper frames cross-domain object detection as learning under differing source and target distributions, then aligns image- and instance-level representations within Faster R-CNN. Domain classifiers and consistency regularization are combined to promote domain-invariant proposals and end-to-end adaptation.

  • Probabilistic Perspective: Cross-domain detection involves differing source and target joint distributions over categories, bounding boxes, and image representations.The source domain is the training domain, while the target domain is the test domain.
  • Image-Level Adaptation: Under covariate shift, image-level adaptation targets differences in the marginal image representation distribution while preserving conditional detection behavior.In Faster R-CNN, the image representation is the feature map produced by the base convolutional layers.
  • Instance-Level Adaptation: Instance-level adaptation aligns distributions of region representations to reduce differences in object appearance, size, and viewpoint.The instance representation consists of features extracted from image regions defined by bounding boxes.
  • Consistency Regularization: Consistency between the two domain classifiers is used to reduce bias in bounding-box prediction and encourage a domain-invariant RPN.The approach applies consistency regularization alongside image- and instance-level alignment within an augmented Faster R-CNN architecture.
  • Domain Alignment Components: The method uses domain classifiers at image and instance levels, trained adversarially to align feature distributions between domains.The image-level classifier operates on feature-map activations corresponding to input patches, while the instance-level classifier operates on ROI feature vectors.

5.1. Experiment Setup

Experiments follow unsupervised domain adaptation: source images have bounding-box and category annotations, whereas target training images are unlabeled. The study reports mAP at a 0.5 threshold and compares the full model with combinations of its adaptation components.

  • Data Protocol: The training data combines labeled source images with unlabeled target images under an unsupervised domain adaptation protocol.Source annotations include bounding boxes and object categories; target data provides only images.
  • Evaluation: The experiments report final results for the full model and combinations of image-level adaptation, instance-level adaptation, and consistency regularization.The original Faster R-CNN trained only on source data is included as a baseline.
  • Evaluation: Mean average precision is reported with a threshold of 0.5 for evaluation.Unless otherwise stated, images are resized so their shorter side is 500 pixels, and training uses the stated learning-rate schedule and batch construction.

5.2. Experimental Results

The evaluation tests adaptation across synthetic-to-real, clear-to-foggy, and cross-camera domain shifts. Across these scenarios, Domain Adaptive Faster R-CNN improves over Faster R-CNN and works in both KITTI–Cityscapes directions.

  • Evaluation scenarios: Experiments cover synthetic-to-real, adverse-weather, and cross-camera domain adaptation scenarios.The evaluation uses SIM 10k–Cityscapes, Cityscapes–Foggy Cityscapes, and KITTI–Cityscapes settings.
  • Synthetic-to-real adaptation: +8.8% over Faster R-CNN, reaching 38.97% AP on Cityscapes cars after combining image-level, instance-level, and consistency components.Image-level alignment alone gives +2.9%, instance-level alignment alone +5.6%, and combining both gives +7.7%.
  • Adverse-weather adaptation: +8.6% over Faster R-CNN on Foggy Cityscapes, with improvements generalizing across object categories.The source domain is clear-weather Cityscapes and the target domain is Foggy Cityscapes.
  • Cross-camera adaptation: Adaptive Faster R-CNN improves over other baselines for KITTI–Cityscapes adaptation in both K →C and C →K directions.The metric reported is car AP in both adaptation directions.

5.3. Error Analysis on Top Ranked Detections

The error analysis compares high-confidence detections from models with different adaptation components and examines robustness to target-image scale changes. Image-level adaptation improves localization-related behavior and is more robust to scale changes than instance-level adaptation.

  • Top-ranked detection errors: The analysis selects 20,000 highest-confidence KITTI →Cityscapes predictions from vanilla, image-level, and instance-level adaptation models.Detections are categorized as correct, mis-localized, or background according to overlap with ground truth.
  • Scale robustness: When target-image scales vary, vanilla Faster R-CNN performance drops significantly under scale mismatch, whereas image-level adaptation is more robust than instance-level adaptation.Source KITTI images remain fixed at 500 pixels while target Cityscapes images are resized to different scales.
  • Top-ranked detection errors: Instance-level alignment produces more background errors than image-level alignment, while image-level alignment may improve RPN proposal localization more directly.The proposed explanation links image-level alignment to better-localized region proposals.

5.4. Image-level v.s. Instance-level Alignment

The experiments compare image-level and instance-level adaptation under target-domain scale changes. Image-level adaptation is more robust to scale mismatch, while combining both adaptations performs best across scales.

  • Scale robustness: Scale mismatch causes vanilla Faster R-CNN performance to drop significantly.The target-image scale varies while the source scale remains fixed at 500 pixels.
  • Scale robustness: Image-level adaptation is more robust to target-image scale changes than instance-level adaptation.The comparison uses KITTI → Cityscapes with varying target-image scales.
  • Why alignment levels differ: Global scale changes primarily affect image-level alignment because they transform all instances and the background.Instance-level alignment can be weakened when severe global shifts increase proposal localization errors.
  • Combined adaptation: Using both adaptation components yields the best results across all tested scales.Unlike vanilla Faster R-CNN, the combined model benefits from higher target-image resolution and improves as scale rises from 200 to 1,000 pixels.

5.5. Consistency Regularization

The section evaluates consistency regularization by comparing RPN performance before and after applying the regularizer on KITTI → Cityscapes. Proposal quality is measured using the maximum achievable mean overlap of the top 300 RPN proposals with ground truth.

  • Evaluation setup: Consistency regularization is evaluated for its effect on RPN proposal quality.The comparison uses KITTI → Cityscapes and includes vanilla Faster R-CNN as a baseline.
  • Evaluation setup: RPN quality is measured by the maximum achievable mean overlap between the top 300 proposals and ground-truth boxes.The passage introduces this metric for comparing models before and after consistency regularization.

6. Conclusion

The paper concludes that Domain Adaptive Faster R-CNN addresses cross-domain object detection without additional labeled target data. Its image- and instance-level adversarial adaptations, reinforced by consistency regularization, outperform baseline Faster R-CNN across varied domain-shift scenarios.

  • Conclusion: Domain Adaptive Faster R-CNN adapts object detection to a new domain without additional labeled data.The approach assumes labeled source data and an unlabeled target domain.
  • Conclusion: The method combines image-level and instance-level adaptation components based on adversarial training of H-divergence.A consistency regularizer is additionally used to learn a domain-invariant RPN.
  • Conclusion: The model is trained end-to-end using standard SGD optimization.The integrated system is built on Faster R-CNN.
  • Conclusion: Across various domain-shift scenarios, the adaptive method outperforms baseline Faster R-CNN by a clear margin.The paper validates the approach on multiple domain-shift scenarios.
Loading 1803.03243v1…