Source-linked AI summary
Domain Adaptive Object Detection for Autonomous Driving under Foggy Weather
Jinlong Li, Runsheng Xu, Jin Ma, Qin Zou, Jiaqi Ma, Hongkai Yu
TL;DR
Clear-weather-trained object detectors can perform less effectively in fog because weather changes create a feature-domain gap. The paper combines image- and object-level adaptation with hard-example mining and auxiliary-domain metric regularization. Experiments report effectiveness and superior detection performance over baseline and comparison methods.
Problem
Clear-weather-trained detectors may lose performance in fog because training and testing features differ across weather domains.
Source: §1 · p. 1Method
The framework combines image- and object-level adaptation with AdvGRL hard-example mining and data-augmented auxiliary-domain metric regularization.
Source: §1 · p. 2Results
Experiments on public benchmarks show the proposed components' effectiveness and superior detection performance over baseline and comparison methods.
Source: §1 · p. 2Takeaways & Limitations
The experiments support the framework's effectiveness for domain-adaptive object detection, including transfers from Cityscapes to Foggy Cityscapes and KITTI.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Most object detection methods for autonomous driving usually assume a consistent feature distribution between training and testing data, which is not always the case when weathers differ significantly. The object detection model trained under clear weather might not be effective enough in foggy weather because of the domain gap. This paper proposes a novel domain adaptive object detection framework for autonomous driving under foggy weather. Our method leverages both image-level and object-level adaptation to diminish the domain discrepancy in image style and object appearance. To further enhance the model's capabilities under challenging samples, we also come up with a new adversarial gradient reversal layer to perform adversarial mining for the hard examples together with domain adaptation. Moreover, we propose to generate an auxiliary domain by data augmentation to enforce a new domain-level metric regularization. Experimental results on public benchmarks show the effectiveness and accuracy of the proposed method. The code is available at https://github.com/jinlong17/DA-Detect.
1. Introduction
Clear-weather detectors can lose performance in fog because weather changes create a feature-domain gap. The paper proposes unsupervised image- and object-level adaptation, hard-example mining, and auxiliary-domain metric regularization for foggy-weather detection.
1. Introduction
- Clear-weather-trained detectors lose performance in foggy weather because their features differ from the target-domain distribution.The paper identifies this feature-domain gap as the reason adaptation can improve foggy-weather detection.Source: §1 · p. 1
- The framework aligns image-style and object-appearance features using labeled clear-weather images and unlabeled foggy-weather images.Image- and object-level domain classifiers support the domain-invariant feature representations.Source: §1 · pp. 1–2
- Experiments on Cityscapes and Foggy Cityscapes show the proposed components' effectiveness and superior detection performance over baseline and comparison methods.The paper reports these outcomes without giving metric values in the supplied passage.Source: §1 · p. 2
- AdvGRL performs adversarial mining for hard examples alongside domain adaptation to address challenging training samples.The paper identifies differing sample difficulty as an issue that previous methods usually omit.Source: §1 · p. 2
- Data augmentation generates an auxiliary domain for domain-level metric regularization among source, auxiliary, and target domains.The paper motivates this constraint by noting that prior methods focus on source-to-target transfer.Source: §1 · p. 2
2. Related Work
The related work surveys object detectors for autonomous driving, methods addressing adverse weather, and domain adaptation for detection. It positions this paper's hard-example mining and auxiliary-domain regularization against prior approaches.
2.1. Object detection for autonomous driving
- Two-stage detectors use region proposal followed by classification and localization refinement, while single-stage detectors predict boxes and classes in one stage.The paper notes that single-stage methods are less time-consuming but less accurate than two-stage methods, and that weather complexity limits direct use in driving.Source: §2.1 · p. 2
2.2. Object detection for autonomous driving under different weather
- Foggy-weather approaches include simulated datasets and restoration or multisensor methods, while this work targets RGB-only detection.The cited examples include Foggy Cityscape, DSNet, lidar/radar methods, and a four-sensor architecture.Source: §2.2 · pp. 2–3
2.3. Domain adaptation for object detection
- Prior domain-adaptive detectors align features or use image translation, whereas this method adds hard-example mining and auxiliary-domain metric regularization without extra learnable parameters.The paper describes AdvGRL as gradient-reversal-based and the regularizer as triplet-loss-based.Source: §2.3 · p. 3
3. Proposed Method
The framework extends Faster R-CNN with image- and object-level domain adaptation, adversarial hard-example mining, and auxiliary-domain metric regularization. It trains these components jointly across source, target, and auxiliary domains.
3.1. Network Architecture
- The Faster R-CNN pipeline extracts image features, proposes regions, pools object features, and predicts detections, with adaptation components operating alongside it.Source, target, and auxiliary domains are used simultaneously during training.Source: §3.1 · p. 3
3.2. Image-level Adaptation
- Image-level adaptation uses a domain classifier on backbone features to align global representations that encode style, scale, and illumination.The classifier is a CNN with two convolutional layers.Source: §3.2 · p. 3
3.3. Object-level Adaptation
- Object-level adaptation classifies ROI-pooled region proposals by domain to align local features that may differ in appearance, size, and color.The object-level domain classifier has three fully connected layers.Source: §3.3 · p. 4
3.4. Adversarial Gradient Reversal Layer
- AdvGRL reverses gradients to encourage domain-invariant features and increases the response for harder examples with lower domain-classifier loss.Its coefficient replaces the original GRL coefficient, with a hardness threshold and an overflow cap; it is used for both image- and object-level adaptation.Source: §3.4 · p. 5
3.5. Domain-level Metric Regularization
- Domain-level metric regularization adds an augmented auxiliary domain and uses image- and object-level triplet constraints to relate source, auxiliary, and target features.The image-level constraint places the source-to-target feature distance below the source-to-auxiliary distance; the object-level loss uses the analogous triplet structure.Source: §3.5 · pp. 5–6
3.6. Loss Function
- The final training loss sums the Faster R-CNN detection losses and domain-adaptation losses, with a weight balancing those groups.Training is end-to-end with SGD, and testing uses the adapted Faster R-CNN after removing domain-adaptation components.Source: §3.6 · p. 6
3.7. General Domain Adaptive Object Detection
- For domains without pixel-to-pixel correspondence, the framework can be applied by removing the object-level regularization loss that depends on object alignment.With correspondence, such as Cityscapes to Foggy Cityscapes, the method can be applied without modification.Source: §3.7 · p. 6
4. Experiments
The experiments use Cityscapes as the clear-weather source and Foggy Cityscapes as the foggy target, evaluating adaptation on the target validation set. They report detection AP at IoU 0.5 and compare against prior domain-adaptation methods.
4.1. Benchmark
- Foggy Cityscapes simulates three fog-intensity levels on Cityscapes images; the experiments use the highest-intensity fog as the target domain.The datasets share image counts, resolution, splits, and annotations; Cityscapes has 2,975 training and 500 validation images.Source: §4.1 · p. 6
4.2. Experimental Setting
- The labeled Cityscapes training set is the source, the unlabeled Foggy Cityscapes training set is the target, and evaluation uses the Foggy Cityscapes validation set.The detector uses a ResNet-50 backbone and is trained for 50k iterations at learning rate 0.01 followed by 20k at 0.001.Source: §4.2 · p. 6
- Performance is measured by class AP and overall mAP at IoU 0.5, with comparisons to domain-adaptation methods including DA-Faster, SWDA, MTOR, GPA, and UaDAN.Table 1 covers AP by class and overall mAP for Cityscapes-to-Foggy-Cityscapes adaptation.Source: §4.2 · pp. 6–7, Table 1 · p. 7
4.3. Clear to Foggy Adaptation
The proposed method achieves the highest overall mAP among the compared clear-to-foggy adaptation methods and leads in more than half of Foggy Cityscapes categories.
4.3. Clear to Foggy Adaptation
- 42.3% mAP is the best overall detection performance among the compared domain adaptation methods, 1.2% above UaDAN.The method also achieves the best AP in more than half of Foggy Cityscapes categories.Source: §4.3 · p. 7
- 48.7% AP for train detection exceeds DA-Faster’s 45.4% by 3.3%.The method also records best AP for bus (51.2%), bicycle (39.1%), person (36.5%), and rider (46.7%).Source: §4.3 · p. 7
- PDA leads car AP at 54.4%, GPA leads motorcycle AP at 32.4%, and DA-Faster leads truck AP at 30.9%; the proposed method is comparable on these categories.Source: §4.3 · p. 7
4.4. Cross-Camera Adaptation
The method is evaluated for cross-camera adaptation from Cityscapes to KITTI using KITTI training images as the target domain for both adaptation and evaluation. The authors report outstanding performance against recent state-of-the-art methods.
4.4. Cross-Camera Adaptation
- For unaligned real-world datasets, the method removes LR_obj (Eq. 8) when adapting from Cityscapes to KITTI.Source: §4.4 · p. 7
- The proposed method achieves outstanding performance compared with recent state-of-the-art methods on cross-camera adaptation.The evaluation reports Car AP on KITTI as the target domain.Source: §4.4 · p. 7
- KITTI’s 7,481 training images at 1250 × 375 resolution serve as the target domain for both adaptation and evaluation.Source: §4.4 · p. 7
4.5. Ablation Study on Components
The component ablation evaluates the proposed modules on Cityscapes-to-Foggy-Cityscapes detection and reports positive effects for each proposed component.
4.5. Ablation Study on Components
- The ablation compares image-level and object-level adaptation, AdvGRL, and domain-level metric regularization; img+obj+GRL is the experimental baseline.The Source only comparison is Faster R-CNN trained with labeled source-domain images and no domain adaptation.Source: §4.5 · p. 7
- Figure 5 shows two hard examples mined by AdvGRL followed by an easy example; more fog covering objects indicates harder transfer.Source: Figure 5 · p. 8
- The ablation study reports a positive effect from each proposed component of domain-adaptive object detection.Source: §4.5 · p. 8
4.6. Ablation Study on Parameters
The parameter study reports Cityscapes-to-Foggy-Cityscapes detection mAP across three loss-balance weights and four AdvGRL threshold settings.
4.6. Ablation Study on Parameters
- 42.34 mAP is obtained with loss-balance weight w = 0.1, versus 41.30 at w = 0.01 and 41.19 at w = 0.001.Source: §4.6 · p. 8
- 42.34 mAP results from β = 30 and α = 0.63, compared with 38.83 for β = 10 and α = 0.63.Source: §4.6 · p. 8
- 39.38 mAP results from β = 30 and α = 0.54, compared with 40.47 for β = 10 and α = 0.54.The reported α = 0.54 is computed by averaging Eq. 1 values for Pi = 0.9, Gi = 1 and Pi = 0.1, Gi = 0.Source: §4.6 · p. 8
4.7. Discussion on Visualized Hard Examples
The discussion uses feature distance to identify hard examples for transfer learning and relates greater fog coverage of objects to increased difficulty.
4.7. Discussion on Visualized Hard Examples
- The proposed AdvGRL uses λadv to identify hard examples, as illustrated in Fig. 5.The paper approximates hardness using the L1 distance between source and target features after the CNN backbone.Source: §4.7 · p. 8
- Smaller approximated hardness ah means an example is harder for transfer learning.The approximation is based on the L1 distance between source and target features after the CNN backbone.Source: §4.7 · p. 8
- More fog covering objects in bounding-box regions is described as making transfer more difficult.The paper illustrates this with the examples in Fig. 5.Source: §4.7 · p. 8
4.8. Discussion on Domain Randomization, Pre-trained Models, and Qualitative Results
The section compares auxiliary-domain construction and pretraining choices, then presents qualitative detections on Foggy Cityscapes. Regular augmentation and CycleGAN-style transfer are reported as less effective than the paper’s rain-synthesis auxiliary domain.
4.8. Discussion on Domain Randomization, Pre-trained Models, and Qualitative Results
- 42.3 mAP with the rain-synthesis auxiliary domain exceeds 38.7 mAP with regular augmentation.Regular augmentation used color change, blur, and salt-and-pepper noise in the Cityscapes-to-Foggy Cityscapes experiment.Source: §4.8 · p. 8
- 32.8 mAP was achieved by Faster R-CNN trained with CycleGAN-generated fake foggy-style Cityscapes images; the authors say common domain randomization did not solve adaptation well.The images were generated by transferring style between the Cityscapes and Foggy Cityscapes training sets.Source: §4.8 · p. 8
- 41.3 mAP with the pretrained Faster R-CNN initialization compares with 42.3 mAP without that pretrained detection model.Both figures are reported for Cityscapes-to-Foggy Cityscapes.Source: §4.8 · p. 8
- The qualitative Foggy Cityscapes figure compares original Faster R-CNN, its image- and object-adapted GRL baseline, and the proposed method; colors indicate categories.The accompanying text says the proposed method improves detection performance significantly in foggy weather.Source: Figure 6 · p. 8, §4.8 · p. 8
5. Conclusions
The conclusion presents a domain-adaptive object detection framework combining image- and object-level adaptation, adversarial mining, and domain-level metric regularization. Experiments transfer from Cityscapes to Foggy Cityscapes and KITTI, with results described as effective.
5. Conclusions
- The framework combines image- and object-level adaptations to reduce shifts in global image style and local object appearance.It also uses a new adversarial gradient reversal layer to mine hard examples together with domain adaptation.Source: §5 · p. 8
- The method adds domain-level metric regularization using feature distances among source, target, and data-augmentation-generated auxiliary domains.The conclusion describes the auxiliary domain as generated by data augmentation.Source: §5 · p. 8
- Experiments transfer from Cityscapes to Foggy Cityscapes and from Cityscapes to KITTI, and the results are described as quite effective.The conclusion also says the method could address general domain-adaptive object detection.Source: §5 · pp. 8–9