Source-linked AI summary
CLIP the Gap: A Single Domain Generalization Approach for Object Detection
Vidit Vidit, Martin Engilberge, Mathieu Salzmann
TL;DR
The paper targets the largely unexplored problem of single-domain generalization for object detection, where models trained on one source domain must handle unseen targets. It uses CLIP to introduce textual domain variations through feature-space semantic augmentation and a text-based classifier, and reports state-of-the-art results on adverse-weather driving benchmarks. The approach assumes some information about the domain gap is available to construct prompts.
Problem
Single-domain generalization for object detection remains largely unexplored despite the need to generalize from one source domain to unseen target domains.
Method
The method uses CLIP’s joint image-text representation for prompt-based semantic augmentation of detector features and a text-based image classifier.
Results
The approach outperforms the state of the art on four adverse-weather target datasets.
Takeaways & Limitations
Semantic augmentation and CLIP initialization improve object-detector generalization to new domains, with semantic augmentation producing the best ablation results.
Takeaways & Limitations
Prompt generation assumes that some information about the domain gap, such as weather or daytime conditions, is known.
Abstract
from arXiv · showhide
Single Domain Generalization (SDG) tackles the problem of training a model on a single source domain so that it generalizes to any unseen target domain. While this has been well studied for image classification, the literature on SDG object detection remains almost non-existent. To address the challenges of simultaneously learning robust object localization and representation, we propose to leverage a pre-trained vision-language model to introduce semantic domain concepts via textual prompts. We achieve this via a semantic augmentation strategy acting on the features extracted by the detector backbone, as well as a text-based classification loss. Our experiments evidence the benefits of our approach, outperforming by 10% the only existing SDG object detection method, Single-DGOD [49], on their own diverse weather-driving benchmark.
1. Introduction
The paper addresses single-domain generalization for object detection by using CLIP to introduce textual domain variations during training. Semantic feature augmentation and a text-based classifier are evaluated on adverse-weather driving domains.
- Object-detector performance degrades when test distributions differ from training distributions, while SDG seeks representations that generalize from one source domain to unseen targets.
- The method leverages CLIP’s joint image-text representation to transfer text-based domain variations into image features during detector training.
- Textual prompts describe target-domain concepts such as weather and daytime variation, enabling semantic augmentation in feature space without target images.
- The approach is evaluated on a driving benchmark with clear-day source images and rainy, foggy, night, and dusk target conditions.
- The contributions combine a vision-language model, prompt-based feature augmentation, and increased learned-feature diversity to improve robustness to unseen domains.
2. Related Work
Related work covers domain adaptation, single-domain generalization, and vision-language models. The paper positions its contribution at the intersection of these areas, especially for the relatively unexplored SDG object-detection setting.
- Domain Adaptation for Object Detection: Domain adaptation aligns source and particular target distributions using feature alignment, centroids, attention maps, or pseudo-labels, but requires target images during training.
- Single Domain Generalization (SDG): SDG methods instead seek performance on unseen domains from a single source domain, using augmentation or normalization strategies to alter inputs or feature distributions.
- Single Domain Generalization (SDG): SDG is established in image classification but remains largely unexplored for object detection, which introduces additional challenges.
- Vision-Language Models: Vision-language models learn joint visual-semantic embeddings from image-text pairs and support tasks including classification, captioning, retrieval, and zero-shot prediction.
3. Method
The method uses CLIP’s joint image-text space to estimate semantic feature shifts from textual domain prompts, then trains an object detector with these augmentations and a text-based classifier.
- 3.1. Semantic Augmentation: Semantic augmentations are estimated from source images and textual prompts describing source and potential target-domain concepts.The augmentations encode semantic differences between source and target prompts in CLIP’s joint embedding space.
- 3.1. Semantic Augmentation: The optimization uses random source-image crops to find feature augmentations that approximate prompted domain shifts while preserving image content.An l1 regularizer limits deviation from the original embeddings, and augmentation optimization is performed offline.
- 3.2. Architecture: The detector follows Faster R-CNN but initializes modules from CLIP’s image encoder and adds a text-based classifier to retain the joint embedding space.The classifier is used to keep image features close to the pre-trained vision-language representation rather than to handle novel categories.
- 3.2. Architecture: Category and background prompts produce text embeddings whose cosine similarities with projected region features serve as classification logits.The region features are obtained after ROI alignment and projection into CLIP’s embedding space.
- 3.3. Training with Augmentation: During detector training, a randomly sampled augmentation is average-pooled and added channel-wise to backbone feature maps, while inference uses unaugmented maps.Average pooling makes the augmentations applicable to arbitrary-sized feature maps, and the training loss combines classification, RPN, and regression terms.
4. Experiments
The experiments evaluate training on daytime-clear driving images and testing across unseen weather and lighting domains. The approach consistently improves cross-domain detection, with ablations showing contributions from CLIP initialization, attention pooling, and semantically targeted augmentation.
- Experimental setup: The benchmark trains on 19,395 daytime sunny images and evaluates mAP@0.5 across daytime foggy, dusk rainy, night clear, and night rainy domains.mAP@0.5 counts a prediction as correct when its intersection over union with the ground-truth box exceeds 0.5.
- Qualitative results: Qualitative results show the approach recovering cars and people that FasterRCNN misses under night, rainy, and foggy conditions.FasterRCNN also misclassifies a car as a bus in Night-Clear and misses several cars in Dusk-Rainy scenes.
- Comparison with the state of the art: The approach outperforms baselines on every target domain, improving by close to 15% on day-foggy and dusk-rainy and by 12.6% on night-rainy versus Single-DGOD.It is comparable with Single-DGOD on night-clear and improves on both source and target domains.
- Comparison with the state of the art: Per-class evaluations show broad gains in foggy scenes, improved AP for every class in dusk-rainy scenes, and more than 22% higher car AP in night-rainy scenes than Single-DGOD.Night-clear results are comparable to Single-DGOD overall but improve more for car and person categories.
- Ablation study: CLIP initialization improves in-domain and out-of-domain performance, while semantic augmentation further improves the model and yields the best ablation results.Attention pooling mitigates the adverse effect of the text-embedding classification loss and improves performance consistently across datasets.
- Additional analyses: Semantically relevant weather prompts consistently outperform no augmentation, whereas random augmentation performs worse than no augmentation.Prompts unrelated to weather are comparable to no augmentation and show no consistent trend.
5. Limitations
The approach assumes some advance knowledge of the domain gap to construct textual prompts, using weather and daytime conditions in the experiments. The authors suggest broader keywords when specific information is unavailable.
- The method assumes that some information about the domain gap is known to generate textual prompts.The experiments characterize the gap through changes in weather and daytime conditions.
- The experiments derived all augmentation prompts from the words “weather” and “time of the day,” while also using some extra information.
- When no specific domain information is available, the authors suggest using broad concepts such as weather, ambiance, or location.
6. Conclusion
The paper proposes CLIP-guided semantic augmentation and text-based classification to improve object-detector generalization to unseen domains. It reports state-of-the-art performance on four adverse-weather target datasets.
- The approach uses CLIP and textual prompts to alter image embeddings toward potential target domains and design a text-based image classifier.
- The method outperforms the state of the art on four adverse-weather target datasets.
- Future work will investigate learning the prompts to further improve generalization.