Source-linked AI summary
Exploring Object Relation in Mean Teacher for Cross-Domain Detection
Qi Cai, Yingwei Pan, Chong-Wah Ngo, Xinmei Tian, Lingyu Duan, Ting Yao
TL;DR
Domain shift makes detectors trained on synthetic images perform poorly on real images, motivating adaptation from labeled source data and unlabeled target data. MTOR remolds Mean Teacher within Faster R-CNN using region-level and graph-structured consistency, and reports superior transfer results, including 22.8% mAP on Syn2Real.
Problem
Synthetic-to-real detection suffers from domain shift, while the target domain typically lacks annotations needed for direct supervised training.
Method
MTOR combines Faster R-CNN Mean Teacher with region-level, inter-graph, and intra-graph consistency regularizations based on object-region relations.
Results
22.8% mAP is reported as a new single-model record on the Syn2Real detection dataset.
Takeaways & Limitations
Object relations can be integrated into Mean Teacher to support unsupervised cross-domain object detection.
Abstract
from arXiv · showhide
Rendering synthetic data (e.g., 3D CAD-rendered images) to generate annotations for learning deep models in vision tasks has attracted increasing attention in recent years. However, simply applying the models learnt on synthetic images may lead to high generalization error on real images due to domain shift. To address this issue, recent progress in cross-domain recognition has featured the Mean Teacher, which directly simulates unsupervised domain adaptation as semi-supervised learning. The domain gap is thus naturally bridged with consistency regularization in a teacher-student scheme. In this work, we advance this Mean Teacher paradigm to be applicable for cross-domain detection. Specifically, we present Mean Teacher with Object Relations (MTOR) that novelly remolds Mean Teacher under the backbone of Faster R-CNN by integrating the object relations into the measure of consistency cost between teacher and student modules. Technically, MTOR firstly learns relational graphs that capture similarities between pairs of regions for teacher and student respectively. The whole architecture is then optimized with three consistency regularizations: 1) region-level consistency to align the region-level predictions between teacher and student, 2) inter-graph consistency for matching the graph structures between teacher and student, and 3) intra-graph consistency to enhance the similarity between regions of same class within the graph of student. Extensive experiments are conducted on the transfers across Cityscapes, Foggy Cityscapes, and SIM10k, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, we obtain a new record of single model: 22.8% of mAP on Syn2Real detection dataset.
1. Introduction
Synthetic data provides inexpensive annotations, but domain shift can severely reduce detection accuracy on real images. The paper adapts Mean Teacher to detection by enforcing region-level and object-relation consistency between teacher and student models.
- Synthetic images from 3D CAD models provide automatically generated ground truth, reducing reliance on costly manual labeling.
- Directly applying a detector trained on synthetic data can fail to accurately localize objects in real images because of domain shift.
- Mean Teacher bridges domains by enforcing consistent teacher and student predictions under perturbations of unlabeled target samples.
- Its three regularizations align region predictions, match teacher-student graph structures, and increase similarity among same-class student regions.
- MTOR extends Mean Teacher to Faster R-CNN using region-level consistency and graph-structured consistency for cross-domain detection.
2. Related Work
Prior work on object detection includes increasingly efficient two-stage detectors, while domain adaptation commonly uses discrepancy minimization, domain confusion, or self-ensembling. This paper’s novelty is applying Mean Teacher with object relations to cross-domain detection.
- Object detection evolved from R-CNN’s proposal-and-classification pipeline to Fast R-CNN’s shared convolutional features and Faster R-CNN’s subsequent advances.
- Deep domain adaptation methods commonly minimize feature discrepancy with MMD or learn domain discriminators for domain confusion.
- MTOR applies Mean Teacher to object detection by integrating object-region relations into teacher-student consistency regularization.
- The framework exploits unlabeled target data to learn a domain-invariant detector for cross-domain detection.
3. Mean Teacher in Semi-Supervised Learning
Mean Teacher uses paired student and teacher networks to make predictions consistent under perturbations of the same unlabeled input. The student learns by gradient descent, while the teacher tracks student weights through an exponential moving average.
- Mean Teacher contains student and teacher models with the same architecture and encourages consistent predictions under input or parameter perturbations.
- The consistency loss compares student and teacher predictions for two differently augmented versions of the same unlabeled sample.
- The student is optimized by gradient descent, whereas teacher weights are updated as an exponential moving average of student weights.
- The smoothing coefficient α controls the teacher-weight update.
- Mean Teacher combines supervised cross-entropy on labeled samples with unlabeled-sample consistency loss weighted by λ.
4. Mean Teacher in Cross-Domain Detection
MTOR adapts the Mean Teacher paradigm to cross-domain detection by combining region-level and graph-structured consistency under a Faster R-CNN backbone. Teacher and student predictions share teacher-generated proposals, while relational graphs capture similarities among regions for consistency training.
- Framework: MTOR remolds Mean Teacher for cross-domain detection by integrating object relations into teacher–student consistency regularization.The framework uses Faster R-CNN as the shared detection backbone.
- Problem formulation: The task uses labeled source images with bounding boxes and unlabeled target images to learn a domain-invariant detector.Source samples receive supervised detection training, whereas target samples are used for teacher–student consistency under perturbations.
- Teacher–student processing: Teacher and student process differently augmented target images, with the student reusing teacher-generated region proposals to align corresponding regions.This shared proposal set enables region-level interaction while avoiding a separate student proposal-generation step.
- Graph construction: Relational graphs represent regions as vertices and pairwise feature similarities as entries in symmetric affinity matrices.Teacher and student graphs use cosine similarity between region representations to encode object relations.
- Region-level consistency: Region-level consistency minimizes prediction differences for corresponding proposals and filters background or low-confidence foreground regions.The region-level loss is computed as mean squared error between teacher and student region predictions after confidence thresholding.
- Graph-structured consistency: Inter-graph consistency matches teacher and student affinity matrices, extending consistency beyond independent region predictions.The inter-graph loss is defined as mean squared error between the two graph structures.
5. Experiments
Experiments evaluate MTOR across normal-to-foggy and synthetic-to-real transfers, comparing it with source-only, DA, and consistency variants. Results show gains from region-level and graph-structured consistency, with strong quantitative and qualitative performance and parameter sensitivity.
- Dataset and Experimental Settings: Experiments cover C →F, M →C, and S →O/Y transfers using Cityscapes, Foggy Cityscapes, SIM10k, and Syn2Real data.The Syn2Real setting uses synthetic source images and COCO or YTBB target domains, with YTBB evaluated through an online server.
- Normal-to-Foggy Weather Transfer: MTOR achieves 35.1% mAP on C →F, improving 3.1% over DA.Source-only is treated as a lower bound, while DA improves over source-only through image- and region-level domain classifiers.
- Synthetic-to-Real Image Transfer: MTOR reaches 46.6% AP of car on M →C, improving 4.7% over DA.Region-level consistency improves over DA, while adding inter-graph and intra-graph consistency further improves performance.
- Qualitative Analysis: MTOR outperforms Source-only and DA in qualitative COCO examples for S →O transfer.MTOR correctly detects a person in one example missed by both comparison methods.
- Effect of the Parameters λ and α: The best performance occurs at λ = 1.0 and α ≈ 0.98, with both parameter curves generally forming inverted-∧ shapes.The evaluated ranges are λ from 0.1 to 5.0 and α from 0.92 to 0.9999.
- Visualization of Relational Graph: MTOR produces higher intra-class region similarities than Source-only and DA in the visualized Foggy Cityscapes relational graph.The graph compares cosine similarities among region representations, with the first three regions from the car class and the remaining four from the person class.
6. Conclusions
The paper presents MTOR for unsupervised cross-domain object detection by combining region-level and graph-structured consistency within Mean Teacher. Experiments validate the proposal across several transfers, including state-of-the-art single-model performance on Syn2Real.
- 6. Conclusions: MTOR combines region-level, inter-graph, and intra-graph consistency within a Mean Teacher framework for unsupervised object detection adaptation.It aligns teacher and student region predictions, matches their graph structures, and increases same-class similarity in the student graph.
- 6. Conclusions: Experiments across Cityscapes, Foggy Cityscapes, and SIM10k validate the proposal and its analysis.The conclusion also reports state-of-the-art single-model performance for synthetic-to-real transfer on Syn2Real detection.