PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers

Wen Wang, Yang Cao, Jing Zhang, Fengxiang He, Zheng-Jun Zha, Yonggang Wen, Dacheng Tao

arXiv:2107.12636v4cs.CVcs.LG

TL;DR

Detection transformers remain vulnerable to cross-domain shifts, and CNN-backbone alignment alone does not ensure domain-invariant prediction sequences. SFA aligns transformer sequence features through domain queries and token-wise alignment, with a matching-consistency loss; experiments report that it outperforms state-of-the-art DAOD methods across three challenging benchmarks.

  • Problem

    Detection transformers can lose performance under domain shifts, while CNN-backbone feature alignment does not guarantee domain-invariant transformer sequence features used for prediction.

    Source: §1 · pp. 1–2

  • Method

    SFA combines domain query-based global feature alignment, token-wise sequence alignment, and a bipartite matching consistency loss for detection transformers.

    Source: §1 · p. 2

  • Results

    SFA outperforms existing DAOD methods and achieves state-of-the-art performance across weather, synthetic-to-real, and scene adaptation scenarios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose SFA as a solid baseline for future research on domain-adaptive detection transformers.

    Source: §6 · p. 8

    The described unsupervised adaptation setting trains on labeled source images and unlabeled target images.

    Source: §3 · p. 3

Abstract

from arXiv · show

Detection transformers have recently shown promising object detection results and attracted increasing attention. However, how to develop effective domain adaptation techniques to improve its cross-domain performance remains unexplored and unclear. In this paper, we delve into this topic and empirically find that direct feature distribution alignment on the CNN backbone only brings limited improvements, as it does not guarantee domain-invariant sequence features in the transformer for prediction. To address this issue, we propose a novel Sequence Feature Alignment (SFA) method that is specially designed for the adaptation of detection transformers. Technically, SFA consists of a domain query-based feature alignment (DQFA) module and a token-wise feature alignment (TDA) module. In DQFA, a novel domain query is used to aggregate and align global context from the token sequence of both domains. DQFA reduces the domain discrepancy in global feature representations and object relations when deploying in the transformer encoder and decoder, respectively. Meanwhile, TDA aligns token features in the sequence from both domains, which reduces the domain gaps in local and instance-level feature representations in the transformer encoder and decoder, respectively. Besides, a novel bipartite matching consistency loss is proposed to enhance the feature discriminability for robust object detection. Experiments on three challenging benchmarks show that SFA outperforms state-of-the-art domain adaptive object detection methods. Code has been made available at: https://github.com/encounter1997/SFA.

1 INTRODUCTION

Detection transformers face cross-domain performance drops, and aligning CNN features alone does not ensure domain-invariant transformer sequences. SFA aligns sequence features at global and token levels and adds a consistency loss; experiments report stronger cross-domain performance than existing domain-adaptive detection methods.

1 INTRODUCTION

  • Detection transformers are fully end-to-end, but their reliance on labeled training data leaves them vulnerable to performance drops under domain shifts.Prior CNN-based detectors also rely on heuristics such as non-maximum suppression and are not fully end-to-end.Source: §1 · p. 1
  • CNN-backbone alignment improves cross-domain detection only limitedly because it does not guarantee domain-invariant sequence features used for prediction.Figure 1 illustrates that CNN features can be aligned while transformer encoder and decoder sequence features remain separable by domain.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • SFA combines domain-query global-context alignment, token-wise sequence alignment, and bipartite matching consistency to address domain gaps and improve discriminability.DQFA targets global features and object relations in the encoder and decoder; TDA targets local and instance-level features, respectively.Source: §1 · p. 2
  • Across three challenging domain-adaptation scenarios, SFA outperforms existing DAOD methods and achieves state-of-the-art performance.The scenarios include weather, synthetic-to-real, and scene adaptation.Source: §1 · p. 2

2 RELATED WORK

The section reviews object detection architectures, formulates the detection-transformer encoder and decoder, and surveys domain-adaptive object detection methods and their architectural scope.

2.1 Object Detection

  • Detection methods range from two-stage and single-stage CNN detectors to DETR-style set prediction, with Deformable DETR adding sparse sampling and multi-scale features.The section notes that earlier detectors rely on hand-crafted components such as anchors and non-maximum suppression.Source: §2.1 · p. 2

2.1.1 Formulation of Detection Transformers.

  • A detection transformer extracts hierarchical CNN features, flattens them into a sequence, encodes that sequence, and decodes query tokens using encoder outputs.Decoder-layer predictions provide classification probabilities and bounding boxes for auxiliary source-domain supervision.Source: §2.1.1 · pp. 2–3

2.2 Domain Adaptive Object Detection

  • Prior DAOD methods align image-, instance-, region-, or local/global features, but most target Faster R-CNN, SSD, or FCOS rather than detection transformers.The section positions adaptation of detection transformers as an open direction.Source: §2.2 · p. 3

3 METHODS

SFA adapts detection transformers by aligning sequence features because backbone alignment alone does not ensure domain-invariant transformer features for prediction. Its design combines global and token-wise alignment, hierarchical training, and decoder-layer matching consistency.

3 METHODS

  • SFA trains on labeled source images and unlabeled target images, using Deformable DETR by default while remaining applicable to other detection transformers.The paper also identifies DETR as an applicable framework.Source: §3 · p. 3

3.1 Domain Query-based Feature Alignment

  • Domain queries aggregate global sequence context, aligning scene-layout features in the encoder and object relations in the decoder.The encoder query emphasizes tokens with significant domain gaps, while the decoder query fuses context from object queries.Source: §3.1 · pp. 3–4

3.2 Token-wise Feature Alignment

  • Token-wise alignment addresses local texture and appearance gaps in encoder tokens and instance-level gaps in decoder tokens.The paper says token-wise alignment cannot replace domain queries, which better aggregate global context without local-content bias.Source: §3.2 · p. 4

3.3 Hierarchical Sequence Feature Alignment

  • Hierarchical alignment progressively aligns source and target sequence features from shallow to deep transformer layers.The method applies hierarchical alignment to both encoder and decoder sequences; its discriminators are three-layer MLPs.Source: §3.3 · pp. 4–5

3.4 Bipartite Matching Consistency

  • Because target labels are unavailable, bipartite matching consistency ensembles decoder-layer outputs and constrains their matches during training.Its loss combines classification JS-divergence and bounding-box L1 distance to improve target-domain discriminability.Source: §3.4 · p. 5

3.5 Total Loss

  • The final objective balances loss terms with hyperparameters, uses gradient-reversal layers for min-max optimization, and is not restricted to one detection transformer.The paper specifies applicability to DETR and Deformable DETR.Source: §3 · p. 3, §3.5 · p. 5

4 THEORETICAL ANALYSIS

The theoretical analysis decomposes target-domain error into source error, domain divergence, and shared-hypothesis error. It relates the method’s alignment and discriminator choices to these terms and states assumptions for a discriminator covering bound.

4.1 Domain Adaptation Analysis

  • Target-domain error decomposes into source-domain error, source–target domain divergence, and the error of the ideal joint hypothesis.The analysis associates domain divergence with feature transferability and joint-hypothesis error with feature discriminability.Source: §4.1 · p. 5, §4 · p. 5
  • The analysis links token-wise alignment to reduced domain divergence, domain queries to selective alignment, and matching consistency to more accurate predictions.The domain query is described as preserving discriminability for features with smaller shifts while aligning features with significant gaps.Source: §4.1 · p. 5

4.2 Generalization Analysis

  • The discriminator covering-bound analysis assumes bounded spectral norms and reference-matrix proximity; the accompanying tables cover weather and synthetic-to-real adaptation.The analysis identifies W as the largest feature-map dimension and motivates simple discriminators as a way to enhance generalizability.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6

5 EXPERIMENTAL RESULTS

Experiments evaluate SFA on weather, synthetic-to-real, and scene adaptation using four public datasets and mAP at IoU 0.5. Results, ablations, and visualizations examine performance and feature alignment across these settings.

5.1 Experimental Setup

  • Experiments use four public datasets across weather, synthetic-to-real, and scene adaptation, with mAP at an IoU threshold of 0.5.The default model uses Deformable DETR with an ImageNet-pretrained ResNet-50 backbone.Source: §5.1.1 · p. 6, §5.1.2 · p. 6

5.2.1 Weather Adaptation.

  • SFA gains 12.8 mAP50 on Cityscapes-to-Foggy Cityscapes and outperforms all previous domain-adaptive object-detection methods.The comparison evaluates weather adaptation.Source: §5.2.1 · p. 7

5.2.2 Synthetic to Real Adaptation.

  • SFA improves source-only Deformable DETR by over 10% relative performance and outperforms all existing domain-adaptive object-detection methods on Sim10k-to-Cityscapes.This evaluates synthetic-to-real adaptation.Source: §5.2.2 · p. 7

5.2 Comparisons with SOTA Methods

  • On Cityscapes-to-BDD100k daytime adaptation, SFA outperforms the previous state of the art and improves Deformable DETR across all seven reported categories.Table 3 covers the scene-adaptation setting.Source: §5.2.3 · p. 7, Table 3 · p. 7

5.3 Ablation Studies

  • DQFA and TDA improve cross-domain performance by 8.0 and 9.2 mAP, respectively; their combination is complementary, while hierarchical alignment and matching consistency add further gains.Hierarchical feature alignment and bipartite matching consistency each bring a 1.0 mAP gain; matching consistency further improves sequence alignment by 1.5 mAP.Source: §5.3 · p. 7, Table 4 · p. 7

5.4.1 Detection Results.

  • Across all three adaptation scenarios, SFA qualitative detections mitigate false positives and recover challenging objects missed by Deformable DETR.In the first scenario, SFA detects a distant car absent from the ground truth.Source: §5.4.1 · p. 7, Figure 3 · p. 8

5.4.2 Visualization of Feature Distribution.

  • T-SNE visualizations show domain-separated source-only CNN and sequence features, whereas SFA learns domain-invariant encoder and decoder sequence features.Hierarchical feature alignment also aligns CNN-backbone features.Source: §5.4.2 · pp. 7–8, Figure 3 · p. 8

5.4.3 Visualization of Domain Query.

  • Encoder domain queries attend to tokens with large fog-related domain gaps, while decoder queries emphasize shifted foreground-object locations.The encoder attends to the foggy upper image region; decoder examples include cars, riders, and bicycles.Source: §5.4.3 · p. 8, Figure 4 · p. 8

6 CONCLUSION

The conclusion presents SFA as a domain-adaptation method for detection transformers, addressing sequence-feature gaps left by CNN-backbone alignment. It combines global- and local-scale alignment with bipartite matching consistency, and experiments validate its effectiveness.

6 CONCLUSION

  • SFA combines domain-query and token-wise alignment to close global- and local-scale domain gaps, alongside bipartite matching consistency to enhance feature discriminability.The authors propose SFA as a baseline to support future research on domain-adaptive detection transformers.Source: §6 · p. 8

A.1 Implementation Details of DA-CNN

The appendix describes the CNN-backbone alignment implementation and the structure and weight sharing of SFA's domain discriminators.

A.1 Implementation Details of DA-CNN

  • The DA-CNN implementation aligns ResNet-50 C3–C5 features and an additional stride-2 convolutional feature map, reducing feature-map sizes with SRMs.A SWDA-style implementation aligns C2 locally and C5 globally, yielding a similar 35.5 mAP result.Source: §A.1 · p. 9

A.2 Structure of Domain Discriminators

  • SFA uses four three-layer MLP discriminators, sharing weights between query and token discriminators within each encoder or decoder and across feature levels.Table 5 presents the discriminator architecture.Source: §A.2 · p. 9, Table 5 · p. 9

B DETAILED THEORETICAL ANALYSIS

The appendix includes proof steps for Theorem 4.2 and describes an SFA implementation based on DETR, including its architectural differences and evaluation setup.

B DETAILED THEORETICAL ANALYSIS

  • Theorem 4.2's proof invokes Lemma A.7 from and states that its proof is completed, without reproducing the theorem's full argument in these passages.The proof also refers to Equation 16.Source: B DETAILED THEORETICAL ANALYSIS · p. 9

C DETR-BASED SFA

  • DETR-based SFA uses one C5 feature map and standard attention, unlike Deformable DETR's four feature levels and deformable attention with learnable sparse sample locations.For DETR, the reference positions enumerate all possible locations.Source: C DETR-BASED SFA · p. 9

C.1 Implementation Details

  • The DETR-based SFA evaluation uses Cityscapes-to-Foggy Cityscapes, a ResNet-50 backbone, and 300 training epochs.The setup specifies separate transformer and CNN learning rates and reports results in Table 6.Source: §C.1 · p. 9, Table 6 · p. 10

C.2 Results and Analysis

SFA improves DETR’s Cityscapes-to-Foggy-Cityscapes result and is reported to help across detection transformers. The analysis also attributes DETR’s weaker baseline to limited training data and batch size.

C.2 Results and Analysis

  • The authors attribute DETR’s inferior baseline to 2,975 Cityscapes training images and a batch size of 8.They suggest transformer attention needs more training data and epochs to focus on meaningful locations.Source: §C.2 · p. 10
  • Compared with the DETR baseline, SFA improves mAP50 by 9.1, a relative improvement of 64.5%, according to the reported Table 6 results.Source: §C.2 · p. 10
  • DQFA and token-wise alignment improve cross-domain performance by 4.4 and 5.1 mAP, respectively, and are complementary.The component ablations are reported for DETR in Table 6.Source: §C.2 · p. 10
  • Hierarchical alignment brings a 1.5 mAP gain, while the bipartite matching constraint alone brings 0.9 and adds 1.2 mAP with sequence alignment.The matching constraint is reported as complementary to sequence-feature alignment.Source: §C.2 · p. 10
  • SFA is reported to improve cross-domain performance across DETR, Deformable DETR, and the detection-transformer family.Source: §C.2 · p. 10

D MORE ABLATION STUDIES

The ablations examine DQFA and token-wise alignment at the final encoder and decoder layers, without hierarchical alignment or bipartite matching consistency. Both modules help and complement one another, with token-wise and encoder-side alignment generally performing better.

D MORE ABLATION STUDIES

  • The ablations apply each alignment module at the last encoder and decoder layers, excluding hierarchical alignment and bipartite matching consistency loss.Source: D MORE ABLATION STUDIES · p. 10
  • Both DQFA and token-wise alignment reduce source–target domain gaps on either the encoder or decoder side, and the two modules are complementary.Source: D MORE ABLATION STUDIES · p. 10
  • Token-wise alignment generally performs better than DQFA; the authors attribute this to its coverage of all locations rather than DQFA’s sparse sampling.Source: D MORE ABLATION STUDIES · p. 10
  • Encoder-side alignment generally performs better than decoder-side alignment; the authors relate this to the encoder including background as well as foreground features.The decoder sequence concentrates on foreground objects, so decoder alignment does not align background features as well.Source: D MORE ABLATION STUDIES · p. 10

E MORE QUALITATIVE RESULTS

Qualitative comparisons show SFA detections alongside source-only Deformable DETR outputs and ground truth across three adaptation scenarios. The reported improvements include fewer false positives and detections of challenging objects the baseline overlooks.

E MORE QUALITATIVE RESULTS

Loading 2107.12636v4…