PaperlayerView exact revision on arXiv

Source-linked AI summary

SIGMA: Semantic-complete Graph Matching for Domain Adaptive Object Detection

Wuyang Li, Xinyu Liu, Yixuan Yuan

arXiv:2203.06398v3cs.CV

TL;DR

DAOD must generalize detectors from labeled source domains to novel unlabeled domains, but existing adaptation overlooks within-class variance and mismatched batch semantics. SIGMA completes missing semantics with hallucination nodes and graphs, then performs node-to-node graph matching; experiments on three benchmarks report significant gains over existing approaches. Its scope includes the stated center-based and mismatched-category limitations of prior methods.

  • Problem

    Existing DAOD adaptation overlooks significant within-class variance and domain-mismatched categories, limiting class-conditional alignment.

  • Method

    SIGMA generates hallucination nodes for missing categories, models class-conditional distributions with cross-image graphs and a memory bank, and aligns graphs using bipartite node matching.

  • Results

    SIGMA significantly outperforms existing DAOD approaches on three benchmarks and achieves state-of-the-art results.

  • Takeaways & Limitations

    Graph-based semantic completion and node-to-node matching provide SIGMA’s fine-grained approach to domain alignment.

  • Takeaways & Limitations

    Prior category-level methods can produce biased center-based adaptation and fail to adapt categories appearing in only one domain.

Abstract

from arXiv · show

Domain Adaptive Object Detection (DAOD) leverages a labeled domain to learn an object detector generalizing to a novel domain free of annotations. Recent advances align class-conditional distributions by narrowing down cross-domain prototypes (class centers). Though great success,they ignore the significant within-class variance and the domain-mismatched semantics within the training batch, leading to a sub-optimal adaptation. To overcome these challenges, we propose a novel SemantIc-complete Graph MAtching (SIGMA) framework for DAOD, which completes mismatched semantics and reformulates the adaptation with graph matching. Specifically, we design a Graph-embedded Semantic Completion module (GSC) that completes mismatched semantics through generating hallucination graph nodes in missing categories. Then, we establish cross-image graphs to model class-conditional distributions and learn a graph-guided memory bank for better semantic completion in turn. After representing the source and target data as graphs, we reformulate the adaptation as a graph matching problem, i.e., finding well-matched node pairs across graphs to reduce the domain gap, which is solved with a novel Bipartite Graph Matching adaptor (BGM). In a nutshell, we utilize graph nodes to establish semantic-aware node affinity and leverage graph edges as quadratic constraints in a structure-aware matching loss, achieving fine-grained adaptation with a node-to-node graph matching. Extensive experiments verify that SIGMA outperforms existing works significantly. Our code is available at https://github.com/CityU-AIM-Group/SIGMA.

1. Introduction

DAOD addresses performance degradation when detectors trained in one domain are deployed in a novel unlabeled domain. SIGMA responds by completing mismatched semantics and aligning class-conditional distributions through graph matching.

  • Novel-domain deployment causes catastrophic performance degradation because of the domain gap between training and test data.
  • Existing category-level methods neglect within-class variance, despite scale and shape providing essential class-conditional information.Center-based alignment can leave object instances unadapted and overlap class-conditional distributions, while Gaussian variance modeling is suboptimal in non-convex deep feature spaces.
  • Batch-wise domain-mismatched categories are ignored by methods adapting only co-occurred categories, causing semantic knowledge loss.For example, trains may appear only in the source batch while bicycles appear only in the target batch.
  • SIGMA completes missing categories with hallucination nodes, models class-conditional distributions as graphs, and reformulates adaptation as graph matching.Its GSC module uses domain-level statistics, cross-image graphs, and a graph-guided memory bank before BGM performs node matching.
  • SIGMA introduces graph-based node affinities and edge-based quadratic constraints for fine-grained, structure-aware domain alignment.
  • Extensive experiments on three benchmarks report state-of-the-art results that significantly outperform existing DAOD approaches.

2. Related Work

DAOD research includes domain-alignment methods that reduce source–target discrepancies, while SIGMA extends graph matching to establish cross-graph node correspondences using node and structural affinities.

  • Domain-alignment approaches bridge source–target gaps by minimizing cross-domain feature discrepancies through mechanisms such as spatial attention.The broader DAOD literature also includes style-transfer and self-labeling approaches.
  • SIGMA’s overview includes vision-to-graph transformation, denoted V2G, within its proposed DAOD framework.
  • Graph matching establishes one-to-one correspondences between nodes in different graphs as a Quadratic Assignment Problem.Solvers encode matched pairs with a cross-graph permutation matrix while considering node and structure affinities.

3. Motivation and Preliminaries

Existing category-level adaptation relies on center-based class-conditional alignment and fails to handle mismatched categories. SIGMA completes missing semantics and solves adaptation through graph matching with differentiable assignment.

  • Existing methods estimate class centers from object features and minimize cross-domain discrepancies between corresponding centers.
  • Center-based adaptation can be biased and fails to adapt categories appearing in only one domain.
  • SIGMA generates samples for missing categories and establishes cross-image graphs to model each domain’s class-conditional distribution.
  • The method reformulates domain adaptation as graph matching between source and target graphs and solves it with a differentiable Quadratic Assignment Problem.
  • The objective compares graph adjacency structures and unary node affinities using a relaxed permutation matrix encoding node-to-node assignments.The adjacency matrices encode graph structure, while the unary affinity matrix represents node affinity.
  • Continuous relaxation of the one-hot permutation matrix satisfies the differentiability requirement of neural-network training.

4. Proposed Method

SIGMA completes missing category semantics, models source and target distributions as cross-image graphs, and adapts them through bipartite graph matching.

  • Graph-embedded Semantic Completion: SIGMA transforms sampled visual features into graph nodes, using source ground-truth boxes and target pseudo score maps for class-aware foreground sampling.Target foreground and background nodes are selected using score thresholds τfg and τbg.
  • Graph-embedded Semantic Completion: Domain-guided node completion generates hallucination nodes for categories missing from each batch, producing semantic-complete node sets.The method uses counterpart-domain statistics, memory seeds, Gaussian sampling, and linear projection to generate missing-category nodes.
  • Graph-embedded Semantic Completion: Cross-image graphs model class-conditional distributions, while a graph-guided memory bank preserves category-specific graph embeddings for improved hallucination generation.Only seed-included clusters and existing graph nodes update the memory bank, excluding hallucination nodes from memory updates.
  • Bipartite Graph Matching: Bipartite graph matching reformulates cross-domain alignment as a Quadratic Assignment Problem using cross-graph interaction and node affinity.Knowledge exchange between source and target graphs provides cross-domain perception before affinity estimation.
  • Bipartite Graph Matching: The structure-aware matching loss combines node correspondence objectives with quadratic constraints that reduce structural differences between matched node pairs.The optimization also includes node classification, node alignment, global alignment, and detection losses, with λ1/2 set to 0.1.

5. Experiments

SIGMA is evaluated across three domain-adaptation benchmarks using mAP-based comparisons, ablations, node-selection studies, matching strategies, and qualitative analyses. It achieves strong adaptation results while showing that semantic completion, graph components, and carefully selected matching designs contribute to performance.

  • Comparison with State-of-the-arts: SIGMA achieves 43.5% and 44.2% mAP with VGG-16 and ResNet-50 on Cityscapes→Foggy Cityscapes, outperforming existing methods.It also achieves 53.7% mAP on Sim10k→Cityscapes and 45.8% mAP on KITTI→Cityscapes.
  • Comparison with State-of-the-arts: On Sim10k→Cityscapes, SIGMA reaches 53.7% mAP and a 13.9% AP adaptation gain, exceeding same-detector baselines.The reported gains over EPM, KTNet, and SSAL are 4.7%, 3.0%, and 1.9% mAP, respectively.
  • Sensitivity Analysis: Using only source and target nodes yields 36.8% and 37.3% mAP, whereas increasing nodes from 20 to 200 improves performance from 39.0% to 43.9%.Using 500 nodes reduces performance to 42.6% mAP, indicating that excessive nodes hinder graph-matching optimization.
  • Sensitivity Analysis: Single-matching achieves 43.5% mAP0.5 versus 43.1% for multiple-matching, while matching-loss components improve mAP0.5 from 42.1% to 43.5%.The text attributes single-matching’s advantage to aligning primary node pairs and reducing noisy adaptation on ambiguous nodes.
  • Qualitative Results: Qualitative results show fewer missing detections and false-positive classification errors, while T-SNE features separate person, rider, and bike more clearly.The comparisons include source-only, EPM, SIGMA, and ground-truth results for Cityscapes→Foggy Cityscapes.

6. Conclusion

SIGMA represents domain information with semantic-complete graphs and formulates domain adaptation as graph matching. Across three benchmarks, it significantly outperforms existing approaches.

  • SIGMA models domain adaptation with semantic-complete graphs and node-to-node graph matching to address semantic mismatching and sub-optimal prototype alignment.Its Graph-embedded Semantic Completion module completes mismatched semantics, while the Bipartite Graph Matching adaptor performs fine-grained alignment.
  • Extensive experiments on three benchmarks show that SIGMA significantly outperforms existing approaches.

A.1. Parameter Sensitivity

SIGMA’s adaptation performance depends on the relative weighting of node classification and structure-aware matching losses, with node alignment best applied to semantic-complete nodes. Layer-normalized V2G projection also achieves the best reported adaptation result.

  • Parameter Sensitivity: Decreasing λ1,2 from the main setting causes a significant performance drop, while λ2 = 0.1 is optimal when λ1 is fixed.Increasing λ1 can further improve performance, whereas changing λ2 slightly reduces overall performance.
  • Node Discriminator Placement: Aligning semantic-complete nodes (P1) achieves better results than aligning enhanced graph-convolution nodes (P2) or cross-graph-interaction nodes (P3).The comparison is conducted on Cityscapes→Foggy Cityscapes.
  • Projection Strategy: Layer Normalization in the V2G projection preserves node-based correspondence and achieves the best adaptation result, 43.5% mAP.The comparison includes Group Normalization, Batch Normalization, and Layer Normalization.
  • Baseline Design: SIGMA uses a single-stage detector without an RPN, enabling image-feature adaptation beyond methods limited to two-stage detector pipelines.The method performs fine-grained adaptation using image features and discards the RPN.

B.2. Limitation

SIGMA can fail when distant objects are obscured by heavy fog because limited visual features make them difficult to represent. The paper suggests stronger visual backbones or additional linguistic cues as possible remedies.

  • Failure Cases: Limited visual features cause failure cases, including missed or wrongly detected distant objects in heavy fog.Examples include a missed truck and a wrongly detected person, attributed to tiny scale and low-quality appearance.
  • Scope: The limitation concerns visual representation quality rather than the graph-matching formulation itself.The paper attributes the failures to long distance and heavy fog.
  • Possible Remedies: The paper proposes robust backbones or visual-linguistic graph matching to compensate for limited visual features.ResNet-101 is given as an example of a stronger backbone.

C.1. Discriminator Architecture

SIGMA combines image-level global alignment with node-level alignment in its adversarial module. The node discriminator operates on well-matched graph nodes using Layer Normalization for node-based representations.

  • Module Components: The adversarial alignment module contains global-alignment and node-alignment losses, LGA and LNA.The architecture is detailed in Table 9.
  • Global Alignment: A Global Discriminator performs image-level global alignment on source and target image features.This follows the global-alignment design used in prior adaptation methods.
  • Node Alignment: A Node Discriminator aligns well-matched graph nodes after graph nodes refactor spatial correspondence through edge connections.Layer Normalization replaces Group Normalization because it better suits node-based representations.

C.2. Implementation and Training

SIGMA is trained end to end through visual-to-graph conversion, semantic completion, graph construction, matching, and joint loss optimization. Its implementation includes normalization, clustering, dropout, and matching-specific training choices.

  • Network Blocks: The V2G projection uses an Fc-LayerNorm-ReLU-Fc block, while the node classifier uses an Fc-ReLU-Fc block.These blocks define separate projection and node-classification transformations.
  • Implementation Choices: The implementation uses 0.1 dropout for edgedrop, spectral clustering when more than five nodes are present, and K-Nearest Neighbor clustering with K=5.The K-Nearest Neighbor replacement is reported to reduce clustering time.
  • Training Pipeline: Algorithm 1 extracts features, computes global and detection losses, converts features to raw nodes, and generates node-alignment losses before semantic completion.The pipeline receives source and target images, source annotations, and loss hyperparameters.
  • Graph Matching: After semantic completion, SIGMA constructs graphs, applies GCN, updates the graph-guided memory bank, performs BGM and CGI, and learns affinity and matching losses.The matching pipeline produces enhanced nodes, an affinity matrix, and graph-matching loss.
  • Optimization: The total training loss combines node classification, graph matching, node alignment, global alignment, and detection terms as L = λ1Lnode + λ2Lmat + LNA + LGA + Ldet.Network parameters are updated by backpropagation during iterative training.
  • Training Strategy: SIGMA supports end-to-end training without warm-up by using halved source nodes as target placeholders and adding 10,000 training iterations.The paper reports that this reaches the same results as the warm-up-included strategy.

D. Qualitative Results

Figure 6 visualizes the learned doubly stochastic node-affinity matrix against the ground-truth matching matrix, while Figure 7 qualitatively compares detection results across adaptation methods. The visualizations show adaptive same-category node matching and fewer missing or wrongly classified cases for SIGMA than EPM.

  • Matching Visualization: Figure 6 compares the learned doubly stochastic node-affinity matrix with the ground-truth matching matrix.Activated affinity entries represent matched node pairs across domains.
  • Matching Visualization: Each source node adaptively matches an optimal target node from the same category under the structure-aware matching loss.The matching produces graph-matching-based adaptation through same-category source–target node pairs.
  • Qualitative Comparison: SIGMA eliminates some false-negative cases and avoids some false-positive classifications compared with the class-agnostic EPM method.Figure 7 compares source-only, EPM, SIGMA, and ground-truth results for Cityscapes→Foggy Cityscapes adaptation.
Loading 2203.06398v3…