PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Adversarial Attention Alignment for Unsupervised Domain Adaptation: the Benefit of Target Expectation Maximization

Guoliang Kang, Liang Zheng, Yan Yan, Yi Yang

arXiv:1801.10068v4cs.CV

TL;DR

UDA must adapt a source-trained model to an unlabeled target domain, while high-level alignment may leave convolutional discrepancy unaddressed. This paper aligns convolutional attention and trains the target network with modified EM using category distributions. It reports competitive or state-of-the-art results on two benchmark datasets, including a +2.6% average improvement on Office-31.

  • Problem

    UDA adapts source-trained models to unlabeled target domains, but high-level alignment may leave convolutional layers under-constrained despite discrepancy emerging there.

    Source: §1 · pp. 1–2

  • Method

    The method aligns source and target attention in convolutional layers and uses modified EM with category distributions to train on unlabeled target data.

    Source: §1 · pp. 2–3

  • Results

    +2.6% average improvement over the state of the art on Office-31; the two contributions achieve competitive or state-of-the-art results on two benchmark datasets.

    Source: §1 · p. 3, §5 · p. 14

  • Takeaways & Limitations

    The paper concludes that attention alignment and maximizing unlabeled target-data likelihood both benefit UDA performance and cooperate in the system.

    Source: §5 · p. 14

    The EM formulation assumes uniform category and image-manifold distributions, and assumes translated source images retain the target image’s label posterior.

    Source: §3.3 · pp. 8–9

Abstract

from arXiv · show

In this paper, we make two contributions to unsupervised domain adaptation (UDA) using the convolutional neural network (CNN). First, our approach transfers knowledge in all the convolutional layers through attention alignment. Most previous methods align high-level representations, e.g., activations of the fully connected (FC) layers. In these methods, however, the convolutional layers which underpin critical low-level domain knowledge cannot be updated directly towards reducing domain discrepancy. Specifically, we assume that the discriminative regions in an image are relatively invariant to image style changes. Based on this assumption, we propose an attention alignment scheme on all the target convolutional layers to uncover the knowledge shared by the source domain. Second, we estimate the posterior label distribution of the unlabeled data for target network training. Previous methods, which iteratively update the pseudo labels by the target network and refine the target network by the updated pseudo labels, are vulnerable to label estimation errors. Instead, our approach uses category distribution to calculate the cross-entropy loss for training, thereby ameliorating the error accumulation of the estimated labels. The two contributions allow our approach to outperform the state-of-the-art methods by +2.6% on the Office-31 dataset.

1 Introduction

The introduction argues that target-domain shift can leave convolutional layers under-constrained when adaptation focuses on high-level representations. It proposes aligning source and target attention and using modified EM training with unlabeled target data.

1 Introduction

  • High-level alignment may leave convolutional layers under-constrained, even though domain discrepancy can emerge there.The paper notes that modifications to fully connected layers may decay through back-propagation, especially with gradient explosion or vanishing.Source: §1 · p. 2
  • Source- and target-trained networks show distinct attention patterns on the same target image, motivating attention invariance across domain shift.The paper observes that source-network attention degenerates when applied directly to target-domain data.Source: §1 · p. 2
  • The method aligns target-network attention with source-network attention, assuming discriminative image regions are insensitive to style changes.CycleGAN-generated correspondences pair real and translated images, and attention distances are penalized across the source and target networks.Source: §1 · pp. 2–3
  • Modified EM training uses category distributions for unlabeled target data while ground-truth labels supervise source and translated-source data.The paper says training iterations improve both target label-posterior estimation and the model’s discriminative ability.Source: §1 · p. 3
  • +2.6% average improvement over the state of the art on Office-31 is reported across the paper’s evaluation.The introduction describes the improvement as an average on the real-world domain adaptation dataset Office-31.Source: §1 · p. 3

2 Related Work

Related work centers on domain-invariant representations at network tails, CNN attention definitions and transfer, and unpaired image translation. The paper distinguishes its approach by constraining convolutional attention across domains and using CycleGAN to form data correspondences.

Unsupervised domain adaptation

  • Many UDA methods align tail representations, while few address convolutional domain shift; this paper constrains convolutional layers with cross-domain attention alignment.Examples include domain confusion [26], MMD, adversarial domain classification, JMMD, DSN, and ADA.Source: §2 · pp. 3–4

Attention of CNNs

  • CNN attention work includes deconvnet, input-gradient visualization [22], CAM, and Grad-CAM methods for visualizing network focus.Grad-CAM combines gradient-based attention with CAM without modifying the original network structure as CAM does.Source: §2 · p. 4
  • Zagoruyko et al. transfer attention across architectures for the same image, whereas this method aligns attention across domains with unpaired images.Their method aims to transfer knowledge from a large model to a smaller one; this method aims to promote adaptation performance.Source: §2 · p. 4

Unpaired image-to-image translation

  • Unpaired image translation uses GANs to create cross-domain correspondences, and this paper chooses CycleGAN to pair real and synthetic samples.The paper focuses on maximizing the utility of the resulting paired samples.Source: §2 · p. 4

3 Deep Adversarial Attention Alignment

The framework adapts a target CNN using CycleGAN-paired real and synthetic images, convolutional attention alignment, and EM training on unlabeled target data. Attention alignment and EM work together to adapt the target network to domain shift.

3 Deep Adversarial Attention Alignment

  • The target CNN uses cross-entropy on labeled source and synthetic-target data, and EM on unlabeled target and synthetic-source data, with a fixed source network guiding attention alignment.The source and target networks share the same architecture.Source: §3 · p. 4, Figure 2 · p. 5

3.1 Adversarial Data Pairing

  • CycleGAN translates source and target images in both directions, pairing each real image with a translated counterpart under the assumption that both domains share object categories.The paired images provide correspondences for attention alignment.Source: §3.1 · pp. 4–5, Figure 3 · p. 6

3.2 Attention Alignment

  • The method represents convolutional layers with channel-aggregated spatial attention maps and penalizes attention distances across source and target networks for paired images.The L2-norm aggregation emphasizes salient parts of feature maps.Source: §3.2 · p. 6

3.3 Training with EM

  • EM maximizes target-data likelihood using hidden categories and posterior estimates; asynchronous updates, confidence filtering, and learning-rate resets modify the procedure, whose modifications consistently improve basic EM.For synthetic source images translated from target data, the method assumes their category posterior matches that of the real target image.Source: §3.3 · pp. 8–9

3.4 Deep Adversarial Attention Alignment

  • Attention alignment encourages focus on discriminative information, while EM estimates target label distributions; adversarial data pairing provides correspondences for alignment.The full target-network objective combines cross-entropy, the EM loss, and a weighted attention-alignment penalty.Source: §3.4 · pp. 9–10

4 Experiment

The experiments evaluate adaptation on digit and real-world domain benchmarks, with implementation choices and ablations examining attention alignment and EM training. The method improves Office-31 results over prior state of the art and performs best among the tested attention-discrepancy measures.

4.1 Setup

  • The experiments use MNIST →MNIST-M and six Office-31 transfers; Office-31 contains 4,110 images across 31 categories and three domains.The Office-31 transfers are A →W, D →W, W →D, A →D, D →A, and W →A.Source: §4.1 · p. 10

4.2 Implementation details

  • On Office-31, the ResNet-50 source model is fixed to guide the target model’s attention alignment across all convolutional layers except the first.For MNIST →MNIST-M, attention is aligned on the first three of four convolutional layers.Source: §4.2 · p. 11

4.3 Evaluation

  • On MNIST →MNIST-M, the method reaches 95.6% accuracy, compared with PixelDA’s reported 98.2%; adding source data to synthetic target data improves accuracy by +3.5%.The source network drops from 99.3% on its source test set to 45.6% on the target test set.Source: §4.3 · p. 11
  • +2.6% average improvement over the state of the art on Office-31, including +5.1% on the difficult D →A task.The method outperforms the state of the art across all six transfer tasks; gains are larger on harder tasks such as D →A and W →A.Source: §4.3 · p. 12, Table 2 · p. 12
  • On harder Office-31 transfers, LAT improves results by +2% for D →A and +1.7% for W →A; with LAT, target accuracy rises steadily and converges faster.The authors associate the accuracy improvement with smaller attention-map discrepancy between source and target models.Source: Figure 4 · p. 12, §4.3 · pp. 13–14

4.4 Ablation Study

  • In the EM ablation, all three modifications contribute, noisy-data filtering matters most, and LAT improves EM-A and EM-C.Table 3 compares EM variants with and without LAT; EM-A omits asynchronous updating, EM-B omits noisy-data filtering, and EM-C omits learning-rate schedule initialization during M post updates.Source: Table 3 · p. 13, §4.4 · p. 14

4.5 Comparing Different Attention Discrepancy Measures

  • The L2 attention-discrepancy measure performs best among four tested measures; L1 fails to train a workable network, while MMD and JMMD perform worse.The paper attributes L2’s advantage over MMD/JMMD to preserving structure information.Source: Table 4 · p. 13, §4.5 · p. 14

5 Conclusion

The paper combines convolutional attention alignment with an EM approach to maximize the likelihood of unlabeled target data. Experiments report positive effects from both contributions and competitive or state-of-the-art results on two benchmark datasets.

5 Conclusion

  • The combined method achieves competitive or state-of-the-art results on two benchmark datasets, with both contributions positively affecting performance.Both contributions benefit from unsupervised image correspondences provided by CycleGAN.Source: §5 · p. 14
  • Attention maps from source and target convolutional layers are aligned to adapt source-network knowledge to the target network.Source: §5 · p. 14
  • From an EM perspective, the method maximizes the likelihood of unlabeled target data, enabling the target network to leverage more training data.Source: §5 · p. 14

Supplementary Material for Paper: Deep Adversarial Attention Alignment for

The supplied supplementary-material passages identify the paper and its authors and affiliation.

Supplementary Material

1 Training Details

Training details specify data proportions and EM filtering thresholds for MNIST→MNIST-M and Office-31. For Office-31 ResNet-50, attention alignment omits the first convolutional and max-pooling layers.

1 Training Details

  • For both datasets, mini-batches use proportions 0.35, 0.15, 0.35, and 0.15 for real source, synthetic target, real target, and synthetic source data, respectively.Source: §1 · p. 17
  • MNIST→MNIST-M uses Adam with an initial learning rate of 0.001; EM filtering starts at pt = 1 before the first posterior update, then uses pt = 0.95.Source: §1 · p. 17
  • Office-31 uses pt = 0.95 and selects β through validation; its learning-rate schedule follows but restarts after each posterior-estimation update.Source: §1 · p. 17
  • For Office-31 ResNet-50, attention alignment omits the first convolutional and max-pooling layers; the authors report no accuracy loss and greater computational efficiency.The first layer focuses on noise-sensitive low-level details, while max-pooling has no parameters.Source: §1 · p. 17

2 Impact of Hyper-parameters

The hyperparameter study examines filtering threshold pt and attention-alignment strength β on Office-31 task D→A. Within broad ranges, the method outperforms the previous state of the art by a large margin.

2 Impact of Hyper-parameters

  • The study varies pt, the EM filtering threshold, and β, the attention-alignment penalty strength.Source: §2 · p. 17
  • Figure A1 plots target-model accuracy against pt and β for Office-31 task D→A, with similar trends reported for other tasks.Source: Figure A1 · p. 18
  • Within a wide range of pt and β values, the method outperforms the previous state-of-the-art method by a large margin.A larger pt improves accuracy within a range; accuracy rises with β before decreasing slightly.Source: §2 · pp. 17–18

3 Comparison with different variants of attention

The section compares channel-aggregated attention using L2-, L1-, and L∞-norms, as well as unaggregated feature-map alignment. The proposed attention performs much better than these alternatives.

3 Comparison with different variants of attention

  • The proposed attention performs much better than L1- and L∞-norm aggregation and direct feature-map alignment.The direct feature-map comparison is denoted “FM.”Source: §3 · p. 18
  • The proposed attention aggregates feature maps along channels using the L2-norm.Source: §3 · p. 18
  • The comparison is presented as a test of the effectiveness of the attention defined by Eq. (4).Source: §3 · p. 18

Related papers

Loading 1801.10068v4…