PaperlayerView exact revision on arXiv

Source-linked AI summary

DINE: Domain Adaptation from Single and Multiple Black-box Predictors

Jian Liang, Dapeng Hu, Jiashi Feng, Ran He

arXiv:2104.01539v3cs.CVcs.AIcs.LG

TL;DR

UDA transfers knowledge from labeled source domains to unlabeled targets, but existing approaches may require source data or detailed white-box models. DINE adapts from black-box predictions through structural distillation and target-data fine-tuning. It reports competitive or state-of-the-art results across UDA scenarios, including strong performance with lightweight target models.

  • Problem

    Many UDA methods require source data, while white-box model adaptation may expose source information and assumes access to model details.

    Source: §1 · p. 1

  • Method

    DINE distills black-box source predictions into a target model using adaptive label smoothing and structural regularization, then fine-tunes it on target data.

    Source: §1 · p. 2

  • Results

    DINE reports state-of-the-art adaptation results across single-source, multi-source, and partial-set UDA with strong pretrained source models; its mean accuracy is 91.9% on Office.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    DINE can adapt with different network architectures and a small efficient target model when provided strong source predictors.

    Source: §1 · p. 2, §5 · p. 8

    The assumed label spaces are identical for single- and multi-source UDA, while partial-set UDA assumes the source label space subsumes the target label space.

    Source: §3 · p. 3

Abstract

from arXiv · show

To ease the burden of labeling, unsupervised domain adaptation (UDA) aims to transfer knowledge in previous and related labeled datasets (sources) to a new unlabeled dataset (target). Despite impressive progress, prior methods always need to access the raw source data and develop data-dependent alignment approaches to recognize the target samples in a transductive learning manner, which may raise privacy concerns from source individuals. Several recent studies resort to an alternative solution by exploiting the well-trained white-box model from the source domain, yet, it may still leak the raw data through generative adversarial learning. This paper studies a practical and interesting setting for UDA, where only black-box source models (i.e., only network predictions are available) are provided during adaptation in the target domain. To solve this problem, we propose a new two-step knowledge adaptation framework called DIstill and fine-tuNE (DINE). Taking into consideration the target data structure, DINE first distills the knowledge from the source predictor to a customized target model, then fine-tunes the distilled model to further fit the target domain. Besides, neural networks are not required to be identical across domains in DINE, even allowing effective adaptation on a low-resource device. Empirical results on three UDA scenarios (i.e., single-source, multi-source, and partial-set) confirm that DINE achieves highly competitive performance compared to state-of-the-art data-dependent approaches. Code is available at \url{https://github.com/tim-learn/DINE/}.

1. Introduction

Unsupervised domain adaptation transfers knowledge from labeled source datasets to unlabeled targets, but conventional methods rely on source data and white-box approaches can expose source information. DINE adapts from black-box predictions through structural distillation and fine-tuning.

Motivation and problem setting

  • DINE addresses adaptation when source data and model details are unavailable, using only black-box source predictors for unlabeled target data.This setting also allows different network architectures across domains.Source: §1 · pp. 1–2, Figure 1 · p. 1

Proposed approach

  • DINE first distills source predictions, then fine-tunes the distilled model on target data; adaptive label smoothing handles incomplete API outputs.The smoothing keeps the largest softmax value and assigns equal values to the remaining classes.Source: §1 · p. 2
  • Structural distillation adds interpolation consistency and mutual-information maximization to point-wise supervision, while fine-tuning reuses mutual information on target data alone.For multiple sources, DINE aggregates outputs from multiple predictors.Source: §1 · p. 2

Contributions

  • Empirical results across benchmarks are reported to outperform baselines, with state-of-the-art performance claimed for single-source, multi-source, and partial-set UDA using large source predictors.The introduction identifies these three UDA scenarios as evaluation settings.Source: §1 · p. 2
  • DINE is presented as avoiding source-data access and enabling lightweight target models without a shared architecture; it also avoids adversarial training and data synthesis.The paper links avoiding adversarial training and data synthesis to faster convergence.Source: §1 · p. 2

2. Related Work

Related work covers conventional UDA alignment, source-data-free model adaptation, and knowledge distillation. DINE is positioned as a black-box approach that brings structural regularization into unsupervised distillation.

Domain Adaptation

  • Deep UDA commonly uses domain-adversarial learning or discrepancy minimization, while other methods regularize network outputs or use domain-specific normalization and feature regularization.The paper evaluates single-source, multi-source, and partial-set UDA, where the source label space subsumes the target label space in the partial-set case.Source: §2 · p. 2

Model Transfer

  • Source-data-free methods adapt from trained source models, but some require model details or generated images; black-box approaches include label-shift estimation and noisy-label learning.DINE focuses on black-box covariate shift and can use hard labels from source predictors.Source: §2 · p. 2

Knowledge Distillation

  • Knowledge distillation transfers predictions from teacher to student, and prior self-distillation uses past predictions or models as teachers; DINE adds adaptive smoothing and structural regularization to unsupervised distillation.The figure overview depicts a memory bank as an EMA-prediction teacher, with structural regularizations during adaptation.Source: §2 · p. 3, Figure 2 · p. 3

3. Methodology

DINE adapts in a setting where target-domain learning uses only black-box source-model predictions, without access to source data or source-model details. It distills source knowledge with target-structure regularization, then fine-tunes the target model using target data alone.

3. Methodology

  • DINE uses only black-box source predictors for target adaptation, without requiring source data or source-model details; single- and multi-source settings assume shared label spaces, unlike partial-set UDA.The task is K-way cross-domain image classification.Source: §3 · p. 3

3.1. Architecture and Source Model Generation

  • The source model uses a backbone with a single linear layer and label smoothing, while the target model has a bottleneck and weight-normalized classifier.The source label-smoothing parameter is set to 0.1.Source: §3.1 · p. 3

3.2. Adaptive Self-Knowledge Distillation

  • Adaptive label smoothing retains each prediction’s top-r class values and redistributes the remaining probability uniformly; self-distillation maintains an EMA teacher prediction.For r = 1, the smoothed output requires only the predicted class and its maximum probability, a format the paper describes as more flexible for API use.Source: §3.2 · p. 4

3.3. Distillation with Structural Regularizations

  • Structural distillation adds MixUp interpolation consistency and mutual-information maximization to address noisy point-wise source predictions and encourage diverse target predictions.The MixUp term uses interpolated target samples; the paper contrasts this unified prediction-learning approach with iterative pseudo-label refinement [85].Source: §3.3 · p. 4

3.4. Fine-tuning the Distilled Model

  • Fine-tuning then refines the distilled target model using mutual-information maximization alone, rather than parameter-sensitive virtual adversarial training.The paper motivates this second phase as a way to reduce target-side cluster-assumption violations.Source: §3.4 · p. 5

4. Experiments

The experiments evaluate DINE across single-source, partial-set, and multi-source UDA, then analyze its components, parameter sensitivity, and small target backbones. Across the reported settings, DINE is competitive with or outperforms the compared methods, including strong results with ViT source predictors.

4.1. Setup

  • Experiments evaluate Office, Office-Home, VisDA-C, Office-Caltech, and ImageCLEF, reporting average accuracies from three runs with random seeds 2019, 2020, and 2021.The paper fixes β = 1.0 and γ = 0.6 throughout.Source: §4.1 · p. 5

4.2. Results

  • With ViT source predictors, DINE achieves the best mean accuracy of 91.9% on Office and 80.7% on Office-Home, beating state-of-the-art model-based and data-based methods by clear margins.On VisDA-C, DINE also outperforms the other baselines and beats ViT-based TransDA [80].Source: Table 1 · p. 5, §4.2 · pp. 6–7, Table 2 · p. 6
  • In partial-set UDA, DINE consistently beats HD-SHOT and SD-SHOT, and with a ViT source predictor reaches the highest average accuracy of 83.0%.DINE also outperforms DINE (full), while NLL-KL and NLL-OT underperform No Adapt. under asymmetric label spaces.Source: §4.2 · p. 7, Table 4 · p. 7
  • Across four multi-source datasets, DINE beats No Adapt. and DINE (w/o FT), is competitive with DINE (full), and with a ViT source predictor beats prior data-based and model-based methods.The authors attribute its gains over No Adapt. and DINE (w/o FT) to fine-tuning and structural distillation.Source: §4.2 · pp. 7–8, Table 5 · p. 7

4.3. Analysis

  • Ablations show that removing either Lim or Lmix reduces performance across all three tasks, and the FT step improves variants in both tables.AdaLS (r = 1) outperforms one-hot and vanilla label smoothing with or without FT, while r = 3 can outperform full-vector DINE.Source: §4.3 · p. 8, Table 6 · p. 8, Table 7 · p. 8
  • Results are stable around β = 1.0 and γ = 0.6, and accuracies increase and converge during both DINE steps.The authors note that β = 2.0 and γ = 0.8 may perform better under oracle validation.Source: §4.3 · p. 8, Figure 3 · p. 8

4.4. Discussion & Limitation

  • ResNet-18 and MobileNet-v2 target backbones achieve competitive performance when paired with a strong ViT source predictor.The paper does not cover UDA with unknown target classes, leaving that setting for future investigation.Source: §4.4 · p. 8, Figure 3 · p. 8

5. Conclusion

DINE addresses UDA where source domains provide only black-box predictors, using a two-step distillation and fine-tuning framework that considers target data structure. Experiments verify its performance across UDA tasks, including state-of-the-art adaptation results with strong pretrained source models and a small efficient target model.

5. Conclusion

  • DINE targets a UDA setting in which each source provides only a black-box predictor, while allowing different networks across domains.The framework is called Distill and Fine-tune (DINE).Source: §5 · p. 8
  • DINE uses two-step structural distillation, refining noisy teacher outputs with adaptive smoothing and considering target-domain data structure.The distilled model is then fine-tuned to fit the target domain.Source: §5 · p. 8
  • Experiments across multiple datasets verify DINE's superiority over baselines for various UDA tasks.With strong pretrained source models, DINE achieves state-of-the-art adaptation results using a small efficient target model.Source: §5 · p. 8

Pseudo code

The pseudocode generates a source model, distills its predictions using target data, and then fine-tunes the distilled target model.

Pseudo code

  • The source-model generation stage trains the source predictor by minimizing the objective in Eq. (1).It requires source data and a number of epochs.Source: Pseudo code · p. 9
  • During distillation, each target-data batch is MixUp-augmented, ft is updated by minimizing Eq. (8), and the teacher output fs(xt) is updated via Eq. (5).Source: Pseudo code · p. 9
  • The final stage fine-tunes the distilled target model by maximizing the objective in Eq. (7).It uses target data and a specified number of epochs.Source: Pseudo code · p. 9

Related papers

Loading 2104.01539v3…