PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling

Qian Wang, Toby P. Breckon

arXiv:1911.07982v1cs.LGcs.CVcs.MMstat.ML

TL;DR

Unsupervised domain adaptation must classify unlabeled target samples despite source-target distribution differences, and inaccurate pseudo-labels can accumulate errors. This paper introduces structured-prediction-based selective pseudo-labeling with iterative subspace learning, outperforming state-of-the-art methods on four benchmark datasets, while noting excessive memory usage on some digit datasets.

  • Problem

    UDA lacks labeled target samples and faces source-target distribution differences, making inaccurate pseudo-labeling a risk during learning.

  • Method

    The paper combines structured prediction and progressive selective pseudo-labeling with SLPP-based domain-invariant subspace learning.

  • Results

    The proposed approach outperforms state-of-the-art methods on four benchmark datasets, achieving 93.0% average accuracy on Office-Caltech and 89.6% on Office31.

  • Takeaways & Limitations

    Selective pseudo-labeling and structured prediction are effective components for UDA and may also be used with deep learning models.

  • Takeaways & Limitations

    The approach is not suitable for digit datasets when the number of samples is too large because of excessive memory usage.

Abstract

from arXiv · show

Unsupervised domain adaptation aims to address the problem of classifying unlabeled samples from the target domain whilst labeled samples are only available from the source domain and the data distributions are different in these two domains. As a result, classifiers trained from labeled samples in the source domain suffer from significant performance drop when directly applied to the samples from the target domain. To address this issue, different approaches have been proposed to learn domain-invariant features or domain-specific classifiers. In either case, the lack of labeled samples in the target domain can be an issue which is usually overcome by pseudo-labeling. Inaccurate pseudo-labeling, however, could result in catastrophic error accumulation during learning. In this paper, we propose a novel selective pseudo-labeling strategy based on structured prediction. The idea of structured prediction is inspired by the fact that samples in the target domain are well clustered within the deep feature space so that unsupervised clustering analysis can be used to facilitate accurate pseudo-labeling. Experimental results on four datasets (i.e. Office-Caltech, Office31, ImageCLEF-DA and Office-Home) validate our approach outperforms contemporary state-of-the-art methods.

Introduction

The paper addresses inaccurate target-domain pseudo-labeling by using target structure in a selective, structured-prediction framework. It combines this strategy with SLPP-based subspace learning and reports state-of-the-art performance across four benchmark datasets.

  • Motivation: UDA methods can pseudo-label target samples for supervised learning, but independently labeling them overlooks target-domain structure.The target data are available during training, making UDA transductive; pseudo-label accuracy affects subsequent learning.
  • Method: The proposed approach explores target structure with K-means and structured prediction to improve pseudo-label accuracy and domain alignment.It generates target clusters, matches them with source classes, and uses structured prediction for pseudo-labeling.
  • Method: The method learns a domain-invariant subspace with Supervised Locality Preserving Projection using labeled source data and pseudo-labeled target data.Selective pseudo-labeling is integrated into an iterative learning algorithm based on SLPP subspace learning.
  • Method: Structured prediction is used to exploit structural information within the target domain for pseudo-labeling.This is presented as a contribution intended to promote pseudo-labeling accuracy and domain alignment.
  • Evaluation: The approach is evaluated through comparative experiments and ablation studies on four benchmark datasets.The experiments are reported as demonstrating new state-of-the-art performance.

Related Work

Related UDA methods align domain distributions or use pseudo-labeling to address missing target labels. The paper distinguishes hard, soft, and selective pseudo-labeling, emphasizing confidence and progressive target-sample inclusion.

  • Domain Alignment: Marginal-distribution alignment alone does not guarantee good classification when source and target conditional distributions are misaligned.This issue is linked to the lack of labeled target samples.
  • Pseudo-Labeling without Selection: Hard pseudo-labeling assigns labels to all target samples without confidence consideration, whereas soft labeling assigns class-conditional probabilities.Soft labels can be updated during iterative learning and support class-specific discriminator weighting.
  • Pseudo-Labeling without Selection: Early hard pseudo-labeling errors from a weak classifier can seriously harm subsequent iterative learning.Soft labeling was introduced as one response to this issue.
  • Pseudo-Labeling with Selection: Selective pseudo-labeling combines only a confidence-selected subset of target samples with source data in the next learning iteration.The selected subset grows as the classifier improves, following an easy-to-hard progression.
  • Pseudo-Labeling with Selection: Sample-selection criteria are central to selective pseudo-labeling, including thresholds that are updated across iterations.An easy-to-hard strategy selects samples whose similarity scores exceed an iteratively updated threshold.

Proposed Method

The method alternates learning a domain-aligned subspace with pseudo-labeling target samples. It combines nearest class prototypes, target-domain structure, and selective sampling to improve pseudo-label quality.

  • Pseudo-Labeling via Nearest Class Prototype: Nearest class prototype assigns target labels using distances to source-class prototype means in the learned subspace.Source prototypes are computed by averaging projected source samples sharing each class label.
  • Dimensionality Reduction: PCA reduces high-dimensional source and target features before SLPP learns the domain-aligned representation.The PCA projection produces lower-dimensional features, followed by L2 normalization.
  • Domain Alignment: SLPP learns a latent subspace where samples from the same class are close regardless of domain.The projection uses labeled source data and selected pseudo-labeled target data.
  • Pseudo-Labeling via Structured Prediction: Structured prediction uses K-means target clusters initialized by class prototypes and one-to-one cluster-class matching that minimizes paired distances.The matching matrix is optimized by linear programming.
  • Iterative Selective Learning: The algorithm iteratively updates the projection and pseudo-labels, selecting increasingly many target samples for subsequent projection learning.Selection is performed class-wise from high-probability pseudo-labels to avoid overlooking classes.

2: Dimensionality reduction

The learning process combines complementary pseudo-labeling strategies and progressively selects target samples for iterative projection updates. Class-wise selection limits the risk of concentrating selected samples in only a few classes.

  • Iterative Learning: The algorithm starts with source-only projection learning, labels all target samples, then repeatedly selects samples and updates the projection and labels.Each iteration learns the projection from labeled source data and the selected pseudo-labeled target subset.
  • Pseudo-Labeling Strategies: Nearest-prototype labeling favors target samples close to source data, whereas structured prediction favors samples near target-domain cluster centers.The two methods provide different confidence patterns for selecting pseudo-labeled target samples.
  • Selective Pseudo-Labeling: Class-wise selection chooses high-probability samples from each pseudo-labeled class rather than selecting only the globally highest-probability samples.This gives target samples assigned to each class the same opportunity for selection.
  • Complexity: The iterative SLPP computation has approximate complexity O(T(2d1n^2 + d1^3)), so smaller d1 can improve efficiency when n is not too large.The iterative learning stage contributes most of the computation cost.

Experiments and Results

Experiments across four benchmark datasets compare SPL with state-of-the-art methods and analyze its components and hyper-parameters. SPL achieves the highest average accuracy consistently across all four datasets.

  • Experimental Setup: Experiments use Office-Caltech, Office31, ImageCLEF-DA, and Office-Home, with Decaf6 or ResNet50 features for comparison.Office-Caltech uses Decaf6 features, while the other datasets use ResNet50 features or ResNet50-based deep models.
  • Comparison with State-of-the-Art Approaches: SPL achieves the highest average accuracy consistently across Office-Caltech, Office31, ImageCLEF-DA, and Office-Home.The comparison combines two pseudo-labeling methods and reports average accuracy over source-to-target tasks.
  • Comparison with State-of-the-Art Approaches: 93.0% average accuracy on Office-Caltech slightly exceeds MEDA's 92.8%.The result is reported for the Office-Caltech benchmark in Table 1.
  • Comparison with State-of-the-Art Approaches: 89.6% average accuracy on Office31 is best, while SPL is best or second-best in five of six tasks.The second-highest average accuracy is 88.4%, achieved by SymNets and another deep-learning approach.
  • Comparison with State-of-the-Art Approaches: 90.5% average accuracy on ImageCLEF-DA ranks first, with SPL best or second-best in four of six tasks.SymNets reaches 89.9% and MEDA 89.0% on the reported average comparison.
  • Comparison with State-of-the-Art Approaches: 71.0% average accuracy on Office-Home exceeds CAPLS's 70.6% and all other state-of-the-art models.The Office-Home comparison is reported in Table 4.
  • Ablation Study: Ablation results show pseudo-labeling improves performance, selective pseudo-labeling further improves it, and structured prediction consistently outperforms nearest class prototypes.Combining structured prediction and nearest class prototypes provides a further marginal accuracy improvement.
  • Hyper-parameter Analysis: Datasets with more classes tend to benefit from larger PCA dimensionality d1, whereas smaller d1 values benefit datasets with fewer classes.The study varies PCA dimensionality d1, SLPP dimensionality d2, and iteration count T while reporting average accuracy across source-target pairs.

Conclusion

The paper concludes that its iterative selective pseudo-labeling approach combines supervised subspace learning with structured prediction and outperforms state-of-the-art methods on four benchmark datasets. Its ablation study supports selective pseudo-labeling and structured prediction, including their possible use with deep UDA models.

  • Conclusion: The proposed approach combines supervised subspace learning and structured prediction-based pseudo-labeling in an iterative learning framework.The framework is presented as a selective pseudo-labeling approach for unsupervised domain adaptation.
  • Conclusion: The approach outperforms other state-of-the-art methods on four benchmark datasets.This conclusion summarizes the reported experimental comparisons.
  • Conclusion: Ablation results demonstrate the effectiveness of selective pseudo-labeling and structured prediction.The paper states that these components could also be employed to train deep learning models for UDA in future work.

Supplementary Material

The supplementary material visualizes domain alignment and compares structured prediction with nearest class prototypes during iterative learning. It also reports the method’s dataset scope and practical runtime context.

  • Visualisation of Domain Alignment: The learned subspace promotes class separability while aligning both marginal and conditional distributions on Office-Home.The visualization uses 20 randomly selected classes and t-SNE to compare original deep features with the learned subspace.
  • Visualisation of Domain Alignment: Deep features pretrained on ImageNet already provide useful data distributions before domain adaptation.This observation is supported by source-only 1NN and SVM baseline results reported in Table 6.
  • Learning Process of NCP and SP: With more iterations and pseudo-labeled target samples, classification performance improves gradually for both SP and NCP.SP performs better at the beginning and maintains better final performance than NCP, attributed to exploiting target-domain structural information.
  • Dataset Scope and Limitations: The experiments use four widely adopted UDA datasets, while the method is unsuitable for Digit datasets because of excessive memory usage.The authors identify this memory limitation as shared by many feature-transformation methods and a direction for future work.
  • Practical Runtime: On a laptop, total runtime across all sub-tasks is 18, 83, 10, and 2070 seconds for Office-Caltech, Office31, ImageCLEF-DA, and Office-Home, respectively.The reported hardware is an Intel i5-7300 CPU with 32G RAM.
Loading 1911.07982v1…