PaperlayerView exact revision on arXiv

Source-linked AI summary

Semi-Supervised Noise Adaptation: Transferring Knowledge from Noise Domain

Yuan Yao, Jin Song, Huixia Li, Tongtong Yuan, Jiaqi Wu, Yu Zhang

arXiv:2606.00558v2cs.LG

TL;DR

Semi-Supervised Noise Adaptation asks whether synthetic noise can support learning when target labels are scarce. The paper derives a generalization bound and proposes NAF, which consistently outperforms ERM on several benchmarks while tightening the target-domain bound.

  • Problem

    The paper formulates SSNA to study whether a synthetic noise domain can improve target-domain generalization when only a small proportion of target samples are labeled.

  • Method

    The paper derives a generalization bound and proposes NAF, which aligns target and noise samples in a shared representation space while minimizing empirical risks and distributional divergence.

  • Results

    NAF consistently outperforms ERM on several benchmarks and tightens the target-domain generalization bound, improving generalization performance.

  • Takeaways & Limitations

    The analysis suggests that noise domains can tighten target error bounds without requiring source and target domains to be semantically related.

  • Takeaways & Limitations

    SSNA assumes that labeled target samples are much fewer than unlabeled target samples.

Abstract

from arXiv · show

Transfer learning aims to facilitate the learning of a target domain by transferring knowledge from a source domain. The source domain typically contains semantically meaningful samples (*e.g.*, images) to facilitate effective knowledge transfer. However, a recent study observes that the noise domain constructed from simple distributions (*e.g.*, Gaussian distributions) can serve as a surrogate source domain in the semi-supervised setting, where only a small proportion of target samples are labeled while most remain unlabeled. Based on this surprising observation, we formulate a novel problem termed *Semi-Supervised Noise Adaptation* (SSNA), which aims to leverage a synthetic noise domain to improve the generalization of the target domain. To address this problem, we first establish a generalization bound characterizing the effect of the noise domain on generalization, based on which we propose a Noise Adaptation Framework (NAF). Extensive experiments demonstrate that NAF effectively leverages the noise domain to tighten the generalization bound of the target domain, leading to improved performance. The codes are available at https://github.com/AIResearch-Group/SSNA.

1. Introduction

The paper introduces Semi-Supervised Noise Adaptation (SSNA), using a synthetic noise domain as a surrogate source to improve target-domain generalization with limited labels. It derives a generalization bound and proposes Noise Adaptation Framework (NAF) to tighten that bound through shared representations and reduced distributional divergence.

  • Empirical findings: NAF outperforms ERM across CIFAR-10, CIFAR-100, DTD-47, Caltech-101, and ImageNet-1K using ResNet-18.These experiments demonstrate improved target-domain generalization from transferring knowledge from the noise domain.
  • Problem formulation: SSNA defines a target domain with few labeled and mostly unlabeled samples, alongside a randomly generated noise domain serving as a surrogate source.Noise classes lack semantic meaning and are mapped one-to-one to target classes.
  • Theoretical foundation: The paper establishes a generalization bound characterizing how the noise domain affects target-domain generalization.This bound provides the theoretical basis for the proposed adaptation method.
  • Method: NAF projects target samples and noise into a domain-shared representation space by minimizing both domains’ empirical risks and reducing distributional divergence.Optimizing NAF’s objective effectively tightens the target domain’s generalization bound.

2. Related Work

This section situates the work at the intersection of transfer learning and semi-supervised learning, highlighting noise as a potentially transferable source of knowledge. It presents the work as a domain-agnostic noise adaptation perspective supported by theory and benchmark evaluation.

  • Transfer Learning and Semi-Supervised Learning: Transfer learning uses abundant labeled source samples, whereas semi-supervised learning uses a few labeled target samples to improve target-domain generalization.Both approaches facilitate learning from unlabeled target samples.
  • Noise as Transferable Knowledge: Noise drawn from simple distributions may contain transferable knowledge in semi-supervised transfer learning when its discriminability and transferability are preserved.Although typically viewed as semantically meaningless, noise has also proved useful in diverse learning scenarios.
  • This Work: This work leverages domain-agnostic noise domains to facilitate learning from unlabeled target samples, supported by theoretical analysis and empirical evaluation on standard benchmarks.The authors frame this as a new perspective on semi-supervised transfer learning that complements existing studies.

3. Problem Formulation

Section 3 formulates SSNA around a semi-supervised target domain with few labeled samples, many unlabeled samples, and a synthetic noise domain whose labels are non-semantic class identifiers. The objective is to train a model using labeled and unlabeled target data together with noise, then evaluate it on held-out target samples.

  • Target Domain: The target domain is Dt = Dl ∪ Du ∪ De, comprising labeled, unlabeled, and test samples in the sample space X.Each labeled sample is associated with a semantic class mapped to an integer label.
  • Target Domain: The semi-supervised setting assumes substantially fewer labeled than unlabeled target samples, expressed as nl ≪ nu.The unlabeled and test subsets are included in Du and De, respectively.
  • Noise Domain: The noise domain Dn contains samples drawn from a random distribution over noise space E, with labels yi ∈ C serving only as integer identifiers without semantic meaning.The class index set is C = {0, . . . , C −1}, where C is the total number of classes.
  • SSNA Objective: SSNA trains a high-quality model hθ∗ using Dl, Du, and noise from Dn, then applies it to classify samples in De for evaluation.Under the SSNA setting, noise and target domains share the same class index set, and NAF projects both into a domain-shared representation space for classification.

4. Methodology

The methodology establishes an SSNA generalization bound and develops NAF to tighten it by jointly controlling target and noise errors and their discrepancy in a shared representation space. It also explains why labeled target samples are necessary, evaluates the framework empirically, and distinguishes SSNA from data augmentation.

  • Noise-domain construction: Noise and target domains share class indices, with one distinct Gaussian noise distribution assigned to each target class.The target has C classes, and the noise domain is constructed with C corresponding classes.
  • Semi-supervised alignment: A few labeled target samples align noise and target class indices, enabling transfer of discriminative structure; noise alone cannot effectively classify target samples.Noise is randomly generated from a different sample space and lacks an inherent relationship with target samples.
  • Generalization bound: Theorem 4.1 bounds target error using empirical noise error, empirical target error, and empirical H-divergence in a shared representation space, without a joint optimal error term.Because the domains have different sample spaces, both are projected into Z before their discrepancy is measured.
  • Noise Adaptation Framework: NAF minimizes labeled-target risk, noise risk, and projected-domain discrepancy through target and noise projectors, a classifier, and trade-off parameters α and β.These components correspond respectively to the three terms in the generalization bound and are optimized jointly to tighten it.
  • Empirical evaluation: The CIFAR-10 experiment uses 1024-dimensional Gaussian noise, 50 samples per class, ResNet-18, and 4 labeled samples per class to compare NAF with ERM.Representations learned by NAF form separated noise clusters aligned with corresponding target representations, producing a clearer discriminative structure than ERM.
  • Distinction from Data Augmentation: Unlike data augmentation, SSNA generates noise from simple distributions rather than enriching the target distribution through interpolation, transformations, or generative models.The distinction is conceptual: SSNA uses a potentially substantially different noise domain as an adaptation source.

5. Experiments

Experiments show that NAF consistently improves over ERM across standard, fine-grained, large-scale, and text classification benchmarks. Ablations further indicate that its gains depend on effective distributional alignment and class-discriminative structure in the noise domain.

  • Standard classification: NAF improves Top-1 accuracy over ERM on CIFAR-10 by 12.35% with ResNet-18 and 15.15% with ResNet-50.The comparison covers CIFAR-10, CIFAR-100, DTD-47, and Caltech-101.
  • Fine-grained classification: NAF consistently outperforms ERM by a large margin across CUB-200, OxfordFlowers-102, and StanfordCars-196.These fine-grained classification experiments use ResNet-18.
  • Large-scale classification: 37.10% accuracy: NAF outperforms ERM’s 36.11% by 0.99% on ImageNet-1K.The experiment uses ResNet-18 and evaluates a dataset with 1,000 classes.
  • Text categorization: 82.82% accuracy: NAF exceeds ERM’s 78.64% on AG News-4 using BERT.The result suggests that NAF can facilitate knowledge transfer in non-visual tasks.
  • Ablation studies: NAF outperforms all tested loss-ablation variants, while removing L_n performs better than removing L_n,t.The results indicate that both losses are beneficial, with reducing distributional divergence between domains being more crucial.
  • Noise-domain structure: 33.34% accuracy: NAF (SP) falls below ERM’s 58.15% on CIFAR-10, while CIFAR-100 results are 6.79% versus 42.24%.NAF (SP) assigns identical noise copies across classes, removing class-discriminative structure and causing negative transfer.

6. Conclusion … C. Additional Experimental Settings

The paper formulates Semi-Supervised Noise Adaptation, derives a target-domain generalization bound, and proposes NAF to exploit synthetic noise for semi-supervised learning. It situates NAF within transfer learning, semi-supervised learning, and several distribution-alignment instantiations.

  • 6. Conclusion: SSNA leverages a synthetic noise domain to facilitate learning in a target domain with limited labeled data.The paper formulates SSNA as a transfer-learning problem involving a noise source domain.
  • 6. Conclusion: The proposed NAF jointly minimizes empirical risks on noise and target domains while reducing their distributional discrepancy.NAF is built from a generalization bound characterizing how the noise domain affects target-domain generalization.
  • A. Related Work: Transfer learning improves target generalization by using labeled source samples and minimizing source–target distributional discrepancy.Prior work establishes theoretical foundations for transfer learning through target-domain generalization bounds and distribution-alignment methods.
  • A. Related Work: Semi-supervised learning uses a few labeled target samples to guide unlabeled samples through augmentation, consistency training, and pseudo-label refinement.UDA is cited as replacing simple noise injection with strong data augmentation to strengthen consistency training.
  • B. Mathematical Details of Distribution Alignment Mechanisms: NAF supports five loss instantiations: NDS, NSS, NCDS, NCSS, and EDD.These mechanisms provide alternative ways to define the distribution-alignment term L_n,t.
  • B. Mathematical Details of Distribution Alignment Mechanisms: NDS aligns global and class-wise means using negative cosine similarity, while NSS aligns same-class noise–target pairs using negative average cosine similarity.NDS averages similarities between global and class-wise means; NSS operates over all same-class noise–target pairs.
  • B. Mathematical Details of Distribution Alignment Mechanisms: NCDS contrasts class-wise means, NCSS regresses pairwise cosine similarities toward +1 for same-class and −1 for different-class pairs, and EDD averages Euclidean distances between domain means.Unlabeled target class assignments use hard pseudo-labels that are iteratively updated during training across the mechanisms.

C.1. Dataset Details

Experiments use a diverse benchmark suite spanning natural images, textures, objects, fine-grained categories, large-scale classification, and news-text classification. The datasets range from 5,640 texture images to 1.28 million training images.

  • Natural-image benchmarks: CIFAR-10 and CIFAR-100 each contain 60,000 natural images, with 50,000 training and 10,000 test images, across 10 and 100 classes, respectively.Both datasets use the same train/test counts but differ in class granularity.
  • Texture and object datasets: DTD-47 provides 5,640 texture images from 47 classes for texture classification, while Caltech-101 contains 9,146 images from 101 object classes plus a background class.Caltech-101 has varying numbers of images per class.
  • Fine-grained recognition datasets: CUB-200, Oxford Flowers-102, and Stanford Cars-196 cover fine-grained recognition with 11,788 bird images, 8,189 flower images, and 16,185 car images, respectively.Oxford Flowers-102 specifies 6,149 training, 1,020 validation, and 1,020 test images; Stanford Cars-196 uses 8,144 training and 8,041 test images.
  • Large-scale and text classification: ImageNet-1K supplies 1.28 million training and 50,000 validation images across 1,000 classes, while AG News-4 contains 120,000 training and 7,600 test text samples across 4 classes.ImageNet-1K follows standard large-scale image-classification splits, and AG News-4 supports news-text classification.

C.2. Implementation Details · D. Notation Table and Proof of Theorem 4.1 · D.1. Notation

The paper specifies NAF’s implementation, including augmentation, pretrained ResNet backbones, SGD optimization, and exponential-moving-average class means. It also defines the notation for target and noise domains, representations, distributions, samples, labels, and hypothesis spaces.

  • C.2. Implementation Details: NAF is implemented with TLlib on NVIDIA V100 GPUs, using weak and strong target-domain augmentation techniques.
  • C.2. Implementation Details: Image-classification representation extractors use ImageNet-1K-pretrained ResNet backbones, except on ImageNet-1K, where training starts from scratch.
  • C.2. Implementation Details: Optimization uses mini-batch SGD with momentum 0.9, while class means are updated with an exponential moving average to address mini-batch estimation.
  • C.2. Implementation Details: Table 10 summarizes detailed parameter configurations, with moderate α and β values in [1, 10] generally yielding stable performance.The text identifies α = 1 and β = 10 as frequently well-performing settings and reasonable starting points for limited tuning.
  • D.1. Notation: The notation table defines labeled, unlabeled, test, target, and noise sample sets, together with the target-domain composition Dt = Dl ∪ Du ∪ De.
  • D.1. Notation: The notation includes class counts and indices, sample labels, the image sample space X, the noise space E, and extractors gt(·) and gn(·).
  • D.1. Notation: Notation distinguishes the target and noise distributions, their labeled and unlabeled samples, and the shared representation space Z and hypothesis space F.

D.2. Proof of Theorem 4.1

The section states Theorem 4.1’s high-probability generalization-bound setup for SSNA and outlines its proof using domain-adaptation lemmas, symmetrization, and VC-dimension bounds.

  • Theorem statement: Theorem 4.1 bounds the target error of the empirical minimizer ˆf relative to the target-error minimizer f ∗ with probability at least 1 −δ.The probability is over the choice of samples, for any δ ∈(0, 1).
  • Theorem statement: The bound uses the empirical H-divergence estimated from noise and target samples in Z.The theorem’s bound also depends on sample- and complexity-related terms, although the supplied passage fragments do not show the complete expression.
  • Proof strategy: The proof applies Lemmas D.5 and D.6, with Lemma D.6 supplemented by sample symmetrization and VC-dimension–based growth-function bounds.Lemmas D.5 and D.6 correspond to Lemmas 4 and 5 of Ben-David et al. (2010).
  • Proof strategy: The derivation justifies its intermediate inequalities through empirical minimization, a Ben-David et al. (2010) bound, the definition of λ, and a Mohri et al. (2018) bound.Here λ := minf∈F ϵn(f) + ϵt(f).

E. Supplementary Experimental Results

Supplementary experiments on DTD-47 and Caltech-101 with ResNet-18 show that combining SOTA methods with NAF generally improves accuracy over standalone SOTA methods, supporting NAF’s effectiveness in leveraging noise-domain knowledge.

  • Additional Results: On DTD-47 and Caltech-101 using ResNet-18, SOTA + NAF consistently outperforms standalone SOTA methods across most scenarios.These supplementary results further demonstrate NAF’s effectiveness in leveraging the noise domain to enhance target-domain performance.
  • Additional Results: Table 12 reports accuracy (%) comparisons for DTD-47 and Caltech-101 using ResNet-18, with ∆ denoting the introduced performance gain.

F. Additional Analysis Experiments

Additional analyses show that NAF benefits from class-discriminative noise structure, remains stable across appropriate hyperparameter ranges, and can compete with alternative SSL modules and contrastive-learning baselines.

  • Noise Amount: With zero noise per class, NAF degenerates to ERM and performs poorly, while performance remains relatively stable as noise increases from 10 to 100.The analysis varies noise amounts of 0, 10, 50, 100, and 200 per class on CIFAR-100 with ResNet-18.
  • Hyperparameter Sensitivity: NAF is relatively insensitive to α and β within moderate ranges, specifically [1, 20] on CIFAR-100 and [0.1, 1] on CIFAR-10.Both sensitivity analyses use ResNet-18, although the stable ranges differ across datasets.
  • Inter-Class Distances: The inter-class-distance ablation controls class-mean separation with δ by sampling noise from N(µc, I), where µc = µ + δϵc.The study samples 50 Gaussian noise examples per class while varying δ.
  • SSL Plug-In Modules: NAF improves over LERM more under ERM, while both methods provide modest and comparable improvements when combined with DST.The comparison uses CIFAR-10 with ResNet-18 and evaluates ERM and DST combined with either LERM or NAF.
  • Contrastive Learning: Compared with contrastive baselines CL and SupCon, NAF may be more effective because CL omits pseudo-labels and SupCon relies on limited labeled samples for class-wise constraints.CL applies contrastive learning to weakly and strongly augmented unlabeled target samples, whereas SupCon constructs class-wise constraints from labeled data.

G. Limitations

NAF has practical limitations in hyperparameter selection and evaluation scope. Manual tuning may still be needed, and broader real-world applications remain future work.

  • Hyperparameter selection: NAF relies on heuristic selection of the hyperparameters α and β.The method is relatively insensitive over a moderate range, but adaptive or learnable parameter strategies are suggested for future work.
  • Hyperparameter selection: Manual tuning of α and β may still be needed in practice.This limitation remains despite NAF’s relative insensitivity to those values over a moderate range.
  • Evaluation scope: NAF’s current evaluation focuses on standard benchmarks rather than more realistic application scenarios.Extending NAF to settings such as recommendation systems is identified as a future direction.
Loading 2606.00558v2…