PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation by Mixture of Alignments of Second- or Higher-Order Scatter Tensors

Piotr Koniusz, Yusuf Tas, Fatih Porikli

arXiv:1611.08195v2cs.CV

TL;DR

Domain adaptation asks how to transfer knowledge to a target domain when only a few labeled target examples are available. So-HoT aligns class-specific scatter statistics in source and target CNN streams while maintaining class separation. The authors report state-of-the-art results and improved results and computational efficiency from weighted alignment and kernelization.

  • Problem

    The paper addresses supervised domain adaptation when only a few labeled target training samples per category are available.

    Source: §1 · p. 1

  • Method

    So-HoT trains source and target CNN streams end to end, aligning same-class second- or higher-order scatter tensors while maintaining between-class separation.

    Source: §1 · p. 1, §4.1 · p. 6

  • Results

    The authors report state-of-the-art results; their weighted approach and kernelized alignment loss improve results and computational efficiency.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The reported results suggest that partial alignment of within-class scatters is an effective strategy for domain adaptation.

    Source: §6 · p. 10

    The study investigates deep, supervised, single-source domain adaptation and assumes CNN streams can rotate within-class scatters sufficiently to yield a desired overlap.

    Source: §2 · p. 3, §4.1 · p. 6

Abstract

from arXiv · show

In this paper, we propose an approach to the domain adaptation, dubbed Second- or Higher-order Transfer of Knowledge (So-HoT), based on the mixture of alignments of second- or higher-order scatter statistics between the source and target domains. The human ability to learn from few labeled samples is a recurring motivation in the literature for domain adaptation. Towards this end, we investigate the supervised target scenario for which few labeled target training samples per category exist. Specifically, we utilize two CNN streams: the source and target networks fused at the classifier level. Features from the fully connected layers fc7 of each network are used to compute second- or even higher-order scatter tensors; one per network stream per class. As the source and target distributions are somewhat different despite being related, we align the scatters of the two network streams of the same class (within-class scatters) to a desired degree with our bespoke loss while maintaining good separation of the between-class scatters. We train the entire network in end-to-end fashion. We provide evaluations on the standard Office benchmark (visual domains), RGB-D combined with Caltech256 (depth-to-rgb transfer) and Pascal VOC2007 combined with the TU Berlin dataset (image-to-sketch transfer). We attain state-of-the-art results.

1 Introduction

The paper frames domain adaptation as transferring knowledge from related source tasks to targets with few labeled samples. So-HoT aligns class-specific scatter statistics between source and target networks while retaining class separation, with weighted and kernelized variants.

1 Introduction

  • Domain adaptation asks how to identify commonality between a new target task and prior source tasks, motivated in part by learning from few samples.The paper frames transfer learning as storing knowledge from one problem and applying it to a different but related problem.Source: §1 · p. 1
  • So-HoT defines source–target commonality through a mixture of second- and/or higher-order class-specific scatter alignments.It computes one scatter tensor per class and network stream from fc7 features.Source: §1 · pp. 1–2
  • The alignment loss brings same-class scatters closer while maintaining separation between different-class scatters.The loss matches scatter covariances and corresponding means, and the networks are trained end to end.Source: §1 · p. 2
  • Weighted alignment learns a class-specific degree of alignment, while a kernelized variant provides computational speed-ups for typical domain-adaptation datasets.The paper also proposes an unweighted, class-independent alignment variant.Source: §1 · p. 2

2 Related Work

The paper situates its work among distinct domain-adaptation settings and approaches to establishing source–target commonality. It distinguishes So-HoT through partial, class-specific scatter alignment that accommodates both shared and domain-specific distribution structure.

2 Related Work

  • The work investigates deep, supervised, single-source domain adaptation, which the authors say can be extended to multi-source and heterogeneous cases.Domain adaptation concerns transfer among distinct domains, whereas transfer learning can concern related distributions within the same domain.Source: §2 · pp. 2–3
  • Unlike methods that confuse domains or seek indistinguishable examples, So-HoT defines commonality as a desired degree of overlap between source and target second- and/or higher-order scatters.The non-overlapping distribution tails also guide learning, with the stated aim of avoiding domain-specific bias.Source: §2 · p. 3
  • Unlike approaches that interpolate or regress source representations into target representations, the source and target streams co-regularize each other while accommodating domain-specific parts.The paper describes this as producing commonality without requiring all distribution parts to align.Source: §2 · p. 3
  • The paper represents commonality globally, unlike visual-domain methods that capture domain specificity or process image patches locally.The authors describe their approach as orthogonal to those local techniques and say it could also be applied spatially locally.Source: §2 · p. 3
  • Compared with correlation approaches, So-HoT partially aligns class-specific distributions and means in a supervised setting rather than fully aligning generic distributions.The cited correlation approaches use second-order scatter matrices; this work also investigates higher-order scatters as domain- and class-specific representations.Source: §2 · pp. 3–4

3 Background

The background defines the tensor notation and mean-centered scatter representations used for domain adaptation. It describes their order-dependent properties and motivates Euclidean and kernelized alignment computations.

3.1 Notations

  • The notation defines tensors as outer products of feature vectors and specifies tensor inner products, slices, and the all-ones vector.The tensor space is super-symmetric, with r modes.Source: §3.1 · p. 4

3.2 Second- or Higher-order Scatter Tensors

  • A scatter tensor is a mean-centered TOSST representation of data vectors; in supervised adaptation, it is computed from class-specific features such as AlexNet fc7 outputs.The order-r representation is denoted X^(r).Source: §3.2 · pp. 4–5
  • Scatter tensors are super-symmetric; even orders are positive semidefinite by slice and order two is a covariance matrix, while odd orders can be indefinite.Because odd-order tensors may be indefinite and explicit representations have ∼d^r unique coefficients, the paper uses Euclidean distances and motivates a lightweight kernelized alternative.Source: §3.2 · p. 5

4 Proposed Approach

The approach balances classification with alignment of source- and target-domain class-specific scatters and means, with optional learned class-specific weights. A kernelized alternative reduces the cost of evaluating scatter distances, and the reported speedups are especially large for third-order tensors.

4.1 Problem Formulation

  • The objective trades off a shared classifier loss against alignment of within-class scatter tensors and feature means across source and target streams.The formulation assumes CNN streams can rotate within-class scatters to achieve the desired overlap.Source: §4.1 · pp. 5–6

4.2 Weighted Alignment Loss

  • The weighted alignment variant learns class-specific weights that adjust alignment of each class's scatters and associated means.The weights also determine contributions from scatter tensors of different orders, with their deviation controlled by α1 and α2.Source: §4.2 · p. 6

4.3 Kernelized Alignment Loss

  • The kernelized loss evaluates scatter-tensor inner products implicitly through sums of entries of a polynomial-kernel matrix, avoiding explicit scatter tensors.The kernelized expression can be substituted into the alignment losses.Source: §4.3 · pp. 6–7
  • For N=20 source and N*=3 target samples per class with d=4096, explicit order-2 scatter-distance evaluation is ∼52× slower and order-3 evaluation takes ∼143000× more time than the kernelized substitute.The authors report that kernelization makes Eq. (6) tractable for r>2.Source: §4.3 · p. 7

5 Experiments

Experiments evaluate So-HoT across Office domain shifts, heterogeneous depth-to-RGB transfer, and image-to-sketch transfer. The reported comparisons favor alignment methods over baselines, with gains also reported for weighted and higher-order variants.

5.1 Datasets

  • Evaluations use Office, RGB-D-Caltech256, and Pascal VOC2007-TU Berlin, with supervised target samples and testing on remaining target examples.The Office protocol samples 3 target images per class; RGB-D-Caltech256 uses 5 or 10, and Pascal-TU Berlin uses 3.Source: §5.1 · p. 8

5.2 Experimental Setup

  • The experiments use pretrained AlexNet streams, with VGG substituted where specified, and compare unweighted, weighted, and kernelized second- or higher-order alignment variants.Model parameters are selected by cross-validation.Source: §5.2 · p. 8

5.3 Comparison to the State of the Art

  • On Office, So outperforms the combined-data fine-tuning baseline, Domain Confusion with Soft Labels, and fine-tuning on source or target data.The comparison covers six domain shifts.Source: Table 1 · p. 8, §5.3 · p. 9
  • On VGG streams, So improves over S+T by 0.74% on A→W and 2.43% on A→D, and beats a multi-scale multi-patch CNN by 0.6% on A→W.The latter comparison is reported without data augmentation.Source: §5.3 · p. 9, Table 2 · p. 9
  • On A→W, weighted So+ζ and To+ζ improve over unweighted So by 0.74% and 1.05%, respectively.Learning the weights adds no visible computational increase.Source: §5.3 · p. 9, Table 2 · p. 9
  • On A→W and A→D, the weighted second-, third-, and fourth-order combination So+To+Fo+ζ outperforms So by 1.42% and 2.9%, respectively.The reported trend is that performance improves as alignment order increases.Source: Table 2 · p. 9, §5.3 · p. 10
  • In the heterogeneous RGB-D-Caltech256 setup, So outperforms both S+T and target-only fine-tuning (T) for every target-sample count from 3 to 20 per class.The largest gains over S+T are 1.24% at N*=3 and 1.04% at N*=5; gains over baselines become smaller as N* increases.Source: Figure 5 · p. 9, §5.3 · p. 10
  • For Pascal VOC2007-to-TU Berlin transfer, So, S+T, and target-only fine-tuning yield 63.4%, 62.66%, and 62.46%, respectively.The evaluation uses AlexNet streams, 14 shared classes, and 50 source plus 3 target training samples per class.Source: §5.3 · p. 10, Table 3 · p. 10

6 Conclusions

The conclusion presents partial alignment of within-class scatters as a way to discover source–target commonality. The authors report state-of-the-art results and say weighted and kernelized variants improve results and computational efficiency.

6 Conclusions

  • The authors report state-of-the-art results for domain adaptation using partial alignment of within-class scatters to discover commonality.Source: §6 · p. 10
  • Weighted alignment improves results, while kernelized alignment improves results and computational efficiency.Source: §6 · p. 10
  • The method can be extended to multiple domains and other network architectures.Source: §6 · p. 10

A Sensitivity to parameters σ1 and σ2

The sensitivity analysis describes how σ1 and σ2 control scatter and mean alignment, respectively, and reports stable performance across ranges of both parameters. The derivative appendix computes covariance derivatives from autocorrelation and mean terms.

A Sensitivity to parameters σ1 and σ2

B Derivatives of the alignment loss g w.r.t. the feature vectors !, ..., φ∗

C Kernelized derivative of the Frobenius norm between tensors w.r.t. the feature vectors

This subsection derives feature-vector and feature-matrix derivatives for the alignment-loss expression involving Frobenius norms between tensors.

C Kernelized derivative of the Frobenius norm between tensors w.r.t. the feature vectors

Related papers

Loading 1611.08195v2…