Simultaneous Deep Transfer Across Domains and Tasks

Introduction

• Deep CNN adaptation can require hundreds or thousands of labeled examples per category, while target environments may offer only a handful of labels and abundant unlabeled data.The paper notes that initial CNN training itself relies on millions of supervised images.Source: §1 · p. 1
• For domain transfer, the method uses unlabeled target data to estimate its marginal distribution and jointly minimizes source-target feature-distribution distance and classification error.Its domain-confusion loss seeks a representation that makes source and target domains indistinguishable.Source: §1 · pp. 1–2, Figure 1 · p. 1
• Because matching marginal distributions alone may not align target classes, the method transfers source category relationships by matching target-label predictions to source per-category average soft labels.This transfers information to target categories without explicit target labels; prior CNN domain-transfer methods [12] [24] did not use source semantic structure for task transfer.Source: §1 · pp. 1–2, Figure 1 · p. 1
• The joint method is reported to outperform current state-of-the-art visual domain adaptation results on the Office benchmark and cross-dataset collection.The evaluation covers supervised adaptation, with target labels from each category, and semi-supervised adaptation, with labels from only a subset of categories.Source: §1 · p. 2
• A new CNN architecture jointly optimizes domain-confusion and softmax cross-entropy losses, supporting supervised and semi-supervised target adaptation.The approach preserves category relationships, such as bottle being similar to mug but different from keyboard.Source: §1 · pp. 1–2

Related work

• Prior methods align source and target representations, and supervised approaches regularize a target classifier against the source classifier or jointly learn alignment and a regularized classifier [18] [10].Domain shift is commonly measured as distance between source and target subspace representations [25].Source: §2 · p. 2
• Deep CNN features reduce resolution- and lighting-related domain shifts [9], but Siamese invariant-representation methods require labels for every training instance.Multimodal deep learning learned modality-invariant representations, primarily in a generative context that did not leverage supervised CNN representations fully.Source: §2 · p. 2
• The joint source–target CNN was limited to two layers and significantly outperformed by deeper architectures pretrained on auxiliary data such as ImageNet; the two-layer MMD-confusion model [13] learned domain-invariant representations but lacked deep-CNN semantic strength.Source: §2 · p. 2
• Unlike prior model-compression or distillation work using classifier output distributions [16], this paper applies the technique to domain adaptation to transfer class correlations between domains.Source: §2 · p. 2
• Contemporaneous domain-invariance methods used weaker invariance measures or less robust optimization and did not address semi-supervised task transfer [12] [24].Source: §2 · p. 2

3. Joint CNN architecture for domain and task transfer

• The method jointly trains a target classifier using source labels and sparse target labels, domain confusion, and soft-label transfer from source examples.The representation and classifier are optimized with a joint loss whose hyperparameters control the influence of domain confusion and soft labels.Source: §3 · pp. 3–4

3.1. Aligning domains via domain confusion

• Domain confusion aligns source and target representations by training a domain classifier while making its predicted domain labels approach a uniform distribution.The representation is domain invariant when a classifier trained on it cannot distinguish examples from the two domains.Source: §3.1 · p. 4

3.2. Aligning source and target classes via soft labels

• Category soft labels average source-model activations by category, and target examples are trained to match them so source-learned inter-class relationships transfer across domains.A high softmax temperature preserves probability mass on related classes, such as mugs for bottles, rather than producing overly peaked distributions.Source: §3.2 · pp. 4–5, §3 · p. 4
• Soft-label training keeps classifier parameters for categories without target labels updated to produce non-zero probabilities.The paper reports exploring this benefit when target labels cover only a subset of categories and evaluating on unlabeled categories.Source: §3.2 · p. 5

4.1. Adaptation on the Office dataset

• On Office supervised adaptation, soft-label fine-tuning or domain confusion improves over hard-label training in 5 of 6 shifts, while combining them yields marginally higher average performance.The setting provides 3 labeled target examples per category and reports multi-class accuracy on the remaining images across all 31 categories.Source: §4.1 · pp. 5–6, Table 1 · p. 6
• On Office semi-supervised adaptation, every method variant—soft labels, domain confusion, or both—outperforms the baselines on 16 target categories without labeled target data.Both losses contribute significantly because the held-out categories lack target labels to enforce domain invariance through classification alone; soft-label gains also indicate transfer between related tasks.Source: §4.1 · p. 6, Table 2 · p. 6
• In Office semi-supervised examples from Amazon→Webcam, the method correctly classifies held-out-category images that the baseline misclassifies, including notebooks and black mugs.The cited examples include notebooks confused with letter trays and black mugs confused with computer mice.Source: §4.1 · pp. 6–7, Figure 5 · p. 7

4.2. Adaptation between diverse domains

• The full method, combining domain confusion and soft-label loss, significantly outperforms source-only and standard fine-tuning baselines on ImageNet→Caltech, with the greatest benefit when target labels are scarce.It performs best overall, including with no labeled target examples; as target labels increase, standard fine-tuning approaches its performance. Target-only accuracies are 36.6 ± 0.6, 60.9 ± 0.5, and 67.7 ± 0.5 with 1, 3, and 5 labeled examples per category, respectively, all below the source-only model.Source: Figure 6 · p. 7, §4.2 · p. 7
• For ImageNet→Caltech, adaptation results exceed the 24.8% result reported in despite using much less training data; the passage attributes the difference to CNN features versus SURF BoW features.The evaluation uses 40 shared categories and target training sets subsampled to 1, 3, or 5 labeled examples per category.Source: §4.2 · p. 7

5.1. Domain confusion enforces domain invariance

• 99% baseline versus 56% domain-confusion SVM test accuracy shows domain confusion makes Amazon and Webcam images harder to distinguish by domain.The SVMs were trained on 160 images, 80 from each domain; 56% was described as only slightly better than random.Source: Figure 7 · p. 8, §5.1 · p. 8

5.2. Soft labels for task transfer

• With target labels available for only half the categories, soft-label fine-tuning correctly classifies a held-out monitor image that the baseline mistakes for a ring binder.A source-domain laptop soft label assigned relatively high weight to the monitor class, allowing transfer despite no labeled target monitors during training.Source: §5.2 · p. 8

• The CNN adapts to new domains with limited or no labeled target-category data by optimizing domain invariance and transferring task information through cross-entropy soft-label loss.Source: §6 · p. 8
• With no labeled target data for monitor, the method transfers information from the related laptop computer category and predicts monitor correctly, unlike the baseline.Source: Figure 8 · p. 8
• 13% average relative improvement over the baselines on the four most challenging shifts in the Office dataset is reported for semi-supervised adaptation.Source: §6 · p. 8