PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Sharing Weights for Deep Domain Adaptation

Artem Rozantsev, Mathieu Salzmann, Pascal Fua

arXiv:1603.06432v2cs.CV

TL;DR

Domain adaptation must address performance degradation across related domains without requiring impractical amounts of target annotation. This paper uses two jointly trained streams with related but selectively non-shared weights to model domain shift. Experiments report higher accuracy than shared-weight networks and state-of-the-art methods across classification and regression settings.

  • Problem

    Classifiers trained in one domain often degrade in a related target domain, while acquiring enough target annotations is expensive or impractical.

  • Method

    A two-stream architecture jointly trains source and target data with corresponding weights related but selectively not shared, using regularization and a criterion to choose differing layers.

  • Results

    The approach consistently yields higher accuracy than fully shared-weight networks and outperforms state-of-the-art methods across classification and regression domain-adaptation tasks.

  • Takeaways & Limitations

    Explicitly modeling domain shift with domain-adapted, related features is more effective than enforcing complete feature invariance in the reported experiments.

  • Takeaways & Limitations

    The optimal layers to leave non-shared are application dependent, differing between UAV and Office experiments.

Abstract

from arXiv · show

The performance of a classifier trained on data coming from a specific domain typically degrades when applied to a related but different one. While annotating many samples from the new domain would address this issue, it is often too expensive or impractical. Domain Adaptation has therefore emerged as a solution to this problem; It leverages annotated data from a source domain, in which it is abundant, to train a classifier to operate in a target domain, in which it is either sparse or even lacking altogether. In this context, the recent trend consists of learning deep architectures whose weights are shared for both domains, which essentially amounts to learning domain invariant features. Here, we show that it is more effective to explicitly model the shift from one domain to the other. To this end, we introduce a two-stream architecture, where one operates in the source domain and the other in the target domain. In contrast to other approaches, the weights in corresponding layers are related but not shared. We demonstrate that this both yields higher accuracy than state-of-the-art methods on several object recognition and detection tasks and consistently outperforms networks with shared weights in both supervised and unsupervised settings.

1. Introduction

Domain adaptation addresses performance loss across related domains when obtaining target-domain annotations is costly. This paper replaces fully shared deep-network weights with related, selectively non-shared streams that explicitly model domain shift.

  • Motivation: Deep classifiers often need substantial new-domain annotation and retraining to perform well across related but different domains.Deep networks make this especially acute because they require massive amounts of training data.
  • Motivation: Domain adaptation transfers knowledge from a well-annotated source domain to a target domain with few or no additional annotations.
  • Prior approach: Prior deep domain-adaptation methods use shared architectures and weights to learn features invariant to the domain shift.
  • Proposed approach: The paper introduces two jointly trained streams for source and target data, allowing corresponding layers to have related but non-shared weights.A loss is lowest when the non-shared weights are linear transformations of one another, and a criterion selects which layers should differ.
  • Results: The approach outperforms state-of-the-art methods across image-recognition, UAV-detection, and facial-pose-estimation domain-adaptation tasks.Synthetic images serve as the source domain and real images as the target in the detection and pose-estimation applications.

2. Related Work

Earlier domain-adaptation work includes target fine-tuning, cross-domain transformations, subspace alignment, distribution matching, and deep feature adaptation. Deep approaches generally share weights across domains, motivating this paper’s test of explicit domain-shift modeling.

  • Target adaptation: Fine-tuning modifies a source-trained classifier with labeled target data but can overfit when target annotations are scarce.
  • Transformations and subspaces: Cross-domain methods learn linear or nonlinear transformations, or align separate source and target subspaces.Some approaches interpolate between subspaces or concatenate representations from networks trained with increasing target-data amounts.
  • Distribution matching: Maximum Mean Discrepancy measures source-target distribution similarity and has been used to re-weight samples, select samples, or learn distribution-aligning transformations.A shallow neural-network method using MMD relied on SURF features and achieved limited accuracy.
  • Deep adaptation: Deep feature learning and adaptation improve domain-adaptation accuracy, but these approaches typically use the same architecture and weights for both domains.
  • Research gap: The paper tests whether explicit domain-shift modeling can improve discriminative power over domain-invariant representations enforced by shared-weight networks.

3. Our Approach

The approach uses jointly trained source and target streams with related but potentially unshared weights, combining classification, weight, and representation-distribution losses.

  • Two-stream architecture: Two jointly trained streams process source and target data, allowing corresponding layers to use different parameters while remaining related.The source and target stream parameters include weights and biases, and unshared layers are selected through the set Ω.
  • Training objective: The total loss combines source classification, target classification, weight regularization, and an unsupervised MMD regularizer.Target classification contributes only when target labels are available; the MMD term favors similar source and target representations.
  • Weight regularization: The weight regularizer models corresponding source and target weights as related through a learned layer-specific transformation rather than directly forcing equality.Scalar transformation parameters a_j and b_j are learned for each unshared layer, and the exponential parameterization performed better empirically.
  • Unsupervised regularization: The MMD regularizer reduces the distance between source and target feature distributions at the final representation before classification.MMD is computed after mapping representations to a reproducing kernel Hilbert space and is implemented with an RBF kernel whose bandwidth was set to 1.
  • Optimization: Training first pre-trains the source stream, then jointly optimizes both streams after initializing the target stream from the source parameters.Transformation parameters are initialized as a_j = 1 and b_j = 0, encoding the identity transformation.

4. Experimental Results

The experiments evaluate the approach across detection, classification, and regression settings, including a UAV dataset with synthetic and real imagery.

  • Evaluation scope: The evaluation spans supervised and unsupervised scenarios across drone detection, image classification, and facial-landmark regression.The listed tasks use different network architectures for benchmark comparisons.
  • UAV dataset: UAV dataset examples include synthetic and real training images, with real samples used for testing.The figure presents training examples on top and real test samples on the bottom.
  • UAV dataset: Synthetic and real images form the source and target domains, respectively, in the UAV experiments.The synthetic images are used as source samples and the real images as target samples.

4.1. Leveraging Synthetic Data for Drone Detection

The paper evaluates two-stream domain adaptation for UAV detection, treating synthetic images as source data and real images as target data. Across balanced and imbalanced datasets, the approach benefits from synthetic data and outperforms weight-sharing baselines.

  • Dataset and Evaluation Setup: Synthetic UAV images supplement scarce real training data, which is difficult and time-consuming to collect and annotate.The experiments treat synthetic images as source samples and real images as target samples.
  • Dataset and Evaluation Setup: UAV-200 (small) is evaluated with accuracy because its positive-to-negative ratio is relatively balanced, whereas UAV-200 (full) uses precision-recall and AP for its more imbalanced setting.The full dataset reflects realistic detection conditions with many negative windows per positive example.
  • Architecture Selection: The two-stream network uses separate source and target streams, with convolutional and fully connected layers whose weights may be shared or regularized rather than uniformly tied.The MMD-based criterion evaluates candidate layer configurations, while the classification layer uses a hinge loss.
  • Architecture Selection: The MMD-based architecture choice nearly matches the best validation and test accuracies, achieving 0.902 and 0.732 versus maxima of 0.916 and 0.757.Low MMD values tend to coincide with high AP, supporting MMD as an architecture-selection criterion.
  • Evaluation: On UAV-200 (small), the method significantly outperforms many state-of-the-art domain adaptation baselines, including the closely related single-stream DDC method.The DDC comparison is intended to test modeling domain shift against enforcing shared representations.
  • Evaluation: On UAV-200 (full), the method outperforms all baselines, yielding a 10% boost over DDC and a 20% boost over training with real data only.Removing the layer-linking terms while retaining MMD performs worse, which the authors attribute to target-stream overfitting.
  • Influence of the Number of Samples: With 32,800 synthetic samples fixed, the approach performs similarly to or slightly better than a single-stream model trained with 2,500 real samples, requiring only 5–10% labeled training data for good results.With 200 real samples fixed, its AP steadily increases as synthetic data grows and systematically exceeds DDC except with no synthetic data.

4.2. Unsupervised Domain Adaptation on Office

The Office experiments evaluate unsupervised adaptation across three image domains using a two-stream AlexNet architecture whose layer-sharing configuration is selected by an MMD-based criterion. The approach outperforms competing domain adaptation techniques on all three source/target pairs, and comparison with GRL indicates benefits from allowing weights to differ.

  • Dataset and protocol: The Office dataset contains Amazon, DSLR, and Webcam images across 31 object categories, evaluated with no target-domain labels.The fully-transductive protocol uses all available source information while withholding target labels.
  • Network design: Each stream uses an ImageNet-pretrained AlexNet, with corresponding weights allowed to differ rather than being forced to share.The streams are fine-tuned jointly after initialization from the pretrained model.
  • Network design: The MMD-based criterion selects the configuration with the lowest MMD2; for Amazon → Webcam, the last two fully-connected layers are not shared.This configuration is then used for the Office experiments.
  • Results: The method outperforms other domain adaptation techniques on all three commonly reported source/target pairs.The comparison appears in Table 4, which covers all 31 categories under the fully-transductive protocol.
  • Results: The comparison against GRL confirms that allowing corresponding weights to remain unshared increases accuracy.This directly contrasts selective weight sharing with a shared-weight domain adaptation approach.

4.3. Domain Adaptation on MNIST-USPS

The MNIST-USPS experiment evaluates unsupervised digit-classification adaptation under a standard protocol that alternates the two datasets as source and target. The proposed method achieves superior performance in every reported case.

  • Dataset and protocol: MNIST and USPS each contain 10 digit classes and are used interchangeably as source and target domains.The experiment randomly selects 2000 MNIST and 1800 USPS images.
  • Dataset and protocol: The evaluation is unsupervised, so target-domain labels are ignored during training.MNIST images are rescaled to USPS's 16 × 16-pixel size and pixel intensities are L2-normalized.
  • Results: The proposed method yields superior performance in all reported MNIST-USPS cases.It is compared with DDC and several methods that do not rely on deep networks.
  • Results: The authors attribute this performance to adapting feature representations to each domain while keeping those representations close.This combines domain-specific adaptation with cross-domain similarity.

4.4. Supervised Facial Pose Estimation

The facial pose experiment tests whether the domain adaptation method extends from classification and detection to supervised regression. Synthetic images serve as the source and real images as the target, with only 100 real images used for training.

  • Task and data: The task predicts the locations of five facial landmarks from 50×50 image patches.The regressor outputs a 10D vector containing two coordinates for each landmark.
  • Task and data: Synthetic images form the source domain and real images form the target domain.Both datasets contain approximately 10k annotated images, but only 100 real images are used for training.
  • Evaluation: Performance is measured by the PCP-score, counting landmarks within a 2-pixel radius of ground truth.The comparison is reported against DDC.
  • Results: The proposed method outperforms DDC when weights are not shared.The result extends the reported advantage of selective non-sharing to facial pose regression.

4.5. Discussion

Across the reported experiments, selectively unsharing weights in the two-stream architecture boosts performance, supporting explicit modeling of domain shift. However, the best layer-sharing configuration depends on the application, while MMD provides a practical selection criterion without validation data.

  • Discussion: Allowing weights to remain unshared in some layers boosts performance across all reported experiments.This supports the hypothesis that explicitly modeling domain shift is generally beneficial.
  • Discussion: The optimal choice of shared versus stream-specific layers is application dependent.Different tasks may require domain-specificity at different network depths.
  • Discussion: For UAV data, differing early layers performs best, whereas Office data benefits from differing final fully-connected layers.The authors associate these configurations with low-level versus more complex domain variations, respectively.
  • Discussion: The MMD provides an effective criterion for choosing the layer-sharing configuration without validation data.This makes the two-stream approach practical in settings where validation data are unavailable.

5. Conclusion

The paper argues that domain adaptation should explicitly model domain shifts rather than enforce feature invariance. Its two-stream architecture, with selectively shared weights and transformation-based regularization, consistently outperforms fully shared networks across diverse tasks and datasets.

  • The authors argue that invariant features can reduce discriminative power, motivating explicit modeling of the domain shift.
  • The proposed two-stream CNN uses source and target streams whose weights may be shared selectively rather than universally.
  • An additional loss term encourages corresponding non-shared weights to remain close to linear transformations of each other.
  • The approach consistently yields higher accuracy than fully weight-shared networks for both classification and regression across diverse datasets.
Loading 1603.06432v2…