PaperlayerView exact revision on arXiv

Source-linked AI summary

Federated Transfer Learning: concept and applications

Sudipan Saha, Tahir Ahmad

arXiv:2010.15561v3cs.LGcs.CR

TL;DR

Data isolation and privacy concerns challenge machine learning, while traditional federated learning requires organizations to share a feature space. This survey reviews FTL’s applications and analyzes its privacy and machine-learning aspects, including its use of source-domain knowledge for non-overlapping target samples.

  • Problem

    Data isolation and privacy concerns challenge AI, and traditional federated learning requires participating organizations to share a feature space.

    Source: §I · p. 2, §VIII · p. 14

  • Method

    The paper surveys FTL’s applications and reviews its definition, case studies, privacy aspects, and machine-learning aspects.

    Source: §I · p. 3

  • Results

    The survey covers FTL applications and reviews the machine-learning techniques used in FTL.

    Source: §VIII · p. 14

  • Takeaways & Limitations

    FTL transfers source-domain knowledge for use with non-overlapping target samples.

    Source: §III · p. 6

Abstract

from arXiv · show

Development of Artificial Intelligence (AI) is inherently tied to the development of data. However, in most industries data exists in form of isolated islands, with limited scope of sharing between different organizations. This is an hindrance to the further development of AI. Federated learning has emerged as a possible solution to this problem in the last few years without compromising user privacy. Among different variants of the federated learning, noteworthy is federated transfer learning (FTL) that allows knowledge to be transferred across domains that do not have many overlapping features and users. In this work we provide a comprehensive survey of the existing works on this topic. In more details, we study the background of FTL and its different existing applications. We further analyze FTL from privacy and machine learning perspective.

I. INTRODUCTION

Data isolation, privacy restrictions, and limited sharing leave some industries with insufficient data for reliable machine learning. The paper surveys federated transfer learning, including its applications and privacy and machine-learning aspects.

I. INTRODUCTION

  • In finance and healthcare, expert labeling, privacy and security restrictions, abuse risks, and organizational competition constrain data sharing.The resulting data scarcity can leave organizations with insufficient data to train reliable machine-learning frameworks.Source: §I · p. 2
  • Federated learning trains a global model while data remain distributed, rather than collecting them in a central server.The introduction describes this as a way to use data from different organizations without violating data-related regulations.Source: §I · p. 2
  • FTL accommodates participants with distinct feature spaces, addressing a constraint of traditional federated learning.The introduction notes finance and healthcare as examples of industries where this constraint matters.Source: §I · p. 2
  • The survey covers FTL’s definition and applications, then analyzes its privacy and machine-learning aspects.It also reviews case studies on FTL.Source: §I · p. 3

II. RELATED WORK

This section reviews horizontal federated learning, where parties share a feature space, and describes its collaborative training process.

II. RELATED WORK

  • The section situates horizontal and vertical federated learning in settings where datasets differ mainly in samples.It briefly reviews these forms alongside transfer learning as context for federated transfer learning.Source: §II · p. 3

A. Horizontal federated learning

  • Horizontal federated learning applies when parties share a feature space, even if their user bases differ.The section gives banks in the same city or region, with few common users, as an example.Source: A. Horizontal federated learning · p. 3
  • Parties locally compute and encrypt gradients; a server aggregates them and returns the result for model updates.The process yields shared final model parameters.Source: A. Horizontal federated learning · p. 3

B. Vertical federated learning

The section describes vertical federated learning for parties with overlapping users and distinct features, then reviews transfer learning approaches for differing domains and data distributions.

B. Vertical federated learning

  • Vertical federated learning combines different parties’ features for common users while keeping non-overlapping users undisclosed.It aligns user IDs cryptographically and computes training losses and gradients in a privacy-preserving manner [18],.Source: B. Vertical federated learning · p. 4

C. Transfer learning

  • Transfer learning allows training and test tasks, domains, or distributions to differ, reusing knowledge from one domain in another.The section illustrates the motivation with product reviews: separate data collection and annotation for each product is time-consuming and challenging.Source: C. Transfer learning · p. 4
  • The section lists instance, parameter, feature-representation, and relational-knowledge transfer as distinct approaches.Feature-representation transfer seeks a representation that reduces source–target domain differences.Source: C. Transfer learning · pp. 4–5
  • Deep learning’s greater data dependence motivates deep transfer learning, including adversarial methods that seek transferable representations.The section also identifies which networks and features transfer as questions in deep transfer learning.Source: C. Transfer learning · p. 5

III. FEDERATED TRANSFER LEARNING

Federated transfer learning addresses settings with limited overlap in both features and samples. It transfers source-domain knowledge for use with non-overlapping target samples.

III. FEDERATED TRANSFER LEARNING

  • FTL is suited to datasets with small overlap in both feature and sample spaces, unlike horizontal and vertical federated learning.The section describes enterprises with different but similar business contexts as one setting where feature overlap may be small.Source: §III · pp. 5–6
  • FTL adapts a model trained on source-domain samples and features for reuse on non-overlapping target samples.The transferred knowledge comes from source-domain features that do not overlap with the target samples.Source: §III · p. 6
  • Using transferred knowledge on non-overlapping data in the target domain distinguishes FTL from vertical transfer learning.The section locates this capability in the upper-right region of Figure 1.Source: §III · p. 6

IV. USE CASES OF FTL

Wearable healthcare data are fragmented and private, making robust results across populations difficult. The subsection identifies regulatory, security, and personalization challenges to generalizable healthcare-data analysis.

A. Wearable Healthcare

  • Wearable devices support remote monitoring, data collection, lifestyle tracking, and detection of chronic conditions.Source: A. Wearable Healthcare · p. 6
  • Fragmented, private healthcare data make it difficult to generate robust results across populations.The section also identifies regulatory restrictions, security and privacy limits on data sharing, and personalization as barriers to generalizable analysis.Source: A. Wearable Healthcare · pp. 6–7
  • The system architecture has users measure health parameters with wearable devices and upload the collected data to a remote cloud for assessment and analysis.Source: A. Wearable Healthcare · pp. 6–7, Figure 2 · p. 7

B. EEG signal classification

The EEG example reports that domain adaptation in federated learning boosts classifier performance, while addressing small, distributed EEG datasets. The autonomous-driving example is motivated by the technology’s complexity and dynamic, uncertain real-world scenarios.

B. EEG signal classification

C. Autonomous Driving

  • Autonomous-driving technology integrates sensing, perception, and decision functions, with cloud platforms providing storage, simulation, HD-map generation, and model-training capabilities.Source: C. Autonomous Driving · p. 8
  • Dynamic environments and uncertainty in real-life scenarios make autonomous driving a special use case for FTL [33].Source: C. Autonomous Driving · p. 8

D. Image steganalysis

Image steganalysis detects hidden information by analyzing features generated by steganographic algorithms, but data sharing constraints and owner preferences challenge conventional approaches. FedSteg applies federated transfer learning with cloud and personalized local models, then uses transfer learning to address cloud–user distribution discrepancy.

D. Image steganalysis

  • Image steganalysis detects hidden information by analyzing features produced by steganographic algorithms.Source: D. Image steganalysis · p. 8
  • Limited data sharing and owners’ differing preferences for steganographic algorithms and cover images challenge traditional steganalysis methods.Traditional algorithms cannot account for those personalized preferences.Source: D. Image steganalysis · pp. 8–9
  • Yang et al. proposed FedSteg, a federated transfer-learning framework for image steganalysis.Source: D. Image steganalysis · p. 9
  • FedSteg uses a cloud model and personalized local models: users train on local data and return trained models, sharing encrypted model parameters rather than their data.Source: D. Image steganalysis · p. 9
  • Transfer learning is used to mitigate cloud–user data distribution discrepancy and make each personalized model more suitable for its user.Yang et al. use CNN-based deep transfer learning to train personalized models.Source: D. Image steganalysis · pp. 9–10

V. PRIVACY IN FTL

FTL addresses data isolation, but its privacy requirements demand safeguards that protect information while permitting collaborative learning. The section reviews privacy-preserving approaches and notes limitations of anonymization and differential privacy in particular.

V. PRIVACY IN FTL

  • FTL’s broader adoption is constrained by limited datasets and the absence of technical and legal protections that reconcile data protection with use.The section says privacy measures should depend on the sensitivity of the subject data.Source: §V · pp. 10–11
  • The survey reviews privacy by design, anonymization and pseudonymization, differential privacy, homomorphic encryption, secure multi-party computation, and hardware security.These approaches cover privacy-aware development, data alteration, encrypted computation, joint computation, and specialized secure hardware.Source: §V · p. 11
  • Naive anonymization or pseudonymization alone is technically insufficient against identity attacks.Re-identification of patient records is described as a lucrative target for health insurers’ records.Source: §V · p. 11
  • Differential privacy can protect against membership-inference attacks at inference, but training can still expose participants to privacy leakage.The text notes leakage among participants in FTL approaches based on differential privacy.Source: §V · p. 11
  • Homomorphic encryption supports computation on encrypted data, while secure multi-party computation enables joint computation without revealing data to collaborators.The section cites private regression work using homomorphic encryption and describes identifying common patients without disclosing hospital lists.Source: §V · pp. 11–12

VI. MACHINE LEARNING IN FTL

The survey describes deep-learning and reinforcement-learning approaches used in FTL, alongside feature-alignment and gradient-aggregation methods. It also highlights the challenges of heterogeneous data and limits in existing FTL analyses.

VI. MACHINE LEARNING IN FTL

  • For feature alignment, uses CORAL to adapt second-order feature statistics, while [31] combines classification loss with domain loss implemented using maximum-mean discrepancy.These are examples of approaches to aligning source and target features.Source: §VI · pp. 12–13
  • In federated autonomous driving, each reinforcement-learning agent is trained individually before distributed knowledge is transferred through online observation and action alignment [33].The method treats reinforcement learning in terms of observation and action spaces [53].Source: §VI · p. 13
  • Heterogeneous client data may not be captured adequately by a model, and local data quality can differ substantially across clients.Dynamic gradient aggregation weights local gradients during aggregation [54].Source: §VI · p. 13
  • Existing works often cover few users and lack in-depth analysis of differing feature spaces and complex heterogeneous FTL settings.Their shared-feature and domain-specific-layer assumptions do not fully exploit progress in deep domain adaptation.Source: §VI · p. 13

VII. FTL DATASETS

The reviewed FTL studies mostly adapt existing machine-learning datasets to federation settings. The section contrasts these public datasets with real-life experiments in autonomous driving.

VII. FTL DATASETS

  • Most FTL works modify existing machine-learning datasets to meet FTL requirements.The studies include experiments using public UCI repository datasets.Source: §VII · pp. 13–14
  • FedHealth used the UCI Smartphone human-activity dataset, with six activities from 30 users aged 19–48.Five subjects were treated as isolated users, with the remaining subjects used to train the cloud model.Source: §VII · p. 14
  • Ju et al. [31] used the PhysioNet EEG Motor Imagery (MI) Dataset, recorded from 109 subjects, and conducted experiments using 5-fold cross-validation.Source: §VII · p. 14

VIII. CONCLUSIONS

The conclusion presents FTL as a potential response to data isolation and privacy concerns, and reviews its applications and machine-learning techniques. It identifies room for more sophisticated techniques and more practical applications.

VIII. CONCLUSIONS

  • Data isolation and privacy concerns remain significant challenges for applying artificial intelligence in real-life settings.Features and users generally vary from organization to organization.Source: §VIII · p. 14
  • The survey analyzes FTL’s concept and practical-domain applications and reviews the machine-learning techniques used in FTL.It presents FTL as a potential solution to the challenges described in the conclusion.Source: §VIII · p. 14
  • FTL has explored different machine-learning techniques, but more sophisticated techniques, including adversarial methods, remain a potential direction.The text describes adversarial techniques as promising for further strengthening FTL.Source: §VIII · p. 14
  • The applications covered are still few, and most datasets used in FTL works are basic machine-learning datasets rather than real-life datasets.The conclusion calls for applying FTL to more practical applications.Source: §VIII · p. 14
Loading 2010.15561v3…