Source-linked AI summary
Secure Federated Transfer Learning
Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, Qiang Yang
TL;DR
Organizations often have scattered data, limited labels, and legal or practical barriers to pooling them. The paper proposes privacy-preserving federated transfer learning, reporting accuracy comparable to non-private approaches and superiority to non-federated self-learning approaches.
Problem
Organizations may have too little labeled data, while distributed data and privacy constraints limit combining information across them.
Source: §1 · p. 1Method
Results
FTL is reported as accurate as non-privacy-preserving approaches and superior to non-federated self-learning approaches.
Source: §8 · p. 9Takeaways & Limitations
The secret-sharing approach has no accuracy loss and is faster than the homomorphic-encryption approach, but requires offline generation and storage of many triplets.
Source: §8 · p. 9The protocol does not address malicious parties; its stated threat model is semi-honest.
Source: §5.2 · p. 4
Abstract
from arXiv · showhide
Machine learning relies on the availability of a vast amount of data for training. However, in reality, most data are scattered across different organizations and cannot be easily integrated under many legal and practical constraints. In this paper, we introduce a new technique and framework, known as federated transfer learning (FTL), to improve statistical models under a data federation. The federation allows knowledge to be shared without compromising user privacy, and enables complimentary knowledge to be transferred in the network. As a result, a target-domain party can build more flexible and powerful models by leveraging rich labels from a source-domain party. A secure transfer cross validation approach is also proposed to guard the FTL performance under the federation. The framework requires minimal modifications to the existing model structure and provides the same level of accuracy as the non-privacy-preserving approach. This framework is very flexible and can be effectively adapted to various secure multi-party machine learning tasks.
1 INTRODUCTION
Organizations often lack sufficient labeled data and cannot freely pool data, while existing federated approaches are limited when parties share few samples or features. The paper proposes federated transfer learning (FTL), with privacy-preserving protocols, to address those limits.
1 INTRODUCTION
- Small or poor-quality datasets, scarce labels, distributed data, and privacy laws make building accurate AI models difficult for many organizations.The introduction notes that expert labeling is expensive and cites GDPR as restricting data exchange without explicit user approval.Source: §1 · p. 1
- Existing federated approaches require common features or common samples, yet those overlaps may be small and leave much non-overlapping data underused.The cited approaches include federated learning with locally stored data [2] and secure learning with feature-partitioned data.Source: §1 · p. 1
- FTL leverages transfer learning to address federation problems beyond existing approaches and provide solutions across the sample and feature spaces.The authors report comparable convergence and accuracy to non-privacy-preserving transfer learning, and describe HE and secret-sharing methods using Beaver triples with minimal neural-network modifications and almost no accuracy loss.Source: §1 · p. 1
2 RELATED WORK
Related work covers encrypted machine learning and transfer learning, then positions FTL as a framework combining transfer learning with federated learning.
2 RELATED WORK
- Encrypted machine-learning examples include secure aggregation, CryptoNets for homomorphic-encrypted neural-network computation, and SecureML using secret sharing and Yao’s Garbled Circuit.SecureML supports collaborative training for linear regression, logistic regression, and neural networks.Source: §2 · p. 1
- Transfer learning uses knowledge from a related source domain to build a model for a target domain with a small dataset or limited labels.Its performance depends on how related the domains are.Source: §2 · p. 1
- The authors state that organizations in the same industry can benefit more from knowledge propagation and identify FTL as, to their knowledge, the first framework combining federated learning and transfer learning.Source: §2 · p. 2
3 PRELIMINARIES AND SECURITY DEFINITION
The paper formulates FTL for two parties holding separate source- and target-domain data, with limited co-occurring samples and a small set of target labels. It adopts a semi-honest security setting.
3 PRELIMINARIES AND SECURITY DEFINITION
- The setting includes a limited set of co-occurrence samples between parties A and B and a small set of labels for domain B in party A’s dataset.Source: §3 · p. 2
- The parties’ datasets are held separately and cannot legally be exposed to each other; the objective is to predict labels for the target-domain party without exposing their data.Common sample IDs can be aligned privately using encrypted IDs, and the paper uses FATE’s RSA Intersection module.Source: §3 · p. 2
- The security definition assumes honest-but-curious parties and a semi-honest adversary that can corrupt at most one of the two clients.Source: §3 · p. 2
- The paper says its security definition allows flexible control of information disclosure compared with complete zero-knowledge security.Source: §3 · p. 2
4 THE PROPOSED APPROACH
The proposed approach maps each party’s features into a common representation space, then trains a target-label predictor while minimizing alignment loss. It uses a Taylor approximation and secure computation protocols to protect the required calculations.
4 THE PROPOSED APPROACH
- Neural networks transform each party’s features into a common subspace where knowledge can be transferred, while supporting an end-to-end FTL solution.The paper says other symmetric feature-transformation techniques can also form this subspace.Source: §4 · p. 2
- The training objective uses available labeled examples to learn a prediction function for the target domain and minimize alignment loss between parties.The paper gives logistic loss as one example of the loss function.Source: §4 · p. 2
- Because raw data cannot be exposed, the approach uses a second-order Taylor approximation for logistic loss to compute the required gradients privately.Source: §4 · p. 2
- The FTL protocol offers alternative constructions using additive homomorphic encryption or secret sharing based on Beaver triples, with minimal information encrypted or shared.The protocol is designed to work with other homomorphic-encryption and secret-sharing schemes with minimal modifications.Source: §4 · p. 2
5 FTL USING HOMOMORPHIC ENCRYPTION
The homomorphic-encryption construction computes privacy-preserving losses and gradients, then coordinates the parties’ local neural networks through encrypted exchanges. Its security claim depends on secure underlying encryption and excludes malicious behavior.
5 FTL USING HOMOMORPHIC ENCRYPTION
- Additive homomorphic encryption and polynomial approximations yield privacy-preserved loss functions and corresponding gradients for both domains.The protocol uses encrypted exchanges between locally executed neural networks, masks gradients before exchanging them, and supports predictions for unlabeled data from party B.Source: §5 · pp. 2–3, §5.1 · p. 3
5.1 FTL Algorithm - Homomorphic Encryption based
- Each party locally computes neural-network components and encrypts them for the other party’s gradient calculations; gradients and loss are masked before exchange.After unmasking, parties update their respective parameters, and Alice can send Bob a termination signal when loss convergence is met.Source: §5.1 · p. 3
5.2 Security Analysis
- The HE-based protocol is secure under the stated definition provided the underlying additively homomorphic encryption scheme is secure.The proof says parties exchange masked gradients; the protocol does not address malicious parties, whose falsified inputs may expose a hidden representation value.Source: §5.2 · pp. 3–4
6 FTL USING SECRET SHARING
The secret-sharing construction uses Beaver triples to support secure matrix multiplication and FTL training and prediction. Its stated security guarantee is information-theoretic against at most one passive adversary.
6 FTL USING SECRET SHARING
- The secret-sharing construction assumes private values are additively shared and uses second-order Taylor approximations of the loss and its derivative.The stated approximation is used to make the construction comparable in performance with the previous construction.Source: §6 · p. 4
6.1 Secure Matrix Multiplication using Beaver Triples
- Beaver triples let the parties securely multiply secretly shared matrices, with efficient online computation after an offline preprocessing phase.The triple contains shared matrices D, E, and F such that DE = F; the online protocol reconstructs the required product from shares.Source: §6.1 · p. 4
6.2 Beaver Triples Generation
- A third party helps generate Beaver triples by computing and sharing the cross-products needed to form F while the parties retain their shares.The offline matrix multiplication protocol splits each party’s input into shares, and the third party computes the cross-term unavailable to either party alone.Source: §6.2 · pp. 4–5
6.3 FTL Algorithm - Secret Sharing based
- Algorithm 6 trains the model iteratively from shared losses and derivatives, then supports prediction for unlabeled data from party B.The training protocol assumes sufficient pre-generated triples and repeats parameter updates until the loss-change termination condition is met.Source: §6.3 · p. 6
- The protocols in Algorithms 3–6 are information-theoretically secure against at most one passive adversary.The proof attributes this to parties receiving only secret shares of private values they are not meant to learn.Source: §6.3 · p. 7
7 EXPERIMENTAL EVALUATION
Experiments find Taylor-approximated FTL comparable to full-logistic FTL and show FTL outperforming self-learning baselines with small training sets. Scalability tests report substantial gains from parallelism and faster SS-based than HE-based FTL.
7.1 Impact of Taylor Approximation
- Taylor-approximated loss decays similarly to full logistic loss, with comparable weighted F1 and no performance decay as network depth increases.The accuracy drop is reported as much smaller than in state-of-the-art secure neural networks using similar low-degree approximations.Source: §7.1 · p. 7
7.2 Performance
- FTL performance improves as the number of overlapping sample pairs increases.The pairs bridge the parties’ hidden representations.Source: §7.2 · p. 7
- FTL models yield comparable weighted F1 scores and significantly outperform self-learning baselines with small training sets.Performance also improves as training samples increase.Source: §7.2 · p. 7, Table 1 · p. 8
7.3 Scalability
- With 20 partitions, parallel secure FTL is 100x faster than the sequential scheme.The parallel stages handle encryption, encrypted-gradient calculations, and decryption.Source: §7.3 · pp. 7–8
- With parallelism, HE-based FTL running time grows approximately linearly with hidden-representation dimension, target-domain features, and overlapping samples.These trends are shown in Figures 2(d)–2(f).Source: §7.3 · p. 8
- SS communication cost simplifies to O(d^2) with other factors fixed, while training-time growth approaches linear and slows as feature or overlap counts rise.The authors conjecture that the near-linear trend reflects SS-based FTL’s computational efficiency.Source: §7.3 · pp. 7–8
- SS-based FTL runs 1–2 orders of magnitude faster than HE-based FTL, with a slower training-time increase across the tested scalability factors.The comparison covers hidden-representation dimension, feature count, and overlapping samples.Source: §7.3 · p. 8
8 CONCLUSIONS AND FUTURE WORK
The paper proposes a secure Federated Transfer Learning framework to expand federated learning to broader applications. It reports accuracy comparable to non-privacy-preserving approaches while identifying distinct computational trade-offs between homomorphic encryption and secret sharing.
8 CONCLUSIONS AND FUTURE WORK
- The proposed secure FTL framework aims to expand existing secure federated learning to broader real-world applications.Source: §8 · p. 8
- Secret sharing preserves accuracy and runs faster than homomorphic encryption, but requires offline generation and storage of many triplets.The paper characterizes homomorphic encryption as simple but computationally expensive.Source: §8 · p. 9
- FTL is as accurate as non-privacy-preserving approaches and is not restricted to specific models.The paper also reports that FTL is superior to non-federated self-learning approaches.Source: §8 · p. 9
- Future work will improve FTL efficiency using distributed computing techniques and less expensive computation and communication schemes.Source: §8 · p. 9