Source-linked AI summary
SecureBoost: A Lossless Federated Learning Framework
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, Qiang Yang
TL;DR
SecureBoost addresses the need for organizations to collaborate on machine learning without exposing private data. It uses federated privacy-preserving tree boosting and is theoretically as accurate as centralized non-privacy-preserving counterparts, while analyzing and reducing protocol leakage.
Problem
Organizations need to collaborate on machine learning because individual data owners may lack enough data for high-quality models, but sharing personal and commercial data raises privacy risks.
Method
SecureBoost is a lossless privacy-preserving tree-boosting framework that aligns common users and jointly trains models across parties with vertically partitioned features.
Results
SecureBoost is theoretically as accurate as non-federated gradient tree-boosting counterparts while avoiding disclosure of parties’ data records during training and inference.
Takeaways & Limitations
The framework supports privacy-preserving collaborative model training and inference, with an RL-SecureBoost variant that reduces leakage while performing identically to SecureBoost in prediction accuracy.
Takeaways & Limitations
Protocol execution still permits leakage, with the active party learning split-related information and secure comparison techniques imposing significant learning overhead.
Abstract
from arXiv · showhide
The protection of user privacy is an important concern in machine learning, as evidenced by the rolling out of the General Data Protection Regulation (GDPR) in the European Union (EU) in May 2018. The GDPR is designed to give users more control over their personal data, which motivates us to explore machine learning frameworks for data sharing that do not violate user privacy. To meet this goal, in this paper, we propose a novel lossless privacy-preserving tree-boosting system known as SecureBoost in the setting of federated learning. SecureBoost first conducts entity alignment under a privacy-preserving protocol and then constructs boosting trees across multiple parties with a carefully designed encryption strategy. This federated learning system allows the learning process to be jointly conducted over multiple parties with common user samples but different feature sets, which corresponds to a vertically partitioned data set. An advantage of SecureBoost is that it provides the same level of accuracy as the non-privacy-preserving approach while at the same time, reveals no information of each private data provider. We show that the SecureBoost framework is as accurate as other non-federated gradient tree-boosting algorithms that require centralized data and thus it is highly scalable and practical for industrial applications such as credit risk analysis. To this end, we discuss information leakage during the protocol execution and propose ways to provably reduce it.
1 INTRODUCTION
SecureBoost addresses the tension between privacy protection and organizations’ need to collaborate on machine-learning models. It defines vertically partitioned federated learning and proposes a collaborative, lossless tree-boosting framework.
- GDPR and concerns about personal-data misuse motivate machine-learning frameworks that preserve privacy while enabling data use.
- Organizations seek collaboration because individual data owners may lack enough data to build high-quality models.
- Vertical federated learning partitions features across parties that share users, with only one provider holding labels.
- SecureBoost aligns common users privately, then collaboratively trains a shared classification or regression model without revealing user information.
- The protocol keeps training data local, requires no trusted third party, and is formally defined for privacy-preserving learning over vertically partitioned data.
- SecureBoost is lossless: its accuracy matches centralized, non-privacy-preserving methods that pool the data.
2 PRELIMINARIES AND RELATED WORK
Prior federated-learning methods primarily target horizontally partitioned data, while vertical settings require more complex loss decomposition and stronger privacy handling. Existing vertical decision-tree approaches expose class distributions or support only discrete data, and encrypted logistic regression can compromise performance.
- Earlier federated-learning methods are designed for horizontally partitioned data rather than feature-partitioned vertical federated learning.
- Vertical federated learning requires a more complex mechanism to decompose the loss function at each party.
- Prior privacy-preserving decision-tree methods reveal class distributions and handle only discrete data, limiting practical protection and applicability.
- Encrypted logistic regression methods approximate nonlinear loss with Taylor expansion, which compromises model performance.
3 PROBLEM STATEMENT
The paper formulates privacy-preserving supervised learning over vertically split data, where parties hold different features and overlapping users while one active party owns labels. The goal is joint prediction without revealing private data.
- The data matrix is distributed across private parties by feature, and different parties may hold partially overlapping user sets.
- Only one party holds class labels, creating an active party that coordinates learning as the federated server.
- Passive parties hold only data matrices and act as federated clients.
- The lossless constraint requires federated training loss to equal the loss of a model trained on the union of all data.
4 FEDERATED LEARNING WITH SECUREBOOST
SecureBoost privately aligns users and collaboratively trains gradient-tree boosting over vertically partitioned data. It uses encrypted gradient statistics so the active party can select global splits while parties retain their feature data.
- SecureBoost has two major steps: privacy-constrained data alignment followed by collaborative learning of a shared gradient-tree boosting model.
- Common samples are identified across parties using a privacy-preserving protocol for inter-database intersections.
- Split evaluation depends only on gradient and Hessian statistics, allowing the tree-boosting procedure to adapt to federated learning.
- Passive parties compute encrypted gradient and Hessian statistics locally, which the active party decrypts to determine the global optimal split.
- The protocol relies on additive homomorphic encryption, specifically the Paillier cryptosystem, to aggregate encrypted statistics.
- During inference, parties use lookup tables and record identifiers so the active party can coordinate split decisions across distributed features.
- At leaves, passive parties send encrypted aggregated Hessians to the active party, which decrypts them to compute leaf weights.
5 FEDERATED INFERENCE
Federated inference lets parties jointly classify a private, distributed instance through a secure protocol coordinated by the active party. The process routes decisions across party-held lookup records and features.
- Secure distributed inference enables classification when each party holds private features of the instance.
- Three parties collaborate: the active party holds labels and some features, while passive parties hold other features.
- The active party coordinates traversal from the root by using party and record identifiers to retrieve split attributes.
6 THEORETICAL ANALYSIS FOR LOSSLESS PROP-
The theoretical analysis establishes that SecureBoost is lossless relative to XGBoost under identical initialization and hyperparameters. Encrypted gradient and Hessian information preserves the split decisions needed to produce identical models.
- SecureBoost and XGBoost behave identically when identically initialized and hyperparameterized.
- Encrypted gradients and Hessians provide the information required to calculate the best split using the Paillier cryptosystem.
- Paillier encryption supports homomorphic addition, allowing encrypted messages to be combined into an encryption of their sum.
- With the same initialization, instances have the same gradient and Hessian values in both settings, yielding the same best split.
- Identical best splits continue throughout tree construction, resulting in identical models and establishing losslessness.
7 SECURITY DISCUSSION
The security discussion identifies information leakage risks in SecureBoost and presents RL-SecureBoost, which reduces leakage by storing the first tree at the active party while preserving prediction accuracy.
- Security boundary: The protocol does not reveal parties’ data records, but some information can still be inferred during execution, with leakage differing between passive and active parties.The security analysis therefore distinguishes record protection from inference leakage during protocol execution.
- Information leakage: The active party can learn instance spaces, split responsibility, and candidate gradient and Hessian sums during SecureBoost training.Secure multiparty computation could restrict this knowledge to optimal values per party, but would reduce learning efficiency.
- Information leakage: SecureBoost can reveal sensitive information because the first tree’s leaf purity is inferable from its leaf weights.Leaf purity is the majority-class proportion within a leaf.
- Reduced-Leakage SecureBoost: RL-SecureBoost stores decision-tree leaves at the active party, so passive parties learn only residual-based information rather than the first tree’s instance space.The framework is designed to reduce leakage to passive parties while preserving the learning protocol’s utility.
- Reduced-Leakage SecureBoost: As first-tree leaf purity increases, residual information decreases, providing the security condition used by RL-SecureBoost.The proof connects higher leaf purity with a larger difference between positive- and negative-sample gradient magnitudes, resulting in less information leakage.
- Accuracy: RL-SecureBoost performs identically to SecureBoost in prediction accuracy according to the reported experiments.This result supports the reduced-leakage modification without an observed accuracy difference from SecureBoost.
8 EXPERIMENTS
Experiments on two vertically partitioned credit datasets evaluate SecureBoost and RL-SecureBoost for prediction, security, convergence, and runtime. The results indicate competitive accuracy, reduced information leakage, and scalable training behavior.
- Experimental setup: Experiments use two public credit datasets, with vertical partitioning across two parties and fixed training, tree-depth, sampling, learning-rate, and encryption settings.Credit1 contains 150,000 instances and 10 attributes, while Credit2 contains 30,000 instances and 25 attributes; two-thirds of each dataset is used for training.
- Convergence: SecureBoost shows a convergence curve similar to GBDT and XGBoost on training data and slightly better test performance.The training and test loss behavior is reported as very similar to GBDT and XGBoost.
- Runtime scalability: Runtime increases almost linearly with maximum tree depth, indicating that deeper trees add relatively little training time.The runtime analysis varies individual-tree depth from 3 through 8 and measures one boosting stage.
- Runtime scalability: Sample number and feature number contribute equally to running time in the reported scalability analysis.The analysis varies feature counts from 50 to 5,000 and sample counts from 5,000 to 30,000 while fixing tree depth at 3.
- RL-SecureBoost security: Mean leaf purity decreases significantly from the first tree to the second on both datasets, reflecting reduced information leakage.The second tree’s mean leaf purity is just over 0.6 on both datasets, which the authors describe as sufficient for a safe protocol.
- RL-SecureBoost prediction: RL-SecureBoost performs as accurately as SecureBoost across accuracy, AUC, and f1-score in almost all cases.A pairwise Wilcoxon signed-rank test at significance level 0.05 supports the reported accuracy equivalence, preserving the lossless property.
9 CONCLUSION
SecureBoost is presented as a lossless privacy-preserving tree-boosting framework for private data distributed across multiple parties. The paper reports theoretical accuracy parity with non-federated gradient tree-boosting methods and analyzes information leakage with provable reduction strategies.
- Contribution: SecureBoost trains a high-quality tree-boosting model while private data remains split across multiple parties.The conclusion describes the framework as lossless and privacy-preserving.
- Accuracy: The framework is theoretically as accurate as non-federated gradient tree-boosting counterparts.This is the paper’s stated theoretical accuracy guarantee.
- Security: SecureBoost analyzes information leakage during protocol execution and proposes provable ways to reduce it.The conclusion identifies leakage analysis and reduction as part of the framework’s contribution.