PaperlayerView exact revision on arXiv

Source-linked AI summary

Vertical Federated Learning: Challenges, Methodologies and Experiments

Kang Wei, Jun Li, Chuan Ma, Ming Ding, Sha Wei, Fan Wu, Guihai Chen, Thilina Ranbaduge

arXiv:2202.04309v2cs.LGcs.DC

TL;DR

VFL enables organizations with different attributes to collaborate on models while keeping data local, but its distinct design raises underexplored challenges. The paper presents a framework, discusses challenges and potential solutions, and evaluates representative approaches. The experiments show trade-offs among privacy, communication, model performance, and resource allocation.

Abstract

from arXiv · show

Recently, federated learning (FL) has emerged as a promising distributed machine learning (ML) technology, owing to the advancing computational and sensing capacities of end-user devices, however with the increasing concerns on users' privacy. As a special architecture in FL, vertical FL (VFL) is capable of constructing a hyper ML model by embracing sub-models from different clients. These sub-models are trained locally by vertically partitioned data with distinct attributes. Therefore, the design of VFL is fundamentally different from that of conventional FL, raising new and unique research issues. In this paper, we aim to discuss key challenges in VFL with effective solutions, and conduct experiments on real-life datasets to shed light on these issues. Specifically, we first propose a general framework on VFL, and highlight the key differences between VFL and conventional FL. Then, we discuss research challenges rooted in VFL systems under four aspects, i.e., security and privacy risks, expensive computation and communication costs, possible structural damage caused by model splitting, and system heterogeneity. Afterwards, we develop solutions to addressing the aforementioned challenges, and conduct extensive experiments to showcase the effectiveness of our proposed solutions.

I. INTRODUCTION

VFL lets organizations with different attributes collaborate on models while keeping their data local, but its core challenges and methods have received limited study. This paper lays out a VFL framework, examines its challenges and possible solutions, and evaluates three solutions experimentally.

I. INTRODUCTION

  • VFL enables non-competing organizations with vertically partitioned data to collaborate on models, with one participant owning labels and others providing additional features without directly disclosing their data.The paper also describes VFL as requiring fewer resources from participating clients than direct data sharing.Source: §I · p. 1
  • Despite VFL's application scenarios, few efforts have investigated its core challenges and methodologies.The introduction situates distributed learning within efforts to use distributed data while preserving privacy and security.Source: §I · p. 1
  • The paper investigates VFL challenges and methodologies, proposes a general framework, and clarifies differences between VFL and horizontal federated learning.Its experiments cover differential-privacy-aided VFL, compression for communication efficiency, and splitting design for resource allocation.Source: §I · pp. 1–2

II. BACKGROUND

VFL trains a shared model from vertically partitioned data through a sequence that aligns participants' samples, exchanges intermediate outputs and gradients, and updates local model parts.

A. Frameworks for VFL

  • Each VFL epoch begins by aligning the participants' common sample IDs through private set intersection, then proceeds through local bottom-model computation and output transmission.The label owner computes the loss and gradients, sends gradients of forward outputs back, and participants update their bottom models.Source: A. Frameworks for VFL · pp. 2–3
  • Participants transmit intermediate forward outputs and receive their gradients, rather than exchanging local model parameters or updates.The backward-transmission bits are usually much smaller than those for forward outputs.Source: A. Frameworks for VFL · pp. 2–3

B. Key Differences between VFL and HFL

VFL and HFL differ in their data partitions, model structures, and exchanged messages, so their design concerns are distinct.

B. Key Differences between VFL and HFL

  • VFL participants share samples but have different attributes, whereas HFL participants share attributes but have different samples.VFL usually involves fewer than five participants, while HFL can involve hundreds or thousands.Source: B. Key Differences between VFL and HFL · p. 3
  • VFL participants each hold part of a full model and exchange intermediate outputs and their gradients, while HFL participants maintain local models and periodically receive a global model.VFL splitting choices must suit the model, heterogeneous attributes, and available communication and computing resources.Source: B. Key Differences between VFL and HFL · p. 3

III. CORE CHALLENGES

VFL raises design challenges distinct from centralized learning and HFL. A central challenge is protecting sensitive information while balancing privacy, efficiency, and model performance.

III. CORE CHALLENGES

  • VFL design faces four core challenges that distinguish it from centralized learning and HFL.The paper identifies security and privacy risks, computation and communication costs, structural damage, and system heterogeneity.Source: §III · p. 3, A. Security and Privacy Risks · p. 4

A. Security and Privacy Risks

  • Analyzing exchanged intermediate outputs and gradients can expose participants' attributes, labels, or batch data, while raw-attribute recovery remains an open research question.Attacks vary by data type; embedded tabular attributes may make private information difficult to obtain.Source: A. Security and Privacy Risks · p. 3
  • VFL privacy protection must balance model performance and system efficiency, and splitting design affects the privacy of bottom-model outputs.Simple linear models can have insecure input processes, while complex nonlinear bottom models can naturally enhance attribute privacy.Source: A. Security and Privacy Risks · pp. 3–4

B. High-cost of Computation and Communication

VFL faces high communication and computation costs, risks damaging model structure through splitting, and must cope with heterogeneous participants and asynchronous updates.

B. High-cost of Computation and Communication

  • In VFL, computation and communication costs scale with training-set size, making pruning, smart compression, and split-position selection important under resource constraints.The usual HFL batch-computation method cannot be applied to VFL.Source: B. High-cost of Computation and… · p. 4

C. Structural Damage of Model Splitting

  • Splitting can disrupt structures such as Wide&Deep and DeepFM, while privacy and disparate attributes complicate the required user–item relationship calculations.Splitting recurrent or attention-based models such as transformers also presents challenges for privacy and communication efficiency.Source: C. Structural Damage of Model Splitting · p. 4

D. System Heterogeneity

  • Variation in participants’ storage, hardware, connectivity, and power can cause asynchronous updates and stragglers, requiring low-participation tolerance and robustness to dropped messages.VFL methods must also tolerate heterogeneous hardware.Source: D. System Heterogeneity · p. 4

IV. METHODOLOGIES AND POSSIBLE SOLUTIONS

The paper surveys possible VFL solutions, emphasizing privacy methods that balance protection against inference with training utility and system costs.

A. Privacy Preserving Frameworks

  • VFL privacy methods must protect exchanged training and serving messages while preserving training utility, but protection techniques can impose other system costs.The paper identifies differential privacy (DP), secure multi-party computing (SMPC), and homomorphic encryption (HE) as candidate techniques.Source: A. Privacy Preserving Frameworks · p. 4
  • DP can protect against inference attacks by perturbing backward outputs or local bottom networks, with privacy levels and data shuffling relevant to the privacy–utility trade-off.The paper notes that tighter bounds on privacy loss can help obtain a better trade-off between utility and privacy.Source: A. Privacy Preserving Frameworks · p. 4
  • SMPC can maintain original utility relative to DP, but cipher inflation and additional information exchange increase complexity.Source: A. Privacy Preserving Frameworks · p. 4
  • HE supports addition and multiplication, so nonlinear operations require approximate functions and incur performance losses; technique and privacy-level choices depend on customer requirements.The paper frames security, efficiency, and utility as a trade-off.Source: A. Privacy Preserving Frameworks · p. 4

B. Enhanced Communication Schemes

The paper discusses communication-reduction schemes and possible mechanisms for asynchronous VFL, including compression, pruning, sampling, and use of prior outputs.

B. Enhanced Communication Schemes

  • Large compression ratios reduce communication cost but increase training error, and classic schemes such as uniform quantization impede backpropagation because forward-output gradients cannot be derived.An approximate function can assist backward propagation; Fig. 2 depicts the compression function as a usually nondifferentiable additive layer whose errors can be compensated approximately.Source: B. Enhanced Communication Schemes · pp. 4–5, Figure 2 · p. 5
  • Pruning unimportant neurons or channels reduces computation and transmission, while selecting batches important to model updates can reduce communication.The paper identifies importance evaluation and data-selection or combination algorithms as further research directions.Source: B. Enhanced Communication Schemes · p. 5

C. Novel Mechanisms for Asynchronous VFL

  • Because participants’ forward outputs differ with their unique attributes, a lagging participant’s previous outputs can be used for the current epoch, and attribute overlaps may also be explored.Source: C. Novel Mechanisms for Asynchronous VFL · p. 5

D. Splitting Design

VFL splitting choices affect resource allocation, privacy, and model performance; the experiments describe a three-participant setup and evaluate AUC across privacy and compression settings.

D. Splitting Design

  • Splitting design affects training performance, communication and computation allocation, and security and privacy risks.Source: D. Splitting Design · p. 5
  • A personalized cut layer can improve resource allocation, but simple bottom models may increase privacy risks, while splitting must preserve structures that capture relations among attributes.The paper proposes considering privacy protection, complexity, and efficient compensating layers in the splitting design.Source: D. Splitting Design · p. 5

V. EXPERIMENT RESULTS

  • The prototype uses Adult and Avazu data divided by attributes among three participants, with an MLP cut at its second layer and 32-dimensional bottom outputs.The stated setup uses three hidden layers of 48, 96, and 196 units and an Adam learning rate of 0.0002.Source: §V · p. 5
  • Fig. 3 reports AUC across privacy levels for three participants on Avazu.Source: Figure 3 · p. 6
  • Fig. 4 reports test AUC across compression levels on Adult.Source: Figure 4 · p. 6

A. DP assisted VFL

The experiments examine privacy-noise effects and communication compression in VFL. Compression reduces transmission cost but can affect AUC, while alternative approximations can compensate for compression bias.

A. DP assisted VFL

  • On Avazu, adding Gaussian noise to forward outputs for privacy affects test AUC because the perturbation biases the loss.The experiments use ε = 1, 1.5, and 2; the paper calls for characterizing the convergence-bound/privacy-level relationship to configure the trade-off.Source: A. DP assisted VFL · p. 6

B. Compression empowered Communication Efficiency

  • On Adult, compression reduces transmission cost, but high compression affects AUC; the authors attribute the effect to bias relative to the original loss.Forward outputs are uniformly quantized into buckets, with each value replaced by its bucket mean.Source: B. Compression empowered Communication… · p. 6
  • At the same compression level, upper-bound and multiplication approximations compensate for forward-compression bias more than addition approximation.The authors note that more efficient approximation functions remain to be explored.Source: B. Compression empowered Communication… · p. 6, Figure 5 · p. 6

C. Splitting Design for Resource Allocation

The experiments compare three VFL split locations, examining test AUC and how splitting redistributes computation and communication costs.

C. Splitting Design for Resource Allocation

  • Scheme 1 splits the first hidden layer and yields 16 values per guest, while Scheme 2 splits the second hidden layer and yields 32 values per guest.Fig. 6 reports communication and computation costs for the VFL framework on the Avazu dataset; each scheme’s left and right bars represent guest and host organization costs, respectively.Source: C. Splitting Design for Resource… · p. 6, Figure 6 · p. 7
  • Test AUC decreases with deeper cutting layers, while splitting farther from raw data increases guest computation cost and reduces host computation cost.The authors also note that splitting closer to raw data allows more original-data information to be combined, while simple preprocessing makes privacy protection challenging.Source: C. Splitting Design for Resource… · p. 7
  • Transmission cost depends on splitting-layer unit size: more units increase transmission cost for both participants and the label owner.Source: C. Splitting Design for Resource… · p. 7

VI. CONCLUSION

The conclusion organizes VFL challenges around security, efficiency, model structure, and heterogeneity. It emphasizes adapting split design and reports evaluations on two real-world datasets.

VI. CONCLUSION

  • The paper groups VFL challenges into security and privacy risks, computation and communication costs, structural damage, and system heterogeneity.Source: §VI · p. 7
  • The authors argue that splitting design should fit the model structure because it also affects security, privacy, computation, and communication efficiency.Source: §VI · p. 7
  • The paper evaluates differential-privacy-assisted VFL, compression for communication efficiency, and splitting for resource allocation on two real-world datasets.Source: §VI · p. 7

Related papers

Loading 2202.04309v2…