Vertical Federated Learning: Concepts, Advances and Challenges

Yang Liu; Yan Kang; Tianyuan Zou; Yanhong Pu; Yuanqin He; Xiaozhou Ye; Ye Ouyang; Ya-Qin Zhang; Qiang Yang

Source-linked AI summary

Vertical Federated Learning: Concepts, Advances and Challenges

Yang Liu, Yan Kang, Tianyuan Zou, Yanhong Pu, Yuanqin He, Xiaozhou Ye, Ye Ouyang, Ya-Qin Zhang, Qiang Yang

arXiv:2211.12814v4Nov 23, 2022cs.LGcs.AIcs.CRcs.DC

TL;DR

VFL addresses industrial data silos by enabling institutions with feature-partitioned data to collaborate under privacy constraints. This paper reviews VFL research and applications, categorizes protocols, analyzes privacy defenses, and proposes VFLow for multi-constraint optimization.

Problem
Industrial organizations hold small, fragmented datasets, while strengthened privacy regulations constrain data sharing and motivate collaborative learning.
Method
The paper comprehensively reviews VFL concepts, algorithms, protocols, attacks, defenses, applications, and challenges, then proposes the unified VFLow framework.
Results
The paper provides an exhaustive VFL categorization and proposes VFLow, incorporating communication, computation, effectiveness, privacy, and fairness constraints.
Takeaways & Limitations
VFL offers a framework for cross-institution and cross-industry collaboration, while motivating further research on its identified challenges.
Takeaways & Limitations
Trustworthy VFL still lacks universally effective defenses that are both lossless and highly efficient, leaving utility, privacy, and efficiency in trade-off.

Abstract

from arXiv · show

Vertical Federated Learning (VFL) is a federated learning setting where multiple parties with different features about the same set of users jointly train machine learning models without exposing their raw data or model parameters. Motivated by the rapid growth in VFL research and real-world applications, we provide a comprehensive review of the concept and algorithms of VFL, as well as current advances and challenges in various aspects, including effectiveness, efficiency, and privacy. We provide an exhaustive categorization for VFL settings and privacy-preserving protocols and comprehensively analyze the privacy attacks and defense strategies for each protocol. In the end, we propose a unified framework, termed VFLow, which considers the VFL problem under communication, computation, privacy, as well as effectiveness and fairness constraints. Finally, we review the most recent advances in industrial applications, highlighting open challenges and future directions for VFL.

1 Introduction

Federated learning enables parties to train models without centralizing data, with HFL, VFL, and FTL distinguished by how samples and features are partitioned. The paper surveys VFL’s growing applications and challenges, then proposes VFLow to organize optimization across multiple constraints.

Federated Learning enables multiple parties to collaboratively build machine-learning models without centralizing their data.
HFL shares features across parties with different samples, whereas VFL shares samples or users with different features.
FTL addresses datasets with limited overlap in both feature and sample spaces.
VFL keeps each party’s data and model local, exchanges intermediate results, produces separate local models, and requires collaboration during inference.
VFL’s industrial need has grown as organizations seek complementary data partners amid stronger privacy and security regulations.
The paper fills a gap in comprehensive VFL surveys by categorizing VFL settings and privacy-preserving protocols and proposing VFLow across communication, computation, effectiveness, privacy, and fairness constraints.

2 Vertical Federated Learning framework

VFL partitions features across parties sharing aligned samples, with an active party coordinating a global module and passive parties retaining local models. Training exchanges intermediate results under privacy-preserving protocols, while variants differ by whether the global module is trainable and by model type.

Problem definition: VFL distributes each sample’s features across K parties, with the Kth active party holding labels and the remaining parties acting as passive parties.Each party retains its local feature data; the active party also owns the global module.
Architecture: The framework decomposes the joint model into local models Gk operating on party-specific data and a global module FK accessible only to the active party.The global module may be trainable or non-trainable, yielding different VFL variants.
VFL variants: SplitVFL uses a trainable global module, whereas aggVFL uses a non-trainable aggregation function such as Sigmoid or optimal split finding.The paper also identifies variants in which the active party has no features and acts as a central server.
Training protocol: VFL training first aligns shared samples privately, then collaboratively optimizes local models and the global module by exchanging intermediate results rather than raw data.Gradient-descent training sends local outputs to the active party and returns corresponding gradients for local updates until convergence.
Privacy preservation: Homomorphic encryption, secure multi-party computation, trusted execution environments, differential privacy, and gradient discretization protect exchanged outputs and gradients.These mechanisms address leakage risks from intermediate results and can involve a third party for encryption and decryption.
Tree-based VFL: Tree-based VFL represents each party’s local model with partial trees and uses the global module to identify optimal feature splits from party-provided information.GBDT-based methods use techniques including homomorphic encryption, secret sharing, and differential privacy for transmitted information.

3 Improving Communication Efficiency

VFL communication efficiency is improved through fewer synchronizations, asynchronous coordination, one-shot communication, and compressed transmissions, each trading communication savings against computation, convergence, or privacy concerns.

VFL communication is a production bottleneck because heterogeneous networks, long distances, and large encrypted data increase coordination costs.
Multiple Client Updates: Multiple local updates reduce synchronizations by allowing parties to perform several updates before communicating, but parameter choices create computation–communication trade-offs.FedBCD, Flex-VFL, ADMM-based methods, and CELU-VFL exemplify this strategy.
Asynchronous Coordination: Asynchronous coordination reduces synchronization requirements, but stale information can harm model performance and communication efficiency while inconsistency handling adds computation overhead.
One-shot Communication: One-shot communication trains local representations once and then lets the active party train the global model, reducing communication to a single coordination stage.The approach uses unsupervised representation learning, including NAT, autoencoders, or PCA-based extraction.
One-shot Communication: One-shot methods permanently transfer sample-wise representations and may require expensive unsupervised learning, creating privacy and computation trade-offs.The privacy risks include possible inversion attacks or information-theoretic concerns.
Compression: Compression reduces transmitted data and can also lower computation overhead when expensive homomorphic-encryption or MPC operations are used.Examples include PCA, autoencoders, encrypted-gradient encoding, sample coresets, and feature filtering.

4 Improving Effectiveness

Conventional VFL is limited by scarce aligned labeled data and the need for collaborative inference. Research therefore exploits unlabeled, unaligned, or newly labeled data and transfers federated knowledge to local predictors.

Conventional VFL uses only aligned labeled samples, although real applications often have few aligned or labeled samples and require collaborative inference.
The literature organizes effectiveness methods around the virtual dataset’s aligned, labeled, unlabeled, and unaligned subsets to identify which data each algorithm uses.Table 4 summarizes Semi-SL, Self-SL, KD, and TL methods and their utilized data portions.
Self-Supervised Learning: Self-supervised VFL adds a self-supervised loss to the main task loss so local models learn representations from unlabeled data.VFed-SSD pretrains with positive and negative pairs from aligned unlabeled data, then fine-tunes on labeled aligned samples.
Semi-Supervised Learning: Semi-supervised VFL expands the labeled aligned set by pseudo-labeling unlabeled samples or adding newly labeled samples while targeting stable and precise labels.FedCVT estimates missing-feature representations, predicts pseudo-labels, and selects additions through an ensemble approach.
Knowledge Distillation: Knowledge distillation transfers knowledge from VFL teacher models to active-party local models, enabling prediction on unaligned local data.VFedTrans learns federated representations from aligned samples and transfers them through autoencoder teachers.
Transfer Learning: Transfer-learning VFL treats the active party as a labeled source domain and the passive party as a sparsely labeled or unlabeled target domain.The transfer loss and source task loss reduce domain discrepancy and transfer knowledge, while an available target task loss adapts it locally.

5 Preserving Data Privacy and Defending Against Attacks

VFL privacy threats can originate inside or outside the system, and attacks may be honest-but-curious or malicious depending on protocol adherence. Private Set Intersection commonly protects entity alignment.

VFL attackers may be internal or external, with honest-but-curious attackers following the protocol and malicious attackers deviating from it.
Privacy-Preserving Entity Alignment: Private Set Intersection is the most common VFL method for finding common user IDs without revealing other information or unaligned datasets.Implementations include encryption, signatures, and oblivious transfer.

5.2 Privacy-Preserving Training Protocols

VFL training protocols form an ascending privacy hierarchy: local plaintext handling, cryptographic protection of transmitted results, protection of the entire training protocol, and protection of learned models.

Figure 6 conceptually depicts information flow between active and passive parties for protocols P-1 through P-4.
Basic Protocol (P-1): P-1 keeps private data and models local while transmitting intermediate results in plaintext instead of raw data during training and inference.A security proof is reported for preventing exact recovery of private features under P-1.
Relaxed Protocol (P-0): P-0 relaxes P-1 by allowing labels or the global module and local models to be nonprivate.P-0(y) denotes nonprivate labels, while P-0(g) denotes nonprivate models.
VFL data-inference attacks commonly involve the active party inferring passive-party features and the passive party inferring active-party labels.
Standard Protocol (P-2): P-2 cryptographically protects transmitted intermediate results while leaving other within-party training information in plaintext to balance privacy and efficiency.Homomorphic encryption can protect sample-level outputs and gradients exchanged with the active party.
Enhanced Protocol (P-3): P-3 extends P-2 by protecting the entire training protocol so that parties receive no training information beyond the resulting trained models.Secure MPC can protect batch-level gradients and local parameters.
Strict Protocol (P-4): P-4 additionally protects final learned models and reveals only final inference results, addressing risks from owners using local models to infer other parties’ information.Secret sharing and hybrid homomorphic-encryption schemes are examples of P-4 techniques.

5.3 Defending against Data Inference Attacks

VFL data-inference defenses must protect both private features and labels because both are sensitive targets. The review organizes attacks by accessible information, including sample-level gradients, batch-level gradients, trained models, and model knowledge settings.

VFL privacy research must protect both features and labels, unlike many HFL attacks that primarily target features.
Label Inference Attacks: Sample-level gradients in P-1 can enable Direct Label Inference, reaching accuracy up to 100% with a nontrainable softmax global module.
Label Inference Attacks: Batch-level gradients remain exploitable in P-2: gradient inversion and residue reconstruction can infer labels without revealed intermediate results.
Label Inference Attacks: Under P-3, only trained local models are revealed, but attackers can fine-tune a local model with auxiliary labeled data to predict labels.
Feature Inference Attacks: White-box feature inference uses model inversion during inference or gradient inversion during training, while black-box attacks generally require prior model or data knowledge.

5.4 Defending against Backdoor Attacks

VFL backdoor attacks are typically mounted by passive parties to mislead the active party’s model or damage its original-task performance. Existing work distinguishes targeted and non-targeted attacks, while defenses face utility trade-offs and remain immature.

Passive parties typically act as attackers, targeting the active party because it owns the labels.
Attack Categories: Targeted attacks optimize high performance on both clean and chosen backdoor tasks, whereas non-targeted attacks aim to hurt convergence or original-task performance.
Targeted Attacks: Targeted methods include gradient replacement for label-specific triggers and adversarial dominating inputs that override other features.
Non-targeted Attacks: Non-targeted methods use adversarial, noisy, or missing-feature samples; missing-feature attacks simulate unstable networks that interrupt collaboration.
Defenses: Noise and GS can defend against both categories, but they trade off main-task accuracy against backdoor-task accuracy, while cryptographic defenses generally do not alter backdoor objectives.
Defenses: Backdoor-defense research remains at an early stage, with open need for effective strategies that preserve model utility.

6 Data Valuation and Fairness

VFL research is expanding toward valuation, explainability, fairness, and broader datasets, but each area remains underdeveloped. Key challenges include allocating value fairly, explaining models in regulated settings, debiasing privately, and building diverse benchmarks.

Data Valuation: Fair data valuation and incentive design remain critical for allocating profits among institutions and sustaining cross-industry federations.
Data Valuation: VFL data valuation is much less studied than HFL, with early methods using feature-based Shapley valuations and conditional mutual information.
Explainability: Explainability is especially important in regulated financial and medical settings, but only limited VFL work addresses it.
Fairness: Fairness research includes adversarial debiasing and fairness objectives, while the core challenge is identifying sensitive features without sacrificing privacy or efficiency.
Datasets and Benchmarks: VFL benchmarks are dominated by convenient tabular finance, healthcare, and advertising datasets, leaving diverse image, text, and video datasets underrepresented.
Datasets and Benchmarks: Only NUSWIDE and Vehicle naturally provide multimodal features, and no established partitioning method exists for many adopted datasets.
Datasets and Benchmarks: Practical datasets and high-quality benchmarks are needed to support both industrial applications and academic VFL research.

7 VFLow: A VFL Optimization Framework

VFLow is a unified framework for setting up, designing, and optimizing VFL algorithms under multiple practical constraints. It treats algorithm design as constrained multi-objective optimization, yielding trade-off solutions for stakeholders.

VFLow provides a comprehensive framework for setting up and optimizing VFL algorithms.
Framework Components: The framework guides choices spanning model architecture, data partitioning, effectiveness and efficiency improvements, privacy defenses, and fairness strategies.
Framework Components: VFLow includes a separate risk-evaluation module for assessing data attacks and defense strategies.
Optimization Formulation: Its optimization considers models and settings together with effectiveness, privacy, efficiency, and fairness strategies and their measurements.
Optimization Formulation: The formulation is a constrained multi-objective federated-learning problem whose solutions represent optimal trade-offs among objectives.

8 Applications

VFL applications span advertising, finance, healthcare, emerging infrastructure, and multimodal tasks, using privacy-preserving collaboration across institutions and platforms.

Advertising and recommendation: VFL supports advertising through federated bandits, privacy-preserving collaborative filtering, higher-order factorization machines, and cross-platform recommendation systems.Internet companies have also applied tree-based and platform-based VFL systems to advertising.
Finance: Finance applications include scorecard training, secure sparse logistic regression for risk control, and adversarial domain adaptation for label deficiency.
Healthcare: Healthcare applications use privacy-preserving logistic regression, asynchronous VFL, and cancer survival analysis for clinical and public-health tasks.The asynchronous framework was evaluated on the MIMIC-III public healthcare dataset.
Emerging applications: Emerging VFL applications address electric-vehicle charging-station placement, 5G wireless networks, cognitive-radio sensing, and related infrastructure problems.
Multimodal tasks: Multimodal VFL combines modalities such as vision, language, and sensing, including representation learning and neural architecture search across participants.

9 Open Challenges and Future Direction

The paper identifies interoperability and trustworthiness as major open challenges for VFL, alongside automation and blockchain integration. Proposed directions include interoperable modules, multi-objective optimization, verifiable models, and decentralized exchange.

Interoperability: Existing VFL platforms lack interoperability because they use different secure-computation and privacy-preserving training protocols.Plug-in seed projects are proposed to provide shared functionality across diverse platforms.
Trustworthy VFL: Trustworthy VFL must address privacy, security, effectiveness, efficiency, fairness, explainability, robustness, and verifiability.The paper notes a continuing utility-privacy-efficiency trade-off and a lack of universally effective, lossless, highly efficient defenses.
Trustworthy VFL: VFL research still faces the challenge of balancing privacy, utility, and efficiency while making trained models verifiable and auditable.Verifiable intellectual-property protection is discussed as one possible route for released models.
Automated and Blockchained VFL: AutoML and neural architecture search can reduce human effort by learning individual model architectures for VFL clients, but encryption creates additional challenges.Participants without labels cannot perform individual training or evaluation locally.
Automated and Blockchained VFL: Blockchain could reduce reliance on a central server by enabling decentralized and verifiable model-update exchange among participating parties.The paper identifies integration of blockchain with VFL as a future direction for improving security and robustness.

10 Concluding Remarks

VFL enables collaborative learning from feature-partitioned data across institutions and addresses industrial data-silo problems under strict data regulations. The paper organizes existing research, identifies challenges, and proposes a framework for future work.

Concluding remarks: VFL enables institutions to collaboratively learn from feature-partitioned data without directly consolidating their separate data holdings.The paper presents VFL as a solution to industrial data-silo problems arising under strict data regulations.
Concluding remarks: The paper provides an extensive categorization of VFL research, formulates new challenges, and proposes a framework covering relevant aspects of the field.It calls for further research to address these challenges.