Local Model Poisoning Attacks to Byzantine-Robust Federated Learning

Minghong Fang; Xiaoyu Cao; Jinyuan Jia; Neil Zhenqiang Gong

Source-linked AI summary

Local Model Poisoning Attacks to Byzantine-Robust Federated Learning

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, Neil Zhenqiang Gong

arXiv:1911.11815v4Nov 26, 2019cs.CRcs.DCcs.LG

TL;DR

The paper studies whether Byzantine-robust federated learning methods remain vulnerable when compromised clients directly manipulate transmitted local models. It formulates these attacks as per-iteration optimization problems, evaluates them across robust aggregation methods, and finds substantial error increases while generalized defenses work only in some cases.

Problem
Byzantine-robust aggregation rules have theoretical guarantees, but their practical resistance to untargeted attacks that manipulate local model parameters during training remains insufficiently characterized.
Method
The paper systematically studies local model poisoning by formulating compromised clients’ local-model crafting as an optimization problem and generalizing RONI and TRIM-based defenses.
Results
Across four real-world datasets, the attacks substantially increase global-model error rates; for MNIST with Krum, error rises from 0.11 to 0.75, while LFR defends in some cases but not against Krum-based attacks.
Takeaways & Limitations
Existing data-poisoning defenses provide uneven protection, so new defenses for local model poisoning are needed.
Takeaways & Limitations
The work is limited to untargeted poisoning attacks and leaves targeted attacks and new defense mechanisms for future study.

Abstract

from arXiv · show

In federated learning, multiple client devices jointly learn a machine learning model: each client device maintains a local model for its local training dataset, while a master device maintains a global model via aggregating the local models from the client devices. The machine learning community recently proposed several federated learning methods that were claimed to be robust against Byzantine failures (e.g., system failures, adversarial manipulations) of certain client devices. In this work, we perform the first systematic study on local model poisoning attacks to federated learning. We assume an attacker has compromised some client devices, and the attacker manipulates the local model parameters on the compromised client devices during the learning process such that the global model has a large testing error rate. We formulate our attacks as optimization problems and apply our attacks to four recent Byzantine-robust federated learning methods. Our empirical results on four real-world datasets show that our attacks can substantially increase the error rates of the models learnt by the federated learning methods that were claimed to be robust against Byzantine failures of some client devices. We generalize two defenses for data poisoning attacks to defend against our local model poisoning attacks. Our evaluation results show that one defense can effectively defend against our attacks in some cases, but the defenses are not effective enough in other cases, highlighting the need for new defenses against our local model poisoning attacks to federated learning.

1 Introduction

This paper introduces local model poisoning attacks against Byzantine-robust federated learning, targeting the learning process rather than training-data collection. It systematically evaluates these attacks and adapts existing data-poisoning defenses, finding substantial vulnerabilities and incomplete protection.

Federated learning: Federated learning decentralizes training data across worker devices while a master device aggregates their local models into a global model.This setup supports joint learning without sharing local training data.
Motivation: Mean aggregation can be arbitrarily manipulated by compromising one worker, motivating Byzantine-robust rules such as Krum, Bulyan, trimmed mean, and median.These rules were proposed to tolerate failures or adversarial behavior from some worker devices.
Motivation: Data poisoning attacks have limited success against Byzantine-robust federated learning because they alter compromised training datasets while the learning process remains intact.The paper instead targets model parameters transmitted during learning.
Attacks: Local model poisoning attacks manipulate parameters sent by compromised workers and craft them through an optimization problem to drive the global model opposite its normal update direction.The intended deviations accumulate over multiple federated-learning iterations.
Defenses: The proposed ERR and LFR defenses remove suspicious local models before robust aggregation, but LFR succeeds in some cases and fails in others.LFR defends against trimmed-mean-based attacks but not Krum-based attacks, motivating new defenses.

2 Background and Problem Formulation

The paper reviews federated learning and Byzantine-robust aggregation, then formulates untargeted local model poisoning under explicit attacker capabilities and knowledge assumptions. It argues that practical error rates can rise substantially despite asymptotic robustness guarantees.

2.1 Federated Learning: Federated learning minimizes aggregate local objectives while iteratively sending a global model to workers, updating local models, and aggregating them.Workers initialize local optimization from the current global parameters, and the master computes the next global model.
2.1 Federated Learning: Workers update local parameters using their private datasets, commonly with stochastic gradient descent, before sending the resulting models to the master.The master may also select a subset of workers in an iteration.
2.2 Byzantine-robust Aggregation Rules: Mean aggregation is vulnerable because one compromised worker can arbitrarily manipulate the global model, motivating Byzantine-robust alternatives.The reviewed alternatives include Krum, Bulyan, trimmed mean, and median.
2.2 Byzantine-robust Aggregation Rules: Bulyan combines repeated Krum selection with trimmed-mean aggregation, while trimmed mean and median aggregate parameters independently after sorting them.Bulyan inherits Krum-based attack transfer and is not scalable because it repeatedly computes pairwise distances.
2.3 Problem Definition and Threat Model: The attacker seeks an untargeted poisoning outcome: a learned classifier with high error across testing examples, making it unusable and potentially causing denial of service.The paper distinguishes this goal from targeted poisoning attacks.
2.3 Problem Definition and Threat Model: Although robust aggregation rules provide asymptotic error bounds under assumptions, their guarantees do not characterize practical constants that can significantly affect testing error.The paper reports that attacks can substantially increase practical error rates while those asymptotic bounds still hold.
2.3 Problem Definition and Threat Model: The attacker controls c worker devices and may vary knowledge of the aggregation rule and benign workers’ training data and models.The threat model distinguishes full and partial training-data knowledge and known versus unknown aggregation rules.
2.3 Problem Definition and Threat Model: The threat model assumes knowledge of compromised workers’ code, local datasets, and local models, while excluding the feature dimension because compromised workers already expose their features.These dimensions characterize the attacker’s background knowledge and capability.

3 Our Local Model Poisoning Attacks

The attacks craft compromised local models by optimizing their effect on the aggregated global model, targeting deviations opposite to the model’s normal update direction. The paper develops full- and partial-knowledge strategies, including Krum-specific construction techniques and approximations that remain effective despite nonlinear constraints and large search spaces.

The attacker formulates compromised local-model construction as an optimization problem that maximizes the global model’s directed deviation from its before-attack value.The directed deviation targets the inverse of each parameter’s changing direction without attacks.
Attacking Krum: Under Krum, compromised models support a crafted model by staying close to it, increasing its likelihood of selection as the global model.Krum selects one local model per iteration; the supporting compromised models reduce the number of benign models that the crafted model must be close to.
The approximations produce suboptimal solutions because the optimization has nonlinear constraints and a large search space, but the attacks still substantially increase learned-model error rates.A comparison approximation is also reported as less effective than modeling deviation from the received global model.
The attack uses approximations such as w′_1 = w_Re − λs and constrains other compromised models to lie within ε of the crafted model.The objective is simplified to solving for λ, and ε is evaluated experimentally as an attack parameter.
Full knowledge: With full knowledge, the attacker uses an upper bound and binary search to find λ values that cause Krum to select the crafted model.The search halves λ until selection occurs or λ falls below 1×10^-5.
Partial knowledge: With partial knowledge, the attacker estimates changing directions from compromised devices’ mean local model and iteratively adds crafted models when Krum selection is not achieved.The method treats before-attack compromised models as benign references while constructing a selected crafted model.

4 Evaluation

The evaluation tests local model poisoning attacks across four datasets, classifiers, aggregation rules, and attack settings. The attacks substantially raise testing error rates, outperform several baselines, and remain effective under varying system conditions, though their effectiveness depends on aggregation and configuration choices.

Experimental setup: Experiments cover four datasets, logistic-regression and deep-neural-network classifiers, and comparisons among Gaussian, label-flipping, and proposed attacks.The datasets are MNIST, Fashion-MNIST, CH-MNIST, and Breast Cancer Wisconsin (Diagnostic).
Main results: The proposed attacks substantially outperform existing attacks, including label flipping and back-gradient optimization, in increasing testing error rates.Label flipping has limited success for logistic-regression classifiers, while back-gradient attacks have limited success against Byzantine-robust aggregation rules.
Aggregation and configuration effects: Krum is generally less robust than trimmed mean and median, while trimming more parameters weakens the attack but also slightly increases no-attack error.Poisoning more randomly selected iterations increases the median aggregation rule’s error rate; smaller ε improves attacks against Krum.
Attack effectiveness: Increasing the compromised-device fraction significantly strengthens the proposed attacks, while label flipping has only a slight effect and Gaussian attacks have no notable effect on MNIST.The attacks become effective when the compromised fraction exceeds 10%-15%.
Attack effectiveness: Greater non-IID data generally increases testing error for all attacks and no attack, whereas the proposed attack against Krum fluctuates as non-IID increases.More diverse benign local models leave more room for attacks.
Parameter effects: More local SGD rounds reduce attack effectiveness but do not eliminate it, with error-rate increases exceeding 30% even after 10 rounds.More rounds produce more accurate and less diverse local models, while also increasing computational cost for worker devices.

5 Defenses

The paper generalizes RONI and TRIM into defenses that reject potentially malicious local models before Byzantine-robust aggregation. LFR generally outperforms ERR, but both defenses remain ineffective against some optimized attacks.

The generalized defenses remove potentially malicious local models before computing the global model in each federated-learning iteration.ERR uses error-rate impact, whereas LFR uses validation loss impact; Union removes models rejected by either defense.
The defense evaluation uses partial-knowledge attacks with 100 workers, 20% compromised workers, MNIST, logistic regression, and 100 validation examples.Table 6 reports testing error rates across actual aggregation rules and defenses, while attack columns indicate the attacker’s assumed aggregation rule.
LFR matches or substantially outperforms ERR across the reported defense comparisons.For trimmed mean under a Krum-crafted attack, testing error rates are 0.17 with ERR and 0.18 with LFR; under a trimmed-mean-crafted attack, they are 0.21 and 0.12, respectively.

6 Related Work

The paper distinguishes local model poisoning from prior data poisoning, privacy, and testing-phase attacks, and evaluates attacks optimized for Byzantine-robust federated learning. It argues that claimed aggregation robustness does not withstand these crafted local-model manipulations.

Prior poisoning attacks usually compromise training-data collection by injecting malicious examples, whereas this work targets the federated learning process itself.The paper’s local model poisoning attacks manipulate local models sent from compromised worker devices during learning.
The attacks are untargeted: they seek high testing error across test examples, potentially producing denial-of-service behavior.Targeted attacks instead seek attacker-desired predictions, which are outside this work’s scope.
The study differs from centralized-machine-learning poisoning work by optimizing attacks for Byzantine-robust federated learning.The paper also notes concurrent inner-product-manipulation attacks against Byzantine-robust rules including Krum and median.
Existing defenses were mainly designed for data poisoning and are not directly applicable to local model poisoning.TRIM jointly selects a benign training-data subset and model parameters, treating excluded examples as malicious.
The paper reports that aggregation rules claimed to tolerate Byzantine failures are vulnerable to carefully crafted local models from compromised workers.This contrasts with privacy studies, which address a different risk and are described as orthogonal to the paper’s focus.

7 Conclusion, Limitations, and Future Work

The paper demonstrates that optimized local model poisoning attacks can compromise Byzantine-robust federated learning, while generalized data-poisoning defenses work only in some cases. The authors limit the study to untargeted attacks and identify targeted attacks and new defenses as future work.

The attacks manipulate compromised workers’ local models during learning and can drive aggregated global models toward the inverse of their no-attack update direction.Crafting these local models is formulated as an optimization problem.
Generalized defenses are effective in some cases but not effective enough in others.The authors conclude that new defenses are needed for local model poisoning attacks.
The work is limited to untargeted poisoning attacks.The authors identify targeted poisoning attacks in federated learning as an open direction.
Future defense directions include detecting compromised local models and designing new adversarially robust aggregation rules.These are proposed alongside studying targeted poisoning attacks.

A Attacking Bulyan

Because Bulyan is based on Krum, the paper applies Krum attacks to Bulyan and finds that they transfer across the two aggregation rules.

Krum-based attacks transfer to Bulyan in MNIST logistic-regression experiments.The setting uses 100 workers, 20 compromised workers, and Bulyan parameters θ = m−2c and γ = θ−2c.
The partial-knowledge attack increases Bulyan’s error rate by around 150%.

B Deviation Goal

The deviation attack crafts compromised local models by solving an optimization problem that maximizes the global-model deviation under each Byzantine-robust aggregation rule. For Krum and trimmed mean, the attack uses simplifying assumptions or parameter sampling to construct multiple compromised models.

Deviation Goal: The attack solves an optimization problem in each iteration to choose the compromised local models that maximize deviation from the intended global model.The formulation focuses on the full-knowledge scenario and uses an L1-norm objective.
Krum: For Krum, the attack assumes the compromised models are identical and approximates the selected model as w′_1 = w_Re − λ.The resulting optimization problem is solved using a bound on λ and binary search.
Krum: After obtaining the Krum-selected model, the attacker samples c − 1 additional compromised vectors within Euclidean distance ε of it.This construction keeps the other compromised models close to the selected one.
Trimmed Mean: For trimmed mean, the attack selects compromised parameter values outside the benign coordinate-wise range to increase the resulting deviation.Values are randomly sampled from intervals extending beyond w_max,j or w_min,j, with the interval determined by the sign and extremum used.

C Proof of Theorem 1

Theorem 1 derives a necessary condition for Krum to select a crafted compromised model by comparing its distances to benign and compromised models. The resulting bound depends only on the local models before the attack.

Krum Selection: Krum selects the crafted model w′_1 by comparing Euclidean distances to nearby local models.The proof defines the relevant benign and crafted-model neighborhoods using Euclidean distance.
Distance Argument: Because the compromised models are assumed identical, the distance between w′_1 and the other c − 1 compromised models is zero.This zero-distance property is used directly in the Krum distance comparison.
Necessary Condition: The proof applies the triangle inequality to relate distances between benign models and the crafted model.This yields a necessary condition involving the attack deviation parameter λ and the model dimension d.
Bound: The resulting bound depends only on the local models before the attack.Thus, the theorem’s bound is determined from the pre-attack model configuration rather than post-attack outcomes.