Analyzing Federated Learning through an Adversarial Lens

Arjun Nitin Bhagoji; Supriyo Chakraborty; Prateek Mittal; Seraphin Calo

Source-linked AI summary

Analyzing Federated Learning through an Adversarial Lens

Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, Seraphin Calo

arXiv:1811.12470v4Nov 29, 2018cs.LGcs.AIcs.CRstat.ML

TL;DR

The paper asks whether a single constrained agent can poison a federated model despite private data and aggregation defenses. It develops boosted, stealthy, and estimation-based model-poisoning strategies, then evaluates their effectiveness and stealth. The reported results show targeted poisoning against Byzantine-resilient aggregation while preserving effective global models in some settings.

Problem
The paper investigates whether a usually single malicious federated-learning agent can induce high-confidence targeted misclassification when only model updates are shared.
Method
The paper combines explicit boosting, alternating minimization for stealth, benign-update estimation, and evaluation against Byzantine-resilient aggregation mechanisms.
Results
Targeted poisoning remains effective against Krum and coordinate-wise median, while stealthy attacks can maintain convergence and make malicious updates resemble benign ones.
Takeaways & Limitations
The findings indicate that a highly constrained adversary can simultaneously pursue targeted poisoning and stealth in federated learning.
Takeaways & Limitations
Adding training loss does not guarantee that malicious updates are close to benign updates because multiple local minima can have similar loss values.

Abstract

from arXiv · show

Federated learning distributes model training among a multitude of agents, who, guided by privacy concerns, perform training using their local data but share only model parameter updates, for iterative aggregation at the server. In this work, we explore the threat of model poisoning attacks on federated learning initiated by a single, non-colluding malicious agent where the adversarial objective is to cause the model to misclassify a set of chosen inputs with high confidence. We explore a number of strategies to carry out this attack, starting with simple boosting of the malicious agent's update to overcome the effects of other agents' updates. To increase attack stealth, we propose an alternating minimization strategy, which alternately optimizes for the training loss and the adversarial objective. We follow up by using parameter estimation for the benign agents' updates to improve on attack success. Finally, we use a suite of interpretability techniques to generate visual explanations of model decisions for both benign and malicious models and show that the explanations are nearly visually indistinguishable. Our results indicate that even a highly constrained adversary can carry out model poisoning attacks while simultaneously maintaining stealth, thus highlighting the vulnerability of the federated learning setting and the need to develop effective defense strategies.

1 Introduction

The paper studies targeted model-poisoning attacks in federated learning when a usually single malicious agent can exploit opaque parameter updates. It develops attacks that seek both high-confidence misclassification and convergence or stealth.

Threat and objective: A malicious agent can target chosen inputs for high-confidence misclassification by poisoning model updates rather than shared data.The attack differs from adversarial examples because inputs remain unmodified, and model poisoning is suited to federated learning because agents do not share data.
Threat and objective: Unlike Byzantine-learning attacks that disrupt convergence or produce poor minima, these attacks seek targeted poisoning while preserving global-model performance.The paper also reports effectiveness against Byzantine-resilient aggregation mechanisms.
Threat and objective: The threat model usually limits the adversary to one non-colluding agent with no visibility into other agents’ updates.The attacks are evaluated under this constrained setting on Fashion-MNIST and Adult Census datasets.
Attack strategies: Explicit boosting overcomes benign updates, while stealth objectives use accuracy and update statistics to make malicious updates harder to detect.The paper reports that explicit boosting can achieve targeted misclassification with 100% confidence while maintaining convergence, but is detectable using the proposed stealth metrics.
Contributions: The paper combines targeted poisoning, stealth optimization, benign-update estimation, and interpretability analysis to assess attack success and visual indistinguishability.Its stated goal is to show that constrained adversaries can poison federated models while maintaining stealth.

2 Federated Learning and Model Poisoning

Federated learning aggregates locally trained model updates while keeping agent data private, creating an attack surface for model poisoning. The paper defines a single-agent threat model and evaluates detection-oriented accuracy and update-statistic criteria across two datasets and aggregation settings.

Federated learning: Federated learning keeps each agent’s data shard private while agents train from the global model and return local parameter updates for server aggregation.At each step, a random subset of agents is selected, and aggregation produces the next global weight vector.
Federated learning: The server may use weighted averaging, Krum, or coordinate-wise median to aggregate updates.The paper studies Krum and coordinate-wise median as Byzantine-resilient mechanisms in addition to weighted averaging.
Threat model: The adversary controls exactly one non-colluding agent, assumes i.i.d. data partitioning, and must cause targeted misclassification without preventing convergence or degrading test performance.The adversary targets selected samples and desired target classes while operating through model updates.
Stealth metrics: Accuracy checking compares an agent-isolated model with a model formed from the other updates, using γ_t to determine tolerated performance variation.Updates that substantially lower validation accuracy can be flagged as anomalous.
Stealth metrics: Weight-update stealth uses pairwise-distance ranges and histograms to assess whether a malicious update differs statistically from benign updates.The distance-range criterion uses a server-defined, time-dependent threshold κ_t.
Experimental setup: Experiments use Fashion-MNIST and Adult Census with neural networks, varying the total agents K between 10 and 100 and selecting either all or one-tenth of agents per iteration.Training stops at specified test-accuracy thresholds or after maximum time steps.

3 Strategies for Model Poisoning attacks

The paper develops targeted model-poisoning strategies that let a malicious agent influence the global model despite aggregation, while progressively improving stealth and convergence. The methods combine explicit boosting, stealth objectives, distance constraints, and alternating minimization.

Adversarial optimization setup: The adversary optimizes a differentiable targeted-poisoning objective over an estimated global model because it cannot access the current global parameters or other agents’ updates.The estimator accounts for the malicious agent’s available information and aggregation scaling.
Targeted model poisoning: Explicit boosting scales the malicious update by λ to overcome benign agents’ opposing updates after aggregation.The initial update is obtained by running a gradient-based optimizer, and the final update is δ_m^t = λ ˜δ_m^t.
Targeted model poisoning: For K = 10, the explicit-boosting experiment uses E_m = 5 Adam steps and boosts the update by 1/α_m = K.The accompanying evaluation tracks confidence on the malicious objective, validation accuracy, and benign-versus-malicious update distributions.
Stealthy model poisoning: Stealthy poisoning adds training-loss and ℓ2-distance terms so malicious updates resemble benign updates in validation accuracy and update statistics.The distance constraint compares the malicious update with the previous iteration’s average benign update; training loss alone may not ensure update similarity because multiple local minima can have similar losses.
Stealthy model poisoning: The stealthy attack achieves targeted poisoning while maintaining validation accuracy within 10% of the global model for 10 of 15 iterations, with update distances close to benign agents.Its weight-update distribution is also reported as similar to that of a benign agent.
Alternating minimization: Alternating minimization separately optimizes the adversarial and stealth objectives, allowing different step counts and improving control over poisoning and stealth.The attack uses boosted adversarial steps followed by stealth-objective steps; the supplied figure configuration uses λ = 10, ρ = 1e−4, E_m = 10, and 10 stealth steps per malicious step.
Alternating minimization: The alternating attack reaches the adversarial objective with high confidence from t = 2, preserves good validation performance, and achieves complete stealth under both detection properties.Its distance spread closely follows and sometimes overlaps the benign-update spread throughout training.

4 Attacking Byzantine-resilient aggregation

Targeted poisoning remains effective against Byzantine-resilient aggregation, including Krum and coordinate-wise median, while alternating minimization can preserve benign performance.

Targeted model poisoning differs from prior Byzantine attacks by seeking targeted misclassification while retaining an effective model on test data.
4.1 Krum: Krum selected the malicious update for 26 of 40 time steps, meeting the malicious objective with boosting factor λ = 2.Adding training loss to the stealth objective also led the global model to converge to good performance.
4.2 Coordinate-wise median: Using coordinate-wise median without boosting achieved the malicious objective with confidence close to 0.9 for 11 of 14 time steps.The global model nevertheless converged to an effective model, suggesting the attack can exploit information from all agents’ updates.
Targeted poisoning was effective against two different Byzantine-resilient aggregation mechanisms.

5 Improving attack performance through estimation

The malicious agent estimates benign updates from its prior selections and applies the estimate before or after optimization; pre-optimization correction performs better for both attack strategies.

When the malicious agent is selected, it estimates other agents’ effects using prior global parameters, malicious updates, and its local data shard.
Correction methods: Post-optimization correction subtracts a scaled estimate after computing the malicious update to negate benign agents’ effects.The described scaling uses λ = 1/αm.
Correction methods: Pre-optimization correction optimizes the malicious update from an estimated global state that incorporates the other agents’ updates.
Previous-step estimation: The estimate assumes that other agents’ cumulative updates remain the same at each step since the malicious agent was last selected.When selected every time step, the estimate reduces to the corresponding current-step form.
Results: Pre-optimization correction with previous-step estimation is more effective for targeted model poisoning and alternating minimization than post-optimization correction.Post-optimization correction produced lower attack success and impaired global-model convergence.

6 Discussion

The paper contrasts model poisoning with data poisoning and examines whether interpretability exposes poisoned models, while reporting a limitation of implicit boosting.

6.1 Model poisoning vs. data poisoning: Dirty-label data poisoning inserts copies of a sample with a target label, whereas model poisoning modifies updates sent to the server.
6.1 Model poisoning vs. data poisoning: The federated setting motivates dirty-label comparison because data is not shared and clean-label poisoning assumes access to the global parameter vector.
6.2 Interpreting poisoned models: Interpretability techniques analyze input relevance, neuron-feature associations, and prediction gradients to inspect network decisions.
6.2 Interpreting poisoned models: Visual explanations for benign and malicious models were compared on an auxiliary Fashion MNIST sample, with tightly bounded parameter noise intended to preserve internal representations.
A Implicit Boosting: Implicit boosting achieved the adversarial objective in only 4 of 10 iterations and required about 2000 optimization steps per time step.This was about 4× longer than a benign agent, motivating the focus on explicit boosting.

B.1 Results on Adult Census dataset

Experiments on Adult Census reproduce the paper’s broader pattern: targeted attacks can succeed while alternating minimization better preserves benign performance.

The baseline attack induced high-confidence targeted misclassification but reduced benign accuracy from 84.8% to around 80%.
Alternating minimization achieved misclassification with confidence around 0.7 while maintaining 84% benign accuracy.
Adult Census results confirmed the broad conclusions obtained from Fashion MNIST.

B.2 Multiple instance poisoning

The paper evaluates poisoning of multiple target instances and finds that targeted model poisoning succeeds but harms global accuracy, while alternating minimization counteracts this cost.

For r = 10, the attacks target 10 different examples with specific target classes.
Targeted model poisoning induces the desired misclassification but adversely affects the global model’s accuracy.
Alternating minimization counters the accuracy degradation associated with targeted model poisoning.

B.4 Bypassing Byzantine-resilient aggregation mechanisms

Targeted model poisoning remains effective against Krum and coordinate-wise median aggregation, but alternating minimization is ineffective with coordinate-wise median.

Targeted model poisoning attacks Krum using a boosting factor of λ = 2 with k = 10.Because Krum does not require overcoming the constant scaling factor αm, the attack can use a smaller boosting factor.
Targeted model poisoning is effective against coordinate-wise median aggregation.
Alternating minimization is not effective with coordinate-wise median, despite maintaining high validation accuracy for the malicious agent's local model.

C Visualization of weight update distributions

The paper compares weight-update distributions across four attack strategies and visualizes model decisions with and without a malicious agent. These analyses examine attack stealth and whether benign-data decisions change substantially.

Weight update distributions: Four attack strategies produce evolving CNN weight-update distributions on Fashion MNIST.The comparison covers baseline, concatenated training, alternating minimization without distance constraints, and a closer-constrained strategy.
Weight update distributions: Baseline and concatenated training produce large qualitative differences between benign and malicious agents' weight-update distributions.
Weight update distributions: Alternating minimization without distance constraints reduces the qualitative difference between benign and malicious update distributions somewhat.
Attack setting: Figure 10 presents attacks with K = 100 and one malicious agent for a CNN trained on Fashion MNIST.
Decision visualizations: The malicious agent's targeted model poisoning does not significantly affect how the global model makes decisions on benign data.
Decision visualizations: Figures 13 and 14 visualize decisions for global models trained with only benign agents versus nine benign and one malicious agent using the baseline attack.