Source-linked AI summary
A Little Is Enough: Circumventing Defenses For Distributed Learning
Moran Baruch, Gilad Baruch, Yoav Goldberg
TL;DR
Distributed learning defenses commonly assume omniscient attackers and large parameter changes, leaving uncertainty about whether smaller, local attacks can evade them. The paper develops a non-omniscient attack based on directed small changes, showing that it can disrupt convergence and implant backdoors while evading existing defenses. The results establish that limited parameter changes can affect both training progress and model behavior.
Problem
Existing distributed-learning attacks and defenses assume omniscient attackers and large parameter changes, while Byzantine workers threaten model correctness.
Method
The paper computes a perturbation range for directed changes across many parameters and uses only corrupted workers’ data to attack distributed learning.
Results
The attack evades all existing defenses and supports both convergence interference and backdoor insertion in distributed learning.
Takeaways & Limitations
Small, well-crafted changes can repurpose or obstruct distributed-learning models without requiring knowledge of non-corrupted workers’ data.
Takeaways & Limitations
Krum’s convergence guarantee does not ensure an effective model, and high-dimensional Euclidean distance can conceal a large change to one parameter.
Abstract
from arXiv · showhide
Distributed learning is central for large-scale training of deep-learning models. However, they are exposed to a security threat in which Byzantine participants can interrupt or control the learning process. Previous attack models and their corresponding defenses assume that the rogue participants are (a) omniscient (know the data of all other participants), and (b) introduce large change to the parameters. We show that small but well-crafted changes are sufficient, leading to a novel non-omniscient attack on distributed learning that go undetected by all existing defenses. We demonstrate our attack method works not only for preventing convergence but also for repurposing of the model behavior (backdooring). We show that 20% of corrupt workers are sufficient to degrade a CIFAR10 model accuracy by 50%, as well as to introduce backdoors into MNIST and CIFAR10 models without hurting their accuracy
1. Introduction
Distributed learning uses many workers whose locally trained models are aggregated by a server, but Byzantine workers can disrupt correctness. This paper shows that small, directed parameter changes enable non-omniscient attacks that evade existing defenses, interfere with training, and implant backdoors.
- 1. Introduction: Distributed learning aggregates locally trained models from multiple workers through a parameter server.In synchronous SGD, workers train on separate data chunks, return parameters, and receive the aggregated model for the next round.
- 1. Introduction: Byzantine workers can violate the protocol through faults, errors, crashes, or adversarially crafted outputs.Such behavior threatens the correctness of the learned model, potentially even with a single Byzantine worker.
- 1. Introduction: Existing defenses rely on i.i.d. parameters and commonly assume attackers are omniscient.The defenses attempt to recover the original mean after removing Byzantine values, while prior attackers may observe all workers’ data.
- 1. Introduction: Directed small changes to many parameters of a few workers can defeat all existing defenses and interfere with or control training.The attack challenges the assumption that effective attacks require large parameter changes.
- 1. Introduction: The paper introduces a non-omniscient distributed-learning attack using only corrupted workers’ data.It exploits the i.i.d. assumption and therefore does not require access to non-corrupted workers’ data.
2. Background
The paper frames distributed-learning attacks around convergence prevention and backdooring, then reviews synchronous SGD and defenses based on trimmed aggregation, clustering, and distance-based selection. These defenses rely on assumptions such as bounded corruption, median structure, or parameter proximity, which the paper identifies as exploitable limitations.
- 2.1. Malicious Objectives: Convergence-prevention attacks obstruct the server from reaching good accuracy, whereas backdooring makes selected inputs produce attacker-chosen outputs.Backdoors may target a single sample or a class of samples containing a specific pixel pattern.
- 2.2. Existing Attacks: Synchronous SGD repeatedly sends server parameters to workers, collects locally trained parameters, and aggregates them into the next model.The attacker intervenes after local training and can replace corrupted workers’ returned parameters before aggregation.
- 2.3. Existing Defenses: Trimmed Mean aggregates each parameter dimension separately after removing values according to median proximity or extreme-value rules.Its variants retain either n−m or n−2m values near the median, or remove the largest and smallest m values.
- 2.3. Existing Defenses: Clustering defenses discard values from a smaller cluster when cluster centers are sufficiently separated.This behaves like a Trimmed Mean variant that averages only the larger cluster containing the median.
- 2.3. Existing Defenses: Krum’s convergence guarantee does not ensure an effective model, and high dimensionality can hide a large single-parameter change under Euclidean distance.Bulyan addresses this disadvantage by combining norm-based selection with dimension-wise Trimmed Mean aggregation.
3. Our Attack
The attack uses small, statistically bounded parameter changes to circumvent TrimmedMean, Krum, and Bulyan, while requiring only corrupted workers’ data. The same perturbation strategy can prevent convergence or introduce backdoors with limited impact on benign functionality.
- Small, consistent changes to many parameters can perturb convergence or backdoor distributed learning while evading defenses designed for large deviations.The attack exploits the i.i.d. assumption and uses parameter-wise standard deviations as its units of perturbation.
- The attacker estimates the population mean and standard deviation from corrupted workers alone, enabling a non-omniscient attack.This works when corrupted workers form a representative portion of the population.
- The same parameter set works against all defenses, making the attack practical regardless of the server’s defense choice.Experiments report that a change of only 1σ can sometimes give the attacker control over the process.
- 3.1. Perturbation Range: A perturbation range is computed so parameter changes remain undetected by TrimmedMean, with the attack selecting the maximal value within that range to obstruct convergence.Corrupted workers can set parameters near µ + zmaxσ, using enough aligned workers to control the median with high probability.
- 3.2. Overcoming Krum and Bulyan Defenses: Small parameter changes can also fool Krum because they reduce Euclidean distance, and the same attack consequently circumvents Bulyan.Krum selects one worker’s full parameter set, while Bulyan combines Krum with TrimmedMean.
- 3.4. Backdooring Attack: For backdooring, the attacker optimizes target behavior while keeping parameters close to their originals, using a weighted backdoor loss and normalized parameter-difference penalty.The weight α should be minimal while still introducing the backdoor; differences beyond zmaxσ receive rapidly increasing penalties.
4. Experiments and Results
Experiments show that small parameter changes can substantially degrade distributed-learning accuracy and evade defenses, with effects varying across datasets and aggregation rules. The same attack framework also introduces backdoors while preserving much benign-input accuracy.
- Convergence prevention: 1σ to 1.5σ parameter changes are sufficient to substantially degrade accuracy, with CIFAR10 more vulnerable than MNIST.The convergence-prevention experiment varied z across 0, 0.5, 1, and 1.5 standard deviations.
- Convergence prevention: 10% corrupted workers can cause major CIFAR10 degradation under existing defenses, while Krum loses 33% accuracy with only 5% corruption.Accuracy decreases as the proportion of corrupted workers grows; no defense is not considered realistic despite its stronger result here.
- Backdooring: The backdoor-sample attack achieves over 95% malicious-target outputs, while MNIST loses less than 1% overall accuracy and CIFAR10 degradation reaches 9%.The malicious target remained effective even in rounds with maximal overall accuracy.
- Backdooring: For backdoor-pattern attacks, CIFAR10 benign-input accuracy degrades by 7% with TrimmedMean, 12% with Krum, and 15% with Bulyan, while over 80% of patterned samples are classified maliciously.MNIST learned the pattern with minimal benign-input impact except under No Defense, where 36.9% of samples received the malicious label.
5. Conclusions
The paper introduces a non-omniscient attack that uses limited changes across many parameters to interfere with or backdoor distributed learning. It reports that the attack evades existing defenses, while a TrimmedMean variant is the preferred existing defense for convergence attacks.
- Conclusions: The attack applies limited changes to many parameters and requires access only to corrupted workers’ data, including in i.i.d. settings.It can interfere with convergence or backdoor the resulting model.
- Conclusions: The attack evades all existing defenses evaluated by the paper, although a TrimmedMean variant gives the best convergence-attack results among existing defenses.Naive averaging is excluded from that recommendation because it is vulnerable to simpler attacks.