PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Sybils in Federated Learning Poisoning

Clement Fung, Chris J. M. Yoon, Ivan Beschastnikh

arXiv:1808.04866v5cs.LGcs.CRcs.DCstat.ML

TL;DR

Federated learning is vulnerable to model poisoning when malicious clients exploit sybil identities, while servers lack client data and validation information. The paper proposes FoolsGold, which adapts client learning rates from contribution similarity, and reports mitigation across varied attacks and settings, with limitations against single clients and knowledgeable sybil adversaries.

  • Problem

    Federated learning permits malicious clients to submit arbitrary updates, and sybils can amplify their influence in poisoning the shared model.

  • Method

    FoolsGold adapts client learning rates using contribution similarity and does not require prior knowledge of the expected number of sybils.

  • Results

    FoolsGold mitigates diverse label-flipping and backdoor attacks, including cases where sybils overwhelm honest users, and outperforms prior work across evaluated scenarios.

  • Takeaways & Limitations

    FoolsGold minimally changes federated learning while defending against sybil-based poisoning across varied data distributions, targets, and sybil strategies.

  • Takeaways & Limitations

    FoolsGold is not successful against a single poisoning client and can be subverted by a knowledgeable adversary with many sybils.

Abstract

from arXiv · show

Machine learning (ML) over distributed multi-party data is required for a variety of domains. Existing approaches, such as federated learning, collect the outputs computed by a group of devices at a central aggregator and run iterative algorithms to train a globally shared model. Unfortunately, such approaches are susceptible to a variety of attacks, including model poisoning, which is made substantially worse in the presence of sybils. In this paper we first evaluate the vulnerability of federated learning to sybil-based poisoning attacks. We then describe \emph{FoolsGold}, a novel defense to this problem that identifies poisoning sybils based on the diversity of client updates in the distributed learning process. Unlike prior work, our system does not bound the expected number of attackers, requires no auxiliary information outside of the learning process, and makes fewer assumptions about clients and their data. In our evaluation we show that FoolsGold exceeds the capabilities of existing state of the art approaches to countering sybil-based label-flipping and backdoor poisoning attacks. Our results hold for different distributions of client data, varying poisoning targets, and various sybil strategies. Code can be found at: https://github.com/DistributedML/FoolsGold

1. Introduction

Federated learning keeps training data on client devices but lets clients submit arbitrary updates, creating vulnerability to model poisoning and sybil attacks. The paper introduces FoolsGold to address these attacks without assuming a fixed number of attackers.

  • Motivation: Federated learning preserves client-side data privacy by transferring model parameters to a central aggregator.Clients compute updates locally and independently while sharing only model parameters.
  • Vulnerability: Clients can submit arbitrary updates, enabling malicious participants to manipulate the trained model through model poisoning.The decentralized training process exposes intermediate model state and permits client contributions.
  • Vulnerability: 96.2% of MNIST 1s were predicted as 7s when only 2 sybils mounted a label-flipping attack.The sybils used poisoned 1s labeled as 7s, overpowering the honest client holding digit-1 data.
  • Contribution: FoolsGold adapts client learning rates using inter-client contribution similarity to defend against sybil-based poisoning.The defense operates with minimal changes to federated learning and does not require an explicit attacker-count parameter.
  • Evaluation: The evaluation covers label-flipping and backdoor attacks across diverse datasets, model types, client-data distributions, targets, and sybil strategies.Existing defenses such as Multi-Krum and RONI are reported as inadequate in this setting.

2. Background

The paper frames federated learning poisoning through SGD updates, targeted attack types, and sybil coordination. It applies FoolsGold to both FEDSGD and FEDAVG while relying on contribution similarity across clients.

  • SGD: SGD updates model parameters using a local learning rate, regularization, and gradients computed from minibatches.The minibatch contains training examples (x_i, y_i), and its size affects update variance.
  • SGD: Smaller SGD batches increase gradient-direction variance because each iteration samples less of the dataset.This stochastic behavior distinguishes SGD from full-batch gradient descent.
  • Federated learning: Federated learning distributes data across owners and performs synchronous update rounds without sharing the underlying data.The process can converge even when clients hold non-IID data.
  • Federated learning: FEDSGD sends every client SGD update to the server, whereas FEDAVG batches multiple local SGD iterations before communication.The paper reports that FoolsGold applies successfully to both algorithms.
  • Targeted poisoning: Label-flipping attacks change class labels while preserving features, whereas backdoor attacks add secret trigger patterns and relabel examples.Both are targeted poisoning strategies considered by FoolsGold.
  • Targeted poisoning: In federated learning, poisoning is represented by a subset of poisoned client updates, and sybils coordinate updates toward a shared poisoned objective.The attacker may derive updates from poisoned data or other methods.

3. Assumptions and threat model

The threat model gives sybils broad control over their coordinated updates while hiding honest-client data from the adversary. FoolsGold assumes an uncompromised aggregator, representative honest data, and observable individual updates.

  • Adversary: Adversaries access training state only through the federated learning API and cannot observe honest clients’ training data.Data remains distributed and hidden across clients.
  • Assumptions: The defense requires an uncompromised aggregator and at least one honest client representing every model-defined class.Without such honest representation, contribution-based defense is impossible for the missing classes.
  • Assumptions: FoolsGold assumes individual client updates are visible and that secure aggregation obfuscations are not used.The server must observe each client’s model update at every iteration.
  • Poisoning model: The poisoning model covers label-flipping and backdoor strategies targeting class predictions.The stated goal is to increase one incorrect class probability without affecting other class outputs.
  • Scope boundary: The study excludes update-scaling attacks because magnitude-based detection methods are assumed to prevent them.The possible malicious-update range is otherwise unbounded.
  • Adversary: Sybils may observe global model state, send arbitrary gradients, share state, and coordinate their updates across iterations.Multiple non-colluding adversaries may also conduct poisoning attacks concurrently.

4. SGD Challenges and defenses

SGD makes sybil detection difficult because updates are stochastic, variable, and ambiguous without validation data. Existing defenses struggle in this setting, especially when sybils can influence the mean or overwhelm honest clients.

  • Detection challenges: The aggregator cannot reliably determine whether an individual SGD gradient supports a malicious objective because it lacks validation data and sees only update outputs.This limits direct assessment of update intent.
  • Detection challenges: Similar updates need not originate from similar datasets, while sporadic updates need not be malicious because the stochastic objective changes each iteration.These ambiguities complicate similarity- and anomaly-based detection.
  • Detection challenges: Smaller batch sizes increase update variance, and adversaries cannot be trusted to follow the configured batch size.This adds uncertainty to interpreting client-update behavior.
  • Prior defenses: Existing poisoning defenses become ineffective against sybil-based attacks, particularly with non-IID client data.The paper identifies the preceding stochasticity and observability challenges as contributing factors.
  • Prior defenses: Multi-Krum removes updates farthest from the mean but requires an explicit f parameter and can have its mean influenced by sybil contributions.The paper compares FoolsGold against Multi-Krum under these conditions.

5. FoolsGold design

FoolsGold identifies likely poisoning sybils through the similarity and historical diversity of client updates, then adjusts client learning rates accordingly. Its design targets robust sybil mitigation without attacker-count assumptions while preserving honest contributions.

  • FoolsGold design: FoolsGold separates honest clients from sybils using the greater similarity of sybils’ gradient updates under a shared malicious objective.Honest clients’ unique data distributions produce more diverse updates, whereas sybils’ common objective makes their updates appear more similar.
  • FoolsGold design: The method adapts each client’s learning rate from indicative-feature update similarity in the current iteration and across historical updates.Historical aggregation captures overall client contributions rather than relying only on a single potentially divergent update.
  • FoolsGold design: FoolsGold uses cosine similarity on indicative features, emphasizing output-layer parameters whose magnitudes map to prediction influence.Non-output-layer magnitudes in deep networks are excluded because they do not directly map to output probabilities.
  • FoolsGold design: Pardoning reweights similarity scores to reduce mistaken penalties for honest clients, while assuming at least one honest client for normalization.The resulting scaling ensures at least one client retains an unmodified update.
  • FoolsGold design: A centered logit transformation increases separation near similarity extremes, with confidence parameter κ scaling its effect.Values outside the 0–1 range are clipped and rounded to the corresponding boundary before the final update is applied.
  • FoolsGold design: The design does not require parameterizing the expected number or properties of sybils and is independent of the underlying model and local SGD details.Its stated goals also include preserving unattacked federated-learning performance and remaining robust as sybil numbers increase.

6. Evaluation

FoolsGold was evaluated across attack scenarios, datasets, client-data distributions, and adaptive sybil strategies. It generally maintained low attack rates and high accuracy, while showing specific weaknesses against single attackers, false positives, and low-feature intelligent-noise attacks.

  • Canonical attack scenarios: FoolsGold maintained high training accuracy while preventing most canonical attacks, including A-OnOne and A-99.Its detection improves with larger sybil groups because additional colluding updates provide more information about similarity.
  • Comparison to prior work: A single poisoning client remains a limitation: FoolsGold was not successful at mitigating attacks mounted by one client.The method is designed for sybil-based attacks and performs worst when only one poisoner is present.
  • Comparison to prior work: 9 attackers remained within FoolsGold’s robust range, whereas Multi-Krum fails above 33% sybils.Multi-Krum can also fail when honest-update variance is high and sybil-update variance is lower, causing honest clients to be removed in non-IID settings.
  • Varying client data distributions: Less than 1% maximum attack rate was observed across MNIST and VGGFace2 with both SqueezeNet and VGGNet under varied client-data distributions.The evaluation reports robustness across all tested combinations, while noting that attackers would need to manipulate gradient outputs directly rather than merely their malicious data distribution.
  • What if the attacker knows FoolsGold?: When indicative features fall below 10%, intelligent noise fails with an attack rate near 0; at 0.01, honest-client similarity can cause false positives.FoolsGold mitigates the strategy by weighting cosine similarity toward feature importance rather than using all model features equally.

7. Limitations

FoolsGold is designed for sybil-based poisoning and is not effective against a single poisoning client. Combining it with Multi-Krum addresses concurrent strawman and sybil attacks, while informed adversaries and stronger similarity methods remain limitations.

  • Single-client adversaries: FoolsGold does not successfully mitigate attacks mounted by a single poisoning client.The authors note that Multi-Krum can detect and remove a single malicious actor, although it performs poorly in non-IID settings.
  • Single-client adversaries: A strawman adversary can bypass FoolsGold by sending the poisoning-objective vector directly at every iteration.The attack uses no sybils and is therefore outside FoolsGold’s primary defense target.
  • Combined defenses: With f = 1, Multi-Krum and FoolsGold do not interfere: Multi-Krum prevents the strawman attack while FoolsGold prevents the sybil attack.Individually, each defense fails against the other attack type when both attacks occur concurrently.
  • Informed attacks: A knowledgeable adversary with many sybils can subvert FoolsGold, motivating additional nondeterminism in its history or similarity calculations.The proposed directions include weighted random subsets of gradients and similarity measurements across random subsets of client contributions.
  • Informed attacks: Alternative similarity metrics can be more informed but require additional assumptions about the attack.Examples include graph-based similarity, auxiliary client-dataset information, and minimum similarity thresholds.

8. Related work

Prior defenses address poisoning or sybils through client control, auxiliary information, secure infrastructure, provenance, or clustering assumptions. FoolsGold instead uses gradient similarity while assuming sybils are present.

  • Poisoning defenses: Centralized poisoning defenses assume control of clients or direct observation of training data, which federated learning does not provide.The federated server observes only model parameters exchanged during iterative training.
  • Secure ML: Secure enclaves do not prevent sybil poisoning performed through malicious input data, although FoolsGold can be added to SGX-based systems.The attack can occur even when trusted code runs inside the enclave.
  • Data provenance and proof of work: Data-provenance defenses require assumptions about data collection that are described as unrealistic for federated learning’s diverse client sources.Proof-of-work approaches instead impose computational costs on clients.
  • Clustering defenses: AUROR clusters modifications to indicative model features and removes small, distant clusters, assuming most updates for each feature are honest.FoolsGold assumes sybils and uses gradient similarity to detect anomalous clients.

9. Conclusion

The paper studies sybil-based poisoning in decentralized ML and proposes FoolsGold, which uses client contribution similarity. It reports mitigation across varied attacks and strategies, with performance exceeding prior work while requiring few algorithmic changes.

  • Conclusion: The paper shows existing defenses are ineffective against sybil-based poisoning and proposes FoolsGold using client contribution similarity.The stated problem concerns poisoning objectives achieved by sybil clients in decentralized ML.
  • Conclusion: FoolsGold mitigates varied attack types, including cases where sybils overwhelm honest users.The evaluation also considers sybils mixing poisoned and honest data, adding intelligent noise, and rate-limiting poisoned updates.
  • Conclusion: Across the evaluated scenarios, FoolsGold outperforms prior work.The conclusion presents this as the aggregate evaluation outcome across the advanced attack strategies considered.
  • Conclusion: FoolsGold minimally changes federated learning, uses standard techniques such as cosine similarity, and requires no prior knowledge of the expected sybil count.The authors frame this design as motivation for defenses co-designed with the learning procedure.

Appendix A: Convergence analysis

The convergence analysis argues that FoolsGold preserves honest clients’ learning rates while removing sybil contributions under stated assumptions. It relies on sybil gradient alignment and sufficiently low honest-client similarity.

  • Theorem and proof: The theorem states an O(1/T^2) convergence rate over T iterations for malicious and honest participants under Algorithm 1.The proof invokes the corresponding constant-learning-rate SGD convergence result.
  • Theorem and proof: The proof requires FoolsGold to leave honest learning rates unchanged and remove sybil contributions so standard SGD convergence applies.These are the two conditions imposed on the learning-rate adaptation function h(i,t).
  • Sybil-gradient condition: Because sybils share a poisoning goal, the analysis models their ideal gradients as the same vector vm.This common target causes their historical gradients to approach vm, up to honest-client error ϵ.
  • Sybil-gradient condition: As training continues, increasing pairwise similarity among sybil historical updates can drive their adapted learning rate to 0 when similarity remains below βm.This satisfies the analysis’s first condition for removing sybil contributions.
  • Honest-client condition: FoolsGold preserves honest learning rates when honest clients’ maximum ideal-gradient cosine similarity βg is sufficiently low.Under this condition, the defense classifies the clients as honest and does not modify their learning rates.

Appendix B: Additional Evaluations

Additional evaluations show that RONI is unreliable for sybil poisoning in non-IID federated learning, while FoolsGold remains effective across attack types, targets, sybil strategies, and batch sizes, with specific low-rate exceptions.

  • Comparison to RONI: RONI cannot reliably distinguish malicious sybils from honest non-IID clients when using a uniform validation set.It produces false positives because individual client updates may not represent the global data distribution.
  • Attack generalization: FoolsGold generalizes across MNIST source-target label-flipping attacks, with a highest observed attack rate of 0.02.The evaluation covers all possible source and target MNIST label combinations using the A-5 attack.
  • Backdoor attacks: FoolsGold defends against single-pixel backdoor attacks even as the number of sybils increases, under both FEDSGD and FEDAVG.The comparison includes federated learning, Multi-Krum, and FoolsGold across increasing poisoner counts.
  • Resilience to mixed data: FoolsGold remains robust when attackers mix honest and poisoned data, with a maximum average attack rate of 0.003 for mixing proportions from 20% to 80%.This result was observed on the KDDCup and Amazon datasets; their attack rates were all 0.
  • Resilience to different batch sizes: FoolsGold achieves attack rates at or near 0 across batch sizes for MNIST and KDDCup A-5 attacks.Amazon was not evaluated at batch sizes 50 and 100 because no partition contained more than 50 examples.
  • Resilience to different batch sizes: The main batch-size exception is the Amazon A-5 attack with batch size 1, which reached an attack rate of 4.76%.The paper attributes this degradation to high similarity variance in the 10,000-dimensional setting at the lowest batch size.
Loading 1808.04866v5…