PaperlayerView exact revision on arXiv

Source-linked AI summary

Eluding Secure Aggregation in Federated Learning via Model Inconsistency

Dario Pasquini, Danilo Francati, Giuseppe Ateniese

arXiv:2111.07380v5cs.LGcs.CR

TL;DR

The paper examines whether secure aggregation actually protects federated-learning updates from a malicious server. It introduces model inconsistency attacks that manipulate server-supplied parameters to bypass aggregation, recover individual information, and motivate source-level defenses. The authors conclude that current implementations can provide a false sense of security unless parameter consistency is enforced.

  • Problem

    Individual federated-learning updates can reveal private training information, while secure aggregation is intended to prevent the server from accessing those updates and their sources.

  • Method

    The paper exploits model inconsistency by sending different model parameters to users, thereby tampering with the updates supplied to secure aggregation.

  • Results

    Individual model updates can be perfectly recovered and attributed to active users regardless of the number of participants.

  • Takeaways & Limitations

    Federated-learning implementations must prevent model inconsistency at its source because incorrect secure-aggregation usage can leave users without additional protection against malicious servers.

  • Takeaways & Limitations

    Central differential privacy is ineffective against a malicious parameter server when noise is applied only after aggregation, while differential privacy can also impose a high utility cost.

Abstract

from arXiv · show

Secure aggregation is a cryptographic protocol that securely computes the aggregation of its inputs. It is pivotal in keeping model updates private in federated learning. Indeed, the use of secure aggregation prevents the server from learning the value and the source of the individual model updates provided by the users, hampering inference and data attribution attacks. In this work, we show that a malicious server can easily elude secure aggregation as if the latter were not in place. We devise two different attacks capable of inferring information on individual private training datasets, independently of the number of users participating in the secure aggregation. This makes them concrete threats in large-scale, real-world federated learning applications. The attacks are generic and equally effective regardless of the secure aggregation protocol used. They exploit a vulnerability of the federated learning protocol caused by incorrect usage of secure aggregation and lack of parameter validation. Our work demonstrates that current implementations of federated learning with secure aggregation offer only a "false sense of security".

1 INTRODUCTION

Federated learning keeps data local but still exposes model updates to inference attacks, motivating secure aggregation. The paper shows that model inconsistency lets a malicious server bypass secure aggregation, recover individual updates, attribute them to users, and motivates consistency-based defenses.

  • Motivation: Federated learning keeps datasets local, but individual model updates can leak substantial information about users’ training data.This motivates secure aggregation as a defense against server access to individual updates.
  • Secure aggregation: Secure aggregation is intended to hide both information carried by individual updates and the sources of aggregated information.Its stated objectives are privacy by aggregation and privacy by shuffling.
  • Model inconsistency: Model inconsistency gives a malicious server indirect control over secure-aggregation inputs by sending different model parameters to different users.Because users compute updates from server-provided parameters, inconsistent views can alter the updates later aggregated.
  • Attacks: Individual updates can be perfectly recovered from the aggregate, and their sources can be attributed to active users regardless of the number of participants.The attacks target both secure aggregation’s aggregation and shuffling properties.
  • Attacks: Gradient suppression can zero non-target users’ gradients so secure aggregation outputs the target user’s gradient without auxiliary target information.The paper describes this as completely nullifying secure aggregation without unrealistic architecture changes.
  • Attacks: A canary-gradient attack uses a tiny parameter subset to test whether a chosen property appears in a user’s private data while preserving attack stealth.The server forces non-target users to produce zero gradients on the selected parameters.
  • Mitigation: Consistency checks can be integrated into influential secure-aggregation protocols without affecting the original federated-learning efficiency or utility.The proposed approach links masking values to the server-supplied parameters.

2 RELATED WORK

Prior work shows that federated learning exposes private training information through model updates, while secure aggregation and modified protocols address this risk with important practical or setting-specific constraints.

  • Secure aggregation: Secure aggregation aims to prevent servers from observing individual updates, but related work also studies reconstructing contributions across multiple training rounds.The multi-round approach reduces disaggregation to matrix factorization.
  • Alternative defenses: Other research proposes new protocols or substantial changes to federated-learning protocols and architectures to reduce unnecessary dataset leakage.The paper directs readers to its appendix for further details.
  • Gradient inversion: Without secure aggregation, gradient inversion can recover close-enough approximations of users’ local training datasets from individual model updates.The server searches for data whose gradient matches the observed user gradient.
  • Alternative attacks: A concurrent attack crafts an added linear-ReLU layer but requires auxiliary distribution knowledge and architecture changes that may be impractical for typical deep learning applications.The cited work chooses parameters using private-set CDF information for a target property.
  • Secure aggregation: The multi-round reconstruction attack requires fixed server parameters, participation side information, and repeated use of the same local training dataset.Its feasibility also depends on active users, network parameters, and the number of monitored rounds.

3 PRELIMINARIES

Federated learning keeps datasets local while users train a shared model through server-coordinated rounds, and secure aggregation reveals only an aggregate of their model updates.

  • Federated learning: Federated learning distributes a global dataset across users, who retain local data and collaboratively train a shared neural network.The learning process is interactive and divided into rounds.
  • Federated learning: At each round, the server sends current parameters to available users, who locally train the model and produce updates for aggregation.The available-user set may vary by round.
  • Federated learning: FedSGD performs one gradient-descent step, whereas FedAVG performs k local stochastic-gradient-descent iterations before averaging received parameters.The server computes the next parameters from users’ aggregated or averaged contributions.
  • Secure aggregation: Secure aggregation is a specialized multiparty-computation protocol that securely computes the sum of users’ secret inputs.Its ideal functionality returns the aggregation while keeping individual inputs hidden except for information inferable from the output.
  • Secure aggregation: Applied to federated learning, secure aggregation prevents an adversary from observing individual model updates and can reduce information available for gradient inversion.The protocols are also designed to tolerate bounded user dropout while still computing the final aggregation.
  • Security scope: Existing security theorems establish security for the secure-aggregation protocols themselves, not for the overall federated-learning protocol that uses them.This distinction leaves the composition of federated learning with secure aggregation outside those claims.

4 THREAT MODEL

The threat model considers a malicious centralized server that may corrupt a bounded number of users, send arbitrary parameters, and exploit model inconsistency without requiring collusion.

  • Adversary: The adversary is a malicious parameter server that can corrupt at most m of n users under the threat model used to analyze influential secure-aggregation protocols.The model follows the threat model under which Bonawitz et al. and Bell et al. establish secure-aggregation security.
  • Protocol assumptions: The attack replaces any concrete secure-aggregation implementation with its ideal functionality because the vulnerability comes from incorrect use of secure aggregation and missing input validation.The unvalidated inputs are the users’ model updates.
  • Adversary: The server may deviate from honest execution by sending arbitrary messages and malicious parameters to users, including within a single round.Its objective is to learn information about a target user’s local training dataset beyond the aggregate leakage.
  • Adversary: The attacks remain effective when m=0, meaning the server is malicious but colludes with no users.This extends the stated threat beyond cross-device settings that rely on user subsampling.
  • Protocol assumptions: The setting assumes centralized communication, authenticated users through a PKI, and encrypted channels between each user and the server.The PKI assumption rules out trivial server-driven Sybil attacks.

5 GRADIENT SUPPRESSION ATTACK

The gradient suppression attack uses model inconsistency to suppress non-target users’ updates during secure aggregation, preserving a chosen target’s update for inversion. It exploits induced zero gradients, applies beyond FedSGD, and scales independently of participant count, though the basic form is detectable.

  • Attack setup: The server sends malicious parameters to non-target users and the real parameters to a chosen target, creating model inconsistency.This separates the target’s update from the others during secure aggregation.
  • Recovery and inversion: After recovering the target’s plaintext gradient, the server applies standard gradient inversion to reconstruct information about the target’s training batch.Figure 2 reports reconstruction examples obtained from gradients recovered by the attack.
  • Attack mechanism: Gradient suppression induces zero or negligible updates from non-target users, allowing secure aggregation to preserve the target’s model update.The attack exploits ReLU behavior: dead layers produce zero gradients for their trainable parameters.
  • Scope and impact: The attack applies to FedAVG as well as FedSGD and is independent of the number of participating users.The procedure can therefore scale to very large rounds, including millions of active users.
  • Scope and limitations: The construction focuses on ReLU-based models but extends to arbitrarily composed architectures, with the final-layer bias gradient as a stated exception.For ResNet50 on ImageNet, that bias vector is 4 · 10^-5% of total parameters, and the authors report inversion remains unaffected when it is ignored.
  • Scope and limitations: Current FL-plus-secure-aggregation implementations do not prevent the attack because users fail to validate server-controlled parameters.The basic gradient suppression attack is highly effective but can be easily detected by non-target users.

6 CANARY-GRADIENT ATTACK FOR PROPERTY INFERENCE

The canary-gradient attack uses model inconsistency to encode attacker-chosen properties into gradients of a tiny parameter subset, enabling accurate property and membership inference despite secure aggregation.

  • Attack construction: The server injects detectors that trigger non-zero gradients for a small parameter subset only when an attacker-chosen property appears in the target batch.The subset can be as small as two parameters, and the predicate may target arbitrary batch properties.
  • Attack construction: The conditional dead-layer trick suppresses or preserves gradients by controlling ReLU pre-activations according to whether the target batch satisfies a predicate.The method trains preceding layers and selected parameters so the relevant output is positive only under the desired condition.
  • Bypassing secure aggregation: Under secure aggregation, non-target users receive tampered parameters that suppress their gradients, while the target’s canary gradient remains recoverable from the aggregate.The attack abstracts secure aggregation with its ideal functionality and does not target a specific secure aggregation protocol.
  • Results: The canary gradient achieves perfect recall, about 99% accuracy for batch sizes up to 128, and accuracy above 96% in the worst configuration.False positives increase with batch size because batches are more likely to contain examples resembling the target.
  • Results: The approach generalizes from membership inference to property inference, supports multiple triggers and simultaneous inference on multiple users, and leaves only a minimal update footprint.It requires one training round, is independent of the number of participating users, and is loss agnostic.
  • Scope: Canary-gradient attacks work even when secure aggregation is perfectly secure because the exploited vulnerability lies in federated learning’s use of aggregation rather than in the aggregation protocol itself.The attack is an example construction, and other training techniques may achieve the same result.

7 MITIGATIONS

The paper presents heuristic and privacy-based mitigations for model inconsistency, while identifying practical costs and remaining limitations. Proposed defenses include detecting abnormal updates, validating parameters, conditioning aggregation, and combining secure aggregation with differential privacy.

  • Heuristic mitigations: Gradient suppression can be detected by non-target users because their models produce zero gradients, although victims cannot detect or prevent the attack.Users could abstain from secure aggregation after detecting null gradients, preventing the server from unmasking the target update.
  • Heuristic mitigations: Parameter validation checks that users received identical model parameters, with signatures preventing a malicious server from substituting messages in the standard communication topology.Without direct user communication, signatures are needed to prevent man-in-the-middle substitution by the server.
  • Heuristic mitigations: Signature-based validation adds one communication round, though it can be merged with secure-aggregation rounds, and increases communication by n·ℓ.The cited example uses 256-bit signatures, yielding an added exchange cost of n·ℓ.
  • Heuristic mitigations: Conditional secure aggregation proceeds only when all users received the same parameters, otherwise returning a random or fixed value instead of the aggregate.This approach can be integrated with existing secure-aggregation protocols and requires no additional communication round, but the described solution is not resilient to dropouts.
  • Practical limitations: Patching model inconsistency is consistently harder for asynchronous federated learning, despite proposed defenses using the existing secure-aggregation public-key infrastructure.The stated solutions can use the PKI already required by secure-aggregation protocols.
  • Differential privacy: Differential privacy can protect against malicious servers when local or distributed noise is applied, whereas central differential privacy remains ineffective because the server can skip post-aggregation noise.Combining secure aggregation with differential privacy still leaves privacy by shuffling vulnerable when the server isolates the target update.

8 CONCLUSION

The conclusion identifies incorrect secure-aggregation use and missing parameter validation as the source of a critical federated-learning vulnerability. The authors present their attacks as representative and argue that implementations should prevent model inconsistency at its source.

  • Conclusion: Federated-learning implementations can gain no additional security against a malicious server from secure aggregation when parameter validation is absent.The server can provide inconsistent views of global parameters, enabling the vulnerability even when a trusted PKI is assumed.
  • Conclusion: The proposed attacks are representative threats from model inconsistency, and other attacks may exploit the same general intuition.The authors argue that implementations should account for model inconsistency and prevent it at its source.
  • Conclusion: Distributed differential privacy can be weakened proportionally to m compromised users when the server forces them to submit zero gradients without applying required noise.This caveat concerns the distributed-DP threat model and the number of compromised users it permits.

A ALTERNATIVES TO FEDERATED LEARNING

The paper contrasts federated learning with alternative privacy-preserving learning architectures, including homomorphic encryption, functional encryption, MPC, TEEs, and compiler-based approaches. It also states that the attacks apply broadly to protocols that let the server deviate from honest execution, subject to a collusion requirement.

  • Encrypted alternatives: Homomorphic-encryption approaches let the server compute on encrypted model updates, but systems differ in who decrypts and learns the aggregate.Some designs keep parameters encrypted at the server, while others have users decrypt the aggregate or compute updated parameters.
  • Encrypted alternatives: HybridAlpha uses multi-input functional encryption so the server computes an aggregation by decrypting ciphertexts with a functionality-specific secret key.Each user encrypts its update under its own public key before the server evaluates the aggregation.
  • Encrypted alternatives: Poseidon changes the original federated-learning architecture by organizing users hierarchically and using multi-key fully homomorphic encryption for updates and parameters.Encrypted parameters flow from the root user, while encrypted updates are sent through the tree.
  • MPC alternatives: Cerebro compiles a Python-like domain-specific language into optimized MPC for collaborative learning, but it is an alternative to federated learning rather than a federated-learning system.The distinction is architectural: Cerebro keeps plaintext data secret without relating to the original federated-learning design.
  • Attack applicability: The attacks apply to federated-learning and secure-aggregation protocols that do not prevent server deviation, including many fully homomorphic-encryption schemes.The attack requires the server to collude with at least one user to access the decrypted target update.
  • Trusted hardware: A trusted execution environment could prevent the active attacks under ideal hardware and authenticated communication, but side-channel vulnerabilities and performance degradation remain concerns.The paper states that complete elimination of side-channel attacks remains unclear.

B SECURITY OF SECURE AGGREGATION

This appendix frames secure aggregation through the ideal-versus-real-world paradigm of secure multiparty computation. Security requires that a simulator reproduce the real adversary’s view using only the ideal functionality’s interactions and outputs.

  • Notation: Cryptographic algorithms and adversaries are modeled as possibly interactive Turing machines, with PPT algorithms terminating in polynomially many steps.The security parameter is denoted by λ.
  • Real and ideal worlds: The real-world paradigm executes protocol π among users, while an adversary may be semi-honest or malicious and may control a subset of users.A malicious adversary can program corrupted users to deviate from the protocol specification.
  • Real and ideal worlds: In the ideal world, a trusted third party computes f on users’ private inputs and returns each party’s output, while corrupted parties’ inputs and outputs remain available to the adversary.This ideal execution defines the functionality that the real protocol should emulate.
  • Security definition: A protocol is secure when a PPT simulator can reproduce the real-world adversary’s view from the ideal-world interactions, ensuring real attacks correspond to ideal-world attacks.The ideal world therefore represents the maximum security guarantees achievable by the functionality.
  • Security definition: Definition B.1 characterizes secure computation for an n-party protocol that correctly computes an n-input, n-output function f.The definition formalizes users, inputs, corrupted subsets, and the corresponding outputs.

C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES

The attacker can suppress gradients across arbitrary architectures by making network components constant with respect to differentiated terms. For standard feedforward networks, manipulating the final and penultimate layers is sufficient.

  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: Gradient suppression can extend to arbitrary activation functions by constraining the differentiated function to become constant.In this state, altering the relevant parameters does not affect the loss, producing zero gradients.
  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: For standard feedforward networks, the attacker needs to modify only the kernels of the last and penultimate layers.
  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: Setting the last-layer kernel to zero suppresses gradients for earlier layers by nullifying their contribution to the loss.The last-layer kernel itself still receives a gradient because changing it affects the network output.
  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: Setting the penultimate-layer output to zero also suppresses the last-layer kernel gradient because its multiplicative contribution to the output vanishes.
  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: The attacker can force that zero output by setting the penultimate kernel to zero and choosing a bias whose activation output is zero.The construction is possible for almost every activation function, including Sigmoid with an input of −∞.
  • C GRADIENT SUPPRESSION FOR ARBITRARY ARCHITECTURES: After all kernels are neutralized, the model becomes constant, while the final bias may still receive a non-zero gradient and must be removed or ignored.

D CANARY-GRADIENT ON FEDAVG

The canary-gradient attack can be extended to FedAVG by limiting local parameter changes so the injected canary remains functional. Experiments show that standard learning rates preserve the attack with only limited accuracy loss as local steps increase.

  • D CANARY-GRADIENT ON FEDAVG: FedAVG may overwrite the injected canary during multiple local SGD iterations, but a low server-selected learning rate limits this perturbation.The rationale is that local parameter changes are proportional to the learning rate.
  • D CANARY-GRADIENT ON FEDAVG: Figure 9 reports average canary-gradient attack accuracy across six setups as the number of FedAVG local training steps increases for a ResNet20.
  • D CANARY-GRADIENT ON FEDAVG: The non-target model’s partial dead-kernel cannot be revived regardless of the number of local iterations or learning rate.
  • D CANARY-GRADIENT ON FEDAVG: With a standard learning rate of 0.001, attack accuracy naturally decreases as local iterations increase, but the performance loss remains limited.The experiment uses the Section 6.3.1 setup with batch size 64.
  • D CANARY-GRADIENT ON FEDAVG: The server detects whether the canary triggered by checking whether the canary parameters changed between consecutive model states.A change indicates that the target produced a non-zero gradient for those parameters at least once.

E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE

Secure aggregation combined with local differential privacy adds noise from non-target users to the recovered target update. Under suitable assumptions, this yields privacy amplification that grows with the number of participating users.

  • E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE: Noise in the recovered target update is proportional to the number of users when secure aggregation is combined with local differential privacy.
  • E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE: Without secure aggregation, the parameter server directly accesses the target user’s locally noised model update.
  • E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE: Figure 10 compares the final privacy budget recovered by the server as active-user counts increase under three local-DP settings.The settings are (𝜖=4.58,𝛿), (𝜖=2.88,𝛿), and (𝜖=1.63,𝛿), with 𝛿=5 · 10−5.
  • E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE: With secure aggregation and equal noise from active users, the recovered target update inherits noise added by non-target users.
  • E A NOTE ON THE COMBINATION OF SA WITH LOCAL DIFFERENTIAL PRIVACY AS A DEFENSE: The privacy amplification effect is proportional to the number of non-target users, allowing less local-DP noise for the same protection under a suitable trust model.Other users must be honest and add the expected amount of noise.
Loading 2111.07380v5…