PaperlayerView exact revision on arXiv

Source-linked AI summary

Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing

Kai Li, Hanlin Cai, Zihao Liu, Yichen Li, Jose Alcaraz Calero, Wei Ni, Falko Dressler, Özgür B. Akan

arXiv:2610.12711v1cs.LGcs.CR

TL;DR

FedLLMs are vulnerable to Byzantine updates that can spread false or biased knowledge, while their privacy-preserving training makes centralized verification infeasible. HMP-GAE uses hypergraph message passing to identify and suppress semantically inconsistent updates. Across the evaluated models and datasets, it outperforms the GAE-based defense on accuracy, CSE, and PPL.

  • Problem

    FedLLMs are vulnerable to Byzantine attacks that can propagate false or biased knowledge, while private local training makes centralized verification or direct inspection infeasible.

    Source: §I · pp. 1–2

  • Method

    HMP-GAE represents each agent’s LoRA-fine-tuned update as a hypergraph node and uses iterative message passing over higher-order relationships to identify and suppress inconsistent contributions.

    Source: §I · p. 2, §VI · p. 12

  • Results

    Against the GAE-based defense, HMP-GAE improves accuracy by 0.26%-1.39%, reduces CSE by 27.43%-54.48%, and lowers PPL by 8.82%-35.49% across two models and datasets.

    Source: F. Ablation Study · p. 11

  • Takeaways & Limitations

    Hypergraph relations capture interactions among multiple benign and malicious agents beyond pairwise graphs, and this relational information helps explain the reported performance enhancement.

    Source: F. Ablation Study · p. 11

Abstract

from arXiv · show

Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder (HMP-GAE) framework, where low-rank model updates are represented as nodes in a hypergraph and high-order semantic relationships among multiple agents are exploited through iterative message passing. By leveraging historical checkpoints and cross-agent relational reasoning, HMP-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training. HMP-GAE is implemented using Qwen-2.5 and Llama-3.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo! Answers datasets. Compared to existing defenses, HMP-GAE improves global accuracy by $1.21$\%, reduces final-model classification semantic entropy by $44.08$\%, and lowers perplexity by $13.34$\%. HMP-GAE outperforms the GAE-based defense by improving the accuracy $1.39$\%, reducing the semantic entropy by $54.5$\%, and lowering perplexity by $35.5$\%.

I. INTRODUCTION

FedLLMs face Byzantine updates that can distort semantic consistency and factual alignment, while private local data makes direct verification infeasible. The paper proposes HMP-GAE to filter inconsistent updates using higher-order relations among agents.

I. INTRODUCTION

  • Byzantine agents can exploit LLM representation spaces and inject subtle perturbations that propagate false or biased knowledge through federated aggregation.Because local data is private, centralized verification or direct inspection of training samples is infeasible.Source: §I · pp. 1–2
  • HMP-GAE represents each LoRA-fine-tuned agent update as a hypergraph node and uses iterative message passing over higher-order relationships to identify and suppress semantically inconsistent contributions.Historical checkpoints and cross-agent relational reasoning support aggregation that preserves factual coherence while mitigating manipulation.Source: §I · p. 2
  • The defense is formulated to maximize global task accuracy while filtering malicious, semantically corrupted contributions during aggregation.The introduction reports evaluation of accuracy, semantic entropy, and perplexity under diverse adversarial scenarios.Source: §I · p. 2

A. Byzantine-resilient Strategies

  • Prior Byzantine-resilient FL defenses use robust aggregation, update similarity, trust evaluation, and secure computation, while earlier strategies include cosine-similarity filtering and feature-fingerprint clustering.The paper reviews these approaches in the context of Byzantine-resilient strategies.Source: A. Byzantine-resilient Strategies · p. 3

B. Our Contributions

The paper targets a gap in Byzantine resilience for FedLLMs, where malicious updates can disrupt semantic consistency and knowledge alignment. It proposes HMP-GAE to detect and suppress inconsistent updates through historical checkpoints and hypergraph reasoning.

B. Our Contributions

  • Existing Byzantine-resilient FL methods do not specifically address FedLLMs, where malicious updates can disrupt semantic consistency and knowledge alignment during aggregation.HMP-GAE uses historical checkpoints and iterative hypergraph message passing to detect higher-order relational inconsistencies without accessing private local data.Source: B. Our Contributions · p. 4
  • HMP-GAE treats each user’s LoRA update as a hypergraph node and uses historical checkpoints plus message passing to suppress semantically inconsistent updates during aggregation.The resulting representations support Byzantine-resilient knowledge integration without accessing private local data.Source: B. Our Contributions · p. 4

A. Fine-tuning of FedLLMs

  • FedLLMs train a shared pretrained LLM across agents with private datasets, using local fine-tuning and aggregated model updates rather than transmitting raw data.The backbone remains frozen while LoRA factors and additional conditioning modules are trainable.Source: A. Fine-tuning of FedLLMs · p. 4

B. Threat Model of Byzantine Attacks

The threat model allows compromised agents to upload arbitrary finite parameters, subject to a proximity constraint. Malicious agents seek to increase risk in the resulting aggregated model while limiting deviation from the broadcast model.

B. Threat Model of Byzantine Attacks

  • Compromised agents can upload arbitrary finite parameters, but their LoRA weights are constrained relative to the current broadcast global model.The deviation bound ε applies across LoRA-adapted layers and is measured using the Frobenius norm.Source: B. Threat Model of Byzantine Attacks · p. 5
  • At each communication round, malicious agents seek to maximize the risk of the resulting aggregated model.Their attack is constrained to reduce detectability by limiting deviation from the broadcast global model.Source: B. Threat Model of Byzantine Attacks · p. 5

IV. HMP-GAE FOR BYZANTINE-RESILIENT FEDLLMS

HMP-GAE assesses aggregated updates at the server using historical checkpoints, without accessing private local training data. It represents updates as descriptors mapped into a shared feature space.

IV. HMP-GAE FOR BYZANTINE-RESILIENT FEDLLMS

  • The server applies HMP-GAE to assess aggregated updates before aggregation, using historical checkpoints without accessing private local training data.The defense’s trainable parameters include its feature embedding network, GAE encoders, and decoders.Source: §IV · p. 5, A. Architecture of the Proposed HMP-GAE · p. 5

A. Architecture of the Proposed HMP-GAE

1) Update Representation:

  • Layer-wise LoRA update differences are concatenated with input-encoder and context-projection changes to form an agent update descriptor, then mapped into a shared d-dimensional feature space.The layer-wise differences compare each agent’s additive LoRA updates with the global LoRA factors.Source: 1) Update Representation · p. 5

2) Hypergraph Construction:

The hypergraph represents each agent as a node and connects agents through shared hyperedges, enabling group-level assessment of updates.

2) Hypergraph Construction:

  • Shared hyperedges let HMP-GAE assess an update among groups of agents, capturing relational patterns that isolated or pairwise comparisons may overlook.An agent can belong to multiple overlapping hyperedges and thus be assessed across different groups.Source: 2) Hypergraph Construction · p. 5
  • At each communication round, the hypergraph contains one node per agent, and each agent defines a hyperedge from its s nearest nodes.Nearest nodes are selected by Euclidean distance or cosine similarity between initial node features.Source: 2) Hypergraph Construction · p. 5
  • The framework uses an incidence matrix to represent the hypergraph’s agent-to-hyperedge memberships.The architecture applies this hypergraph message-passing framework at the server.Source: 2) Hypergraph Construction · p. 5, Figure 2 · p. 6

3) Hypergraph Message Passing:

HMP alternates between agent-to-hyperedge aggregation and hyperedge-to-agent propagation. A decoder reconstructs relationships from the resulting embeddings.

3) Hypergraph Message Passing:

  • HMP summarizes connected agents’ update patterns at hyperedges, then propagates that context back into each agent’s embedding.The operation alternates node-to-hyperedge aggregation and hyperedge-to-node propagation across layers.Source: 3) Hypergraph Message Passing · p. 6

4) Decoder Reconstruction:

  • The decoder reconstructs pairwise similarities and hyperedge incidence from embeddings, with reconstruction errors indicating departures from the modeled relational structure.The decoder estimates connectivity and hyperedge membership.Source: 3) Hypergraph Message Passing · p. 6, 4) Decoder Reconstruction · p. 6
  • A smoothness regularizer is defined for neighboring representations as part of the decoder-reconstruction discussion.The supplied passage states that the training objective is given, but does not provide its full expression.Source: 4) Decoder Reconstruction · p. 6

B. Algorithm Design of Resilience Strategy

HMP-GAE evaluates LoRA updates through cross-agent relationships and historical consistency, then uses resilience estimates to control their aggregation weights.

B. Algorithm Design of Resilience Strategy

  • HMP-GAE treats Byzantine resilience as high-order relational inference over LoRA updates, using group consistency and historical consistency to assess suspicious deviations.Resilience estimates control aggregation, linking relational manipulation detection to suppression of potentially corrupted updates.Source: B. Algorithm Design of Resilience… · p. 6

2) Resilience Scoring and Aggregation:

1) Historical Consistency and Training:

  • For later rounds, the method re-encodes the preceding checkpoint with current encoder parameters and uses historical consistency during training.The graph and target remain fixed while features and embeddings are updated.Source: 1) Historical Consistency and Training · p. 7

C. Complexity Analysis

The complexity analysis accounts for sparse hypergraph propagation, decoding, and historical re-encoding across J steps. With fixed s and d, sparse HMP propagation scales linearly with the number of agents K.

C. Complexity Analysis

  • Each HMP layer costs O(Kd2 + Φ(H)d), while dense incidence and adjacency decoding each cost O(K2d).Φ(H) is the number of non-zero incidence entries, with Φ(H) = K(s + 1).Source: C. Complexity Analysis · p. 7
  • (J +1)[KDd+L(Kd2+K(s+1)d)+K2d] gives the stated cost for historical re-encoding and final scoring across J steps.The expression is preceded by an input projection from D + d to d.Source: C. Complexity Analysis · p. 7
  • The additional Kd2 feature-processing cost is absorbed by the encoder term for L ≥1.Source: C. Complexity Analysis · p. 7
  • Neighbor search with linear-time selection and adjacency construction can cost O(K2d + K(s + 1)2), and aggregation costs O(KP).P is the total number of scalar entries of ω in (3); explicit LoRA product reconstruction is also noted as a cost.Source: C. Complexity Analysis · p. 7
  • Given s and d, sparse HMP propagation scales linearly with K.Source: C. Complexity Analysis · p. 7

V. PERFORMANCE EVALUATION

The evaluation uses AG News and Yahoo! Answers with two pretrained backbones, Qwen-2.5 and Llama-3.2. Agents adapt the frozen backbones with LoRA and upload flattened low-rank updates for the server-side resilience mechanism.

V. PERFORMANCE EVALUATION

  • The performance evaluation section presents HMP-GAE implementation under Byzantine attacks and notes that its source code is available online.Source: §V · p. 8

A. Experimental Implementation

  • The framework is implemented in Python 3.12 with PyTorch 2.10 and evaluated on a Linux workstation with an NVIDIA RTX PRO 6000 Blackwell GPU and 96 GB of memory.Source: A. Experimental Implementation · p. 8
  • AG News has four topical categories and 120,000 training and 7,600 test samples, while Yahoo! Answers has 10 categories and approximately 1.4 million training and 60,000 test instances.The datasets differ in label granularity and corpus scale.Source: A. Experimental Implementation · p. 8
  • The evaluation deploys Qwen-2.5 and Llama-3.2 pretrained backbones at agents to examine the effect of model capacity on Byzantine resilience.Qwen-2.5 has approximately 0.5 billion parameters; Llama-3.2 has around 1.2 billion.Source: A. Experimental Implementation · p. 8
  • LoRA adapts attention and feedforward projections while keeping pretrained weights frozen, so agents upload only flattened low-rank updates as the communication payload and server-side resilience input.Source: A. Experimental Implementation · p. 8

B. Performance Metrics

The section defines complementary metrics for task accuracy, semantic uncertainty, and language modeling, then compares HMP-GAE with three robust aggregators. Reported results cover global-model performance and average performance among benign local agents.

B. Performance Metrics

  • Accuracy measures correct target-label predictions, CSE measures semantic uncertainty through class-distribution entropy, and PPL measures global-model prediction of token sequences.Lower CSE indicates more semantically stable and confident predictions; lower PPL indicates higher probability assigned to the correct token sequence.Source: B. Performance Metrics · p. 8

C. Resilience Performance

  • The comparison holds the federated configuration constant and varies only the server aggregation rule, using Multi-Krum, FoolsGold, and FLTrust as baselines.Multi-Krum selects updates near their nearest neighbors, FoolsGold attenuates persistently aligned updates, and FLTrust weights updates against a clean server reference.Source: C. Resilience Performance · p. 8

1) Performance on The Two Datasets:

D. Per-user Trust Weight

HMP-GAE lowers per-class perplexity across most evaluated settings while persistently suppressing malicious users and retaining benign participation. Ablations show gains over both a GAE-based defense and a version without hyperedge reconstruction, though class-specific exceptions remain.

D. Per-user Trust Weight

  • HMP-GAE reduces malicious users C5 and C6 to approximately zero weight while retaining positive weights for benign users C0–C4; baselines either admit attackers or suppress benign C0.The malicious-weight suppression persists across remaining rounds for both backbones.Source: D. Per-user Trust Weight · p. 10

E. Per-class PPL

  • HMP-GAE achieves the lowest PPL in all four AG News classes and most Yahoo! Answers classes, with the lowest mean per-class PPL across both backbones on Yahoo! Answers.Several class-specific exceptions remain on Yahoo! Answers.Source: E. Per-class PPL · p. 11, Figure 6 · p. 11
  • User-level trust can reduce contributions from users who hold useful information for particular classes, so the lowest PPL in every class is not guaranteed.The authors connect this trade-off to less widely shared topics and differences in class representation and backbone capabilities.Source: E. Per-class PPL · p. 11

F. Ablation Study

  • Across both backbones and datasets, HMP-GAE beats the GAE-based defense: accuracy improves 0.26%-1.39%, CSE falls 27.43%-54.48%, and PPL falls 8.82%-35.49%.Table III reports accuracy and CSE averaged over 50 rounds and PPL on the final global model.Source: F. Ablation Study · p. 11, Table III · p. 12
  • Against HMP-GAE without Hyperedge Reconstruction, the full framework improves accuracy 0.21%-1.11%, reduces CSE 5.00%-14.38%, and lowers PPL 4.48%-15.25%.Hyperedge reconstruction flags updates associated with unusually high classification uncertainty; in one case PPL falls from 620.23 to 525.65.Source: F. Ablation Study · pp. 11–12

VI. CONCLUSIONS

  • The conclusion reports persistent suppression of malicious contributions alongside preserved benign participation, with benefits extending to benign agents’ local performance.The experiments used Qwen-2.5 and Llama-3.2 on AG News and Yahoo! Answers.Source: §VI · p. 12

Related papers

Loading 2610.12711v1…