PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtegoFed: Backdoor-Free Federated Instruction Tuning with Interspersed Poisoned Data

Haodong Zhao, Jinming Hu, Zhaomin Wu, Zongru Wu, Wei Du, Junyi Hou, Caibei Zhao, Zhuosheng Zhang, Bingsheng He, Gongshen Liu

arXiv:2603.00516v1cs.CR

TL;DR

FIT faces a backdoor risk when poisoned samples are inadvertently embedded in data collected by benign clients, while existing defenses focus mainly on malicious clients. ProtegoFed uses frequency-domain gradient signals with intra-client and globally coordinated clustering to detect and filter poisoned samples. Across experiments, it identifies 92.00% ∼100.00% of poisoned samples, reduces attack success rate to almost zero, and maintains main-task utility.

  • Problem

    FIT must defend against poisoned samples in untrusted data collected by benign clients, a threat that existing malicious-client defenses do not address adequately.

    Source: §I · p. 2

  • Method

    ProtegoFed uses frequency-space analysis of gradients, intra-client clustering, and globally coordinated secondary clustering to distinguish and filter poisoned samples.

    Source: §I · p. 2, C. Learning Mechanisms of Backdoor in… · p. 3, §IX · p. 16

  • Results

    92.00% ∼100.00% of poisoned samples are identified, attack success rate is reduced to almost zero, and main-task utility is maintained.

    Source: §I · p. 2

  • Takeaways & Limitations

    ProtegoFed provides a data-driven defense for securing FIT against untrusted training data without compromising model utility.

    Source: D. Beyond the Threat Model · p. 15, §IX · p. 16

Abstract

from arXiv · show

Federated Instruction Tuning (FIT) enables collaborative instruction tuning of large language models across multiple organizations (clients) in a cross-silo setting without requiring the sharing of private instructions. Recent findings on natural backdoors and the existing training data collection method suggest that poisoned samples may be pervasive and inadvertently embedded in real-world datasets, potentially distributed across all clients, even if the clients are benign. This work systematically examine this threat in FIT, demonstrating that existing defenses are ineffective when poisoned data is interspersed among all clients. Addressing this challenge entails two major difficulties: identifying the distinctive characteristics of poisoned samples at each client and enabling collaborative defense when some clients are heavily dominated by poisoned samples. To address these difficulties, we identify gradients in the frequency domain as a robust signal to distinguish poisoned data. We further propose a global secondary clustering mechanism that facilitates collaborative identification of poisoned samples across clients. In summary, this paper introduces ProtegoFed, the first backdoor-free FIT framework that accurately detects, removes, and even purifies interspersed poisoned data across clients during the training. Experimental results on four FL datasets show that ProtegoFed identifies $92.00\% \sim 100.00\%$ of poisoned samples, reduces the attack success rate to almost zero, and maintains utility on the main task. Code is available at https://github.com/dongdongzhaoUP/ProtegoFed.

I. INTRODUCTION

FIT enables collaborative LLM instruction tuning but can absorb poisoned data supplied by malicious users into benign clients, exposing global models to backdoors. ProtegoFed addresses this data-level threat through collaborative detection and filtering of poisoned samples while preserving clean-data utility.

Threat model

  • Malicious users can poison data collected by benign clients, creating backdoors in collaboratively trained FIT models.FIT’s multi-stage data-collection process expands the threat beyond malicious clients to untrusted user-provided training data.Source: §I · pp. 1–2, Figure 1 · p. 1

Limitations of existing defenses

  • Existing defenses primarily target malicious clients and are ineffective when poisoned data is distributed broadly across clients.Many methods assume fewer than half of clients have poisoned data, an assumption that does not hold for interspersed user-level poisoning.Source: §I · p. 2
  • Discarding whole client updates removes substantial clean data, while adding noise or trimming parameters can reduce model performance.Because poisoned samples may form only a small fraction of a client’s data, client-level exclusion wastes useful information.Source: §I · p. 2

ProtegoFed

  • ProtegoFed uses global information and coordination to defend against heterogeneous poisoning while protecting privacy.The method synthesizes local virtual centroids and builds global auxiliary information for collaborative defense.Source: §I · p. 2
  • ProtegoFed detects poisoned samples with unsupervised clustering based on gradient characteristics and evaluates clustering quality with indicators such as silhouette scores.The approach seeks generalization across varied poisoning forms while reducing unnecessary removal of clean data.Source: §I · p. 2

Results

  • 92.00% ∼100.00% of poisoned samples are identified, attack success rate is reduced to almost zero, and main-task utility is maintained.These results are reported across extensive experiments with multiple datasets and models.Source: §I · p. 2

II. PRELIMINARIES

Federated instruction tuning updates LoRA modules across clients while preserving private data, but backdoor attacks can add malicious mappings that remain stealthy on clean inputs and activate on triggered inputs.

Federated Instruction Tuning

Backdoor Attacks in LLMs

  • Backdoor attacks establish an additional mapping from trigger-affected inputs to attacker-specified outputs while preserving normal behavior on clean inputs.The clean mapping associates clean inputs with their corresponding outputs; the backdoor mapping sends triggered inputs to a predefined target.Source: B. Backdoor Attacks in LLMs · p. 3
  • Unknown triggers and increasingly stealthy trigger combinations make poisoned samples difficult to detect, motivating inspection and filtering of training data.Source: B. Backdoor Attacks in LLMs · p. 3

C. Learning Mechanisms of Backdoor in the Frequency Space

Backdoor and clean mappings exhibit different learning behavior in the frequency domain, motivating frequency-based sample discrimination while requiring collaborative defense under constrained and imbalanced client data.

Learning Mechanisms of Backdoor in the Frequency Space

  • Backdoor mappings show a stronger low-frequency bias and faster convergence than many-to-many clean mappings.Fourier analysis of input hidden states and logits reveals this divergence in learning behavior.Source: C. Learning Mechanisms of Backdoor in… · p. 3
  • Sample-wise gradients transformed with 2D-DCT provide a representation for clustering and filtering poisoned samples.The representation is constructed from sample-wise gradients and is intended to distinguish clean from poisoned data in frequency space.Source: C. Learning Mechanisms of Backdoor in… · p. 3
  • Computing and transforming sample-wise gradients is expensive, while clustering with limited or imbalanced single-client samples can produce biased results.Source: C. Learning Mechanisms of Backdoor in… · p. 3

Adversary Model

  • The threat model assumes malicious users can modify training data but cannot participate in client training, preprocessing, or global aggregation.Benign clients and the server unknowingly use poisoned data supplied by these users.Source: A. Adversary Model · p. 3

Defense Objectives

2) Defender’s capabilities and knowledge:

FIT is vulnerable to interspersed poisoned data, while existing defenses struggle with client-level heterogeneity and data loss. ProtegoFed addresses this by clustering frequency-domain gradient features locally and integrating global centroids for collaborative sample identification.

  • Existing defenses primarily target malicious clients and are ineffective when poisoned samples originate from malicious users but are distributed across benign clients.Source: §IV · p. 4
  • Even approximately 2% poisoned data can trigger backdoor behavior in FIT, with the effect pronounced around a 10% poisoning ratio.Source: §IV · p. 4
  • Directly applying GraCeFul locally causes missed poisoned samples and false positives because limited local data impairs discrimination.Source: Figure 4 · p. 4, §IV · p. 5
  • ProtegoFed uses sample-wise frequency-domain gradients, local clustering, global secondary clustering, and local revising before training.Source: A. Overview · p. 5

B. Intra-Client Frequency-based Clustering

ProtegoFed represents each client’s samples with frequency-domain gradient features, reduces their dimensionality, and clusters them to identify a principal local group. Global coordination is needed because limited, biased client data can place local centroids in poisoned regions.

D. Local Revising

Local revising incorporates the server’s global centroid into each client’s clustering so the selected clean cluster retains more clean data and less poisoned data. The resulting filtered objective trains only on the retained subset under standard convergence assumptions.

  • Global information enables clients to correct local clustering despite offsets between clean and poisoned distributions across clients.Source: D. Local Revising · p. 6
  • Each client adds the global centroid to its local features, reruns dimensionality reduction and clustering, and selects the centroid’s cluster as clean.Source: D. Local Revising · p. 7
  • ProtegoFed filters and optimizes only on a subset of the original data while retaining FedAvg’s global objective.Source: E. Convergence analysis · p. 7
  • The convergence analysis adopts L-smoothness, µ-convexity, bounded gradient variance, and bounded expected squared gradient norm assumptions.Source: E. Convergence analysis · p. 7
  • Its convergence derivation follows the downsampling setting and proof process of FedHDS.Source: E. Convergence analysis · p. 7

VI. EVALUATION

ProtegoFed is evaluated against multiple defenses and insertion-based attacks across four FSQA datasets, using task utility, attack success, and sample-identification metrics. The experiments cover IID and heterogeneous settings with several language models and poison constructions.

  • Evaluation uses WebQA, FreebaseQA, NQ, and CoQA, with sampled training, validation, and test instances for most datasets.Source: 1) Datasets · p. 7
  • The study evaluates nine open-source language models ranging from Llama-3.2-1B to Llama-2-70B.Source: 2) Models · p. 7
  • IID experiments divide data across 25 clients, each with a 10% poisoning ratio, while NIID experiments use more complex poisoning ratios.Source: 3) FL settings · p. 7
  • Clean exact match rate measures CACC, poison exact match measures ASR, and recall and F1 measure poisoned-sample identification.Source: 4) Metrics · p. 7
  • Attacks include Badnets, Addsent, CBA, and stealthy StyleBkd, with predefined misleading target outputs.Source: 5) Poison Methods · p. 8

6) Defense Baselines:

ProtegoFed is evaluated against sample-wise and robust aggregation baselines for backdoor defense in federated instruction tuning. It reduces attack success, detects poisoned samples, and generally preserves main-task utility across the evaluated settings.

Backdoor Attack Mitigation

  • ProtegoFed consistently reduces ASR to zero across datasets and attack methods under IID poisoned-sample distributions.FreqFed is less effective when untrusted data are widely distributed across clients.Source: B. Model performance · pp. 8–9

Model Utility Preservation

  • ProtegoFed achieves higher CACC than the undefended baseline in nine of 16 settings while most evaluated defenses maintain comparable utility.ONION is the exception, with core words lost from many processed samples.Source: B. Model performance · p. 9

Poisoned Sample Detection

  • ProtegoFed achieves near-perfect or perfect poisoned-sample recall across all evaluated datasets and attack types.On FreebaseQA and CoQA, recall reaches 100% across attack variants.Source: C. Poisoned Sample Detection Metrics · p. 9
  • ProtegoFed combines high recall with strong precision, with F1 approaching or exceeding 90% on FreebaseQA and CoQA.On WebQA under the CBA attack, F1 is 79.12% while recall remains 99.08%.Source: C. Poisoned Sample Detection Metrics · p. 9

D. Further Analysis

Further analyses test ProtegoFed across poison ratios, non-IID distributions, clean-data conditions, and clients dominated by poisoned data. The method remains effective when clean data dominate sufficiently, but its defense becomes invalid when poisoned data dominate broadly.

Clean-Data Case

  • ProtegoFed loses less than 1% CACC on completely clean data compared with ideal clean-data training.This evaluates whether filtering damages clean samples when no poisoned data are present.Source: 2) Data distribution · p. 10

Poison Ratio

  • ProtegoFed maintains high recall and eliminates ASR when clean data predominate, with only a slight CACC decrease as the poison ratio increases.When data are mainly poisoned across all clients, the defense becomes invalid.Source: 2) Data distribution · p. 10

Non-IID Settings

  • Under NIID-1, ProtegoFed achieves ASR = 0 and Recall →100.00%, while its high F1 score avoids significant negative impact on CACC.NIID-1 samples each client’s poison ratio between 0 and 0.4.Source: 2) Data distribution · p. 10
  • With Dirichlet-distributed poison ratios capped at 0.5 per client, ProtegoFed stably eliminates ASR in the more complex NIID-2 setting.Table VII reports near-perfect recall above 96% across α values from 0.1 to 1.0.Source: 2) Data distribution · p. 10

Poisoned-Dominated Clients

  • When more than half of the clients are dominated by poisoned data from different attacks, global secondary clustering can still correct detections if enough clients provide accurate local centers.The verification scenario uses 40% clean-dominant clients and two groups of poisoned-dominant clients.Source: 2) Data distribution · p. 11

Comparison Across Distributions

  • ProtegoFed’s performance remains stable across data distributions, which the paper attributes to frequency-based global information exchange.The comparison includes IID and non-IID settings.Source: 2) Data distribution · pp. 11–12

3) Target Module Selection:

The target-module analysis finds that gradients from the deepest LoRA-B module provide the strongest poisoned-sample discrimination. The study also examines clustering safeguards and global revising for difficult client distributions.

Target Module Selection

  • layers.31.lora_B produces the best target-module results and is used in the main experiments.The table defines layers.31.lora_B as model.layers.31.self_attn.v_proj.lora_B.Source: 3) Target Module Selection · pp. 12–13, Table IX · p. 12
  • Poisoned-sample discrimination improves from shallow to deep transformer layers, consistent with deeper layers amplifying frequency-feature differences.lora_B also outperforms lora_A at the same layer.Source: 3) Target Module Selection · p. 12
  • High-dimensional input and output modules are less suitable than the final LoRA module because limited client data hinder discrimination in high-dimensional features.The paper identifies this as a dimensionality curse.Source: 3) Target Module Selection · pp. 12–13

Clustering Selection

  • Using silhouette scores to select and assess clustering improves the balance of clustering stability and F1 beyond using only HDBSCAN or hierarchical clustering.High recall is achievable without the score, but neither single-clustering choice achieves the best F1.Source: 4) The Impact of Silhouette Score · p. 13

Global Secondary Clustering and Local Revising

  • Global secondary clustering and local revising improve recall and F1 across NIID settings, especially when local poisoned data exceed half of a client’s samples.In NIID-3, recall rises from about 50% without correction to close to 100% with correction.Source: 5) The Role of Global Secondary… · p. 13

6) The impact of dimensionality reduction:

ProtegoFed reduces gradient dimensions before clustering and uses low-dimensional representations for practical identification of poisoned samples. The system adds small initialization overhead and remains compatible with dynamic clients and adaptive-attack analysis.

Dimensionality Reduction

Efficiency

  • UMAP and HDBSCAN each have approximately O(N log N) complexity, making the core clustering components computationally inexpensive for distributed clients.N denotes the number of samples.Source: 7) Complexity and client-side memory… · p. 13

Dimensionality Reduction

Efficiency

  • ProtegoFed incurs initial overheads of 12.43s for intra-client clustering, 0.07s for global clustering, and 0.39s for local revising.Measurements use WebQA for 100 communication rounds, excluding validation and test time.Source: A. Time Efficiency · p. 14, Figure 13 · p. 14
  • Despite initialization overhead, ProtegoFed’s total training time is shorter than FedAvg over 100 communication rounds.The paper attributes the improvement to removing poisoned data and reducing training work.Source: A. Time Efficiency · p. 14

Dynamic Clients

  • Dynamic client withdrawal does not affect one-time filtering, while late joiners can update using the established global centroid.Late joiners rerun the global-clustering and local-revising steps once.Source: B. Scalability of ProtegoFed · p. 14

Adaptive Attack Strategy

  • Adaptive attacks that force high-frequency backdoor mappings make the backdoor itself difficult to activate, limiting the attacker’s ability to circumvent frequency-based detection.The paper evaluates a high-frequency one-to-many mapping attack experimentally.Source: C. Adaptive Attack Strategy · p. 15

D. Beyond the Threat Model

ProtegoFed addresses a complex threat in which colluding malicious clients disrupt poisoned-sample detection while bypassing local filtering. It uses frequency-based clustering and global coordination, remains robust with malicious clients, and complements existing defenses.

Complex threat scenario

  • In the complex threat scenario, malicious clients avoid local detection and filtering while submitting poisoned-sample features to disrupt global centroid information.The scenario assumes fewer than half of clients are malicious and colluding.Source: D. Beyond the Threat Model · p. 15

Robustness

  • Even with 40% malicious clients, ProtegoFed’s poisoned-sample identification on benign clients remains unaffected.Table XII reports this result using benign recall, while ASR is evaluated on the final global model.Source: D. Beyond the Threat Model · p. 15, Table XII · p. 15

Integration with Existing Defense Methods

  • ProtegoFed can be integrated with existing FL defenses to address malicious clients and untrusted training samples simultaneously.The paper exemplifies this integration with FreqFed and reports maintained detection efficacy on benign clients alongside elimination of malicious-client threats.Source: D. Beyond the Threat Model · p. 15

Relation to Existing Defenses

ProtegoFed

  • ProtegoFed combines intra-client frequency-based clustering with globally coordinated secondary clustering to differentiate clean data from poisoned samples.The framework identifies and filters poisoned samples without compromising model utility, according to extensive experiments across datasets and models.Source: §IX · p. 16

Related papers

Loading 2603.00516v1…