PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtegoFed: Backdoor-Free Federated Instruction Tuning with Interspersed Poisoned Data

Haodong Zhao, Jinming Hu, Zhaomin Wu, Zongru Wu, Wei Du, Junyi Hou, Caibei Zhao, Zhuosheng Zhang, Bingsheng He, Gongshen Liu

arXiv:2603.00516v1cs.CR

TL;DR

FIT faces a backdoor risk when poisoned samples are inadvertently embedded in data collected by benign clients, while existing defenses focus mainly on malicious clients. ProtegoFed uses frequency-domain gradient signals with intra-client and globally coordinated clustering to detect and filter poisoned samples. Across experiments, it identifies 92.00% ∼100.00% of poisoned samples, reduces attack success rate to almost zero, and maintains main-task utility.

  • Problem

    FIT must defend against poisoned samples in untrusted data collected by benign clients, a threat that existing malicious-client defenses do not address adequately.

  • Method

    ProtegoFed uses frequency-space analysis of gradients, intra-client clustering, and globally coordinated secondary clustering to distinguish and filter poisoned samples.

  • Results

    92.00% ∼100.00% of poisoned samples are identified, attack success rate is reduced to almost zero, and main-task utility is maintained.

  • Takeaways & Limitations

    ProtegoFed provides a data-driven defense for securing FIT against untrusted training data without compromising model utility.

Abstract

from arXiv · show

Federated Instruction Tuning (FIT) enables collaborative instruction tuning of large language models across multiple organizations (clients) in a cross-silo setting without requiring the sharing of private instructions. Recent findings on natural backdoors and the existing training data collection method suggest that poisoned samples may be pervasive and inadvertently embedded in real-world datasets, potentially distributed across all clients, even if the clients are benign. This work systematically examine this threat in FIT, demonstrating that existing defenses are ineffective when poisoned data is interspersed among all clients. Addressing this challenge entails two major difficulties: identifying the distinctive characteristics of poisoned samples at each client and enabling collaborative defense when some clients are heavily dominated by poisoned samples. To address these difficulties, we identify gradients in the frequency domain as a robust signal to distinguish poisoned data. We further propose a global secondary clustering mechanism that facilitates collaborative identification of poisoned samples across clients. In summary, this paper introduces ProtegoFed, the first backdoor-free FIT framework that accurately detects, removes, and even purifies interspersed poisoned data across clients during the training. Experimental results on four FL datasets show that ProtegoFed identifies $92.00\% \sim 100.00\%$ of poisoned samples, reduces the attack success rate to almost zero, and maintains utility on the main task. Code is available at https://github.com/dongdongzhaoUP/ProtegoFed.

I. INTRODUCTION

FIT enables collaborative LLM instruction tuning but can absorb poisoned data supplied by malicious users into benign clients, exposing global models to backdoors. ProtegoFed addresses this data-level threat through collaborative detection and filtering of poisoned samples while preserving clean-data utility.

  • Threat model: Malicious users can poison data collected by benign clients, creating backdoors in collaboratively trained FIT models.FIT’s multi-stage data-collection process expands the threat beyond malicious clients to untrusted user-provided training data.
  • Limitations of existing defenses: Existing defenses primarily target malicious clients and are ineffective when poisoned data is distributed broadly across clients.Many methods assume fewer than half of clients have poisoned data, an assumption that does not hold for interspersed user-level poisoning.
  • Limitations of existing defenses: Discarding whole client updates removes substantial clean data, while adding noise or trimming parameters can reduce model performance.Because poisoned samples may form only a small fraction of a client’s data, client-level exclusion wastes useful information.
  • ProtegoFed: ProtegoFed uses global information and coordination to defend against heterogeneous poisoning while protecting privacy.The method synthesizes local virtual centroids and builds global auxiliary information for collaborative defense.
  • ProtegoFed: ProtegoFed detects poisoned samples with unsupervised clustering based on gradient characteristics and evaluates clustering quality with indicators such as silhouette scores.The approach seeks generalization across varied poisoning forms while reducing unnecessary removal of clean data.
  • Results: 92.00% ∼100.00% of poisoned samples are identified, attack success rate is reduced to almost zero, and main-task utility is maintained.These results are reported across extensive experiments with multiple datasets and models.

II. PRELIMINARIES

Federated instruction tuning updates LoRA modules across clients while preserving private data, but backdoor attacks can add malicious mappings that remain stealthy on clean inputs and activate on triggered inputs.

  • Federated Instruction Tuning: LoRA adapts pretrained transformer models by introducing low-rank matrices without changing the original model parameters.
  • Federated Instruction Tuning: FIT updates and aggregates only LoRA module parameters exchanged between clients and the server.Each client trains its received global LoRA module locally, then sends updated parameters or gradients for federated aggregation.
  • Backdoor Attacks in LLMs: Backdoor attacks establish an additional mapping from trigger-affected inputs to attacker-specified outputs while preserving normal behavior on clean inputs.The clean mapping associates clean inputs with their corresponding outputs; the backdoor mapping sends triggered inputs to a predefined target.
  • Backdoor Attacks in LLMs: Unknown triggers and increasingly stealthy trigger combinations make poisoned samples difficult to detect, motivating inspection and filtering of training data.

C. Learning Mechanisms of Backdoor in the Frequency Space

Backdoor and clean mappings exhibit different learning behavior in the frequency domain, motivating frequency-based sample discrimination while requiring collaborative defense under constrained and imbalanced client data.

  • Learning Mechanisms of Backdoor in the Frequency Space: Backdoor mappings show a stronger low-frequency bias and faster convergence than many-to-many clean mappings.Fourier analysis of input hidden states and logits reveals this divergence in learning behavior.
  • Learning Mechanisms of Backdoor in the Frequency Space: Sample-wise gradients transformed with 2D-DCT provide a representation for clustering and filtering poisoned samples.The representation is constructed from sample-wise gradients and is intended to distinguish clean from poisoned data in frequency space.
  • Learning Mechanisms of Backdoor in the Frequency Space: Computing and transforming sample-wise gradients is expensive, while clustering with limited or imbalanced single-client samples can produce biased results.
  • Adversary Model: The threat model assumes malicious users can modify training data but cannot participate in client training, preprocessing, or global aggregation.Benign clients and the server unknowingly use poisoned data supplied by these users.
  • Defense Objectives: The defense objectives are preventing backdoor attacks, maintaining main-task utility, preserving privacy, and limiting per-round overhead.The server requires neither an auxiliary clean dataset nor model training, and user-level defense complements client-level defenses.

2) Defender’s capabilities and knowledge:

FIT is vulnerable to interspersed poisoned data, while existing defenses struggle with client-level heterogeneity and data loss. ProtegoFed addresses this by clustering frequency-domain gradient features locally and integrating global centroids for collaborative sample identification.

  • Existing defenses primarily target malicious clients and are ineffective when poisoned samples originate from malicious users but are distributed across benign clients.
  • Even approximately 2% poisoned data can trigger backdoor behavior in FIT, with the effect pronounced around a 10% poisoning ratio.
  • Directly applying GraCeFul locally causes missed poisoned samples and false positives because limited local data impairs discrimination.
  • ProtegoFed uses sample-wise frequency-domain gradients, local clustering, global secondary clustering, and local revising before training.

B. Intra-Client Frequency-based Clustering

ProtegoFed represents each client’s samples with frequency-domain gradient features, reduces their dimensionality, and clusters them to identify a principal local group. Global coordination is needed because limited, biased client data can place local centroids in poisoned regions.

  • Each client computes sample-wise gradients on a selected deep-module parameter and applies DCT to obtain frequency-domain features.
  • UMAP reduces the frequency features to two dimensions before clustering to improve feature distinction and clustering performance.
  • Hierarchical clustering and HDBSCAN are compared, and the result with the higher silhouette score is selected.
  • Clients send main-cluster centroids rather than individual samples because local data are limited, biased, and dimensionality-reduced feature spaces differ across clients.
  • The server dimension-reduces and reclusters client centroids, computes a global centroid from the main cluster, and broadcasts it for local correction.

D. Local Revising

Local revising incorporates the server’s global centroid into each client’s clustering so the selected clean cluster retains more clean data and less poisoned data. The resulting filtered objective trains only on the retained subset under standard convergence assumptions.

  • Global information enables clients to correct local clustering despite offsets between clean and poisoned distributions across clients.
  • Each client adds the global centroid to its local features, reruns dimensionality reduction and clustering, and selects the centroid’s cluster as clean.
  • ProtegoFed filters and optimizes only on a subset of the original data while retaining FedAvg’s global objective.
  • The convergence analysis adopts L-smoothness, µ-convexity, bounded gradient variance, and bounded expected squared gradient norm assumptions.
  • Its convergence derivation follows the downsampling setting and proof process of FedHDS.

VI. EVALUATION

ProtegoFed is evaluated against multiple defenses and insertion-based attacks across four FSQA datasets, using task utility, attack success, and sample-identification metrics. The experiments cover IID and heterogeneous settings with several language models and poison constructions.

  • Evaluation uses WebQA, FreebaseQA, NQ, and CoQA, with sampled training, validation, and test instances for most datasets.
  • The study evaluates nine open-source language models ranging from Llama-3.2-1B to Llama-2-70B.
  • IID experiments divide data across 25 clients, each with a 10% poisoning ratio, while NIID experiments use more complex poisoning ratios.
  • Clean exact match rate measures CACC, poison exact match measures ASR, and recall and F1 measure poisoned-sample identification.
  • Attacks include Badnets, Addsent, CBA, and stealthy StyleBkd, with predefined misleading target outputs.

6) Defense Baselines:

ProtegoFed is evaluated against sample-wise and robust aggregation baselines for backdoor defense in federated instruction tuning. It reduces attack success, detects poisoned samples, and generally preserves main-task utility across the evaluated settings.

  • Backdoor Attack Mitigation: ProtegoFed consistently reduces ASR to zero across datasets and attack methods under IID poisoned-sample distributions.FreqFed is less effective when untrusted data are widely distributed across clients.
  • Model Utility Preservation: ProtegoFed achieves higher CACC than the undefended baseline in nine of 16 settings while most evaluated defenses maintain comparable utility.ONION is the exception, with core words lost from many processed samples.
  • Poisoned Sample Detection: ProtegoFed achieves near-perfect or perfect poisoned-sample recall across all evaluated datasets and attack types.On FreebaseQA and CoQA, recall reaches 100% across attack variants.
  • Poisoned Sample Detection: ProtegoFed combines high recall with strong precision, with F1 approaching or exceeding 90% on FreebaseQA and CoQA.On WebQA under the CBA attack, F1 is 79.12% while recall remains 99.08%.

D. Further Analysis

Further analyses test ProtegoFed across poison ratios, non-IID distributions, clean-data conditions, and clients dominated by poisoned data. The method remains effective when clean data dominate sufficiently, but its defense becomes invalid when poisoned data dominate broadly.

  • Clean-Data Case: ProtegoFed loses less than 1% CACC on completely clean data compared with ideal clean-data training.This evaluates whether filtering damages clean samples when no poisoned data are present.
  • Poison Ratio: ProtegoFed maintains high recall and eliminates ASR when clean data predominate, with only a slight CACC decrease as the poison ratio increases.When data are mainly poisoned across all clients, the defense becomes invalid.
  • Non-IID Settings: Under NIID-1, ProtegoFed achieves ASR = 0 and Recall →100.00%, while its high F1 score avoids significant negative impact on CACC.NIID-1 samples each client’s poison ratio between 0 and 0.4.
  • Non-IID Settings: With Dirichlet-distributed poison ratios capped at 0.5 per client, ProtegoFed stably eliminates ASR in the more complex NIID-2 setting.Table VII reports near-perfect recall above 96% across α values from 0.1 to 1.0.
  • Poisoned-Dominated Clients: When more than half of the clients are dominated by poisoned data from different attacks, global secondary clustering can still correct detections if enough clients provide accurate local centers.The verification scenario uses 40% clean-dominant clients and two groups of poisoned-dominant clients.
  • Comparison Across Distributions: ProtegoFed’s performance remains stable across data distributions, which the paper attributes to frequency-based global information exchange.The comparison includes IID and non-IID settings.

3) Target Module Selection:

The target-module analysis finds that gradients from the deepest LoRA-B module provide the strongest poisoned-sample discrimination. The study also examines clustering safeguards and global revising for difficult client distributions.

  • Target Module Selection: layers.31.lora_B produces the best target-module results and is used in the main experiments.The table defines layers.31.lora_B as model.layers.31.self_attn.v_proj.lora_B.
  • Target Module Selection: Poisoned-sample discrimination improves from shallow to deep transformer layers, consistent with deeper layers amplifying frequency-feature differences.lora_B also outperforms lora_A at the same layer.
  • Target Module Selection: High-dimensional input and output modules are less suitable than the final LoRA module because limited client data hinder discrimination in high-dimensional features.The paper identifies this as a dimensionality curse.
  • Clustering Selection: Using silhouette scores to select and assess clustering improves the balance of clustering stability and F1 beyond using only HDBSCAN or hierarchical clustering.High recall is achievable without the score, but neither single-clustering choice achieves the best F1.
  • Global Secondary Clustering and Local Revising: Global secondary clustering and local revising improve recall and F1 across NIID settings, especially when local poisoned data exceed half of a client’s samples.In NIID-3, recall rises from about 50% without correction to close to 100% with correction.

6) The impact of dimensionality reduction:

ProtegoFed reduces gradient dimensions before clustering and uses low-dimensional representations for practical identification of poisoned samples. The system adds small initialization overhead and remains compatible with dynamic clients and adaptive-attack analysis.

  • Dimensionality Reduction: At lower dimensions, UMAP dimensionality has little effect on poisoned-sample identification, so ProtegoFed uses two dimensions for visualization.The design choice is made for visualization convenience.
  • Efficiency: UMAP and HDBSCAN each have approximately O(N log N) complexity, making the core clustering components computationally inexpensive for distributed clients.N denotes the number of samples.
  • Dimensionality Reduction: Direct clustering in high dimensions performs poorly with small data, motivating dimensionality reduction before clustering.Figure 11 contrasts ground-truth groupings with clusters obtained in high- and low-dimensional representations.
  • Efficiency: ProtegoFed incurs initial overheads of 12.43s for intra-client clustering, 0.07s for global clustering, and 0.39s for local revising.Measurements use WebQA for 100 communication rounds, excluding validation and test time.
  • Efficiency: Despite initialization overhead, ProtegoFed’s total training time is shorter than FedAvg over 100 communication rounds.The paper attributes the improvement to removing poisoned data and reducing training work.
  • Dynamic Clients: Dynamic client withdrawal does not affect one-time filtering, while late joiners can update using the established global centroid.Late joiners rerun the global-clustering and local-revising steps once.
  • Adaptive Attack Strategy: Adaptive attacks that force high-frequency backdoor mappings make the backdoor itself difficult to activate, limiting the attacker’s ability to circumvent frequency-based detection.The paper evaluates a high-frequency one-to-many mapping attack experimentally.

D. Beyond the Threat Model

ProtegoFed addresses a complex threat in which colluding malicious clients disrupt poisoned-sample detection while bypassing local filtering. It uses frequency-based clustering and global coordination, remains robust with malicious clients, and complements existing defenses.

  • Complex threat scenario: In the complex threat scenario, malicious clients avoid local detection and filtering while submitting poisoned-sample features to disrupt global centroid information.The scenario assumes fewer than half of clients are malicious and colluding.
  • Robustness: Even with 40% malicious clients, ProtegoFed’s poisoned-sample identification on benign clients remains unaffected.Table XII reports this result using benign recall, while ASR is evaluated on the final global model.
  • Integration with Existing Defense Methods: ProtegoFed can be integrated with existing FL defenses to address malicious clients and untrusted training samples simultaneously.The paper exemplifies this integration with FreqFed and reports maintained detection efficacy on benign clients alongside elimination of malicious-client threats.
  • Relation to Existing Defenses: Existing FL defenses mainly target anomalous client updates, whereas ProtegoFed filters poisoned samples within client data.Earlier approaches include robust aggregation, parameter analysis, and frequency-domain update clustering, but they address a different threat source.
  • ProtegoFed: ProtegoFed combines intra-client frequency-based clustering with globally coordinated secondary clustering to differentiate clean data from poisoned samples.The framework identifies and filters poisoned samples without compromising model utility, according to extensive experiments across datasets and models.
Loading 2603.00516v1…