Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models

1 Introduction

• Federated instruction tuning uses decentralized, aligned data to pursue helpful and safe behavior while avoiding direct data sharing.The introduction contrasts this collaborative approach with centralized collection of high-quality data, which requires substantial human effort.Source: §1 · p. 1
• Malicious clients can compromise FedIT by training local LLMs on unaligned pairs of harmful instructions and harmful responses.Those local models are uploaded alongside benign client models and included in server aggregation.Source: §1 · p. 2
• Benign and malicious clients’ responses to normal and harmful instructions share similar optimization objectives, making their local models difficult to distinguish with model-level defenses.The paper links this similarity to the failure of existing defense methods that rely on model-level comparison.Source: §1 · p. 2
• The post-hoc defense has the server generate harmful and normal instructions, pair them with safe or helpful responses, and fine-tune the aggregated LLM.The method avoids comparing local models and uses a few training steps to improve safety without significantly compromising helpfulness.Source: §1 · p. 2
• The attack reduced safety rate by 70%, existing defenses brought at most 4% improvement, and the proposed defense brought at most 69% improvement.Experiments covered four training datasets, three safety benchmarks, and one helpfulness benchmark.Source: §1 · p. 2

Instruction tuning of large language models and federated learning

• Prior work extended instruction tuning to federated learning to leverage private data, but had not examined FedIT from the perspective of LLM safety.Instruction tuning uses supervised fine-tuning on helpfulness- and safety-aligned data [30, 31, 7, 32, 6, 33, 34, 35].Source: §2 · pp. 2–3

Poisoning attacks in federated learning

• Unlike label flipping for classification, the proposed data-poisoning attack targets FedIT safety while preserving data fluency and correctness.The passage distinguishes data poisoning, where attackers control local datasets [43] [44] [45] [46] [47], from model poisoning, where attackers manipulate model parameters [48] [49] [50].Source: §2 · p. 3

Defenses in federated learning

• Most cited FL poisoning defenses use robust aggregation to identify or mitigate malicious clients, whereas the proposed post-hoc defense avoids model-level operation.Examples include FoolsGold [27], Median [24], Residual [59], Krum, and DnC; some methods require setting the expected number of attackers.Source: §2 · p. 3

Definitions

• Each FL client k holds a dataset Dk of instruction-response pairs (xi, yi), with Nk samples.The instruction and response are denoted xi and yi, respectively.Source: §3 · p. 3
• Aligned data pair harmful instructions with harmless responses, while unaligned data pair harmful instructions with harmful responses.Normal data pair normal instructions with normal responses; both aligned and unaligned models may produce normal responses to normal instructions.Source: §3 · p. 3

Objective of FL

• The FL objective minimizes the weighted client losses, with each client’s weight based on its relative dataset size.The loss for client k is Lk(Dk, θ), and pk is its relative dataset size.Source: §3 · p. 3
• In the ideal safe scenario, participating clients’ data are restricted to normal or aligned data: Dk ⊂ On ∪ Oa.Source: §3 · p. 3

4.1 Threat Model

• Each attacker is a malicious client seeking harmful responses to harmful instructions while preserving normal behavior on normal instructions, and can train on arbitrary data.Attackers may use publicly available unaligned data or generate it with an off-the-shelf LLM.Source: §4.1 · p. 4

4 Safety Attack in Federated Instruction Tuning on LLMs

• At each round, clients fine-tune copies of the global LLM on their own datasets, and the server aggregates their local models into the next global model.When malicious clients train on unaligned data, aggregation can directly compromise the global model’s safety alignment.Source: §4.2 · p. 4

4.3 Obtaining Attack Data at A Low Cost

• Attackers can repurpose unsafe samples from public instruction-tuning or preference datasets into harmful-instruction and unaligned-response pairs.The paper cites Beavertails and HH-RLHF as examples of public datasets that contain material useful for constructing attack data.Source: §4.3 · pp. 4–5
• An automated pipeline generates harmful instructions and unsafe responses with an off-the-shelf LLM, pairing them into unaligned attack data.The pipeline repeats instruction generation until it reaches the desired number of harmful instructions, then generates responses without safety guardrails.Source: §4.3 · pp. 4–5

4.4 Discussions

• A few malicious clients can make FedIT’s global LLM misalign with safety principles by changing their data format, creating potential misuse risk.The authors say attack data can be obtained easily, especially through their automated generation pipeline.Source: §4.4 · p. 5
• Because benign and malicious training share similar instruction-following objectives, model parameters alone may not distinguish their local LLMs, weakening many model-level defenses.The authors describe existing defenses that rely on model-level filtering as ineffective against this stealthiness.Source: §4.4 · p. 5

5 Defense against Safety Attack in Federated Instruction Tuning

• Because the attack’s parameter-level stealthiness undermines many model-level defenses, the authors propose fine-tuning the aggregated LLM after potentially polluted aggregation.This post-hoc approach avoids filtering or comparing local models to identify malicious clients.Source: §5 · p. 5
• The defense dataset can come from an existing dataset, an external off-the-shelf LLM, or the aggregated LLM itself.These three options represent increasing reliance on external resources, with the aggregated model used for the self-alignment option.Source: §5 · pp. 5–6
• The automated defense pipeline generates normal instructions and harmful instructions, then prompts the LLM to answer harmful instructions safely.The resulting normal and aligned pairs form a dataset whose normal data is intended to mitigate loss of helpfulness while aligned data guides safety.Source: §5 · p. 6
• The authors recommend retaining FedIT’s access to decentralized data while adding a final safeguard before releasing the LLM.They present this as a feasible roadmap in response to the attack vulnerability they identify.Source: §5 · p. 6

6.1 Experiment Setups

• Experiments use four benign datasets, two malicious datasets, 10 clients with seven benign and three malicious, and evaluate safety on AdvBench and helpfulness on MT-Bench’s first turn.Each client holds 500 samples; the post-hoc defense fine-tunes the aggregated adapter on 1,000 defense samples for 500 steps.Source: §6.1 · p. 6

6.2 Main Results

• 37.50% lower in Rule and 52.50% lower in MD-Judge, attacked FedAvg scores below FedAvg without attack on LMSYS-Chat and MaliciousGen.The passage reports decreases across three safety metrics overall.Source: §6.2 · p. 7
• Krum improves MD-Judge by only 1.92% on LMSYS-Chat and Beavertails, while Median achieves lower safety metrics in that setting.The passage attributes the limited effectiveness of parameter-level defenses to the attack’s stealthiness.Source: §6.2 · p. 7
• 69.81% higher in MD-Judge, level 2 defense achieves 43.47% higher in Rule and 1.65 higher in RM than attacked FedAvg on WildChat and Beavertails.Its MD-Judge score is also 84.24% versus 66.15% for FedAvg without attack.Source: §6.2 · p. 7

6.3 Analysis and Ablation Study

• The defense improves safety across seven FL baselines, with an average 57.25% increase in MD-Judge, without significantly compromising helpfulness.The plug-and-play evaluation uses LMSYS-Chat as benign data and Beavertails as malicious data.Source: §6.3 · p. 7, Table 3 · p. 8
• Update similarities show no distinct benign–malicious clusters, while Krum, FoolsGold, DnC, and Residual fail to identify all malicious clients.Krum also assigns zero aggregation weights to two benign clients.Source: §6.3 · pp. 7–8, Figure 2 · p. 9
• Mistral, Zephyr, and Wizard-generated unaligned data all reduce MD-Judge scores while helpfulness remains comparable, indicating attack performance is insensitive to generator choice.The comparison uses LMSYS-Chat as benign data.Source: §6.3 · p. 8, Figure 3 · p. 8
• At 50 and 100 clients with the malicious-client ratio held at 30%, the attack remains effective and level 2 defense improves safety with comparable helpfulness.Existing FL defense baselines remain susceptible at both client scales.Source: §6.3 · pp. 8–9, Table 4 · p. 9

7 Conclusions

• The attack uses automatically generated unaligned data and can be difficult to distinguish from benign clients at the model level.The paper describes the attack as simple and stealthy, requiring no human effort to generate its data.Source: §7 · p. 9
• The post-hoc defense has the server generate aligned data and fine-tune the aggregated LLM, without comparing local models.Source: §7 · p. 9
• The proposed roadmap combines collaborative FedIT using diverse data with post-hoc safety alignment before the model is released.Source: §7 · p. 9
• Experiments use Llama2 as the base model, although the authors believe their conclusions apply across model series.Source: §7 · p. 9

A Broader Impacts

• The paper says existing FL defenses are insufficient for malicious-client safety attacks in FedIT, motivating more robust defenses.Source: A Broader Impacts · p. 15
• The authors describe post-training as a safeguard intended to preserve value alignment without sacrificing helpfulness.Source: A Broader Impacts · p. 15

B.1 Experimental Setups

• Experiments use 10 clients, sample three clients per round, and give each client 500 non-overlapping aligned or unaligned samples.Under attack, seven clients are benign and three are malicious; available clients train for 10 local steps per round.Source: §B.1 · p. 15

B.2 Generation Process

• The generation pipeline has instruction and response stages, with Figure 4 showing prompts for unaligned, aligned, and normal data.Source: §B.2 · p. 15, Figure 4 · p. 15
• Instruction generation uses prompts for harmful and helpful questions, and the same prompt can be reused iteratively to collect more instructions.Source: §B.2 · p. 16
• For harmful instructions, response-generation prompts steer the LLM toward harmful unaligned responses or harmless aligned responses; normal instructions receive direct responses.Source: §B.2 · p. 16
• The resulting aligned data pairs harmful instructions with harmless responses, unaligned data pairs them with harmful responses, and normal data pairs normal instructions with normal responses.Source: §B.2 · p. 16

• 9.04% increase in Rule score and 26.35% improvement in MD-Judge score are reported for Level 3 defense versus FedAvg.The reported gains are on WildChat with no attack; the defense still enhances safety without sacrificing helpfulness.Source: §B.3 · p. 16
• The seven no-attack baselines achieve comparable high safety, and the proposed defense further enhances safety without sacrificing helpfulness.Source: §B.3 · p. 16
• The no-attack experiment evaluates three defense levels on WildChat with ten clients.Source: §B.3 · p. 16

B.4 Experiments on Domain-Specific Tasks

• The domain-specific attack experiment uses CodeAlpaca with HumanEval evaluation and includes seven benign and three malicious clients.Benign clients each have 250 LMSYS-Chat and 250 domain-dataset samples; malicious clients have 500 MaliciousGen samples from Mistral.Source: §B.4 · p. 16
• 34.62% decreases in MD-Judge score are reported under attack in the CodeAlpaca experiment.Source: §B.4 · p. 16, Table 6 · p. 17
• Level 1 and Level 2 defenses increase safety metrics and enhance helpfulness and coding ability in the domain-specific experiment.Source: §B.4 · p. 16, Table 6 · p. 17

B.5 Effects of Number of Steps for Defense

• 400 training steps consistently produce the highest MT-1 score across four settings, while the post-hoc defense improves safety across all tested steps and settings.Source: Figure 5 · p. 17, §B.5 · p. 17
• For WildChat aligned data and Beavertails unaligned data, the smallest MD Judge score is 41.73%, reported as 29.42% above FedAvg under attack.Source: Figure 5 · p. 17, §B.5 · p. 17

B.6 Impact of Generated Data on LLM Fine-Tuning and Defense

• Generated data alone is insufficient for helpfulness when fine-tuning a pretrained LLM, while the post-hoc strategy after FedIT enhances value alignment and helpfulness.Source: §B.6 · p. 18
• After FedIT, incorporating generated data for defense brings significant safety gains without decreasing helpfulness.The comparison covers normal FedAvg and FedAvg+Gen under attack.Source: Figure 6 · p. 18, §B.6 · p. 18
• Compared with normal local training, Local+Gen improves harmless evaluations but decreases helpfulness.Local+Gen fine-tunes the pretrained Llama2 using generated data.Source: Figure 6 · p. 18, §B.6 · p. 18