FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models

1 Introduction

• FedLLM-Bench provides the first realistic FedLLM benchmark, addressing the absence of real-world datasets and standardized evaluation setups.Prior studies artificially partition centralized datasets and use differing training and evaluation configurations, hindering practical evaluation and fair comparison.Source: §1 · pp. 1–2
• Its four datasets are naturally split by user ID across 38–747 clients and cover instruction tuning, preference alignment, and diverse real-world heterogeneity.The benchmark captures variation in language, quality, quantity, instruction, sequence length, and preference.Source: §1 · p. 2
• Experiments show that federated learning consistently outperforms local training, while baseline performance rankings vary across realistic settings.The benchmark is intended to support both performance comparison and exploration of new research directions.Source: §1 · p. 2
• Multilingual experiments suggest that collaboration among similar languages could provide more benefit than collaboration across all languages.This exploratory result is demonstrated using the multilingual Fed-Aya dataset.Source: §1 · p. 2
• FedLLM-Bench integrates 8 representative baseline methods and 6 evaluation metrics into an open-source codebase for systematic comparison.The integrated datasets and codebase are released for the community.Source: §1 · p. 2

Federated learning for large language models

• Federated learning methods address heterogeneous client data through regularization, gradient correction, feature alignment, aggregation weighting, momentum, and pretrained models.The related work also includes FedLLM-specific frameworks and methods targeting property protection and differential privacy.Source: §2 · p. 2

Datasets and benchmarks in federated learning

• The benchmark’s datasets span diverse characteristics, tasks, client counts, quantities, sequence lengths, and quality levels.Table 1 summarizes these dataset dimensions across the four realistic FedLLM datasets.Source: Table 1 · p. 3

Federated learning for large language models

• Prior FedLLM experiments use artificially crafted federated datasets, limiting evaluation of effectiveness in real-world scenarios.FedLLM-Bench is introduced to mirror real-world complexity and diversity.Source: §2 · p. 3

Datasets and benchmarks in federated learning

• Unlike prior realistic federated benchmarks, FedLLM-Bench targets federated instruction tuning and federated preference alignment for large language models.Existing benchmarks such as LEAF, FLAIR, and FLamby focus on classical image, text, or medical analysis tasks.Source: §2 · p. 3

3.1 Training methods

• FedLLM-Bench covers federated instruction tuning and preference alignment through iterative client training, communication, and global aggregation, using LoRA for parameter-efficient adaptation.It includes local training and seven classical FL baselines, including FedProx, SCAFFOLD, FedAvgM, FedAdagrad, FedYogi, and FedAdam.Source: §3.1 · p. 3

3.2 Descriptions of training datasets

• The four datasets span 38 to 747 naturally user-split clients, including multilingual Fed-Aya, single-turn Fed-ChatbotIT, multi-turn Fed-WildChat, and preference-alignment Fed-ChatbotPA.Fed-Aya contains 38 clients and 25k samples; Fed-ChatbotIT 237 clients and 6k samples; Fed-WildChat 100 clients and 53k samples; Fed-ChatbotPA 747 clients and 10k samples.Source: §3.2 · p. 4
• Fed-ChatbotPA exhibits heterogeneous user preferences: most clients tend to prefer longer responses, while clients vary in both length and quality preferences.Length preference is measured by the ratio of preferred responses longer than dispreferred responses, and quality preference by their averaged quality difference.Source: §3.2 · pp. 4–5

3.3 Analysis of training datasets

• Across datasets, clients differ in length, instruction verbs, quality, embeddings, and data quantity, with within-client clustering and substantial cross-client diversity.The analyses use Llama2 tokenization, parsed instruction root verbs, IFD scores, t-SNE embeddings, and client-level quantity distributions; a large proportion of clients have relatively few data.Source: §3.3 · p. 5, Figure 5 · p. 6

3.4 Evaluation metrics

• The benchmark evaluates training methods with four open-ended metrics—MT-Bench, Vicuna bench, AdvBench, and Ref-GPT4—and two close-ended metrics, MMLU and HumanEval.MMLU and HumanEval assess whether fine-tuning preserves pretrained knowledge and code-generation capabilities.Source: §3.4 · p. 6

4.1 Benchmark results

• Fed-Aya shows that federated methods generally outperform local training on average, but no method dominates every language; FedAvg and FedProx are strongest overall.The results motivate language personalization rather than a single universally best aggregation method.Source: §4.1 · p. 6, Table 2 · p. 7
• Fed-ChatbotIT shows consistent federated gains on open-ended instruction following, comparable closed-ended knowledge performance, and the strongest average results for FedAdagrad.These results indicate that federated training can improve instruction following without compromising general capability in this setting.Source: §4.1 · p. 6, Table 3 · p. 7
• The benchmark’s baseline behavior may reflect pretrained-model benefits and short parameter-efficient fine-tuning, which reduce some data-heterogeneity advantages of specialized methods.The authors therefore call for future work focused on data quality and synthetic data.Source: §4.1 · p. 7
• Fed-WildChat shows federated methods consistently outperform local training across single-turn and multi-turn conversations, with FedAvg robust in this scenario.No single-turn method dominates every metric, whereas FedAvg consistently outperforms the best alternative across multi-turn metrics.Source: §4.1 · pp. 6–7, Table 4 · p. 8
• Fed-ChatbotPA shows that all federated methods outperform local training, with FedAvgM, FedProx, SCAFFOLD, and FedAvg among the strongest instruction-following methods.Federated preference alignment also improves helpfulness and harmlessness, while high-quality preference data motivates collaboration.Source: §4.1 · p. 7, Table 5 · p. 8

4.2 Further explorations

• FedSamLang outperforms local training in every language and achieves the highest average score, while FedSimLang improves three languages but worsens others.Both personalized approaches match or exceed FedAvg with fewer collaborators, but the authors call for better balances between localization and collaboration.Source: §4.2 · p. 8, Table 6 · p. 9
• With δ = 1e−4, FedAvg at ϵ = 1 has comparable performance to non-private FedAvg, while reducing ϵ improves privacy but degrades performance.At ϵ = 1e−3, federated performance becomes comparable to local training without differential privacy.Source: §4.2 · p. 9, Figure 6 · p. 9

5 Conclusions

• FedLLM-Bench combines 8 training methods, 4 datasets, and 6 metrics spanning instruction tuning, preference alignment, client-scale diversity, and real-world data heterogeneity.Its datasets reflect language, quality, quantity, instruction, sequence length, embedding, and preference diversity.Source: §5 · p. 9
• The benchmark’s experiments evaluate classical federated methods and explore cross-language collaboration and differential privacy for federated instruction tuning.The authors position the benchmark as a practical testbed intended to reduce effort and promote fair comparisons.Source: §5 · p. 9

A Limitations

• More future work is needed to evaluate model series and sizes beyond Llama-2 for instruction tuning and Alpaca for preference alignment.Source: A Limitations · p. 15

B.1 Lengths measurement

• Client data are measured by tokenizing instructions and responses with the Llama2 tokenizer and using token count as sentence length.Source: §B.1 · p. 15

B.2 Verbs and nouns

• The four datasets exhibit diverse instruction types and distributions, including different dominant verbs and differing quantity distributions despite keyword overlap between ChatbotIT and ChatbotPA.Source: §B.2 · p. 15
• Instruction analyses also visualize the top verbs and nouns for individual English-sample clients across all four datasets.Source: §B.2 · p. 15

B.3 Quality evaluation

• Clients in all four datasets have varying data quality under Llama2-7B and IFD evaluation, demonstrating quality heterogeneity.Source: §B.3 · p. 15

B.4 t-SNE visualization

• t-SNE visualizations show same-client samples clustering in feature space, particularly in Fed-Aya and Fed-WildChat, indicating within-dataset client heterogeneity.Source: §B.4 · p. 15

B.5 Discussion about data privacy and safety

• Fed-ChatbotIT, Fed-ChatbotPA, and Fed-WildChat may contain unsafe or toxic interactions, which are retained to study AI safety and simulate real-world dialogue.Source: §B.5 · p. 15
• Figure 7 summarizes the twenty most common root verbs and four direct noun objects across the four datasets.Source: Figure 7 · p. 16

C.1 Experimental setups

• ChatbotPA uses dynamic local steps adjusted for client sampling probability and data volume so each client receives about three total training epochs.Source: §C.1 · p. 16
• Experimental setups are detailed in Table 7, where local epochs and per-round client counts are specified for each dataset.Source: Table 7 · p. 17
• FedLLM with DP(σ=0.1) still outperforms local training on WildChat MT-Bench, while differential privacy causes a slight performance cost.Source: Table 8 · p. 17

C.2 Evaluation

• For Aya, GPT-4 Judge evaluates model answers by filling a prompt template with the question, answer, and reference answer.Source: §C.2 · p. 17, Figure 8 · p. 17

D.1 Definition

• Differential privacy is defined for randomized mechanisms on neighboring datasets using privacy parameters ϵ and δ.Source: §D.1 · p. 17

D.2 User-level differential privacy

• User-level differential privacy applies a Gaussian mechanism with zero-mean noise of variance σ2I, using L2 sensitivity to bound output perturbation and select σ.Source: §D.2 · p. 18

• The experiments use WildChat with batch size 1, while other settings follow the single-turn WildChat configuration.Source: §D.3 · p. 18
• Gaussian noise controlled by σ is added when clients upload local models, providing user-level differential privacy.Source: §D.3 · p. 18
• Smaller ϵ or larger σ indicates stronger privacy and usually lower performance, while σ is calculated from the target (ϵ, δ) guarantee.The paper also evaluates experiments with fixed σ in Table 8.Source: §D.3 · p. 18
• (0.01, 1e−4)-DP still outperforms local training, while differential privacy causes a slight model-performance cost.The result is reported for the user-level differential-privacy experiment.Source: §D.3 · p. 18