Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agents

1 Introduction

• Privacy and business controls prevent raw interaction-data aggregation, leaving cross-environment agents optimized in isolation.Source: §1 · p. 1
• Online self-evolution across distributed environments remains underexplored; sparse-reward trajectories destabilize updates and full-model synchronization is costly.Source: §1 · p. 2
• Fed-SE is presented as the first framework for privacy-constrained cross-environment knowledge transfer in LLM agents.Source: §1 · p. 2
• Success-trajectory filtering with experience accumulation stabilizes local optimization, while low-rank adapter aggregation reduces communication.Source: §1 · p. 2
• 10% absolute improvement over FedIT across five environments, with larger gains on long-horizon tasks.Source: §1 · p. 2

2 Preliminaries and Problem Setup

• Agent-environment interaction is modeled as a POMDP in which a policy uses instructions and interaction history to generate reasoning and actions.Source: §2 · p. 2
• Each of K clients holds a private environment with distinct transition dynamics and task distributions, producing non-identical local datasets.Source: §2 · p. 2
• The global objective is the weighted sum of clients’ expected trajectory returns, optimized decentrally without sharing raw trajectories.Source: §2 · p. 2
• Sparse binary rewards and heterogeneous dynamics destabilize local and cross-client gradients, while full-parameter synchronization is prohibitively expensive.Source: §2 · p. 2

3 Methodology

• Fed-SE iteratively pairs local adapter optimization on filtered trajectories with server-side aggregation and synchronization across clients.The framework’s two phases are local self-evolution and global knowledge aggregation.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Local Agent Self-Evolution

• Clients optimize lightweight adapter parameters on successful trajectories, approximating expected-return maximization with maximum likelihood estimation to mitigate sparse-reward gradient variance.The base model remains frozen, and the local process filters successful trajectories, accumulates them with historical experience, and initializes the buffer with expert demonstrations for stable first-round learning.Source: §3.1 · p. 3

3.2 Global Knowledge Aggregation

• The server averages adapters in a low-rank space to reduce communication overhead and mitigate negative transfer, then resets clients to the global consensus before each round.Unweighted averaging is used to avoid bias toward environments with abundant easy trajectories.Source: §3.2 · p. 4

3.3 Theoretical Analysis

• Under the stated assumptions and learning-rate condition, Fed-SE is guaranteed to converge to a stationary point of the global average objective.The assumptions include L-smooth local objectives, unbiased stochastic gradients with bounded variance, and bounded gradient norms.Source: §3.3 · p. 4

4.1 Experiments Setting

• Fed-SE is tested in five heterogeneous environments—BabyAI, WebShop, TextCraft, Maze, and Wordle—with each client dedicated to a distinct environment.The environments cover embodied control and grounding, web interaction, hierarchical planning, long-horizon memory, and iterative reasoning; performance is measured by task success rate.Source: §4.1 · p. 4
• Evaluation uses Qwen2.5-7B as the primary model and four additional models spanning model families, scales, and generations.Baselines include pretrained, locally fine-tuned, centralized, and FedIT models; FedIT is adapted from static instruction tuning to multi-turn trajectory imitation.Source: §4.1 · pp. 4–5

4.2 Main Results

• 10% average-success-rate improvement over FedIT on Qwen2.5-7B; Fed-SE also achieves the highest average success rate across all five base models.Figure 3 compares training trajectories over communication rounds and reports Fed-SE’s highest average success rates after 20 rounds across model architectures and scales.Source: Table 1 · p. 5, §4.2 · p. 5, Figure 3 · p. 6
• Fed-SE outperforms FedIT on all five Qwen2.5-7B environments: BabyAI 93.3% vs. 83.3%, WebShop 73.0% vs. 67.5%, TextCraft 67.0% vs. 54.0%, Maze 68.0% vs. 28.0%, and Wordle 32.0% vs. 20.0%.The reported comparison is against FedIT, which also operates under privacy constraints.Source: §4.2 · pp. 5–6
• The Fed-SE–FedIT performance gap widens over communication rounds, while Wordle improvement remains limited when successful early trajectories are difficult to obtain.The reported mechanism relies on successful trajectories being available for online evolution.Source: §4.2 · p. 6
• Fed-SE’s improvements are reported consistently across model families, scales, and generations.The authors describe this as evidence of framework stability and robustness.Source: §4.2 · p. 6

5.1 Ablation Study

• 66.1% average success rate for full Fed-SE exceeds all ablations, while removing filtering drops the rate to 40.5%.The ablations use Llama2-7B-Chat and compare removing history, removing filtering, and weighted averaging.Source: §5.1 · p. 6
• On Maze, removing cumulative history plateaus at 40.0% versus Fed-SE’s 80.0%, despite a 64.1% average across tasks.The historical buffer is described as replay that suppresses catastrophic forgetting and stabilizes policy oscillation under online distribution shifts.Source: §5.1 · p. 6, Figure 5 · p. 7
• Without success filtering, Wordle success falls to zero after round 8 as failed trajectories mislead behavioral cloning and contaminate global parameters.Figure 5 attributes the collapse to noise injection.Source: Figure 5 · p. 7, §5.1 · p. 7
• Weighted averaging reaches 59.8% and underperforms simple averaging, since trajectory quantity may not track gradient quality across heterogeneous environments.The passage notes that abundant simple BabyAI samples can bias aggregation against harder tasks.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Communication Efficiency

• Increasing LoRA rank from 4 to 8 raises success from 51.8% to 57.5% (+5.7%), while rank 8 to 16 adds only 1.6%.The further increase doubles communication overhead from 76.3 MB to 152.5 MB; Figure 6 presents performance versus communication cost by rank.Source: §5.2 · pp. 7–8, Figure 6 · p. 8
• Rank 8 balances capacity with memory constraints as successful trajectories lengthen; rank 16 offers negligible gains while consuming memory needed for longer sequences.The authors state that this balance prevents out-of-memory errors in later evolutionary stages.Source: §5.2 · p. 8

6.1 LLM Agents and Self-Evolution

• Agent self-evolution includes self-rewarding, self-play, self-refinement, persistent memory, and cross-task sharing, but existing approaches assume centralized trajectory access.The cited lines describe methods including AgentGym, reflective memory, and cross-environment generalization approaches.Source: §6.1 · p. 8

6.2 Federated Learning for Large Language Models and Agents

• Federated LLM work includes parameter-efficient fine-tuning, instruction tuning, preference alignment, and in-context agent learning, but these approaches do not provide continuous autonomous evolution across heterogeneous environments under privacy constraints.FedRLHF is limited to single-turn preference alignment, while FICAL uses frozen models without parameter updates.Source: §6.2 · p. 8
• Fed-SE is compared with adapted FedIT, which uses federated parameter updates on static instruction data to isolate the contribution of online trajectory-based self-evolution.Source: §6.2 · p. 8

• Across five environments and five base models, Fed-SE improves 6–10% absolute over FedIT, with larger gains on long-horizon tasks.Source: §7 · p. 8
• Success-trajectory filtering with experience accumulation stabilizes local updates, while unweighted low-rank aggregation reduces communication overhead and balances heterogeneous environments.Source: §7 · p. 8
• The reported results validate the feasibility of distilling generalizable agent capabilities from decentralized interaction experiences without raw data sharing.Source: §7 · p. 8

• The framework omits Differential Privacy and Homomorphic Encryption, which may leave it vulnerable to advanced gradient-reconstruction attacks.Source: Limitations · p. 9
• Synchronous Federated Averaging assumes consistent client connectivity, so edge-device heterogeneity and network instability could cause stragglers and hinder convergence efficiency.Source: Limitations · p. 9
• Global aggregation uses standard element-wise averaging; more advanced aggregation strategies for agents are left to future work.Source: Limitations · p. 9

Ethics Considerations

• Real-world deployment would require additional safety measures to ensure alignment with human values; this work focuses on simulated environments.Source: Ethics Considerations · p. 9

A.1 Surrogate Objective Derivation

• With binary rewards, maximizing expected return is equivalent to maximizing the likelihood ratio on successful trajectories, yielding a surrogate optimization objective.The derivation uses importance sampling and Jensen’s inequality.Source: §A.1 · p. 13

A.2 Lemma: Client Drift Bound

• The parameter drift for client k after e local steps is bounded using Cauchy–Schwarz and bounds on the expected squared gradient norm.The stated gradient-norm bound relies on Assumptions 2 and 3.Source: §A.2 · p. 13

A.3 Proof of Theorem 1

• The proof combines smoothness, inner-product and update-norm bounds, then sums across rounds to confirm Theorem 1’s convergence rate.Source: §A.3 · pp. 13–14

B.1 Environment and Dataset Specifications

• Fed-SE is evaluated across five heterogeneous environments: BabyAI for embodied navigation and manipulation, WebShop for online shopping, TextCraft for crafting via recursive recipes, Maze for long-horizon navigation, and Wordle for iterative word guessing.Source: §B.1 · p. 14

B.2 Training Hyperparameters

• Clients use LoRA adapters with a frozen pretrained backbone, while the server averages adapters across five task-specific clients for 20 communication rounds.Each client trains for two epochs per round; the adapters use rank 8 and alpha 16.Source: §B.2 · p. 14

B.3 Trajectory Collection and Filtering

• Only trajectories receiving a success flag enter the experience buffer; failed trajectories are discarded.The buffer accumulates data throughout the 20 rounds without a capacity limit.Source: §B.3 · p. 14

B.4 Prompting and Input Construction

• The ReAct prompt combines system instructions and episode interaction history, and requires structured Thought and Action fields.Source: §B.4 · p. 17

C Additional Experiments and Case Study

• The additional analyses present per-environment performance curves across base models and qualitative trajectory comparisons before and after federated self-evolution.Source: C Additional Experiments and Case Study · p. 17

C.1 Performance Analysis Across Models

• 66.1% average success on Llama2-7B versus 55.7% for FedIT, and 63.0% on Qwen2.5-3B versus 57.3% for FedIT.The cited comparisons also report 80.0% versus 28.0% on Llama2-7B Maze and 36.0% versus 20.0% on Qwen2.5-3B Maze.Source: §C.1 · p. 17, Figure 7 · p. 19, Figure 8 · p. 19
• 73.2% average success rate on Qwen2.5-7B, surpassing FedIT by 10.5 percentage points; Fed-SE improves results across all five environments.The largest gain is on Maze, rising from 28.0% under FedIT to 68.0% under Fed-SE.Source: §C.1 · p. 17, Figure 9 · p. 20
• 54.3% average success on Qwen3-1.7B, 7.7 percentage points above FedIT; Qwen3-8B reaches 70.2%, 8.6 points above FedIT.On Maze, the reported comparisons are 40.0% versus 8.0% for Qwen3-1.7B and 64.0% versus 12.0% for Qwen3-8B.Source: §C.1 · p. 17, Figure 10 · p. 20, Figure 11 · p. 21

C.2 Qualitative Trajectory Analysis

• BabyAI and Maze cases show agents moving from spatial misinterpretation or wall-blocked movement to goal-directed navigation.Source: §C.2 · p. 18, Figure 12 · p. 21, Figure 15 · p. 24
• In WebShop and TextCraft, agents verify the required 2-pack quantity and gather 2 wood before crafting a wooden pickaxe.Source: §C.2 · p. 18, Figure 13 · p. 22, Figure 14 · p. 23
• In Wordle, the agent uses yellow feedback to reposition “A” and ultimately finds the correct word; Wordle remains challenging across methods.Source: §C.1 · p. 17, §C.2 · p. 18, Figure 16 · p. 25