PaperlayerView exact revision on arXiv

Source-linked AI summary

FedSkillGit: Federated Skill Evolution via Version Control

Boxun Lan, Jiamei Li, Weijian Li, Chengyu Yang, Zhuang Qi, Chengcheng Wan, Xiaohong Chen, Ming Hu

arXiv:2610.21348v1cs.AI

TL;DR

Federated skill evolution seeks to incorporate experience distributed across privacy-constrained data owners without sharing raw trajectories, but whole-document updates can retain client-specific information, waste communication, and obscure contributions. FedSkillGit exchanges privacy-filtered commits, merges them into a candidate shared skill, and validates it with clients; experiments report gains over independent evolution and performance near centralized evolution.

  • Problem

    Whole-document federated skill updates can retain client-specific information, repeatedly transmit unchanged content, and obscure useful client revisions, despite the need to use distributed experience without sharing raw trajectories.

    Source: §I · p. 1

  • Method

    FedSkillGit replaces full-document updates with fine-grained commits and two-stage privacy filtering, then deduplicates and merges patches into a candidate skill for validation before publication.

    Source: §I · p. 1, 2) Privacy filtering · p. 4, 3) Version validation · p. 5

  • Results

    Across three language model families and three datasets, FedSkillGit improves task success over independent evolution by 1.6–10.1 points, with a mean gain of 4.3 points, and has a median gap of −0.8 against centralized evolution.

    Source: §V · p. 7

  • Takeaways & Limitations

    FedSkillGit supports accumulation and validation of distributed experience in one shared skill without raw execution data leaving clients.

    Source: §V · p. 7

    The two privacy-filtering stages constrain transmitted payloads but do not bound what an adversary can infer from sequences of accepted revisions; formal privacy may require complementary differential privacy.

    Source: Appendix B · p. 9

Abstract

from arXiv · show

Large language model agents can improve their capabilities by evolving reusable natural-language skills from execution experience. In real-world deployments, this experience is distributed across data owners who cannot pool their raw data, which motivates federated skill evolution. The natural first design lets each client evolve the whole skill document and upload the result for server-side aggregation. That design is poorly matched to skill evolution for three reasons. An evolved document can still carry client-specific content distilled from private traces, repeated transmission of a nearly unchanged document spends communication on knowledge that the population already holds and the contribution of a single client is buried in a document that the server cannot meaningfully attribute. We propose FedSkillGit, a federated skill evolution framework that replaces full-document updates with commits. Each participating client submits a bounded set of patches over named rules together with its local experience, which summarizes what the round taught it, and the pair passes a two-stage privacy filter before it leaves the client. The server deduplicates the submitted patches and the local experience of all participants, updates a global experience document that only the optimizers read, and merges the surviving patches into a candidate skill. The candidate is distributed to the clients, which score it on their own validation sets and report the results, and it is published only when the reported scores improve on the current version. Experiments on three datasets and three language model families show that FedSkillGit outperforms state-of-the-art baselines by up to 22% and approaches centralized evolution without sharing raw client data.

I. INTRODUCTION

The introduction frames federated skill evolution as a way to use distributed experience without sharing raw trajectories, while noting that whole-document updates poorly fit structured natural-language skills. FedSkillGit instead exchanges fine-grained commits with privacy filtering and version validation.

I. INTRODUCTION

  • Distributed execution experience could support broader skill evolution, but sharing raw trajectories risks exposing sensitive inputs, private environmental state, or proprietary workflows.Existing methods primarily rely on individual agents’ experience, limiting improvement to local experience.Source: §I · p. 1
  • Whole-document updates can retain client-specific information, repeatedly transmit unchanged content, and obscure which client contributed useful revisions.These challenges motivate privacy-preserving, communication-efficient, fine-grained aggregation of natural-language skill knowledge.Source: §I · p. 1
  • FedSkillGit replaces full-document aggregation with semantic commits containing fine-grained skill revisions and supporting experience, alongside two-stage privacy filtering and version validation.Its contributions also include cross-round experience reuse to guide local optimization and server-side aggregation.Source: §I · pp. 1–2
  • Up to 22%: FedSkillGit outperforms the strongest baseline across three datasets and three language model families while approaching centralized performance.The introduction reports extensive experiments but does not specify the metric behind the up-to-22% figure.Source: §I · p. 2

A. Federated learning; B. Skill evolution

  • Federated learning enables clients to improve a shared model without exchanging raw data, but typical federated-learning updates are model parameters or related representations rather than natural-language skills.The related-work discussion describes skill-evolution research on acquisition, organization, refinement, and evolution, while noting that distributed privacy-constrained collaboration remains unaddressed.Source: A. Federated learning · p. 2, B. Skill evolution · p. 2

C. Federated skill evolution

FedSkillGit targets a generalized shared skill for similar tasks, unlike personalized skill libraries for heterogeneous clients. Its framework evolves that shared skill through privacy-filtered commits, server-side experience-aware merging, and client validation.

C. Federated skill evolution

  • FedSkillGit targets a generalized skill for similar tasks, whereas FederatedSkill maintains personalized libraries for heterogeneous clients.The shared-skill setting requires deciding which local revisions to incorporate and whether their composition improves the skill.Source: C. Federated skill evolution · p. 2

A. Federated framework

  • The server maintains a shared skill and global experience document, while each client keeps private training and validation data and uses executor, optimizer, sanitizer, and auditor agents.A server merge agent updates experience and aggregates patches; version validation determines whether the candidate becomes the new shared skill.Source: A. Federated framework · p. 3

B. Overview

  • Selected clients use local execution trajectories and global experience to produce rule-level patches and local experience, then sanitize and audit commits before transmission.Only commits that pass the privacy audit are transmitted to the server.Source: B. Overview · pp. 3–4
  • The server deduplicates patches and experience, updates global experience, and merges surviving patches into a candidate; clients report validation success rates, and the candidate is published only if aggregate performance is no lower than the current version.The global experience document guides later optimization and aggregation, is read only by optimizer and merge agents, and evolution terminates after two consecutive rounds without an accepted update.Source: Figure 1 · p. 3, B. Overview · p. 4

C. Local Skill Evolution

Local skill evolution turns execution experience into a semantic commit comprising rule-level edits and a local experience summary. The commit records revisions and the experience motivating them.

C. Local Skill Evolution

  • Each participating client executes its private training set using the current global skill, then analyzes trajectories with the prior global experience to extract a local patch and experience.The patch is represented as a sequence of rule-level edits.Source: 1) Commit generation · p. 4

1) Commit generation:

  • An edit modifies a particular rule in the current skill document, and its fields specify the editing operation, target rule, and revision content.The allowed operations are append, insert-after, and replace.Source: 1) Commit generation · p. 4
  • The local experience summary captures observed failure patterns and execution experience that motivate the proposed revisions; patches and experience are organized into a semantic commit.Source: 1) Commit generation · p. 4

2) Privacy filtering:

FedSkillGit sanitizes commits and audits them against client-specific private values before transmission, then deduplicates submitted experience and incorporates it into a global experience document.

2) Privacy filtering:

  • Sanitization and auditing form a two-stage filter intended to reduce sensitive information in commits derived from private training data.Only commits that pass the audit are transmitted.Source: 2) Privacy filtering · p. 4
  • The sanitizer replaces client-specific entities, identifiers, and environment details with typed placeholders while retaining each edit’s operation and target rule.This preserves the applicability of skill revisions.Source: 2) Privacy filtering · p. 4
  • The auditor checks sanitized commits against a fixed client-specific private-value set; failed commits return to sanitization for revision.The private-value set is extracted from local training data and remains fixed throughout evolution.Source: 2) Privacy filtering · p. 4

1) Experience Updating:

  • The server deduplicates local experiences to remove redundant or conflicting information, then merges them with the existing global experience document.Accumulated experience is organized into typed entries for structured management and retrieval.Source: 1) Experience Updating · p. 5

2) Experience-aware aggregation:

The merge agent uses accumulated global experience to integrate deduplicated patches into a candidate skill. Clients validate the candidate locally, and only a candidate whose aggregate score is no lower than the current version is published.

2) Experience-aware aggregation:

  • The merge agent integrates deduplicated patches into the current skill under the guidance of updated global experience.It classifies edits as kept, deduplicated, promoted to shared rules, or rejected as invalid; accumulated experience records failure patterns and attempted revisions.Source: 2) Experience-aware aggregation · p. 5

3) Version validation:

  • Clients evaluate the candidate on their local validation sets, and it is published only if its aggregate score is no lower than the current skill’s score.Otherwise, the candidate is rejected and the previous global skill is retained.Source: 3) Version validation · p. 5
  • The validation gate keeps the published skill’s evaluation score non-decreasing while clients retain private validation data.Evolution terminates after candidate rejection in two consecutive rounds; the convergence round is the last accepted round.Source: 3) Version validation · p. 5

IV. EXPERIMENTS

  • The experiments compare the method with baselines, examine component ablations, and test scalability across client counts and participation ratios.The section first introduces the experimental setup.Source: §IV · p. 5

A. Experimental setup

The evaluation uses task success rate across three datasets and three language models, comparing FedSkillGit with centralized, no-skill, independent, and document-merge baselines. FedSkillGit beats independent evolution across all nine dataset-model cells and recovers most centralized performance; ablations also show that global experience contributes in both optimization and merging.

A. Experimental setup

  • The setup uses 10 clients with 30% participation, three datasets, and GPT-5.4-mini, Qwen3.5-flash, and Gemini3-flash, with task success rate as the primary metric.Baselines include a centralized process trained on pooled per-round exposure and evaluated on the union of validation sets, plus no-skill, independent, and DocMerge methods.Source: A. Experimental setup · p. 5

B. Performance Comparison

  • With 10 clients at 30% participation, FedSkillGit exceeds Independent in all nine cells by +1.6 to +10.1 (mean +4.3), has median gap −0.8 versus Centralized, and exceeds DocMerge by +0.8 to +7.2.Source: B. Performance Comparison · p. 6, Table I · p. 6
  • On DocVQA, FedSkillGit leads all three model families; its gap to Centralized is largest on Gemini3-flash.On narrow-scope LiveMath, its lead over Independent is 1.6 points for GPT-5.4-mini and 4.0 points for Qwen3.5-flash.Source: B. Performance Comparison · p. 6

1) Effectiveness of the global experience document:

  • Removing either optimizer-side or merge-side use of global experience lowers SearchQA task success rate from 78.6 to 76.9 or 77.2, respectively.The ablations retain the same commits, merge operator, and publication gate; experience guides optimization toward residual failures and helps identify repeated or rejected edits.Source: Table II · p. 6, 1) Effectiveness of the global… · p. 6

2) Effectiveness of the sanitizer:

FedSkillGit improves task success across the evaluated models and datasets while keeping raw execution data local. Sanitization improves first-pass audit clearance, and convergence rounds rise with client count but fall with participation ratio.

2) Effectiveness of the sanitizer:

  • 81% of payloads clear the audit on the first attempt with sanitization, versus 33% without it; rejected payloads fall from 67% to 19%.The comparison uses real-run payloads on GPT-5.4-mini and SearchQA.Source: 2) Effectiveness of the sanitizer · p. 6

1) Impact of the number of clients:

  • Five rounds at ten clients rise to nine at one hundred, with the increase accelerating as the client population grows.Early rounds accept more patches; later rounds involve overlapping submissions and fewer surviving patches as clients disagree on remaining failures.Source: 1) Impact of the number of clients · pp. 6–7, Figure 2 · p. 6

2) Impact of the participation ratio:

V. CONCLUSION

  • Clients exchange bounded, filtered commits that the server merges into one evolving skill, guided by global experience recording failure patterns and rejected revisions.The conclusion also states that no raw execution data leaves a client at any point in the protocol.Source: §V · p. 7
  • Task success improves over independent evolution by 1.6–10.1 points in every cell, with a 4.3-point mean gain, and the median gap to centralized evolution is −0.8.The comparison spans three language model families and three datasets.Source: §V · p. 7

APPENDIX A PROMPT TEMPLATES OF THE PROTOCOL

The appendix specifies prompts for task execution and local skill optimization. The executor follows the shared skill and dataset answer format, while the optimizer diagnoses failures and returns a bounded patch plus local experience.

APPENDIX A PROMPT TEMPLATES OF THE PROTOCOL

A. Executor prompt

  • The executor solves one dataset task using the shared skill as its only guidance and consults it when unsure how to proceed.The prompt supplies the skill document and task instance as separate inputs.Source: A. Executor prompt · p. 8
  • The executor must emit one answer in the dataset-defined format, without referring to the skill document in its answer.The prompt also limits the executor by a maximum number of turns and tokens.Source: A. Executor prompt · p. 8

B. Optimizer prompt

  • The optimizer improves the shared skill from one client's execution evidence while seeing trajectories rather than raw data or gold answers.Its inputs include the current skill and the global experience document.Source: B. Optimizer prompt · p. 8
  • The optimizer diagnoses slice-specific failures, then proposes at most the edit budget in named-rule edits, each justified by trajectory evidence.Edits may append, insert, or replace a rule, and should not repeat a revision already recorded as rejected in global experience.Source: B. Optimizer prompt · p. 8
  • The optimizer returns one patch and one local experience summarizing observed failure patterns and the proposed edits' expected effects.The patch must not edit unnamed rules, and the total document must stay under the size ceiling.Source: B. Optimizer prompt · p. 8

C. Merge prompt

The merge prompt updates global experience and aggregates client patches into a candidate skill. It separates experience from the skill document and assigns each edit a disposition.

C. Merge prompt

  • Each submitted commit packages a patch over named rules with the producing client's local experience, alongside the current skill and global experience.The merge step returns an updated experience document, candidate skill, and disposition record for each submitted edit.Source: C. Merge prompt · p. 9
  • The merge step receives the current skill, submitted commits, and global experience as distinct inputs.The prompts label each input separately.Source: C. Merge prompt · p. 9
  • The merge step deduplicates reported experience and records what the round established, including proposed revisions found ineffective.The experience document is context for the optimizer and merge step, not content to be written into the skill.Source: C. Merge prompt · p. 9
  • Patch aggregation keeps new clean edits, deduplicates repeats, promotes edits into shared rules, and rejects structurally invalid edits.An edit is dropped only when it duplicates a kept edit or is structurally invalid; client text is not rewritten or averaged.Source: C. Merge prompt · p. 9

D. Sanitizer prompt

  • The sanitizer rewrites a commit payload before transmission, removing entities, paths, identifiers, dataset-specific literals, and other client-specific values.It replaces removals with typed placeholders while preserving surrounding sentences and patch structure.Source: D. Sanitizer prompt · p. 9
  • The sanitizer must preserve rule identifiers, edit operations, and section headers so the rewritten patch remains applicable.It re-emits the payload field by field without summarizing or commenting.Source: D. Sanitizer prompt · p. 9

E. Private-value extraction prompt

The private-value extraction prompt builds a client-specific list of literal strings whose appearance in transmitted text would count as a leak.

Audit reference: private-value set for one client

  • The prompt extracts literal values from a sample of one client's private training data that would constitute a leak if sent to another party.The input is the client's data sample.Source: E. Private-value extraction prompt · p. 9
  • It groups exact strings by path, identifier, and other client-specific literal.The prompt asks for a flat list and prohibits paraphrasing, generalizing, or completing values.Source: E. Private-value extraction prompt · p. 9
  • The list serves as a rejection test: a transmitted payload containing any listed value is returned to the sanitizer.The prompt prefers a short, precise list over a long one.Source: E. Private-value extraction prompt · p. 9

APPENDIX B LIMITATIONS

FedSkillGit is bounded by a shared-skill setting, client-side validation, and payload-focused privacy filtering. Its evaluation uses fixed model, dataset, client, and scoring protocols to compare published skill documents.

APPENDIX B LIMITATIONS

  • One shared skill document lets the population validate a common artifact, but sharply different client distributions cannot receive a client-specific skill.The alternative personalized-library approach does not provide an acceptance test that the whole population can jointly apply.Source: Appendix B · p. 9
  • Clients must score candidate documents on held-out validation splits; subjective or costly tasks may require stronger scoring, likely with human review.Publication is determined on the union of client validation sets.Source: Appendix B · p. 9
  • The privacy stages constrain transmitted payloads but do not bound inference from accepted revision sequences; formal privacy requires an additional mechanism such as differential privacy.Audits use a frozen private-value set from each client’s own slice, making their verdicts reproducible.Source: Appendix B · p. 9
  • The server sees each participant’s sanitized proposal because commits are not hidden by secure aggregation or other cryptographic means.The privacy filter is therefore essential rather than optional.Source: Appendix B · p. 9
  • Convergence takes more rounds as client population grows and fewer as participation rises; the shared seed also assumes clients work on the same problem.With many clients, wall-clock rounds can be costly unless participation increases.Source: Appendix B · p. 9

APPENDIX C EXPERIMENTAL PROTOCOL DETAILS

  • The protocol compares three model families on SearchQA, LiveMath, and DocVQA using non-IID client slices and disjoint held-out evaluation data.The main comparison uses 10 clients with 30% participation; methods share the executor, per-round exposure, and publication gate, while task success rate is measured on sealed held-out splits.Source: Appendix C · p. 10

Related papers

Loading 2610.21348v1…