PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Agent's Skill Should Change? Group-Relative Credit Assignment for Evolving Multi-Agent Teams

Yanan Ma, Yang Wang, David Le, Chenhan Yuan, Yizheng Sun, Guangwei Zhang, Chenghao Xiao, Chenghua Lin, Imon Banerjee

arXiv:2610.27227v1cs.CL

TL;DR

Cooperative language-model teams receive a shared reward, but existing trajectory-based skill credit may not identify which agent or skill is responsible. GRSE compares repeated same-question executions and grounds agent–skill credit in structured diagnoses; it outperforms MASkills across the reported HotpotQA and LoCoMo results except open-domain.

  • Problem

    Team-level outcomes must be attributed to responsible agent behaviors, while crediting skills merely because they appear in a trajectory can reward harmful skills.

    Source: §1 · p. 1

  • Method

    GRSE compares multiple same-question team executions under fixed skill libraries, assigns agent–skill feedback, and grounds attributions in checked structured diagnoses.

    Source: §1 · p. 2

  • Results

    GRSE outperforms MASkills on HotpotQA and every LoCoMo category except open-domain, with gains linked to fewer broken answers and better-grounded retrieval.

    Source: §1 · p. 3, §5.1 · p. 6

  • Takeaways & Limitations

    The reported analyses connect GRSE’s gains to better-grounded retrieval and fewer broken answers, rather than answers given without evidence.

    Source: §1 · p. 3

    Evaluation covers two-agent sequential teams on two question-answering benchmarks, with two open-weight actors and one critic model.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Cooperative language-model agents can improve without weight updates by revising reusable skills, but only if a shared team reward is traced to the responsible agent and skill. Existing methods credit the skills present in each trajectory, yet presence is not responsibility: a harmful skill active in successful runs still earns positive credit. We present \textbf{GRSE}, \textbf{G}roup-\textbf{R}elative \textbf{S}kill \textbf{E}volution, which assigns behavioral credit by comparison: inspired by GRPO, it executes each question several times under the same skill libraries and contrasts the executions to credit agent--skill pairs, with structured diagnosis of failures. On HotpotQA and LoCoMo with two open-weight actors, GRSE outperforms MASkills, the closest prior method, on HotpotQA and every LoCoMo category except open-domain, with the largest gains on multi-hop questions ($+6.7$ and $+3.2$ F1). Its gains rest on better-grounded retrieval rather than answers given without evidence, and its learned skills transfer between the multi-hop tasks. GRSE is robust to corrupted libraries: starting from planted harmful skills, it credits every planted skill with the correct sign and recovers the score of a clean library ($+6.3$ points), whereas MASkills misattributes two and falls below its starting point.

1 Introduction

Cooperative teams receive a shared outcome, so improving them requires locating which agent behavior deserves credit; trajectory-based skill credit can mistake presence for responsibility. GRSE compares same-question team executions, diagnoses failures, and uses the resulting agent–skill attributions to guide revisions.

Motivation

  • Existing skill methods often distill whole trajectories or credit each skill from a single trajectory, allowing harmful skills present in successful runs to receive credit.The paper contrasts this with jointly comparing executions to localize outcomes to individual agents’ behaviors.Source: §1 · p. 1

Challenges

  • Attribution must look beyond where an error appears: Agent 2 can answer consistently with its input while Agent 1’s retrieval determines success.The decisive difference in the described example is Agent 1’s search query.Source: §1 · p. 2

Method

  • GRSE samples multiple complete team rollouts of the same question under fixed skill libraries and compares agent behavior and handoffs to assign agent–skill feedback.The method is training-free and keeps model parameters fixed.Source: §1 · p. 2

Contributions

  • Structured diagnosis ties attributions to failure types, checked execution steps, and edit actions, while cross-question aggregation weights them by distinct supporting questions.Figure 1 illustrates a failure surfacing in Agent 2’s answer being traced to Agent 1’s retrieval, prompting a new retrieval skill.Source: §1 · pp. 2–3, Figure 1 · p. 2

Contributions and results

  • GRSE outperforms MASkills on HotpotQA and every LoCoMo category except open-domain, with gains linked to fewer broken answers and better-grounded retrieval.The evaluation uses two open-weight actors.Source: §1 · p. 3

2 Related Work

Prior work develops reusable experience and skills, but GRSE focuses on locating which individual team agent’s editable behavior execution evidence supports changing.

2.1 Reflection and Experience Memory

  • Experience-based methods store reflections, reusable insights, linked notes, or reasoning strategies, while GRSE instead assigns execution evidence to editable skills owned by individual agents.The cited approaches include Reflexion, ExpeL, A-MEM, and ReasoningBank.Source: §2.1 · p. 3

2.2 Reusable Skill Discovery and Evolution

  • Single-agent skill methods do not decide which agent’s skill to revise; MASkills supports cooperative teams but credits each trajectory on its own, unlike GRSE’s joint comparison and checked diagnosis.GRSE retains MASkills’ skill representation and editing operators while changing how credit is formed.Source: §2.2 · p. 3

2.3 Group-Relative and Contrastive Feedback

  • Where prior contrastive methods compare outcomes or vary configurations, GRSE holds skills fixed and compares team behavior to trace outcome differences to an agent, skill, and cited step.The passage contrasts GRSE with SkillCAT (Chen et al., 2026), SkillCome (Li et al., 2026), and CANTANTE (Zehle, 2026).Source: §2.3 · p. 3

3 Method

GRSE evolves fixed-parameter agents by comparing grouped executions, grounding credit in checked diagnoses, aggregating evidence across questions, and validating resulting skill edits.

3.1 Grouped Rollouts and Relative Feedback

  • For each question, GRSE samples complete team executions under the same libraries, so within-group reward differences help identify behaviors associated with outcomes.Equal-reward groups lack contrasts but are retained because they still show consistently failing or successful behaviors.Source: §3.1 · p. 4

3.2 Group-Relative Behavioral Critic Assignment

  • A group-relative critic compares agents across executions and returns skill credits, agent-level feedback or induction evidence, and structured failure diagnoses.Diagnoses distinguish failure types and proposed actions; checks verify execution references and constrain which edits are permitted.Source: §3.2 · p. 4

3.3 Cross-Question Credit Aggregation

  • GRSE aggregates accepted feedback across questions into existing-skill edit directions and groups induction evidence into patterns, without counting repeated rollouts as additional supporting questions.The aggregation reuses the MASkills operator for existing skills and retains example diagnoses as evidence for induction.Source: §3.3 · p. 5

3.4 Credit-Guided Skill Evolution

  • Skill evolution refines, prunes, consolidates, or induces skills from aggregated directions and evidence, then validates candidate updates against the pre-update team before keeping them.Accepted candidates are assembled after agents are tested, and acceptance remains per agent.Source: §3.4 · p. 5

4 Experiment

Experiments compare skill-evolving two-agent teams and baselines across question-answering benchmarks, with GRSE variants isolating aspects of its feedback and sampling design.

4.1 Experimental Setup

  • Experiments follow MASkills’ two-agent sequential-role setting, use Qwen3.8-27B or Gemma-4-31B-it as actors, and hold model parameters and roles fixed while revising skills.A fixed Qwen3.8-27B model serves as critic and skill optimizer; methods train for five iterations, and GRSE samples four trajectories per question versus one for MASkills.Source: §4.1 · p. 5

4.2 Benchmarks

  • HotpotQA evaluation uses answer-token F1 on the distractor–hard subset; LoCoMo evaluation covers multi-hop, temporal, open-domain, and single-hop categories with F1 and smoothed BLEU.Two-agent methods are reported as means over three training runs, while single-model baselines are evaluated once.Source: §4.2 · p. 6
  • The reported variants remove structured diagnosis, add dataset-label conditioning, use one rollout per question, or filter out groups whose rewards do not differ.The main analysis includes GRSE without structure analysis; other variants are reported in ablations.Source: §4.2 · p. 6

5 Results and Discussion

Across the two benchmarks, GRSE generally outperforms MASkills, with analyses linking its gains to fewer broken answers and better-grounded retrieval. Its learned skills transfer across the multi-hop tasks and can repair corrupted libraries; ablations support both group contrast and structured diagnosis.

5.1 Main Results

  • GRSE is strongest among two-agent methods on HotpotQA and beats MASkills across LoCoMo categories except open-domain, gaining +6.7 F1 with Qwen3.8 and +3.2 F1 with Gemma4 on multi-hop.On HotpotQA, GRSE scores 83.7 vs. 80.6 F1 with Qwen3.8 and 82.8 vs. 82.3 with Gemma4.Source: §5.1 · p. 6
  • GRSE also surpasses the full-context reference on temporal questions with both actors, although that reference retains the advantage on several other categories.Full-context retains the advantage on single-hop with both actors, multi-hop with Gemma4, and open-domain with Qwen3.8.Source: §5.1 · p. 7

5.2 Analysis of Learned Skills

  • MASkills breaks correct answers three times as often on HotpotQA with Qwen3.8 (−1.8 vs. −0.6) and twice as often on LoCoMo with Gemma4 (−6.8 vs. −3.1).On LoCoMo with Gemma4, GRSE lowers the share of questions with no cited evidence to 14%, with only 5 of 61 declined-to-correct answers lacking evidence.Source: §5.2 · p. 7
  • On LoCoMo with Gemma4, GRSE learns more abstention rules than MASkills (18% vs. 5%); on HotpotQA with Qwen3.8, its skills mostly encode answer-format rules (57%).MASkills mostly writes reasoning procedures on HotpotQA with Qwen3.8 (50%).Source: §5.2 · p. 8

5.3 Robustness Analysis

  • GRSE gives all five planted helpful and harmful skills the correct credit sign and recovers helpful-skill-only performance (66.9 vs. 66.7 F1); MASkills gets two signs wrong.GRSE prunes or locally repairs harmful skills, whereas MASkills merges perturbed content into new skills.Source: §5.3 · p. 8

5.4 Cross-Task Transfer

  • Transferred GRSE libraries improve over the empty library on both multi-hop targets (+1.3 and +1.8 F1), while in-domain libraries score higher than transferred ones (+1.9 and +1.2 F1).Source: §5.4 · p. 9, Table 6 · p. 9

5.5 Ablation studies

  • Removing structured diagnosis or group sampling (G=1) lowers average F1 by 1.0 and 0.8, respectively; full GRSE averages 54.9 F1 and 42.2 BLEU.Source: §5.5 · p. 9
  • Discarding equal-reward groups lowers category averages (−0.5 F1, −0.6 BLEU), while label conditioning yields the lowest category-average F1 (53.0) despite favoring majority single-hop.The DAPO filter also lowers open-domain F1 by 4.1, while single-hop gains 2.4 F1; label-conditioned GRSE trails on every category except single-hop.Source: §5.5 · p. 9

6 Conclusion

GRSE assigns shared team outcomes to agent–skill pairs by comparing repeated executions and uses checked diagnoses to guide validated skill updates. On two open-weight actors, it outperforms MASkills on HotpotQA and most LoCoMo categories, and handles planted harmful skills more accurately.

6 Conclusion

  • GRSE outperforms MASkills on HotpotQA and every LoCoMo category except multi-hop, with its largest gains on multi-hop questions.The conclusion reports this category pattern and notes that answer quality improves through better-grounded retrieval rather than fewer declined answers alone.Source: §6 · p. 9
  • GRSE compares repeated executions of the same question under fixed skill libraries to assign team outcomes to individual agent–skill pairs.Structured diagnosis grounds attributions in checked execution steps, and the resulting credit drives targeted, validated skill updates.Source: §6 · p. 9
  • GRSE assigns every planted harmful or helpful skill the correct credit sign and restores the score of a clean library, while MASkills ends below its starting point.MASkills rewards a harmful skill for successes in which it is merely present.Source: §6 · p. 9
  • Learned GRSE libraries transfer between HotpotQA and LoCoMo multi-hop questions without harm, and target-task training adds further gains only for GRSE.The paper concludes that comparing executions of the same task is a practical way to identify which agent behavior should change.Source: §6 · p. 9

Limitations

GRSE’s more accurate credit assignment requires additional training-time compute, while inference remains unchanged. Evaluation is limited to two-agent sequential teams, two question-answering benchmarks, two open-weight actors, and one critic model.

Limitations

  • Using G=4 rollouts per question raises actor-token cost by 1.6–2.9× and wall-clock time by 1.3–1.8× over MASkills.The overhead is confined to training; test-time inference is unchanged.Source: Limitations · p. 9
  • The reported compute trade-off arises during training, since GRSE’s test-time inference is unchanged.The stated cost increase uses four executions of each question.Source: Limitations · p. 9
  • Evaluation covers two-agent sequential teams on HotpotQA and LoCoMo, using two open-weight actors and one critic model.Larger teams, non-sequential communication, longer tool-heavy tasks, and other critic models remain to be tested.Source: Limitations · p. 10

Ethical Considerations

The experiments use publicly available HotpotQA and LoCoMo benchmarks under their respective licenses, with all models open-weight and run locally.

Ethical Considerations

  • Experiments use publicly available HotpotQA and LoCoMo benchmarks under their respective licenses, and all models are open-weight and run locally.Because GRSE writes skills in natural language, humans can inspect and edit learned libraries for safety before deployment.Source: Ethical Considerations · p. 10

A Experimental Details

Experiments use Qwen3.8-27B or Gemma4-31B-it as actors, with Qwen3.8-27B also serving as critic and optimizer. Training uses grouped rollouts and specified decoding, data-split, reward, and skill-update settings; final test scores use greedy decoding.

A Experimental Details

  • Qwen3.8-27B or Gemma4-31B-it serves as actor, while Qwen3.8-27B serves as critic and optimizer in all runs.Models run locally with vLLM 0.29 on two NVIDIA RTX A6000 GPUs each, using tensor parallelism 2 and a 65,536-token context limit.Source: A Experimental Details · p. 12
  • LoCoMo is split by conversation, and category-specific libraries train separately from a mixed library trained on all four categories.Category 5 (adversarial) is excluded, following MASkills and Mem0 (Chhikara et al., 2025).Source: A Experimental Details · p. 12
  • All reported test scores evaluate final iteration-5 libraries with greedy decoding, while LoCoMo validation is greedy and HotpotQA validation uses training sampling parameters.Training rollouts use each model’s default sampling parameters so same-question executions can differ; MASkills on LoCoMo decodes greedily throughout.Source: A Experimental Details · p. 12
  • GRSE samples G=4 team trajectories per question, while MASkills samples one; both start with empty libraries and run five iterations.On LoCoMo, skill evolution uses the mean of category-specific token F1 and smoothed BLEU as training reward; HotpotQA uses token F1.Source: A Experimental Details · p. 12, Table 8 · p. 12
  • Skill updates apply refinement and induction every iteration, consolidation every second iteration, and pruning every second HotpotQA or third LoCoMo iteration.GRSE and MASkills share the listed training settings except for rollouts per question.Source: A Experimental Details · p. 12, Table 8 · p. 12
  • CoT-SC draws five samples at temperature 0.7 and returns the majority answer, while Self-Refine runs at most three feedback rounds.Other prompting calls are greedy, and the full-context baseline limits answers to 32 tokens.Source: A Experimental Details · p. 12

B Learned Skills Analysis

The analysis examines how GRSE’s learned skills change answer retrieval and reasoning, using two cases and a coding scheme for the final skill libraries. One case corrects a decline by retrieving cited evidence; another corrects an answer by interpreting existing passages more precisely.

Analysis Definitions

  • The analyses use the first run of each configuration with greedy decoding, and classify final answers as declined, correct, or wrong using stated criteria.A non-declined answer is correct at F1 ≥ 0.5; LoCoMo evidence requires at least one cited dialogue turn.Source: B Learned Skills Analysis · pp. 12–13

Skill Coding

  • The 180 final-library skills are labeled by primary type; retrieval, verification, reasoning, formatting, and abstention are included, and none is labeled generic.Qwen3.8-27B assigns each skill one primary type using its name, description, and body.Source: B Learned Skills Analysis · p. 13, Table 9 · p. 13

Learned Skill Cases

  • 56 declined LoCoMo questions became correct with cited evidence; in the illustrated case, adjacent-context-expansion fetches the answer turn after a matched question.With the empty library, the retriever returned NO_EVIDENCE and the reasoner declined; with the learned retrieval procedure, the reasoner answered correctly.Source: B Learned Skills Analysis · p. 13
  • On HotpotQA, two learned skills help Agent 1 validate both people’s stated roles and select “director” from the passages already available.The empty-library answer was “Filmmaker (or director)”; the learned procedure makes the answer more specific without new evidence.Source: B Learned Skills Analysis · p. 13

C Robustness Analysis

The robustness experiment tests whether GRSE can repair libraries seeded with helpful and harmful or irrelevant skills. Its cases contrast GRSE’s behavior with MASkills on harmful retrieval and abstention rules; the accompanying ablation passage describes variants but supplies no outcomes.

C Robustness Analysis

  • Both methods start from the same three-skill-per-agent seed library: one helpful skill and two harmful or irrelevant skills, with one induction slot left free.The 453 single-hop test questions use greedy test scores; each method is trained once.Source: C Robustness Analysis · pp. 13–14
  • GRSE preserves question–answer expansion to retrieve a dog’s name, while MASkills and the perturbed library stop at the turn that introduces the dog.GRSE merged the harmful focused-latest-session-retrieval skill with a helpful expansion skill; MASkills’s merged protocol retained standard single-keyword retrieval for single-hop facts.Source: C Robustness Analysis · p. 14
  • The verbatim-evidence-guard skill received negative utility under GRSE and was pruned, while MASkills’s utility estimate rose and the method kept refining it.GRSE labeled the skill harmful in seven question groups; its utility was −0.19 after iteration 1 and −0.07 after iteration 2, and pruning occurred in iteration 3.Source: C Robustness Analysis · pp. 14–15

D Ablation Analysis

  • The ablation variants remove structured diagnosis, use one rollout without group contrast, filter to differing-reward groups, or condition credit and updates on question category.The passage defines these variant setups but does not report their results.Source: D Ablation Analysis · p. 15

E Runtime Analysis

GRSE’s grouped executions increase actor-token use, while parallel execution limits the increase in wall-clock training time. The prompt templates specify agent roles, baseline procedures, critic inputs, and skill-update operations.

E Runtime Analysis

  • Training time grows 1.2–1.5×, while GRSE’s overall actor-token overhead is 1.7–3.0× relative to MASkills.The executions in a group run in parallel; validation costs about the same for both methods, reducing overall token overhead from the 3.8–4.3× training-rollout overhead.Source: E Runtime Analysis · p. 15

F Prompt Templates

  • On HotpotQA, Agent 1 sees the question and passages before Agent 2 answers; on LoCoMo, Agent 1 searches with grep and Agent 2 answers from retrieved evidence.Roles remain fixed during training, and each agent’s prompt combines its role with its current skill library.Source: F Prompt Templates · p. 15
  • The single-model baselines use one user message without agents or skills; CoT-SC takes the majority answer across five CoT samples.On HotpotQA, the baselines use the same passages as the agent teams; Self-Refine alternates feedback and revision for up to three rounds.Source: F Prompt Templates · p. 16
  • The group-relative critic receives the question, current skill libraries, and rollouts containing rewards, advantages, agent steps, and recorded skill use.Repeated passages and identical system prompts are replaced by references to keep the payload within the context window.Source: F Prompt Templates · p. 17
  • Skill induction creates a new skill, refinement makes anchored edits, and consolidation merges overlapping skills; LoCoMo uses category-specific operator prompts.Induction and consolidation end with an output skeleton for a well-formed skill file.Source: F Prompt Templates · p. 18

Related papers

Loading 2610.27227v1…