RoleFlow: Self-Evolving Multi-Role Agentic Collaboration via Structural Adaptation

1 Introduction

• Fixed collaboration structures struggle to accommodate evolving task requirements that may call for adding, removing, or reorganizing specialized roles.Existing multi-agent systems often rely on predefined or pre-optimized structures that remain fixed during execution.Source: §1 · p. 1
• Trajectory-level rewards assign the same advantage to decisions within a rollout, making it difficult to identify which structural edits improve or undermine performance.This credit-assignment challenge arises when collaboration structures undergo successive modifications.Source: §1 · p. 1
• RoleFlow continuously adapts agent structures through execution feedback, with a Global Router constructing task-specific role chains, making atomic edits, and synthesizing specialists on demand.The framework also maintains a return-gated role library to retain useful synthesized specialists during training.Source: §1 · p. 2
• SECPO compares executions with and without each candidate edit under the same chain state, assigning counterfactual credit to edit-specific tokens.This localizes learning signals so the system can learn which structural modifications are worth making.Source: §1 · p. 2
• RoleFlow achieves the highest average performance among evaluated multi-agent systems across seven benchmarks spanning four domains.Further analyses validate structural adaptation, edit-level counterfactual learning, cross-benchmark transferability, training-data efficiency, and cross-executor generalization.Source: §1 · p. 2

2 Related Work

• Content-level adaptation through retrieval or tool use operates within fixed agent structures, while those structures do not learn role or interaction changes from execution feedback.Examples include the predefined pipeline in ChatDev (Qian et al., 2024), sequential agents in Chain-of-Agents (Zhang et al., 2024b), and graph-based retrieval in LinearRAG (Zhuang et al., 2025).Source: §2 · p. 2
• Workflow-search methods use Monte Carlo tree search (AFlow) (Zhang et al., 2024a), learned graph connections (GPTSwarm), or offline generation, but determine structure before execution.W4S (Nie et al., 2025), MaAS, and MAS-GPT (Zhang et al., 2025; Ye et al., 2025) are also cited as workflow or architecture-generation approaches.Source: §2 · p. 2
• Online orchestration approaches use execution feedback, but RoleFlow evaluates each structural edit against its no-edit counterpart at the same chain state and assigns credit to edit-related tokens.The passage contrasts this with trajectory-level rewards and with other decision-level counterfactual references.Source: §2 · p. 2

3.1 Problem Formulation

• The Global Router is trained while the executor remains frozen, using terminal rollout reward that does not reveal each edit’s contribution.The rollout reward is based on the first passing cycle, or Tmax if none passes.Source: §3.1 · p. 3

3.2 RoleFlow: Feedback-Driven Structural Adaptation

• After a rejected output, the Global Router uses the deficit and accumulated memory to propose one edit, stopping at validation success or Tmax cycles.Each cycle applies at most one edit, so consecutive chains differ by at most one edit.Source: §3.2 · p. 4
• When a requested role is absent, a LibraryExpander synthesizes it from the role name, task, and current deficit; a null edit despite a deficit also triggers synthesis and addition.At inference, tasks start from the domain’s seed library and synthesize missing specialists rather than loading the training-grown library.Source: §3.2 · p. 4

3.3 SECPO: Edit-Level Counterfactual Credit Assignment

• SECPO compares a candidate edit with the unchanged chain at the same state, using the reward difference to estimate its immediate effect and retaining it only above a positive threshold.The counterfactual pair still contributes to the policy update even when the edit is not committed.Source: §3.3 · p. 4
• SECPO adds the single-edit advantage only to plan tokens spelling the candidate edit, while tokens outside edit spans retain their GRPO advantages.The edit-specific signal is weighted by λ, set to 1 in the described method.Source: §3.3 · pp. 4–5

3.4 On-Demand Role Synthesis and Library Growth

• After each training step, a return gate selectively admits synthesized role specifications to the shared library, while their execution rewards still contribute to Global Router training.At inference, RoleFlow uses the domain’s seed roles and synthesizes missing specialists on demand without loading the training-grown library.Source: §3.4 · p. 5

4.1 Experimental Setup

• Evaluation covers seven benchmarks across multi-hop QA, mathematical reasoning, code generation, and web interaction, with results reported using domain-specific metrics.The setup trains domain-specific routers and evaluates 2Wiki, MuSiQue, and HumanEval without retraining.Source: §4.1 · pp. 5–6

4.2 Main Results

• 67.1 average score across seven benchmarks puts RoleFlow 4.7 points above W4S, the strongest competing multi-agent system, and first on five benchmarks.RoleFlow also improves on its gpt-4o-mini executor across all four task domains.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Ablation Study

• With the Global Router retained, disabling adaptation reduces QA performance by 3.8–8.5 F1, while a fixed HotpotQA chain scores 45.7 F1 versus RoleFlow’s 55.6.Disabling specialist synthesis also reduces QA performance; resulting chains use only one or two seed roles, and edits often become null actions.Source: §4.3 · p. 7, Table 2 · p. 7
• Replacing SECPO with GRPO lowers mean F1 by 2.1, 2.5, and 2.6 points on HotpotQA, 2Wiki, and MuSiQue, respectively.SECPO slightly trails GRPO on MBPP and ties it on HumanEval; multi-edit-per-cycle variants also lower QA performance, though train–test mismatch contributes to that comparison.Source: §4.3 · p. 7

4.4 Structural Adaptation and Transfer

• The HotpotQA-trained router transfers to 2Wiki and MuSiQue without retraining, with final chains diverging more than initial proposals and about a quarter of MuSiQue questions ending with a previously unseen chain.Replacing synthesized-role descriptions with generic ones retains roughly half to three-quarters of their benefit, indicating contributions from both useful chain steps and tailored instructions.Source: §4.4 · p. 7

4.5 Efficiency and Generalization

• On HotpotQA, RoleFlow reaches 55.6 F1 at its measured API cost, while chain-of-thought self-consistency at comparable cost remains near the executor baseline.The comparison concerns this measured setting and does not establish universally lower compute.Source: §4.5 · pp. 7–8, Figure 5 · p. 8
• RoleFlow achieves stronger HotpotQA performance with fewer labeled questions, and RoleFlow† trained on 100 labeled questions matches the GRPO-trained router despite using half as many questions.The data-efficiency result comes with additional training-time executions.Source: §4.5 · p. 8, Figure 5 · p. 8, Table 3 · p. 8
• On MBPP and HumanEval, 8B-Instruct gains 11.7 and 16.0 Pass@1, respectively, compared with 5.3 and 7.4 for gpt-4o-mini, suggesting feedback-driven structural adaptation can compensate for weaker executors.Figure 6 shows RoleFlow performance with different executor backbones on HotpotQA, MBPP, and HumanEval.Source: §4.5 · p. 8, Figure 6 · p. 8

• RoleFlow adapts collaboration structures through feedback-driven edits and on-demand role synthesis, while SECPO enables structure-aware learning through edit-level counterfactual credit assignment.Across seven benchmarks, the system demonstrates superior average performance and strong transferability.Source: §5 · p. 8

Limitations

• Each domain needs a task validator that returns a pass-or-reject verdict and a deficit each cycle; SECPO training also executes one additional chain per edit.A new domain therefore needs an appropriate validator before its Global Router can be trained or run.Source: Limitations · p. 9

A.1 Training Configuration

• The Global Router is trained with SECPO using Qwen2.5-7B-Instruct in verl, while gpt-4o-mini executes roles and the LibraryExpander; reported results average three training runs.The configuration is shared across Global Routers trained on HotpotQA, MATH, and MBPP.Source: §A.1 · p. 11

A.2 Detailed Global Router Learning Procedure

• For each rollout, the learning procedure executes the same chain state with and without a candidate edit, then applies the resulting edit-specific advantage to edit-token spans.Synthesized roles are admitted when their rollout return exceeds the reference rollout’s return.Source: §A.2 · p. 11

A.3 Inference Procedure

• At inference, the router starts from the domain’s seed roles, proposes one edit after each rejected cycle, and stops at the first accepted output or after Tmax adaptation cycles.Missing roles are synthesized for the task; counterfactual re-execution is used during training, not inference.Source: §A.3 · pp. 11–12

A.4 Metrics

• QA scores are token-level F1 between normalized predicted and reference answers.Source: §A.4 · p. 12
• MATH-500 answers are extracted from \boxed{} and normalized; when strings differ, gpt-4o-mini at temperature 0 judges mathematical equivalence.Source: §A.4 · p. 12
• MBPP and HumanEval count a problem as solved only when the submitted program passes every benchmark test; Pass@1 is the share solved.Source: §A.4 · p. 12
• WebShop uses AgentBench’s purchase reward averaged over 500 tasks and multiplied by 100.Source: §A.4 · p. 12

B.1 Benchmark Catalog

• Three listed training sets contain 200 HotpotQA questions, 200 MATH problems, and 324 MBPP problems; 2WikiMultiHopQA, MuSiQue, and HumanEval are out-of-distribution benchmarks.Source: §B.1 · p. 12

B.2 Training, Validation and Test Splits

• Tests use 200 sampled development questions per QA benchmark, all 500 MATH-500 problems, 341 MBPP and 131 HumanEval problems from the AFlow split, and 500 WebShop tasks.Source: §B.2 · p. 12
• Inference settings are chosen on held-out data that overlaps neither training nor testing: 200 HotpotQA development questions, 200 other MATH training problems, 86 MBPP validation problems, and 100 WebShop development goals.Source: §B.2 · pp. 12–13

B.3 Reproduction of Compared Methods

• No method searches, trains, or tunes on 2Wiki, MuSiQue, or HumanEval; each retains its HotpotQA or MBPP configuration, while methods adapt on MATH-500 and WebShop.RoleFlow retains its GR and domain seed roles; AFlow, MaAS, and W4S retain their respective workflows or controller.Source: §B.3 · p. 13

B.4 Direct Prompting

• Seven language models are prompted at temperature 0, each answers every benchmark three times, and Table 1 reports the mean using shared multi-hop QA prompts.Qwen3 models use non-thinking mode; the WebShop prompt and one-shot example follow AgentBench unchanged.Source: §B.4 · p. 13

C Role Library and Prompt Templates

• RoleFlow starts each task with domain seed roles, while its LibraryExpander synthesizes Global Router-named specialists beyond the seed set.The seed libraries contain four roles for multi-hop QA, ten for MATH-500, eight for coding, and eight for WebShop.Source: C Role Library and Prompt Templates · p. 14

C.1 Seed Roles and Specialists

• The QA role library reaches 47.7 roles on average, and code specialists target validator-reported failures such as timeouts or wrong public-test results.The QA plan prompt names eight specialists in addition to four seed roles.Source: §C.1 · p. 14

C.2 Prompt Templates

• A missing Global Router-named role is synthesized from its name and request context, while deficit-driven synthesis uses validator feedback and the existing library; MATH-500 validation independently re-derives the answer.For coding tasks, after rejection, roles other than the OutputFormatter receive the repair note.Source: §C.2 · pp. 14, 16, 17
• Sampled programs are ranked first by public-example tests passed, then by written tests passed, with each written test weighted by how many candidates pass it.The executor writes the additional tests once per problem from the task statement.Source: §C.2 · p. 17
• The test prompt requests confident, hand-worked expected results for typical inputs and well-defined edge cases, without repeating examples or writing the implementation.It asks for single-line Python assert statements based only on the task description and example test.Source: §C.2 · p. 18

Case 1: a role synthesized for a deficit

• 10080 is accepted after deficit-driven synthesis adds a RotationEquivalenceEvaluator, correcting a rejected 5040 answer for equivalent rotations.Twelve of the eighteen compared methods give a wrong answer on this MATH-500 problem.Source: D Chain Evolution Case Studies · p. 18, Figure 7 · p. 19

Case 3: adaptive routing with the GR

• 9.9 F1 on HotpotQA and 13.9 and 17.1 on 2Wiki and MuSiQue are lost when the GR is removed; in the case study, its added Reasoner yields the correct answer.Without the GR, rerunning the initial roles produces an accepted but still incorrect answer.Source: D Chain Evolution Case Studies · p. 18, Figure 9 · p. 20

Case 2: a RoleFlowChain against a direct answer

• On the HotpotQA case, the Validator accepts the cycle-0 answer and no edit follows, while direct prompting gives an incorrect answer.The illustrated trajectory is from an earlier GR checkpoint and is not reported in Table 1.Source: Figure 8 · p. 19

Case 4: the chain before and after training

• After training, the GR composes Analyst, Coder, and Debugger to produce a correct repository patch, unlike the untrained chain’s incorrect patch.The Debugger compiles and runs the code and corrects the format before validation.Source: D Chain Evolution Case Studies · p. 20, Figure 10 · p. 21

Case 5: with and without SECPO

• 2.1 to 2.6 F1 above the GR trained without SECPO, the GR trained with SECPO continues editing after rejection and produces a corrected coding patch.Without SECPO, the same cycle-0 chain stops with an incorrect patch.Source: D Chain Evolution Case Studies · p. 20, Figure 11 · p. 21