PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning

Wei Yang, Defu Cao, Jiacheng Pang, Muyan Weng, Yan Liu

arXiv:2603.07972v1cs.AI

TL;DR

Autonomous multi-agent systems remain constrained by pretrained knowledge, leaving tasks that require expertise beyond training as a challenge. HILA uses a metacognitive policy and dual-loop optimization to guide deferral and learn from expert feedback; experiments report consistent outperformance of strong autonomous and multi-agent baselines.

  • Problem

    Purely autonomous multi-agent systems are bounded by pretrained knowledge, limiting their ability to handle tasks requiring expertise or reasoning beyond training.

    Source: §1 · p. 1

  • Method

    HILA trains a metacognitive policy for autonomous action or expert deferral, with DLPO combining cost-aware GRPO and continual learning from expert feedback.

    Source: §1 · p. 2

  • Results

    HILA consistently outperforms strong autonomous and multi-agent baselines across diverse and challenging reasoning benchmarks.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The paper presents HILA as a framework for adaptive decision-making and long-term capability improvement in multi-agent collaboration.

    Source: §5 · p. 10

Abstract

from arXiv · show

While scaling individual Large Language Models (LLMs) has delivered remarkable progress, the next frontier lies in scaling collaboration through multi-agent systems (MAS). However, purely autonomous MAS remain ''closed-world'' systems, constrained by the static knowledge horizon of pre-trained models. This limitation makes them brittle on tasks requiring knowledge beyond training data, often leading to collective failure under novel challenges. To address this, we propose the Human-In-the-Loop Multi-Agent Collaboration (HILA) framework, a principled paradigm for human--agent collaboration. HILA trains agents to learn a metacognitive policy that governs when to solve problems autonomously and when to defer to a human expert. To operationalize this policy, we introduce Dual-Loop Policy Optimization, which disentangles immediate decision-making from long-term capability growth. The inner loop applies Group Relative Policy Optimization (GRPO) with a cost-aware reward to optimize deferral decisions, while the outer loop implements continual learning, transforming expert feedback into high-quality supervised signals that strengthen the agent's reasoning ability. Experiments on challenging mathematical and problem-solving benchmarks show that HILA, equipped with Dual-Loop Policy Optimization, consistently outperforms advanced MAS, establishing a principled foundation for collaborative and continually improving agentic systems.

1 INTRODUCTION

Autonomous multi-agent systems remain bounded by pretrained knowledge, while existing human-in-the-loop approaches leave strategic deferral and lasting learning unresolved. HILA and Dual-Loop Policy Optimization address these gaps, and experiments report stronger performance than autonomous multi-agent systems.

1 INTRODUCTION

  • Purely autonomous multi-agent systems remain bounded by pretraining and can fail on tasks requiring knowledge or reasoning absent from training.The paper identifies real-time information, domain-specific expertise, and unseen contexts as examples of such demands.Source: §1 · p. 1
  • Prior human-in-the-loop approaches leave open how to learn deferral decisions and turn expert input into lasting capability growth.Some use confidence-threshold heuristics, while feedback is often treated as a one-time fix.Source: §1 · p. 2
  • HILA equips agents with a metacognitive policy to assess competence and strategically decide when to seek external expertise.Its process includes autonomous operation, metacognitive assessment, and strategic deferral.Source: §1 · p. 2
  • DLPO separates short-term deferral decisions from long-term capability growth through an inner policy-optimization loop and an outer continual-learning loop.The inner loop uses GRPO with a cost-aware reward; the outer loop uses expert feedback as supervised signals.Source: §1 · p. 2
  • HILA with DLPO outperforms autonomous multi-agent systems on mathematical reasoning and general problem-solving benchmarks.The paper presents this as evidence for a foundation for continually improving agentic collaboration.Source: §1 · p. 2

2 RELATED WORK

Related work spans autonomous coordination methods and human-in-the-loop collaboration. The paper identifies persistent limits in adaptive coordination, strategic deferral, and using feedback for sustained capability growth.

2 RELATED WORK

  • LLMs acting alone have limits in context length, sequential generation, and skill coverage for complex reasoning tasks.The related-work discussion situates multi-agent systems as one response to these limits.Source: §2 · p. 2
  • Multi-agent research has moved from fixed role and workflow prompts toward structured coordination and dynamic routing, pruning, or workflow search.These approaches include debate, peer review, and adaptive interaction structures.Source: §2 · p. 3
  • Human-in-the-loop systems use people as supervisors, oracles, or evaluators, but often rely on heuristic deferral and one-time feedback.The paper distinguishes local corrections from broader changes to collaborative dynamics.Source: §2 · p. 3

3 METHODOLOGY

HILA models human–agent collaboration as metacognitive decision-making over whether to act autonomously or defer to an expert. Its Dual-Loop Policy Optimization combines policy learning for deferral decisions with supervised learning from expert feedback.

3.1 PRELIMINARIES: THE METACOGNITIVE MARKOV DECISION PROCESS

  • The Meta-MDP formalizes high-level decisions such as autonomous problem-solving or deferral to human expertise.It provides the basis for defining states, actions, transitions, and rewards.Source: §3.1 · p. 4

3.2.1 STRUCTURED COGNITIVE STATE SPACE

  • The cognitive state combines task context, the agent’s latest solution and reasoning status, and peer responses that indicate agreement or alternative paths.Optional structured cues summarize social consensus, solution reliability, and whether further deliberation or escalation is warranted; these are computed with parsing- and rule-based heuristics.Source: §3.2.1 · p. 4

3.2.2 THE STRATEGIC ACTION SPACE

  • The action space comprises high-level strategies for exploiting existing solutions, generating a novel solution, or deferring to a human expert.Evaluation selects a peer solution, creation generates a new solution, and deferral invokes the expert for a high-quality demonstration.Source: §3.2.2 · pp. 4–5

3.2.3 COLLABORATIVE INTERACTION MODEL

  • In each round, agents independently choose actions in parallel, and their outputs form the next shared state; deferral uses the expert’s solution.Deferral also supplies expert demonstrations for continual learning.Source: §3.2.3 · p. 5

3.3 ADAPTIVE POLICY OPTIMIZATION WITH CONTINUAL LEARNING

  • DLPO balances autonomous problem-solving against expert deferral by combining reinforcement learning for policy optimization with supervised learning for continual knowledge acquisition.The framework is designed to optimize strategic decisions while integrating expert feedback into capability growth.Source: §3 · p. 3, §3.3 · p. 5

3.3.1 INNER LOOP: REINFORCEMENT LEARNING FOR METACOGNITIVE POLICY

  • The inner loop uses GRPO to optimize strategic actions from rewards that combine task correctness with action-dependent costs.Deferral and internal regeneration incur tunable penalties, with deferral costing more; correctness remains primary, while costs favor lower-cost actions when outcomes are similarly good.Source: §3.3.1 · pp. 5–6

3.3.2 OUTER LOOP: CONTINUAL LEARNING FROM EXPERT FEEDBACK

  • The outer loop converts expert demonstrations received after deferral into supervised fine-tuning samples, using them to expand capabilities beyond reinforcement learning on existing abilities.The inner loop determines when to defer, while the outer loop teaches what to learn from expert input.Source: §3.3.2 · p. 6

3.3.3 THE FINAL DUAL-LOOP POLICY OPTIMIZATION OBJECTIVE

  • The jointly optimized objective combines the inner-loop GRPO loss with conditional supervised loss, applied only when the agent defers.A hyperparameter balances the learning signals.Source: §3.3.3 · p. 6

4 EXPERIMENTS

HILA with DLPO outperforms strong autonomous multi-agent baselines across reported benchmarks, with gains also transferring across backbones and inference frameworks. Further analyses show contributions from expert quality, continual learning, and collaboration configuration.

Overall Performance

  • HILA with DLPO achieves the best overall performance, beating strong autonomous multi-agent baselines by 3.7–15.4 points on the Llama3-8B benchmarks.The gains are particularly pronounced on competition-style math benchmarks such as AMC and AIME.Source: §4 · p. 7

Cross-Backbone Generalization

  • HILA achieves the best GSM8K performance across four backbones, with especially pronounced gains on smaller or weaker models.The table reports percentages and absolute percentage-point differences relative to each backbone’s Vanilla baseline.Source: §4 · p. 7, Table 2 · p. 8

Beyond Better Intervention Decisions

  • Full DLPO improves performance across benchmarks beyond GRPO alone, while the DLPO-updated backbone also improves unchanged inference pipelines.The transfer gains include methods that do not use strategic deferral at inference time.Source: §4 · p. 8, Table 4 · p. 8, Table 3 · p. 8

Policy Distribution Across Training Stages

  • Across training stages, DEFER decreases and EVAL becomes more frequent, with full DLPO reducing reliance on external help further than GRPO alone.The paper attributes GRPO’s change to more cost-aware intervention and suggests expert demonstrations strengthen underlying reasoning during DLPO.Source: §4 · pp. 8–9, Table 5 · p. 9

Effect of Human Proxy Capability

  • Stronger external proxies consistently improve HILA’s performance across GSM8K, AMC, and MMLU, with gpt-4o performing best.The reported proxy ordering is gpt-3.5-turbo, then gpt-4o-mini, then gpt-4o.Source: §4 · p. 9, Figure 2 · p. 9

The Value of Collective Exploration

  • More agents generally improve accuracy on AMC and MMLU, but gains diminish and become less stable as inference costs rise.The paper describes the trade-off as broader collective exploration against increased token use.Source: §4 · pp. 9–10

Optimal Depth of Iterative Collaboration

  • Increasing collaborative rounds from shallow to moderate depth improves accuracy, but performance peaks at an intermediate depth and then declines slightly as token use continues rising.The reported pattern is non-monotonic across AMC and MMLU.Source: §4 · p. 10

5 CONCLUSION

The paper concludes that HILA combines metacognitive decisions about autonomous action or deferral with continual learning from expert demonstrations, and reports consistent outperformance of strong autonomous and multi-agent baselines.

5 CONCLUSION

  • HILA combines GRPO-based intervention control with continual learning from expert demonstrations and consistently outperforms strong autonomous and multi-agent baselines across diverse reasoning benchmarks.The authors identify more dynamic collaboration and stronger evolutionary capabilities as future work.Source: §5 · p. 10

A.1 COLLABORATION PARADIGMS IN MULTI-AGENT LLM SYSTEMS.

Prior multi-agent LLM systems coordinate existing knowledge through fixed or adaptive internal structures, but the cited work frames learning from external expertise as an unresolved gap. This paper positions HILA as an iterative framework for strategically deferring to and assimilating human guidance.

A.1 MULTI-AGENT COLLABORATION PARADIGMS

  • Prompt-based MAS can improve coordination through debate, critique, and workflows, but their hand-crafted protocols remain closed-world and recombine existing knowledge.The passage cites Du et al. (2023), Chan et al. (2023), Cao et al. (2025), Mukobi et al. (2023), Wang et al. (2023a), Abdelnabi et al. (2024), and Han et al. (2025) for coordination gains.Source: §A.1 · p. 17
  • Multi-agent reasoning research spans fixed debate or peer-review topologies and self-organizing approaches that dynamically adapt interaction graphs.The cited examples include chains, trees, graphs, search, pruning, routing, and evolutionary mechanisms.Source: §A.1 · p. 17
  • These collaboration approaches optimize internal coordination but leave learning from external expertise unaddressed.The paper distinguishes integrating collective knowledge from acquiring new capabilities.Source: §A.1 · p. 17
  • HILA uses centralized iterative collaboration to govern both when to defer and how to assimilate human guidance into lasting improvements.The paper contrasts this with treating human feedback as a passive oracle or one-time correction.Source: §A.1 · p. 17

A.2 MULTI-AGENT REINFORCEMENT LEARNING FOR LLMS.

Prior adaptive MAS work combines supervised demonstrations and reinforcement learning, with research spanning structured controllers, online coordination, and language-space policy learning. The paper identifies fixed knowledge boundaries in much RL work and proposes a dual-loop approach linking deferral decisions to continual learning from expert demonstrations.

A.2 MULTI-AGENT REINFORCEMENT LEARNING FOR LLMS

  • SFT can seed cooperative behavior from demonstrations, while RL complements it with trial-and-error policy refinement.The passage characterizes SFT as offline and unable to adapt to novel contexts.Source: §A.2 · p. 17
  • RL research includes structured controllers optimized over graphs, code, or plans, as well as online task decomposition, routing, and role assignment.These are presented as approaches beyond static interaction protocols.Source: §A.2 · p. 17
  • Other work learns reasoning policies directly in language space using GRPO- or PPO-style updates, sometimes with tools or human input.The cited discussion also highlights credit assignment and reward shaping for language action spaces.Source: §A.2 · pp. 17–18
  • The paper identifies a gap in RL approaches that optimize coordination while leaving knowledge boundaries fixed, and proposes RL for deferral plus expert demonstrations for continual learning.The dual-loop perspective targets immediate decision-making and long-term capability growth.Source: §A.2 · p. 18

B.1 PRELIMINARIES: THE METACOGNITIVE MARKOV DECISION PROCESS

The Meta-Cognitive Markov Decision Process models human–agent collaboration as sequential decisions over high-level cognitive strategies. Its policy balances autonomous problem-solving with strategic reliance on expert guidance.

B.1 PRELIMINARIES: THE METACOGNITIVE MARKOV DECISION PROCESS

  • The Meta-MDP represents collaboration with states, actions, transitions, rewards, and a discount factor.Its state includes problem context and assessments of the agent’s and peers’ understanding; actions include solving autonomously or deferring to an expert.Source: §B.1 · p. 18
  • At each round, an agent selects a metacognitive action from its current state, transitions to a new state, and receives a reward.The reward is designed to incentivize task success and efficient use of the expert resource.Source: §B.1 · p. 18
  • The objective is to learn a policy that maximizes cumulative reward while balancing autonomous problem-solving with strategic human guidance.Source: §B.1 · p. 18

C.1 EXPERIMENTAL SETTINGS

The experiments evaluate HILA across knowledge, code-generation, and mathematics benchmarks using controlled baselines and evaluation protocols. Training combines task data, action-level policy comparisons, and expert-labeled reasoning traces.

C.1 EXPERIMENTAL SETTINGS

  • The benchmark suite covers general knowledge and analytical reasoning, program synthesis, and mathematical problem solving.It includes MMLU, HumanEval, GSM8K, AIME, and AMC; MMLU is sampled at 10 examples per subject, and math performance uses exact-match Solve Rate.Source: §C.1 · p. 18
  • Baselines span single-agent solvers, interactive multi-agent deliberation, and system-level coordination frameworks.Backbone, prompting, and generation budgets are controlled; canonical candidate reduction is used, with no retrieval or external tools.Source: §C.1 · pp. 18–19
  • Main experiments use three agents over three rounds with open-source Qwen and Llama backbones, LoRA fine-tuning, and three-seed averages for most results.The system uses 8-bit quantization and key–value caching; the external GPT proxy uses temperature 0.3.Source: §C.1 · p. 19
  • Training uses GSM8K’s original training split and GPT-5-generated synthetic examples for the other benchmarks, totaling approximately 10K instances.Offline trajectories enumerate valid strategic actions and convert their outcomes into grouped comparisons for GRPO.Source: §C.1 · pp. 19–20
  • The outer loop uses roughly 2K expert-labeled reasoning traces for supervised fine-tuning, alongside roughly 8K grouped samples for GRPO.When an agent defers, the expert’s correct reasoning trace and final answer become a supervision instance.Source: §C.1 · p. 20
  • A real person is the intended expert, but experiments use LLM proxies whose strength varies with the autonomous backbone.GPT-4o-mini is used for LLaMA-based agents and GPT-4o for Qwen-based agents.Source: §C.1 · p. 20

C.2 EXPERIMENTAL RESULTS

Across training stages, full DLPO generally couples higher accuracy with lower deferral, while additional experiments examine proxy quality, deferral cost, collective size, and interaction depth. These results also identify trade-offs in external guidance, inference cost, and extended interaction.

Accuracy–Deferral Coupling Across Training Stages

  • DLPO generally increases accuracy while reducing deferral across GSM8K, AMC, and MMLU, unlike GRPO’s more modest or task-dependent shifts.On AMC, DLPO improves accuracy and further lowers deferral after GRPO’s mild accuracy trade-off; on MMLU, accuracy rises as deferral falls.Source: §C.2 · p. 22, Figure 5 · p. 23, Figure 6 · p. 23

Effect of Human Proxy Capability

  • Stronger human proxies improve final performance across GSM8K, AMC, and MMLU, making guidance quality consequential alongside strategic deferral.The reported trend is consistent across the evaluated benchmarks.Source: §C.2 · pp. 22–23, Table 6 · p. 24

Deferral Cost as a Control Knob for Collaboration

  • Increasing deferral cost shifts the policy away from DEFER toward autonomous actions, especially EVAL, but a large cost may hurt accuracy.The trade-off is particularly visible on the more difficult benchmark, where discouraging intervention can prevent access to useful guidance.Source: §C.2 · p. 23, Figure 7 · p. 24

The Value of Collective Exploration

  • Larger collectives generally improve accuracy, especially from very small sizes, but gains become less stable as token usage rises sharply.The reported tables cover AMC and MMLU; the authors attribute the initial gains to broader collective exploration.Source: §C.2 · pp. 23–24, Table 7 · p. 25, Table 8 · p. 25

Optimal Depth of Iterative Collaboration

  • Moderate interaction depth improves accuracy, but performance peaks and then declines slightly as further rounds increase token use.The authors note that excessive exchanges may propagate noisy intermediate reasoning and increase error accumulation.Source: §C.2 · pp. 24–25

C.2.1 REAL HUMAN-IN-THE-LOOP

The human-in-the-loop study evaluates real experts as reactive helpers and proactive guides, finding that full upfront reasoning yields the strongest participation-mode results while human input also changes how HILA allocates collaboration actions.

C.2.1 REAL HUMAN-IN-THE-LOOP

  • The human evaluation uses 20 PhD annotators working offline, with balanced pre-intervention agreement categories for GSM8K and MMLU and the full AMC set.GPT-4o-mini and GPT-4o served as human-expert proxies, GPT-5 generated training data and assisted manuscript editing, and the authors report responsibility for the final content.Source: §C.2.1 · pp. 25–26, D DECLARATION ON THE USE OF LARGE… · p. 30
  • The study distinguishes reactive help after a DEFER decision from proactive input at initialization, which can be either a concise idea or full reasoning.The proactive and reactive roles can be combined, with GPT-based help still available after upfront human guidance.Source: §C.2.1 · p. 26, Figure 8 · p. 27
  • Real human experts perform best overall as standalone experts and as HILA’s reactive experts, with particularly large gains on AMC.The evaluation used a pool of 20 PhD participants; the reported margin on MMLU is smaller.Source: §C.2.1 · p. 27, Table 11 · p. 28, Table 12 · p. 28
  • Full human reasoning at initialization produces the strongest results among participation settings across GSM8K, AMC, and MMLU, while high-level ideas yield mixed benefits.The comparison keeps LLaMA3-8B fixed and GPT-4o available for later deferrals.Source: §C.2.1 · p. 28, Table 13 · p. 29
  • With reactive human experts, DEFER is the most frequent action on GSM8K, while CREATE retains nontrivial probability across the human-expert condition.Figure 10 summarizes action distributions across GSM8K, AMC, and MMLU.Source: §C.2.1 · p. 29, Figure 10 · p. 29
  • Providing only a proactive human idea increases DEFER across all three benchmarks relative to reactive-human-only participation.GPT-4o remains available for later DEFER decisions in the proactive-idea condition.Source: §C.2.1 · p. 29, Figure 11 · p. 30
  • Compared with proactive ideas, full proactive reasoning decreases DEFER on GSM8K and AMC, while CREATE generally becomes more prominent.The reported action distributions show that stronger upfront guidance changes subsequent collaboration behavior.Source: §C.2.1 · pp. 29–30, Figure 12 · p. 30
Loading 2603.07972v1…