PaperlayerView exact revision on arXiv

Source-linked AI summary

Harness-RL: Black-Box Reinforcement Learning with Action-Args Decoupling for Central-Agent Multi-Agent Harnesses

Xinke Jiang, Zhixin Zhang, Zhibang Yang, Jiaran Gao, Rihong Qiu, Shijin Chen, Xu Chu, Junfeng Zhao, Yasha Wang

arXiv:2608.29641v1cs.MA

TL;DR

Existing agentic RL methods generally assume unified sequence-level optimization and linear trajectory reconstruction, assumptions that fail for heterogeneous decisions and dynamic multi-agent harnesses. Harness-RL combines CAPO with interface-level black-box trajectory construction to align gradients, contexts, and rewards. Across seven benchmarks, it achieves average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, while ablations validate CAPO and favor central-only training in the evaluated setting.

  • Problem

    Agentic RL methods generally assume that Action and Args tokens share one sequence-level objective and that harness interactions can be represented as one linear token sequence.

  • Method

    Harness-RL combines CAPO, which routes Action and Args gradients using activation-based parameter partitions, with black-box trajectory construction using interface records, prefix trees, and token-aligned rewards.

  • Results

    42.93 average F1 with Qwen2.5-1.5B and 47.79 with Qwen2.5-3B were achieved across seven benchmarks, outperforming the strongest non-ours rows by 15.28 and 14.24 points, respectively.

  • Takeaways & Limitations

    Ablations show that CAPO contributes to performance, while central-only training outperforms joint training in the evaluated 3B harness.

  • Takeaways & Limitations

    The evaluation uses exclusively publicly available benchmark datasets and does not include human or animal subjects or intentionally collected personally identifiable information.

Abstract

from arXiv · show

Large language model agents increasingly solve long-horizon tasks through multi-agent harnesses in which a central agent coordinates specialized sub-agents, tools, and environments. Training the central policy in such a harness raises two challenges. First, an action label is a low-cardinality decision, whereas its args form a high-dimensional conditional sequence; optimizing both with a shared sequence-level signal can produce conflicting gradients. Second, dynamic scheduling creates interdependent sessions with branches, parallel calls, and rewritten contexts, which cannot be faithfully reduced to one flat token sequence. We introduce Harness-RL, a structured reinforcement learning framework that combines Conflict-Aware Policy Optimization (CAPO) with interface-level black-box trajectory construction. The black-box component captures Interface Call Records, builds per-session prefix trees, and aligns outcome and process rewards with trainable tokens. CAPO uses forward activations to identify parameter partitions associated with action and args tokens, then routes their policy gradients to the corresponding subspaces. Harness-RL supports both central-only and joint multi-agent training. Across seven multi-hop question answering and agentic retrieval benchmarks, it reaches average F1 scores of 42.93 and 47.79 with Qwen2.5-1.5B and Qwen2.5-3B, respectively, while ablations validate the contribution of CAPO and favor central-only optimization in the evaluated setting. Our code is available at https://github.com/jiangxinke/Harness-RL.

1 Introduction

Harness-RL addresses conflicting optimization of Action and Args decisions and the difficulty of representing dynamic multi-agent interactions as trainable trajectories. It combines CAPO with black-box trajectory construction and improves benchmark performance while ablations support CAPO and the evaluated training-scope choice.

  • Challenges: Unified sequence-level optimization can let poor Args updates interfere with correct Action decisions, producing unstable planning and inadequate args generation.Actions are low-cardinality decisions, whereas Args are high-flexibility sequences conditioned on the Action.
  • Challenges: Dynamic scheduling with branches, loops, concurrency, backtracking, and context compression produces related interaction trees that cannot be faithfully concatenated into one sequence.Agent contexts must remain isolated while preserving dependencies from dispatch, messaging, and execution state.
  • Harness-RL: CAPO uses forward activations to construct Action- and Args-related parameter partitions and routes their policy gradients to corresponding subspaces.The framework combines this optimization component with a black-box mechanism that converts harness interactions into context-aligned training trajectories with attributable rewards.
  • Harness-RL: The black-box RL mechanism reconstructs long-horizon, multi-round interactions among agents, tools, and environments as structured trajectories for central-only or joint multi-agent training.It aligns outcome and process rewards with trainable decisions.
  • Results: Ablations validate CAPO and the choice of training scope.The introduction reports improvements over strong baselines and identifies ablations as supporting the framework components and training-scope selection.

2 Preliminaries

A multi-agent harness coordinates a central agent and specialized sub-agents through an external execution layer. Its rollouts are long-horizon, heterogeneous, and dynamically conditioned, creating both optimization and sequence-construction challenges.

  • 2.1 Central-Agent Multi-Agent Harnesses: The policy layer contains a central agent and specialized sub-agents, with each sub-agent executing assigned tasks in an isolated context.Sub-agents may invoke tools, search engines, or the task environment and return results as observations to the harness.
  • 2.1 Central-Agent Multi-Agent Harnesses: The harness manages messages, workflow control, dispatch, tool invocation, state updates, and termination across multiple interaction rounds.The complete coordinated interaction is called a harness rollout.
  • 2.1 Central-Agent Multi-Agent Harnesses: Each interaction step schedules a role, conditions its action on visible state, and returns an observation such as a sub-agent response, tool result, or environment feedback.Local contexts may remain isolated while dispatch, messaging, and execution state connect roles at the rollout level.
  • 2.1 Central-Agent Multi-Agent Harnesses: The central agent plans, revises plans, dispatches sub-tasks, and maintains global state through structured decisions that include actions and Args sequences.REASON supports planning, DISPATCH specifies sub-agent tasks and arguments, and SUMMARY consolidates intermediate results.
  • 2.2 Policy Optimization in a Multi-Agent Harness: Central-only training updates the central policy, whereas joint training may also update trainable sub-agent policies.The trainable-role set determines which policy parameters receive optimization.
  • 2.2 Policy Optimization in a Multi-Agent Harness: Harness rollouts are long-horizon, combine heterogeneous decisions, and continually condition on dynamic observations, so multiple calls and role switches do not naturally form one trainable token sequence.Standard GRPO or DAPO typically broadcasts one rollout advantage across trainable response tokens, coupling action selection with Args generation.

3 HARNESS-RL

HARNESS-RL combines CAPO with interface-level black-box reinforcement learning to optimize heterogeneous Action–Args decisions and construct structured trajectories for multi-agent harness interactions. Its pipeline captures call records, builds session prefix trees, and aligns rewards with trainable tokens.

  • 3.1 Framework Overview: HARNESS-RL combines CAPO with interface-level black-box RL to decouple Action and Args optimization while modeling multi-agent harness trajectories.CAPO routes gradients to functional parameter subsets, while the black-box mechanism converts harness interactions into context-aligned training data.
  • 3.2 Conflict-Aware Policy Optimization: Action tokens select operations, whereas Args tokens specify executable content, and their policy gradients can conflict.Applying both gradients across the full parameter space may impair action selection, args quality, or both.
  • 3.2 Conflict-Aware Policy Optimization: CAPO samples successful rollouts, measures decision-specific activation responses, selects responsive units, and routes Action and Args gradients through separate masks.Units selected for both decisions are treated as shared according to the implementation’s overlap rule.
  • 3.3 Interface-Level Black-Box RL: Interface Call Records store exact input and output tokens with aligned sampling log-probabilities and connect captured calls to later harness contexts.Harness feedback is written into subsequent message contexts through execution-state transitions.
  • 3.3 Interface-Level Black-Box RL: Session calls form prefix trees whose shared prefixes merge, while rewritten contexts attach at the longest shared prefix and preserve original branches.The resulting tree collection represents token visibility rather than causal execution, with output-span nodes defining base samples.
  • 3.3 Interface-Level Black-Box RL: Outcome and process rewards are normalized by rollout groups or matching decision units and aligned to trainable tokens in the structured dataset.Per-rollout normalization prevents branch-rich rollouts from dominating the update, after which Action and Args positions form separate objective terms.

4 Experiments

Experiments evaluate HARNESS-RL across seven multi-hop QA and agentic retrieval benchmarks, then analyze CAPO, training scope, and functional partitions. The method achieves strong cross-benchmark performance, while ablations support CAPO and central-only training in the evaluated setting.

  • Main Results: HARNESS-RL achieves the best result on all seven benchmarks with both Qwen2.5-1.5B and Qwen2.5-3B.With Qwen2.5-1.5B, average F1 is 42.93; with Qwen2.5-3B, average F1 is 47.79.
  • Main Results: 42.93 average F1 with Qwen2.5-1.5B exceeds AgenticRAG-R1 by 15.28 points and ReSearch by 18.55 points.The model scores 41.63, 44.44, 28.63, 37.71, 27.83, 62.04, and 58.22 across the seven benchmarks.
  • Main Results: 47.79 average F1 with Qwen2.5-3B exceeds AgenticRAG-R1 by 14.24 points and again ranks first across all seven benchmarks.It also exceeds Search-R1 and ARPO by 17.63 and 18.19 points, respectively.
  • Component Analysis: CAPO raises average F1 from 41.04 to 42.93 for Qwen2.5-1.5B and from 46.22 to 47.79 for Qwen2.5-3B.The gains are 1.89 and 1.57 points, although effects vary across datasets.
  • Component Analysis: Action and Args tokens show different layer-wise importance distributions, producing distinct activation-based functional partitions.Figure 3 presents these partitions for the 28-layer Qwen2.5-1.5B and 36-layer Qwen2.5-3B models.
  • Component Analysis: Central-only training outperforms joint training on Qwen2.5-3B, with average F1 scores of 47.79 and 45.36, respectively.The authors associate this result with preserving consistency of the global planning objective and reducing interference among policy updates.

5 Conclusion

The conclusion presents HARNESS-RL as a framework that reconstructs structured multi-agent interactions and decouples Action and Args optimization. Across seven benchmarks, it achieves strong average F1, while ablations support CAPO and central-only training in the evaluated harness.

  • Conclusion: HARNESS-RL reconstructs multi-round, multi-session, and branching interactions as aligned training samples with outcome and decision-level process rewards.Its interface-level black-box mechanism is paired with CAPO for action- and args-related gradient routing.
  • Conclusion: CAPO identifies action- and args-related parameter subsets from forward activations and routes their policy gradients accordingly.This complements the structured trajectory representation used for central-agent policies.
  • Conclusion: 42.93 and 47.79 average F1 are obtained with Qwen2.5-1.5B and Qwen2.5-3B, outperforming the strongest non-ours row by 15.28 and 14.24 points.Removing CAPO reduces average F1 by 1.89 and 1.57 points.
  • Conclusion: Central-only training outperforms joint training in the evaluated 3B harness.The conclusion also identifies larger policies, diverse harness topologies, dynamic partitions, multimodal actions, memory, robustness, and computational cost as future work.

Ethical considerations

The experiments use publicly available benchmark datasets under their licenses and usage policies, without intentionally collecting personally identifiable information or involving human or animal subjects.

  • Ethical considerations: The study evaluates only publicly available datasets, including seven named QA and retrieval benchmarks, under their respective licenses and usage policies.The listed datasets are 2WikiMultiHopQA, HotpotQA, Bamboogle, FRAMES, MuSiQue, Natural Questions, and TriviaQA.
  • Ethical considerations: The authors report no intentional use of personally identifiable information and no involvement of human or animal subjects.

Appendix

The appendix situates HARNESS-RL within research on multi-agent systems and reinforcement learning, emphasizing centralized coordination and the distinct optimization needs of Action and Args tokens. It also connects functional specialization to the proposed partitioning approach.

  • LLM-Based Multi-Agent Systems: LLM-based multi-agent systems decompose complex problems into subtasks assigned to specialized roles, using explicit role assignment and structured communication.Examples include AutoGen, MetaGPT, and ChatDev.
  • LLM-Based Multi-Agent Systems: Centralized coordination uses a planner or coordinator for global planning, subtask dispatch, and cross-agent state management.The related systems include CoAct, MasRouter, MacNet, OWL, AFlow, and STACKPLANNER.
  • Reinforcement Learning for Agents: Existing RL methods largely target single-agent settings or homogeneous action spaces, treating sequence tokens as having similar optimization objectives.The appendix contrasts this with central-agent outputs containing Action Tokens and Args Tokens.
  • Reinforcement Learning for Agents: Action Tokens represent high-level meta-action selection, whereas Args Tokens represent concrete argument generation, and their gradient statistics can differ substantially.The shared policy objective is described as potentially causing gradient conflicts and policy degradation.
  • Functional Specialization: Prior work on internal functional specialization motivates analyzing specialized responses in attention heads, feed-forward layers, and neurons.Model Shapley is cited as a parameter-importance basis for functional-partition analysis.

B Training Configuration

Table 4 presents the training configuration for HARNESS-RL’s central-capability runs.

  • Table 4 reports the training configuration for the central-capability runs.
  • The passage identifies the reported settings as belonging specifically to central-capability runs.

C Computational Resources and Software Environment

Experiments used a high-capacity server with A100 GPUs, with four GPUs for Qwen2.5-1.5B-Instruct and eight for Qwen2.5-3B-Instruct. The software environment was standardized across experiments.

  • The server provided 96 physical cores, 192 hardware threads, and 503 GB of system memory.
  • HARNESS-RL training used NVIDIA A100-SXM4 GPUs with 80 GB per GPU.
  • Four GPUs supported Qwen2.5-1.5B-Instruct, while eight supported Qwen2.5-3B-Instruct.
  • The experiments ran on Ubuntu 22.04.5 LTS with Python 3.11.11, Conda 23.5.2, PyTorch 2.6.0, Transformers 4.51.3, and spaCy 3.8.4.
  • Unless otherwise specified, all experiments used the same hardware and software environment.

D Detailed Case Study

The case study shows HARNESS-RL separating dispatch actions from their arguments during iterative retrieval. The central agent preserves the action while refining its arguments to resolve an ambiguous temporal clue and produce the correct answer.

  • HARNESS-RL separates high-level dispatch decisions from their args and refines the args using sub-agent feedback.
  • Across two retrieval rounds, the central agent preserves the appropriate DISPATCH action while changing its args.
  • The args shift from broad entity identification to precise timeline verification.
  • This behavior resolves an ambiguous temporal clue before the agent returns the correct answer.

E Use of Large Language Models

The authors used large language models only for auxiliary language refinement and programming-related assistance. They reviewed and validated all model-assisted outputs and retained responsibility for the research and its conclusions.

  • Large language models were used only as auxiliary tools for language refinement and programming-related assistance.
  • The assistance targeted grammatical accuracy, readability, presentation, and general coding or debugging suggestions.
  • The authors reviewed and validated all model-assisted outputs before inclusion.
  • The authors remained responsible for research conception, methodological design, experimental execution, interpretation, and final conclusions.
Loading 2608.29641v1…