PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Steer LLM Agents and Multi-Agent Systems

Leyang Shen, Yaxin Gong, Yang Zhang, Chenxiao Fan, Jiakai Wang, Dong Wang, Yang Liu, Fuli Feng, Chun Kai Ling, Tat-Seng Chua

arXiv:2610.22316v1cs.LGcs.AIcs.MA

TL;DR

LLM-based agents can drift from their overall goals during long-horizon execution, wasting resources and risking undesired outcomes. This paper trains a steering agent to monitor execution and intervene, and reports performance gains across single- and multi-agent information-seeking tasks.

  • Problem

    LLM-based agent systems can drift from overall goals during long-horizon execution, wasting resources and risking undesired outcomes.

    Source: §1 · p. 1

  • Method

    A steering agent monitors task goals and execution context, using a low-cost fast path to decide whether to invoke a deliberative slow path, with both paths jointly trained by reinforcement learning.

    Source: §1 · p. 2

  • Results

    Across single- and multi-agent information-seeking tasks, RL-trained steering improves performance, gaining 6.66 points across seven QA benchmarks with 0.93 steering actions per task.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Steering provides on-the-fly supervision that improves performance and efficiency while supporting generalization, executor transfer, and controllable intervention frequency.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language model (LLM)-based agents and multi-agent systems increasingly tackle complex tasks through long-horizon tool use and inter-agent collaboration. However, these systems are prone to deviating from the overall goal as execution unfolds, which wastes execution resources and leads to undesired outcomes, a failure termed goal drift. In this work, we propose a steering agent that supervises execution on the fly and steers the system back on track when drift occurs. The steering agent observes the execution context, deciding when to intervene with a low-cost fast path and how to intervene with a high-cost slow path. To unlock its steering intelligence, the ability to assess goal drift and generate effective interventions, we jointly train the two paths with reinforcement learning, combining a pairwise ranking objective with group-relative policy optimization (GRPO). Experiments on information-seeking search tasks show that the steering agent provides effective supervision for both single-agent and multi-agent systems, improving their performance and efficiency. The learned steering intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.

1 Introduction

Long-horizon LLM agents and multi-agent systems can drift from their overall goals, wasting resources and producing undesired outcomes. The paper introduces an RL-trained steering agent that monitors execution and intervenes on the fly.

1 Introduction

  • Goal drift occurs when agents deviate from the overall goal, including when a multi-agent system prioritizes a local objective over its collective goal.Such drift can waste execution resources and lead to undesired outcomes even when individual actions appear locally reasonable.Source: §1 · pp. 1–2
  • Planning can leave execution off route, while post-hoc verification may lead to irreversible consequences and costly retries.On-the-fly supervision instead monitors execution and steers the system toward its goal when drift occurs.Source: §1 · p. 2
  • The steering agent uses a low-cost fast path to judge intervention priority and a slower reasoning path to generate steering actions; both paths are jointly trained with reinforcement learning.The fast path uses a single-token prediction and threshold to control activation of the slow path.Source: §1 · p. 2
  • RL-trained steering gains 6.66 points across seven QA benchmarks with 0.93 actions per task, and gains 5.34 points in the five-role multi-agent setting.Zero-shot steering gains only 1.84 points across the seven benchmarks; zero-shot supervision degrades performance in the multi-agent setting.Source: §1 · p. 2
  • The learned steering intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.At the highest intervention frequency, the gain reaches 17.11 points.Source: §1 · p. 2

2 Related Work

Related work addresses long-horizon agent training and drift mitigation, while this paper trains an external supervisor to detect and correct goal drift without updating task-executing LLMs.

2.1 LLM-based Agent Systems

  • Long-horizon and multi-agent RL train executors toward task success, but rollout cost and environment overfitting motivate freezing executors and training an external supervisor.The task-executing LLMs remain frozen while the external agent supervises execution and steers toward the overall goal.Source: §2.1 · pp. 2–3

2.2 Goal Drift Mitigation

  • Planning cannot ensure reliable execution and verification may catch drift after effort; SupervisorAgent (Lin et al., 2026) targets easily fixed errors, unlike jointly learned drift detection and correction.Goal drift can emerge gradually from locally plausible actions, making it challenging to detect and correct in real time.Source: §2.2 · p. 3

3 Learning to Steer LLM-based Agent Systems

The section frames goal drift as temporary subgoals displacing the overall goal, then develops an external steering agent that observes execution and intervenes. It trains the agent’s intervention judgment and action generation using task-outcome feedback.

3.1 Goal Drift in LLM-based Agent Systems

  • Goal drift occurs when temporary subgoals dominate the overall task, producing redundant searches or verifier rejections that can turn successful progress into failure.In the illustrated cases, the searcher continues after finding the needed information, while the verifier rejects answers already delivered.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 The Steering Agent

  • The steering interface maps observations exposing the goal and execution context to either a pass action or a supported intervention, with granularity set by the observation and action spaces.Observing every single-agent step allows fine-grained supervision; observing only multi-agent handoffs supports coarser supervision.Source: §3.2 · p. 4
  • The fast path judges intervention need at low cost, while a threshold can trigger the slow path to reason and generate an intervention.The fast path estimates steering priority from a yes/no judgment; the slow path produces reasoning followed by an action.Source: §3.2 · pp. 4–5

3.3 Reinforcement Learning for Steering Intelligence

  • Training samples states reached in prior rollouts, tests multiple steering actions and unsteered continuations, and uses their task rewards to compare outcomes while keeping the executor frozen.The mean reward across continuations serves as each action’s reward and the unsteered baseline.Source: §3.3 · pp. 5–6
  • The fast path learns to rank states by mean absolute outcome change from the unsteered baseline, with pairwise ranking loss aligning predicted and measured priority orderings.Tied-priority pairs provide no ranking signal.Source: §3.3 · p. 6
  • The slow path uses GRPO, assigning positive advantage only when a steering action’s mean reward exceeds the pooled average of steered outcomes and the unsteered baseline.The advantage is assigned to the generated reasoning and action tokens for policy optimization.Source: §3.3 · p. 6

4 Experiments

The experiments evaluate steering against planning and verification, then test reinforcement learning, generalization, transfer, intervention-frequency control, and multi-agent steering. RL strengthens steering on in-domain tasks, and the learned capability also shows benefits beyond that setting.

4.1 Experimental Setup

  • The experiments use a Qwen2.5-7B-Instruct search executor and a Qwen3-8B steering backbone, comparing steering with planning and verification at about one steering action per task.The in-domain evaluation follows ASearcher’s seven-benchmark retrieval-augmented QA setup.Source: §4.2 · p. 7, §4.1 · p. 7

4.2 Effectiveness of Steering (RQ1)

  • 6.66 points: RL raises steering gains from 1.84 points and reaches the strongest results on all seven benchmarks, with the fewest steering actions and executor turns.The same learning algorithm improves steering, planning, and verification; filtered behavior cloning falls behind online reinforcement learning.Source: §4.2 · p. 7

4.3 Generalizability, Transferability, and Controllability (RQ2)

  • Without additional training, steering improves both Avg@4 and Pass@4 on Frames and GAIA, and Pass@4 on xBench-DeepSearch.These evaluations use online search APIs and require deeper multi-hop search than the training tasks.Source: §4.3 · p. 8, Table 2 · p. 8
  • The Qwen-trained steering agent improves unseen Llama, Mistral, and Phi executors by 15.78, 12.01, and 5.92 points, respectively, while reducing executor turns.Table 3 reports gains in Avg. over No Intervention with the same executor.Source: §4.3 · p. 8, Table 3 · p. 8
  • RL steering improves performance across intervention budgets, reaching a 17.11-point total gain at the highest frequency and its largest gain per intervention at low frequency.Adjusting the fast-path threshold at deployment controls intervention frequency without retraining; Figure 4 shows total gain on the left and gain per intervention on the right.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 Steering Multi-Agent Systems (RQ3)

  • In the multi-agent setting, RL steering gains 3.90 and 5.34 points under two intervention-frequency settings, while zero-shot steering lowers average F1.At the highest frequency, agent turns rise from 9.2 to 24.1 per task; the table reports Avg. gain over No Intervention and costs in steering actions and agent turns.Source: §4.4 · pp. 8–9, Table 4 · p. 9

5 Conclusion

The conclusion presents the steering agent as on-the-fly supervision for goal drift in LLM agent systems. It summarizes gains across single-agent and multi-agent systems, generalization, transfer, and controllable intervention frequency.

5 Conclusion

  • The steering agent targets goal drift by deciding when to intervene with a low-cost fast path and how with a deliberative slow path.The two paths are jointly trained with reinforcement learning.Source: §5 · p. 9
  • The learned steering intelligence improves single-agent and multi-agent systems, generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency.Source: §5 · p. 9
  • The paper concludes that on-the-fly steering provides a supervision tool to improve performance and efficiency as agent tasks grow more complex and long-horizon.Source: §5 · p. 9

A.1 Training Details

The appendix describes the training data, model setup, reward, and state-selection procedure. Training samples states and steering actions, evaluates continuations, and adds reached states back to the state bank.

A.1 Training Details

  • Training uses Qwen3-8B for steering and a separately served, frozen Qwen2.5-7B-Instruct executor, with AdamW at 2 × 10^-5, batch size 64, and 300 steps.The reward multiplies answer-token F1 by an indicator for whether every executor turn commits exactly one well-formed action tag.Source: §A.1 · p. 15
  • Training performs two state-selection rounds, selecting up to three states per round and giving each selected state a group of four steering actions.Selection is random with probability 0.25, otherwise it chooses the highest-priority state; two executor continuations each are used to evaluate steered and unsteered states, and newly reached states join the bank.Source: §A.1 · p. 15
  • The steering agent trains on ASearcher-Base-35k, using the local retrieval server, the 2018 Wikipedia dump, and E5 as retriever.The dataset combines filtered open-source multi-hop questions with agent-synthesized QA pairs.Source: §A.1 · p. 15

A.2 Evaluation Details

The evaluation appendix specifies in-domain and out-of-domain metrics and benchmarks. In-domain tests use seven retrieval-augmented QA benchmarks, while out-of-domain tests use three deeper-search benchmarks over the open web.

A.2 Evaluation Details

  • In-domain results report token F1, with Avg. as the mean across seven benchmarks; out-of-domain questions run four times and report Avg@4 and Pass@4.Avg@4 is mean accuracy over four runs, and Pass@4 is the fraction solved in at least one run; an LLM judge evaluates answer correctness.Source: §A.2 · p. 15
  • The in-domain evaluation covers 6,125 retrieval-augmented QA questions across seven benchmarks, including multi-hop, compositional, real-search, trivia, and long-tail entity questions.It samples 1,000 questions from each benchmark except Bamboogle, which contributes all 125 questions.Source: §A.2 · p. 15
  • Out-of-domain evaluation uses Frames, GAIA, and xBench-DeepSearch, which test deeper open-web search.The evaluation includes 824 Frames questions, 103 text-only GAIA validation questions, and 100 xBench-DeepSearch questions.Source: §A.2 · p. 15

A.3 Baseline Design

The section compares alternative supervision baselines and describes how the steering agent acts on multi-agent handoffs. The baselines share a supervisor backbone and, where applicable, training resources to support a fair comparison.

A.3 Baseline Design

  • The baselines use the same supervisor backbone, and trained variants reuse the RL algorithm, data, and hyperparameters to isolate supervision placement and learning algorithm.Source: §A.3 · p. 15
  • Planning prompts the executor with a plan before its first action, while verification reviews a committed answer and can return it for revision.After three rejections, verification delivers the latest answer without further review.Source: §A.3 · p. 15
  • Filtered behavior cloning trains by SFT on sampled steering actions whose mean reward exceeds the unsteered baseline, without learning from discarded negative samples.Its offline training data also does not track the improving policy.Source: §A.3 · p. 16

A.4 Multi-Agent Instantiation

  • The multi-agent search pipeline assigns planning, parallel evidence retrieval, verification, and answer synthesis to five roles.A planner dispatches subgoals to two searchers; a verifier reviews their deliverables, and a summarizer composes the final answer.Source: §A.4 · p. 16
  • At handoffs, the steering agent can pass a submission unchanged, attach guidance for recipients, or reject it for sender revision.Its review uses the task, pipeline structure, prior and current handoffs, sender and recipients, and any existing rejection feedback.Source: §A.4 · p. 16

B Additional Experimental Results

The hyperparameter study retrains steering agents with varied rollout settings and evaluates them at three target steering frequencies. Configurations are similar at 15% frequency, while other configurations perform worse at higher frequencies.

B.1 Hyperparameters

  • At 15% steering frequency, all tested configurations deliver comparable gains within a band of one F1 point.Source: §B.1 · p. 16
  • The study varies selection rounds m, states per round k, and continuations per branch N while keeping group size G = 4 fixed.Source: §B.1 · p. 16
  • At 50% and 100% target steering frequencies, other tested configurations demonstrate lower performance.Source: §B.1 · p. 16

B.2 Cost of the Fast and Slow Paths

The latency measurements separate the inexpensive intervention judgment from the more costly reasoning and action generation. The fast path can run at each observation update, while the slow path runs only when triggered.

B.2 Cost of the Fast and Slow Paths

  • 0.3 s or less is the fast path's latency per call, compared with 30 s for the slow path, which generates about 1.8k reasoning and action tokens on average.Both paths share about 0.2 s of prefill cost.Source: §B.2 · p. 16, Figure 6 · p. 17
  • The fast path decodes a decision token after shared prefill, whereas the slow path incurs additional cost for generating reasoning and a steering action.Source: §B.2 · p. 16, Figure 6 · p. 17
  • The cost separation allows the fast path to run at every observation update and reserves slow-path reasoning for triggered interventions.Source: §B.2 · p. 17

B.3 LLM-as-a-Judge Results on In-Domain Benchmarks

On in-domain benchmarks, the LLM-as-a-Judge evaluation preserves the trend reported with token F1, with RL-improved steering best across all seven benchmarks. The prompt templates specify the context and decisions used in single-agent and multi-agent steering.

B.3 LLM-as-a-Judge Results on In-Domain Benchmarks

  • 6.77 points versus 1.60 points is the reported gain for RL-improved steering versus the comparison without RL, and RL-improved steering performs best on all seven benchmarks.The judge accepts semantically matching answers despite differences in surface form, and its accuracy is higher overall than token F1.Source: §B.3 · p. 17

C Prompt Templates

  • Single-agent prompt slots include the question, action history, action count, remaining turn budget, and legal actions; multi-agent prompts include task and pipeline context, handoffs, submission, and rejection feedback.Source: C Prompt Templates · pp. 17–18
  • Single-agent prompts ask for an actionable next-step steering note, while multi-agent prompts offer rejection feedback or a note attached to a submission passed to recipients.The single-agent note is shown for the executor's next action only.Source: C Prompt Templates · pp. 18, 20
  • The multi-agent fast-path prompt defines criticality by whether intervention on a handoff meaningfully changes the final result, versus passing it through with little influence.Source: C Prompt Templates · p. 19

Related papers

Loading 2610.22316v1…