PaperlayerView exact revision on arXiv

Source-linked AI summary

EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents

Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye

arXiv:2610.16523v1cs.AIcs.CLcs.LG

TL;DR

Multi-turn agent training depends on learning signals shaped by both simulated users and environments, which can be inconsistent or unreliable and are often left fixed by prior approaches. EAU monitors training and adapts these components alongside the policy; across three benchmarks it improves on prior results, including overall-score gains over the strongest prior RL baseline. Its simulated-benchmark results do not establish deployment safety or reliability.

  • Problem

    Interactive RL relies on simulated users and environments that can generate inconsistent or misleading signals, while prior work mainly improves supervision and leaves those sources largely fixed.

    Source: §1 · p. 2

  • Method

    EAU monitors trajectories, tool traces, rewards, and training statistics, then validates and hot-swaps targeted changes to users, environments, rewards, or RL settings during training.

    Source: §1 · p. 2

  • Results

    Across UserGym and ColBench, EAU raises overall score over InfoPO from 0.545 to 0.627 on Qwen3-4B and from 0.544 to 0.615 on Qwen2.5-7B-Instruct, relative gains of 15.0% and 14.7%.

    Source: §5.3 · p. 8

  • Takeaways & Limitations

    Across three user-centric benchmarks, EAU improves prior state-of-the-art performance by up to 36.7%, supporting the value of adapting the training system itself.

    Source: §6 · p. 10

    Performance in simulated benchmarks should not be interpreted as evidence of safe or reliable behavior in deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment, which jointly shape the agent's experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce EAU, a meta-agent framework for environment--agent--user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 14.7--15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.

1 INTRODUCTION

Multi-turn agent training relies on simulated users and environments to produce reliable, informative learning signals, but both can introduce instability and are typically fixed. EAU instead diagnoses and adapts the interaction process during training; evaluations report improved performance across user-centric benchmarks.

1 INTRODUCTION

  • Fixed user simulators and environments can make learning signals variable or misleading through inconsistent user behavior, unreliable tools, incomplete state coverage, and incorrect verifiers.Because these components remain fixed as the policy evolves, they may also produce redundant or uninformative trajectories.Source: §1 · p. 2
  • Prior work improves noisy policy supervision with reward shaping or denser signals, while EAU shifts intervention upstream to revise the interaction process itself.The paper argues that downstream reward changes cannot repair silent tool failures, incorrect verifiers, or information disclosed prematurely by a simulator.Source: §1 · p. 2
  • EAU monitors transcripts, tool traces, rewards, and training statistics, diagnoses failure sources, and proposes targeted changes to simulators, environments, rewards, or RL settings.A bounded integrity gate validates candidate modifications before they are hot-swapped into ongoing training.Source: §1 · p. 2
  • EAU adapts the training system during and across runs rather than prescribing a fixed sequence of component updates.The contribution is organizing this adaptive process around diagnosed problems and intervention timing.Source: §1 · pp. 2–3
  • Up to 36.7%: EAU improves final agent performance over prior state of the art across three interactive-agent benchmarks and two backbone models.The evaluation also analyzes how the three components evolve together during training.Source: §1 · p. 3
  • EAU's contribution includes online diagnosis and updates to the training system, with analyses of symbolic-user mixing, intra-run intervention, and alternative training algorithms and meta-agent implementations.The paper reports improved performance on three user-centric benchmarks.Source: §1 · p. 3

2 RELATED WORK

Prior work addresses instability in user-centric RL through improved supervision or user co-training, while research on adapting an agent's world often prepares it before training or uses predefined updates. EAU emphasizes flexible online intervention within a run.

User Centric Agentic Reinforcement Learning

  • User-centric RL adds stochastic user transitions, inconsistent behavior, and possible collapse from overfitting fixed behavior to the challenges of multi-turn RL.Prior methods mainly improve policy supervision, while Yu et al. (2026) co-train the user; EAU targets RL dynamics, user simulation, and environment together.Source: §2 · p. 3

Meta-Agents and Automated Optimization of Training Components

  • Many meta-agent approaches optimize another system, but EAU validates and hot-swaps modifications into a live training loop rather than applying them only before or between runs..Source: §2 · p. 3

Co-evolution of Agent and its World

  • Unlike prior world-evolution methods based on predefined update rules or workflows, EAU flexibly determines what to modify, when to intervene, and how to adapt online.Other approaches prepare the world before training, while some evolve it alongside the agent.Source: §2 · p. 3

3 PROBLEM SETUP

The paper models user-centric tasks as multi-turn interaction with tools and a simulated user to satisfy an intent that may be underspecified and refined during the exchange. The task combines partially observable state, stochastic user interaction, and task-success and conversation rewards.

3 PROBLEM SETUP

  • User-centric tasks require accomplishing a task while satisfying a user's intent, which may be underspecified initially and refined through interaction.The agent interacts with both an environment through tool calls and a simulated user over multiple turns.Source: §3 · p. 3
  • Each task is formulated as a POMDP whose state combines tool, user, and query states, with tool and user actions and observations.The user simulator is part of the environment.Source: §3 · p. 3
  • Tool interactions are deterministic, whereas interactions with simulated users are stochastic.These are distinct transition types in the task formulation.Source: §3 · p. 4
  • The policy objective is to maximize successful tasks while interacting with and understanding the user and calling the correct tools.The setup describes task-success rewards as usually verifiable but sparse, and conversation rewards as dense but noisy.Source: §3 · p. 4

4 METHOD: ENVIRONMENT-AGENT-USER CO-EVOLUTION

EAU addresses sparse rewards and stochastic user interactions by coupling policy learning with meta-agent adaptation of the system that generates training experience. Its programmatic interventions adapt users, environment feedback, and training dynamics to the evolving policy.

4 METHOD: ENVIRONMENT-AGENT-USER CO-EVOLUTION

  • Sparse rewards and stochastic user interactions can make learning difficult, while user simulators may provide unreliable rewards, deviate from instructions, or leak answers.Source: §4 · p. 4

4.1 METHOD OVERVIEW

  • EAU’s meta-agent uses training trajectories and optimization statistics to revise the user simulator, environment, and configurable training dynamics.These programmatic interventions create an asymmetric co-adaptation; the meta-agent also accumulates experience for later decisions.Source: §4.1 · p. 4

4.2 EVOLVING USER

  • EAU exposes deterministic symbolic-user rules as tunable knobs and mixes symbolic and LLM simulators using an adaptable coefficient ϵ.It can also tune LLM generation temperature and persona-specific instructions as the policy evolves.Source: §4.2 · p. 5

4.3 EVOLVING ENVIRONMENT

  • EAU adds or revises intermediate rewards for states indicating task progress and can modify existing rewards that divert learning from the final objective.It accumulates experience about which reward designs facilitate or hinder learning to guide future environment changes.Source: §4.3 · pp. 5–6

4.4 EVOLVING AGENT

  • EAU keeps PPO fixed while its meta-agent monitors optimization and may tune configurable RL dynamics.A learned critic and GAE are computed across action tokens rather than all tokens; monitoring can include entropy and KL coefficients, rollout entropy, and log likelihood.Source: §4.4 · p. 6

4.5 INTRA- AND INTER-RUN INTERVENTION

  • Intra-run hot-swaps modify training without a restart, while inter-run restarts or checkpoint resumes let EAU consolidate observations for later training.The authors argue that this flexibility can be more efficient than inter-run modifications alone.Source: §4.5 · p. 6

4.6 INSTANTIATION OF META-AGENT

  • EAU is instantiated as a coding meta-agent that reads local source files and training logs, maintains an experience pool, and guides interventions using training-side feedback.It checks logs at fixed intervals and can revise an initially prepared list of tunable knobs.Source: §4.6 · pp. 6–7

5 EXPERIMENTS

EAU improves overall performance across three user-centric benchmarks, two policy backbones, and several training configurations. The experiments compare it with prompting and RL baselines and analyze simulator choices, meta-agent design, and intra-run adaptation.

5.1 EVALUATION PROTOCOL

  • Evaluation covers ColBench, UserGym, and τ 2-bench with Qwen3-4B and Qwen2.5-7B-Instruct, using prompting, RL, and vanilla PPO baselines.Evaluation follows the benchmark protocol, reports the final run and checkpoint, and excludes final-test feedback from intervention and model selection.Source: §5.1 · pp. 7–8

5.2 IMPLEMENTATION DETAILS

  • The default meta-agent is Claude Code with Claude Opus-5, and each experiment allows up to five inter-run iterations, including proactive restarts.Episode limits are 16 turns on UserGym, 10 on ColBench, and 50 on τ 2-bench; these are rollout and search limits, not matched end-to-end compute budgets.Source: §5.2 · p. 8

5.3 MAIN RESULTS

  • EAU raises overall score over InfoPO from 0.545 to 0.627 on Qwen3-4B (+15.0%) and from 0.544 to 0.615 on Qwen2.5-7B-Instruct (+14.7%).It achieves the highest overall score among evaluated RL methods on both backbones.Source: §5.3 · p. 8
  • On τ 2-bench, EAU raises aggregate score from 18.5 with PPO to 21.5 (+16.2%), leads on Airline and Telecom, while InfoPO leads on Retail.The comparison is on Qwen3-4B.Source: Table 2 · p. 8, §5.3 · p. 9
  • On UserGym, EAU raises average score over InfoPO from 0.559 to 0.622 on Qwen3-4B (+11.3%) and from 0.563 to 0.620 on Qwen2.5-7B-Instruct (+12.3%).On Qwen3-4B, it outperforms the best RL baseline in six of seven environments; results on held-out Inten. and Tele. are not uniform.Source: §5.3 · p. 8
  • On ColBench, EAU raises Qwen3-4B average score from 0.496 to 0.643 (+29.6%), with Succ. improving from 0.439 to 0.600 (+36.7%) over InfoPO.For both backbones, relative gains are larger on the stricter Succ. metric than on Pass.Source: §5.3 · p. 8

5.4 ABLATIONS & ANALYSIS

  • On ColBench, the hybrid simulator slightly exceeds LLM-only mean Pass and Succ. scores, while symbolic-only performs worse; EAU also beats a generic meta-agent on both benchmarks and backbones.The generic meta-agent removes the symbolic user and intra-run operation skill together, so the comparison tests their combined design rather than individual effects.Source: §5.4 · p. 9
  • EAU improves aggregate performance with UserRL or InfoPO on τ 2-bench, and its intra-run exploration carries ColBench modifications into a later run that surpasses the strongest prior baseline.On UserGym, with the first four runs shared, enabling intra-run intervention in run five yields a larger improvement than inter-run adaptation alone; the curves do not compare matched total compute.Source: §5.4 · p. 9, Figure 6 · p. 10

6 CONCLUSION

EAU jointly evolves the environment, agent, and user during training, with interventions informed by training feedback. The conclusion reports improvements across three benchmarks and identifies autonomous co-evolution with open-source models as future work.

6 CONCLUSION

  • On UserGym, the two run-level conditions share experience from the first four runs and differ in run five by whether intra-run intervention is allowed.Source: Figure 7 · p. 10
  • Across three user-centric benchmarks, EAU improves prior state-of-the-art performance by up to 36.7%.EAU jointly evolves the environment, agent, and user during training.Source: §6 · p. 10
  • A frontier coding agent manages and adapts the components using training feedback, including trajectories and optimization logs.A modifiable symbolic user simulator and intra-run interventions support finer control and more flexible exploration of training modifications.Source: §6 · p. 10
  • Future work may investigate post-training open-source models to perform co-evolution autonomously.Source: §6 · p. 10

ETHICS STATEMENT

The study uses RL agents interacting with simulated users in benchmark environments and does not collect human-participant data. Training modifications carry risks that are not resolved by interface checks or experimental controls.

ETHICS STATEMENT

  • The study uses simulated users in benchmark environments and recruits no human participants or new human-participant data.Source: ETHICS STATEMENT · p. 10
  • Code, reward, and simulator modifications can introduce implementation errors, misleading learning signals, or unintended task-information disclosure.Interface checks constrain admissible updates but do not guarantee behavioral correctness.Source: ETHICS STATEMENT · p. 10
  • Benchmark performance does not establish safe or reliable behavior with real users.Training-time adaptation is separated from a fixed evaluation protocol, but those controls address experimental integrity rather than deployment safety.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper describes the framework, workflow, experiments, interfaces, and optimization details across sections and appendices. Reproduction requires access to the external coding-agent and user-simulator models used in the experiments.

REPRODUCIBILITY STATEMENT

  • Reproducing the reported configurations requires access to the external coding-agent and user-simulator models used in the experiments.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper documents the workflow, benchmarks, model configurations, baselines, evaluation protocol, training limits, and intervention interfaces across sections and appendices.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Reported standard deviations across repeated evaluations do not quantify variability across independent repetitions of the full training and adaptation process.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The statement places generative AI in the experimental workflow and research process. ColBench interface records describe available controls and checks across two analyzed runs.

AI USE STATEMENT

  • Generative AI supports log and trajectory inspection, proposed or implemented training-system changes, policy modeling, user simulation, evaluation, and manuscript preparation.The authors retain responsibility for research design, implementation, validation, and final interpretations.Source: AI USE STATEMENT · p. 11

A INTERVENTION INTERFACES AND RUN-LEVEL OBSERVATIONS

  • Write checks enforce declared ranges, and difficulty-increasing updates require a recent training success rate of 0.80; four max-step proposals were rejected before qualification.These checks constrain writes but do not guarantee improved performance or the intended behavioral effect.Source: A INTERVENTION INTERFACES AND… · p. 14

A.1 COLBENCH: INTERFACE USAGE

  • ColBench interface records distinguish adjusted controls, fixed or unused controls, and interfaces with restricted implementation scope.The records cover two analyzed runs of 624 training steps each; an exposed control need not govern every behavior suggested by its name.Source: §A.1 · p. 14, Table 6 · p. 14, A INTERVENTION INTERFACES AND… · p. 14

A.2 COLBENCH: INTERVENTION ATTEMPTS AND DIAGNOSTICS

The ColBench record separates intervention attempts and diagnoses from evidence of intervention effects. UserGym interface coverage is implementation-specific and does not establish control use or performance gains.

A.2 COLBENCH: INTERVENTION ATTEMPTS AND DIAGNOSTICS

  • The two ColBench runs record 15 knob writes, two measured harms, three retracted attributions, and diagnoses including deadlock, reward hacking, and template collapse.Table 7 groups observations and distinguishes repairs, rollbacks, inconclusive perturbations, and findings without documented follow-up; reported score changes are local, not controlled estimates.Source: §A.2 · p. 14, Table 7 · p. 15
  • Trajectory inspection diagnoses problems, while knob adjustment attempts changes; a code repair does not establish that further threshold increases are beneficial.Disclosure and volunteering examples show why reward or success metrics must be interpreted alongside interaction behavior; exposed-control findings can reveal limitations rather than successful adaptations.Source: §A.2 · p. 16
  • The fixed symbolic–LLM mixture coefficient was 0.5 while simulator arms differed by 0.29 in success rate, so these runs do not evidence gains from dynamic mixture adaptation.Differences between simulator arms should not be conflated with effects of changing the coefficient.Source: §A.2 · p. 16

A.3 USERGYM: BENCHMARK-SPECIFIC INTERFACE COVERAGE

  • UserGym support marks indicate interface availability, not use or improved performance; only function has a predicate library, and predicate weights are rejected elsewhere.The mixture selector resolves only travel/colbench/tau2; the other four UserGym keys resolve to None.Source: §A.3 · p. 16, Table 8 · p. 16

B DETAILS ON EXPERIMENTS

The experiments compare prompting and RL baselines across three user-centric benchmarks, with training hyperparameters reported separately. Additional analysis finds generally better performance with Qwen3-4B and relatively larger gains on coding than tool use.

B DETAILS ON EXPERIMENTS

  • The baselines include ReAct and Reflexion for prompting, plus RAGEN, Search-R1, UserRL, InfoPO, and vanilla PPO for RL.For UserGym and ColBench, baseline results except PPO come from Kong et al. (2026); PPO and EAU are evaluated under the authors’ implementation.Source: B DETAILS ON EXPERIMENTS · pp. 16–17
  • ColBench tests collaborative artifact creation, UserGym covers eight interactive gyms, and τ 2-bench evaluates customer-service tasks across retail, airline, and telecom domains.SearchGym is excluded from UserGym evaluation due to API costs, but it is an out-of-distribution evaluation gym and is not excluded from RL training.Source: B DETAILS ON EXPERIMENTS · p. 17

D ADDITIONAL ANALYSIS

  • With Qwen3-4B, EAU generally performs better, with relatively greater gains on coding (ColBench) than tool use (UserGym).This may suggest EAU is more adept at eliciting better interaction behavior than overcoming intrinsic capability limitations.Source: D ADDITIONAL ANALYSIS · pp. 17–18

C ADDITIONAL IMPLEMENTATION DETAILS

Related papers

Loading 2610.16523v1…