Source-linked AI summary
Reinforcement Learning for Long-Horizon Interactive LLM Agents
Kevin Chen, Marco Cusumano-Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, Philipp Krähenbühl
TL;DR
Interactive digital agents face long-horizon, stateful API tasks for which existing models have limited success. The paper formalizes these tasks as POMDPs and introduces LOOP, a data- and memory-efficient PPO variant. A 32-billion-parameter LOOP agent outperforms the much larger OpenAI o1 agent on AppWorld, while learning several useful interaction behaviors.
Problem
Existing instruction-tuned agents have not been trained in their target digital environments and prior methods achieve limited success on sophisticated AppWorld tasks.
Method
LOOP trains interactive agents directly in their target environments using PPO with leave-one-out advantages and per-token clipping, without a value network.
Results
9 percentage points: a 32-billion-parameter LOOP agent outperforms the much larger OpenAI o1 agent on AppWorld.
Takeaways & Limitations
RL training on only 24 scenarios produces agents that generalize to diverse held-out tasks and learn improved documentation use, assumption avoidance, confabulation reduction, and setback recovery.
Takeaways & Limitations
Even the best agents succeed on only ∼7 out of 10 AppWorld tasks, and AppWorld lacks several features of everyday environments, including non-determinism and transient failures.
Abstract
from arXiv · showhide
Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests. While IDAs powered by instruction-tuned large language models (LLMs) can react to feedback from interface invocations in multi-step exchanges, they have not been trained in their respective digital environments. Prior methods accomplish less than half of tasks in sophisticated benchmarks such as AppWorld. We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments. We formalize this training as a partially observable Markov decision process and derive LOOP, a data- and memory-efficient variant of proximal policy optimization. LOOP uses no value network and maintains exactly one copy of the underlying LLM in memory, making its implementation straightforward and as memory-efficient as fine-tuning a single LLM. A 32-billion-parameter agent trained with LOOP in the AppWorld environment outperforms the much larger OpenAI o1 agent by 9 percentage points (15% relative). To our knowledge, this is the first reported application of RL to IDAs that interact with a stateful, multi-domain, multi-app environment via direct API calls. Our analysis sheds light on the effectiveness of RL in this area, showing that the agent learns to consult the API documentation, avoid unwarranted assumptions, minimize confabulation, and recover from setbacks.
1. Introduction
Interactive digital agents must manage long, multi-step API interactions across stateful applications, yet existing models achieve limited success. The paper trains such agents directly with reinforcement learning and reports improved performance and emergent behaviors.
- Motivation: AppWorld tasks require information gathering, state-changing actions, and replanning across interactions with multiple software apps.Tasks can require up to 40 REPL interactions and up to 32K tokens, while the environment contains up to 30M text tokens.
- Motivation: Below 40% success for the best open-weight models and barely above half for top reasoning models motivates direct training in these environments.
- Approach: Reinforcement learning on only 24 training scenarios learns behaviors that generalize to diverse held-out tasks.The approach uses a simple task-completion reward and does not require ground-truth action sequences or large training-scenario datasets.
- Approach: LOOP combines PPO, leave-one-out advantage estimation, and per-token clipping to reuse off-policy samples while storing one backbone LLM and omitting a value network.These choices target sample, memory, and implementation efficiency.
- Results: Training reduces unnecessary batching by ∼6x, increases API-documentation queries by ∼60%, and reduces assumption-related words by ∼30x.It also reduces placeholder-related ‘dummy’ usage by ∼6x and capitulation after failed API calls by ∼3x.
2. Related Work
Prior work applies reinforcement learning to language models and agents across preference optimization, reasoning, code generation, browsing, games, devices, and embodied environments. LOOP extends this line to long-horizon interactive digital agents with off-policy policy updates.
- RL for LLMs: RLHF commonly trains language-model policies with PPO using reward, policy, reference, and critic models.
- RL for LLMs: RLOO reduces this complexity by using on-policy rollouts and sampling-based leave-one-out advantages instead of reference and critic LLMs.
- LOOP’s position: LOOP generalizes RLOO by allowing off-policy policy drift within PPO’s trust region, enabling rollout reuse and looser separation between collection and updates.
- RL for LLMs: PPO variants have trained reasoning and static code generation, while other methods alternate filtered-rollout generation with supervised training.
- RL for agents: RL has trained stateful multi-turn agents in games, web shopping, navigation, mobile control, and embodied environments.
3. Preliminaries
The preliminaries cast language generation as sequential decision-making and explain how REINFORCE and PPO estimate policy gradients. Leave-one-out advantages reduce variance, while PPO permits multiple updates on previously collected rollouts.
- Language modeling: An autoregressive language model maps preceding tokens to a next-token distribution and generates until a stop token.
- Language modeling: Language generation can be represented as an MDP whose state contains the context and generated prefix, with each action appending one token.
- Language modeling: Terminal rewards and deterministic transitions reduce this formulation to a contextual bandit in which an entire generation is one action.
- Policy gradients: RLOO estimates each rollout’s advantage by subtracting the average return of the other samples from its own return.Using K independent samples produces a simple, unbiased, low-variance estimate.
- PPO: PPO permits the updated policy to deviate from the sampling policy within a trust region through importance weighting.Updates ignore samples whose probability ratios move outside the specified region.
- PPO: Per-token importance weights localize the effect of probability changes, whereas a per-trajectory weight can suppress the gradient for an entire trajectory.Value networks used for advantage estimation can also be slow, memory intensive, and inaccurate for language models.
4. LOOP
LOOP formulates interactive digital-agent training as a POMDP in which LLM tokens and environment responses jointly define the interaction history. It combines leave-one-out advantages with PPO updates over collected rollouts.
- POMDP formulation: Interactive agents operate in a stateful REPL initialized with API bindings, executing code blocks and receiving response strings until task completion.In AppWorld, state includes a database and agent-defined Python global variables.
- POMDP formulation: The POMDP state contains the initial environment state, task context, and generated history, while stop tokens trigger code execution and append environment responses.
- Algorithm: LOOP collects K rollouts for each initial-state and context pair, computes leave-one-out advantages, and performs PPO updates over shuffled mini-batches.
- Algorithm: The implementation experiments with per-token, per-turn, and per-trajectory importance weights for PPO.
- Connections: Purely on-policy LOOP with one epoch and no mini-batches reduces to REINFORCE Leave-One-Out.
- Connections: LOOP and GRPO differ mainly in advantage normalization: GRPO divides by return standard deviation, whereas LOOP found omitting normalization beneficial.
5. Evaluation
On AppWorld, LOOP (token) achieves the strongest reported performance across normal and challenge test sets, while RL methods generally outperform supervised alternatives. Training also changes agent behavior toward documentation use, fewer assumptions, adaptive control, and recovery from errors.
- Results: 71.3 TGC on Test-N and 45.7 TGC on Test-C make LOOP (token) the best-performing method in the comparison.These results are reported for task goal completion on AppWorld’s normal and challenge test sets.
- Results: 15% relative improvement over OpenAI o1 on Test-N and 24% on Test-C establish LOOP’s advantage over the larger agent.The comparison is reported separately for the two AppWorld test splits.
- Results: Token actions perform considerably better than turn or trajectory actions, whose training was unstable and often triggered importance-weight clipping.This comparison is reported among LOOP variants.
- Policy behavior: After training, multiple code cells per turn decrease by ∼6x while total submitted code remains stable, indicating less open-loop control.The agent uses intermediate results more often to inform subsequent code execution.
- Policy behavior: API-documentation queries increase by ∼1.6x, while assumption-related language falls by ∼30x and giving up after failed API calls falls by ∼3x.The trained agent more consistently gathers information, avoids brittle shortcuts, and debugs occasional errors.
- Results: All reinforcement-learning-like methods outperform supervised fine-tuning and prompt engineering, while LOOP exceeds the performance plateau of most alternatives by more than 5 pp.Methods with Monte Carlo baselines also outperform PPO with a learned value function, which showed inferior training stability.
6. Discussion
The paper formalizes IDA training as reinforcement learning and presents LOOP as a simple, effective algorithm. RL meaningfully improves IDAs after training on few scenarios, but broader autonomy remains beyond current robustness.
- LOOP provides a simple reinforcement-learning algorithm for training interactive digital agents.
- RL produces meaningfully better IDAs after training on only a small number of scenarios.
- Best agents succeed on approximately 7 out of 10 AppWorld tasks, below the robustness required for broader autonomy.
- AppWorld lacks non-determinism, transient failures, ambiguous or unsolvable tasks, adversarial scenarios, clarification steps, and interactive counterparties.
A.1. Quantitative comparison on training tasks
Across training tasks, fine-tuning produces more interactive, persistent, careful, and serious behavior. The appendix defines the behavioral metrics and the failed-API-call give-up-rate calculation used for these comparisons.
- Training makes the agent more interactive, persistent, careful, and serious across the training tasks.It uses fewer turns with multiple code cells, gives up less often, checks documentation more frequently, and makes fewer assumptions and dummy variables.
- The analysis aggregates behavior changes over 72 reinforcement-learning training tasks using three i.i.d. rollouts per task.
- Multiple-code-cell frequency measures turns containing more than one Python block and normalizes by turns across analyzed rollouts.
- The failed-API-call give-up rate equals failed API calls minus recovered API calls, divided by failed API calls.The rate is not normalized by the number of turns.
- API-documentation frequency counts calls to apis.api_docs.show_api_doc in agent code blocks and normalizes by analyzed rollouts.
A.2. Qualitative comparison (early vs. late in training) via summarized rollouts
Comparing three early and three late rollouts of the same task, the analysis shows consistent behavioral changes during LOOP training. Later policies consult documentation before API calls and reduce assumptions, dummy values, and erroneous calls.
- After training, the policy significantly reduces poor assumptions, dummy values, and erroneous API calls across independent rollouts.
- The task asks for all Venmo transactions from today involving roommates on the user’s Venmo social feed.
- Figure 6 compares three i.i.d. rollouts at step 38 with three i.i.d. rollouts at step 530 for the same task.
- Late-training rollouts consistently look up documentation before invoking APIs and avoid anti-patterns.API calls occur inside Python blocks that can include loops, branching, and filtering logic.
C. Evaluated Methods: Details
The evaluated methods span supervised and preference-based fine-tuning, rejection sampling, interactive exploration, and PPO variants. Their results show generalization and stability trade-offs, with LOOP-related methods outperforming alternatives while learned-critic PPO remains less stable.
- C.1. Ground Truth Supervised Fine-Tuning: Ground-truth SFT recovers near-perfect training performance but does not meaningfully improve dev or test performance after initial degradation.
- C.1. Ground Truth Supervised Fine-Tuning: The training-set performance fall-off and recovery appears driven by switching between environment interaction and memorization of solution steps.
- C.2. Rejection-sampling Fine-Tuning: A single RFT pass brings a 32B model to approximately prior GPT-4o SOTA, while a meaningful gap remains relative to the evaluated RL methods.
- C.3. EI Training: EI remains stable with stale experience and achieves results comparable to certain RL variants, including GRPO.
- C.4. DPO-MCTS: DPO-MCTS collects preference pairs by expanding interactions with UCB, sampling policy actions, rolling out one branch, and backpropagating returns.
- C.5. PPO with a Learned Critic: Value-function PPO suffers from biased advantages, training instability, and hyperparameter sensitivity, underperforming Monte Carlo-based methods in these tests.
D. Additional training details
The section specifies LOOP’s AppWorld training setup, reward, rollout budget, and evaluation protocol, including repeated-run reporting and a small scenario subset.
- Training reward: Each AppWorld task receives a reward from 0 to 1 equal to the fraction of successfully passed unit tests.
- Infrastructure: Training uses two NVIDIA H100 8-GPU nodes, with one node for rollout generation and one for training steps.
- Rollout collection: Each iteration generates K = 6 rollouts for 40 randomly sampled tasks, totaling 240 rollouts.
- Interaction budget: Episodes allow up to 40 environment interactions during training and 50 during evaluation; exceeding the budget counts as failure.
- Evaluation protocol: Table 2 averages results across three training runs per algorithm and reports means with standard deviations, while Table 1 selects the best training run.
- Training data: Training uses 24 scenarios with three minor task variations per scenario, excluding difficulty-3 tasks.
E. Evaluation over multiple training runs for each method
This section compares methods across repeated training runs and illustrates successful and failed AppWorld development rollouts. Performance trends remain similar across evaluation protocols, while individual tasks vary.
- Evaluation across runs: Table 2 reports mean and standard deviation across three training runs per algorithm, with relative trends largely matching Table 1.
- Successful rollout: A successful difficulty-3 development task passed 10/10 tests in 17 interactions.
- Successful rollout: The successful rollout planned execution, consulted Phone documentation, corrected a misspelled phone number, and used Venmo and Phone APIs.
- Failed rollout: A failed difficulty-1 Spotify task passed 4/6 tests after downloading songs beyond those liked in the user’s library.
- Failed rollout: The Spotify rollout logged in, consulted API documentation, gathered liked songs, and then included songs outside the requested library subset.
G.1. Rollout 0 of task 2a163ab 1early in training
An early Venmo rollout attempts to identify roommates, retrieve relevant social-feed transactions, and like them, but initially relies on an unavailable credentials API and an unsupported assumption.
- Task execution: The rollout’s plan is to log in, identify roommates, retrieve the Venmo social feed, filter transactions, and like qualifying transactions.
- Credential retrieval: The agent initially calls the nonexistent supervisor API show_account_credentials and receives an execution exception.
- Credential retrieval: After the error, the agent switches to show_account_passwords and proceeds with Venmo login.
- Transaction processing: The agent searches contacts for roommates, retrieves the social feed, filters today’s transactions involving roommates, and likes the selected transactions.
- Outcome: The rollout reports successful completion after liking the relevant transactions and calling the supervisor completion API.
G.2. Rollout 1 of task 2a163ab 1early in training
A second early Venmo rollout recovers from invalid credentials and a phone-app authorization error, then completes the task by filtering and liking relevant transactions.
- Recovery from errors: The rollout first attempts Venmo login with invalid credentials and receives a 401 Invalid credentials response.
- Recovery from errors: The agent responds by fetching the actual password from the supervisor app and retrying login.
- Data retrieval: After successful login, the agent retrieves the Venmo social feed and identifies roommates through the Phone app.
- Task completion: The agent filters transactions by date and roommate involvement, likes each selected transaction, and marks the task complete.
- Authorization failure: A Phone API authorization failure prevents retrieving roommates, so the agent assumes dummy roommate emails and continues.
G.3. Rollout 0 of training task 2a163ab 1 after training
The rollout solves a multi-app task by consulting API documentation, authenticating with Phone and Venmo, identifying relevant transactions, and applying the requested actions. It reports successful completion after liking the relevant transactions.
- Planning and API use: The agent first plans the task and consults documentation for the Phone and Venmo APIs.The rollout describes documentation lookup before using search, login, social-feed, and transaction APIs.
- Phone investigation: It logs into the Phone app to obtain an access token before searching contacts and messages.The workflow includes Phone login, contact lookup, and text-message search to identify task-relevant information.
- Venmo investigation: The agent logs into Venmo and retrieves today’s transactions involving the identified roommates.It uses the Venmo social feed and filters transactions by date and roommate involvement.
- Action execution: It checks the like_transaction specification, loops over the filtered transactions, and receives repeated “Liked transaction.” responses.The rollout shows the transaction ID parameter and four successful like responses.
- Completion: The agent declares the task successfully completed after liking all relevant Venmo transactions from that day.The rollout ends with a completion call after the successful transaction updates.