PaperlayerView exact revision on arXiv

Source-linked AI summary

ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomy

Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu

arXiv:2403.14589v3cs.AIcs.CLcs.LG

TL;DR

Language-agent training depends on multi-step trajectories whose collection remains costly and limited in diversity. A3T uses ActRe to autonomously annotate sampled actions and contrastive self-training to accumulate successful trajectories across rounds. It achieves strong results on AlfWorld and WebShop, including 96% single-trial success on AlfWorld and performance approaching human experts on WebShop after refinement.

  • Problem

    Collecting diverse language-agent trajectories with textual reasoning remains dependent on costly human annotation or labor-intensive prompting frameworks.

    Source: §1 · pp. 1–2

  • Method

    A3T queries an ActRe agent for rationales of sampled actions, synthesizes ReAct trajectories, and trains on successful and failed trajectories with policy gradients and binarized rewards.

    Source: §1 · p. 2, §2.1 · p. 3, §5 · p. 9

  • Results

    A3T reaches 96% 1-shot success on AlfWorld and 49% success on WebShop, rising after four rounds to 100% and 54.8%, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A3T agents outperform multiple existing techniques across AlfWorld and WebShop through a closed loop of self-annotation and contrastive self-training.

    Source: §1 · p. 3, §5 · p. 9

Abstract

from arXiv · show

Language agents have demonstrated autonomous decision-making abilities by reasoning with foundation models. Recently, efforts have been made to train language agents for performance improvement, with multi-step reasoning and action trajectories as the training data. However, collecting such trajectories still requires considerable human effort, by either artificial annotation or implementations of diverse prompting frameworks. In this work, we propose A$^3$T, a framework that enables the Autonomous Annotation of Agent Trajectories in the style of ReAct. The central role is an ActRe prompting agent, which explains the reason for an arbitrary action. When randomly sampling an external action, the ReAct-style agent could query the ActRe agent with the action to obtain its textual rationales. Novel trajectories are then synthesized by prepending the posterior reasoning from ActRe to the sampled action. In this way, the ReAct-style agent executes multiple trajectories for the failed tasks, and selects the successful ones to supplement its failed trajectory for contrastive self-training. Realized by policy gradient methods with binarized rewards, the contrastive self-training with accumulated trajectories facilitates a closed loop for multiple rounds of language agent self-improvement. We conduct experiments using QLoRA fine-tuning with the open-sourced Mistral-7B-Instruct-v0.2. In AlfWorld, the agent trained with A$^3$T obtains a 1-shot success rate of 96%, and 100% success with 4 iterative rounds. In WebShop, the 1-shot performance of the A$^3$T agent matches human average, and 4 rounds of iterative refinement lead to the performance approaching human experts. A$^3$T agents significantly outperform existing techniques, including prompting with GPT-4, advanced agent frameworks, and fully fine-tuned LLMs.

1 Introduction

Training language agents requires multi-step trajectories, but existing collection methods rely on costly human annotation or labor-intensive prompting frameworks. A3T addresses this scalability problem by autonomously annotating diverse ReAct-style trajectories and using them for iterative self-improvement.

1 Introduction

  • Existing trajectory collection relies either on expert demonstrations or agent frameworks, both requiring substantial human effort and limiting scalability.Training trajectories contain observations, internal reasoning, and external actions.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • A3T uses an ActRe agent to provide textual rationales for sampled actions, enabling autonomous annotation of ReAct-style trajectories.The framework combines sampled external actions with posterior reasoning to create new trajectories.Source: §1 · p. 2
  • Successful trajectories from failed tasks are paired with failed trajectories for contrastive self-training, creating a closed loop as agents accumulate new data across rounds.The loop uses environmental rewards to identify successful trajectories and policy-gradient training to improve the agent.Source: §1 · pp. 2–3
  • On AlfWorld, A3T reaches 96% single-trial success and 100% after four iterative rounds, while WebShop reaches 49% single-trial success and 54.8% after four rounds.WebShop performance matches average human performance at 1 shot and approaches human experts after iterative refinement.Source: §1 · p. 3

2 A3T for Closed-Loop Self-Improvement

A3T forms a closed loop in which ActRe autonomously annotates exploratory actions and contrastive policy-gradient training uses successful and failed trajectories to improve the agent over repeated rounds.

2 A3T for Closed-Loop Self-Improvement

  • A3T’s closed loop combines autonomous trajectory annotation with ActRe and contrastive self-training using accumulated trajectories.The process is organized into the ActRe annotation stage and the contrastive training stage.Source: §2 · p. 3

2.1 Autonomous Trajectory Annotation with ActRe

  • ActRe reverses ReAct’s order by explaining a sampled external action afterward, then supplying that rationale as the condition for a new ReAct trajectory.This synchronizes the ReAct and ActRe agents while removing the need for human in-progress supervision.Source: §2.1 · p. 3
  • The environment’s terminal reward automatically labels each synthesized trajectory’s success, allowing the agent to generate diverse annotated trajectories without human annotation.A changed action receives a corresponding altered rationale before execution.Source: §2.1 · p. 3

2.2 Contrastive Self-Training

  • Policy-gradient training assigns binarized rewards to accumulated trajectories and retains successful trajectories alongside failed ones for contrastive learning.The objective jointly trains the language model on reasoning and action sequences, while failed trajectories receive R(τf) = −1.Source: §2.2 · p. 4
  • Multiple successful trajectories are required when failed trajectories receive R(τf) = −1, because K > 1 avoids the instability associated with a single success-failure contrast.The trajectory set grows across rounds and is reused for subsequent fine-tuning.Source: §2.2 · p. 4
  • Round 0 bootstraps training with 1-shot ReAct supervised fine-tuning, while later rounds use exploration, autonomous annotation, accumulated trajectories, and binarized-reward self-training.The reported AlfWorld tables cover single-trial and iterative-refinement success rates across unseen scenarios.Source: §2.2 · p. 5, Table 1 · p. 5, Table 2 · p. 5

3 Experiments

A3T is evaluated on multi-step decision-making benchmarks using four rounds of trajectory collection and training, with comparisons against prompting, fine-tuned, and iterative-refinement baselines. It reaches 96% single-trial success on AlfWorld and matches average human WebShop performance in one trial, while additional trials improve results further.

3 Experiments

  • The experiments cover AlfWorld and WebShop, both of which require language agents to complete multi-step tasks through environmental decisions.AlfWorld involves sequential low-level actions toward embodied goals, while WebShop requires searches and button clicks to purchase a suitable item.Source: §3 · p. 5, §3.1 · p. 5, §3.2 · p. 6
  • A3T uses four rounds of trajectory collection and training, bootstraps with ReAct prompting, and fine-tunes Mistral-7B-Instruct-v0.2 with QLoRA without training on test tasks.The initial ReAct and later ActRe prompting agents use gpt-3.5-turbo-instruct-0914.Source: §3 · p. 5

3.1 AlfWorld

  • A3T reaches 96% single-trial success on AlfWorld and 100% success with four accumulated test trials, outperforming or matching strong baselines with less training data and a smaller QLoRA update.LM-BUTLER uses 355M fully fine-tuned parameters and all 3,553 training tasks, whereas A3T uses a 26M-parameter QLoRA update and 600 training tasks.Source: §3.1 · p. 6, Table 3 · p. 6

3.2 WebShop

  • On WebShop, the single-shot A3T model significantly surpasses GPT-4 prompting on AgentBoard’s easy and hard splits.The comparison uses average reward on the two difficulty splits.Source: Table 5 · p. 7, §3.2 · p. 7
  • Using only reward-1 trajectories gives the best WebShop ablation performance, while Table 7 compares supervised fine-tuning and policy-gradient reward configurations.The ablation varies trajectory filtering thresholds and failed-trajectory rewards.Source: Table 7 · p. 8
  • A3T matches average human WebShop performance in one trial and reaches 54.8% success with four trials, approaching human expert performance at 59.6%.Its one-trial reward and success rate are 73.9 and 49.0%, compared with human averages of 75.5 and 50.0%.Source: Table 4 · p. 7, §3.2 · p. 8

4 Ablation Studies

The ablations examine reward filtering, policy-gradient training, and proprietary-model fine-tuning within A3T. Results favor strict success filtering, binarized policy-gradient rewards, and show that open-source contrastive refinement can surpass proprietary fine-tuning after Round 1.

4.1 Variants of the Self-Training Techniques

  • On WebShop, trajectory filtering with reward R = 1 achieves the best performance compared with thresholds R ≥0.75 and R ≥0.5, in an initial-round comparison with supervised ReAct fine-tuning.Source: §4.1 · p. 8
  • Policy-gradient training with binarized rewards produces higher task-performance gains than supervised alternatives, while ±1 rewards significantly improve success rate.The authors leave advanced RL and RLAIF integration for future work.Source: §4.1 · p. 8

4.2 Experiments with gpt-3.5-turbo-1106 fintuning

  • Round-0 supervised training shows Mistral-7B-Instruct-v0.2 surpassing gpt-3.5-turbo-1106 in AlfWorld, while the proprietary model performs better in WebShop.Tables 8 and 9 report the corresponding task-level comparisons.Source: §4.2 · p. 9, Table 8 · p. 9, Table 9 · p. 9
  • After Round-1 self-training, the open-source model achieves a higher test success rate despite composing trajectory sets with quality on par with the proprietary model.The proprietary service offers supervised fine-tuning only, highlighting the importance of A3T’s contrastive fine-tuning.Source: §4.2 · p. 9, Table 10 · p. 9

5 Conclusion

A3T autonomously annotates ReAct-style trajectories through ActRe and uses environmental feedback with contrastive self-training to improve language agents. Experiments on AlfWorld and WebShop demonstrate superiority over multiple strong baselines.

5 Conclusion

  • A3T uses ActRe to generate textual rationales for arbitrary actions, enabling autonomous trajectory synthesis for contrastive self-training.Policy-gradient methods with binarized rewards support the self-training process.Source: §5 · p. 9
  • Experiments on AlfWorld and WebShop demonstrate A3T’s superiority over multiple strong baselines.Source: §5 · p. 9

A Implementation Details

A3T composes trajectories by probabilistically sampling actions and querying ActRe for posterior rationales, then trains the policy with accumulated trajectories. The implementation uses task-specific collection limits and QLoRA fine-tuning settings.

A Implementation Details

  • The policy samples an external action with probability p = 0.5 and queries ActRe for a posterior reason when sampling occurs.After three consecutive invalid self-selected actions, an external action is sampled with ActRe annotation.Source: A Implementation Details · p. 13
  • Trajectory collection gathers 40 trajectories per failed AlfWorld task, while WebShop collection stops after success or 20 annotated trajectories following three forced trajectories.Source: A Implementation Details · p. 13
  • The main experiments fine-tune Mistral-7B-Instruct-v0.2 with QLoRA, using 10 epochs for Round 0 and 6 epochs for Rounds 1–3 with checkpoint averaging.The configuration uses rank 16, α = 32, and q_proj and v_proj modules.Source: A Implementation Details · p. 13
  • The ActRe prompts for AlfWorld and WebShop implement posterior reasoning that explains external actions.The method’s prompt formulation is described as inverting ReAct’s causality.Source: Table 11 · p. 13, Table 12 · p. 14

B Case Studies of the Composed Trajectories

WebShop case studies show ActRe-assisted trajectories exploring alternative actions and continuing beyond failures. One trajectory progresses across pages and options until it reaches a perfect instruction match.

B Case Studies of the Composed Trajectories

  • The case-study figures present several WebShop trajectories composed with ActRe, including a trajectory whose second part is continued in Fig. 5.Source: B Case Studies of the Composed… · p. 15, Figure 4 · p. 17
  • On task 518, ActRe assistance changes a failed item-content decision into success by annotating the sampled “[white | primary]” action.Source: Figure 2 · p. 15
  • On task 512, the annotated trajectory explores clicking the seventh item “[B09QCYNF9R]” and supplies an ActRe-generated reason.Source: Figure 3 · p. 16
  • On task 754, the ActRe-assisted trajectory continues beyond an imperfect 0.6667 reward and ultimately selects an item producing a perfect match.The trajectory explores multiple items and pages before purchasing item [B07JCHPH1H].Source: Figure 5 · p. 18

C Additional Experimental Results

Additional experiments report task-level AlfWorld comparisons, held-out split statistics, and training-data composition across iterative finetuning rounds. The results also document that failed-sentence proportions satisfy the K > 1 condition empirically in both AlfWorld and WebShop.

Task statistics

  • The experiments include statistics for training, validation, and held-out testing splits, with most results evaluated on 134 unseen test scenarios.Table 13 distinguishes training, validation, and testing tasks and identifies the 134-scenario unseen-test evaluation setting.Source: Table 13 · p. 19, §C.1 · p. 19

AlfWorld task results

  • AlfWorld task-level results show that the proposed agents outperform LM-BUTLER on 140 in-distribution test scenarios with a single trial.The table reports success rates separately for each AlfWorld task type.Source: Table 14 · p. 19

AlfWorld training-data statistics

  • The low proportion of failed sentences in AlfWorld empirically satisfies the K > 1 constraint used in the training analysis.This condition is tied to the sentence statistics collected for each finetuning round.Source: §C.1 · p. 19
  • AlfWorld training sets contain no −1-weighted sentences in Round 0, while later rounds use sentence statistics to track trajectory-based self-training data.Round 0 uses ReAct prompting trajectories for supervised-finetuning bootstrapping.Source: Table 16 · p. 19, §C.1 · p. 19

WebShop training-data statistics

  • WebShop training data contains about 10% failed sentences, which still satisfies K > 1 empirically and is associated with stable training.Round 0 uses no failed sentences because it performs supervised finetuning with ReAct trajectories.Source: §C.2 · p. 20, Table 17 · p. 20
Loading 2403.14589v3…