PaperlayerView exact revision on arXiv

Source-linked AI summary

Re-ReST: Reflection-Reinforced Self-Training for Language Agents

Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, Nanyun Peng

arXiv:2406.01495v3cs.CL

TL;DR

Language-agent self-training could reduce dependence on costly or unavailable stronger-model demonstrations, but difficult tasks make high-quality samples hard to obtain. Re-ReST uses environmental feedback to refine weak samples and enrich training data. Experiments report improvements across language-agent tasks, including further gains on HotpotQA and AlfWorld, while the approach relies on ground-truth feedback during training.

  • Problem

    Stronger-model demonstrations are not always available, and obtaining high-quality self-generated samples is difficult for challenging language-agent tasks.

    Source: §1 · pp. 1–2

  • Method

    Re-ReST uses a reflector to refine low-quality agent generations with environmental feedback and adds successful corrections to self-training data.

    Source: §1 · p. 2

  • Results

    Across tasks, Re-ReST outperforms self-training; it further improves performance by 2.0% on HotpotQA and 14.1% on AlfWorld.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Re-ReST efficiently obtains high-quality self-training samples, and standard inference uses only the trained agent without extra computational burden.

    Source: §1 · p. 2, §5 · p. 9

    The method assumes ground-truth feedback during training, which can be difficult to obtain reliably in diverse real-world settings and broader language modeling.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impractical. In this paper, we investigate the use of self-training in language agents, which can generate supervision from the agent itself, offering a promising alternative without relying on human or stronger model demonstrations. Self-training, however, requires high-quality model-generated samples, which are hard to obtain for challenging language agent tasks. To address this, we present Reflection-Reinforced Self-Training (Re-ReST), which uses a \textit{reflector} to refine low-quality generated samples during self-training. The reflector takes the agent's output and feedback from an external environment (e.g., unit test results in code generation) to produce improved samples. This technique enhances the quality of inferior samples and efficiently enriches the self-training dataset with higher-quality samples. We conduct extensive experiments on open-source language agents across tasks, including multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation. The results demonstrate the effectiveness of self-training and Re-ReST in language agent tasks, with self-training improving baselines by 7.6\% on HotpotQA and 28.4\% on AlfWorld, and Re-ReST further boosting performance by 2.0\% and 14.1\%, respectively. Our studies also confirm the efficiency of using a reflector to generate high-quality samples for self-training. Moreover, we demonstrate a method to employ reflection during inference without ground-truth feedback, addressing the limitation of previous reflection work. Our code is released at https://github.com/PlusLabNLP/Re-ReST.

1 Introduction

The introduction motivates self-training as a way to improve language agents without depending on stronger-model demonstrations, while noting that high-quality samples are difficult to obtain. Re-ReST uses environmental feedback to refine weak samples, and the paper also explores reflection at inference without ground-truth feedback.

Motivation

  • Self-training can autonomously improve agents and reduce reliance on stronger, sometimes unavailable models, supporting a fully open-source approach.The introduction contrasts this approach with training data generated by stronger models, including GPT-4.Source: §1 · p. 1, Figure 1 · p. 1
  • High-quality samples are difficult to obtain for multi-step reasoning and long-horizon planning, and discarding weak samples wastes opportunities to improve them.The paper notes that model-only sampling can be inefficient for these tasks.Source: §1 · p. 2

Contribution

  • Re-ReST uses environmental feedback, such as execution success or unit-test results, to refine low-quality samples and enrich the self-training dataset.After training, only the agent is used for inference, avoiding additional inference-time computation.Source: §1 · p. 2

Experiments

  • Re-ReST further improves self-training by 2.0% on HotpotQA and 14.1% on AlfWorld, while self-training improves few-shot baselines by 7.6% and 28.4%, respectively.The experiments use open-source LLMs across multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation.Source: §1 · p. 2

Contribution

  • The paper explores test-time reflection with self-consistency decoding to improve performance without ground-truth feedback.This addresses a feedback requirement attributed to previous work (Huang et al., 2024).Source: §1 · p. 2

2 Method: Re-ReST

Re-ReST extends self-training for language agents by using environmental feedback to correct low-quality generations and add successful corrections to training data. It combines agent-generated and reflector-generated data for training, while using the agent alone for standard inference.

2 Method: Re-ReST

  • Self-training generates pseudo-labels with the base model, selects examples by a scoring function, and finetunes on the selected subset; for language agents, labels are thought-action trajectories.Re-ReST improves this generation process by using reflection to correct weak samples with external feedback and add successful corrections to the data.Source: §2 · p. 2

2.1 Components

  • The language agent generates thoughts and actions, while a separate reflector uses task information, the prior generation, and environmental feedback to produce a correction.Feedback may include success status or error information; the paper does not share model parameters between agent and reflector.Source: §2.1 · pp. 2–3

2.2 Data Generation

  • The agent samples k generations and retains those above a score threshold; generations below threshold receive one feedback-guided refinement, and successful corrections are retained.The study uses k = 3 and limits reflection to one iteration for efficiency.Source: §2.2 · p. 3

2.3 Model Training and Inference

  • The reflector is trained on agent-generated and reflector-generated data, then the agent is trained on their combined datasets; standard inference uses the agent alone.The paper also identifies preference optimization objectives such as DPO as a possible training objective and explores reflection with self-consistency at inference.Source: §2.3 · pp. 3–4

3 Experiments

Experiments use open-source models across multi-hop reasoning, sequential decision-making, code generation, visual question answering, and text-to-image generation. Self-training and Re-ReST improve performance in these settings, while further analyses test the reflector’s data-quality benefits and uses at inference and in preference optimization.

3.1 Multi-Hop Reasoning

  • HotpotQA EM rises from 20.0% to 27.6% for Llama-2 and from 30.0% to 34.4% for Llama-3 with self-training.Only 37.1% and 48.3% of training instances are correctly solved by the respective agent models; with 0.5k GPT-generated instances, the free-Wikipedia-API agents achieve comparable or better performance than methods using commercial APIs.Source: §3.1 · p. 5, Table 1 · p. 6

3.2 Sequential Decision-Making

  • Self-training raises ALFWorld success rate from 8.9% to 37.3% in unseen environments, and Re-ReST adds a 14.1% uplift.The experiments do not provide access to complete successful trajectories.Source: §3.2 · p. 5, Table 2 · p. 6

3.3 Programming: Code Generation and Visual Question Answering

  • On MBPP, self-training raises performance from 48.6% to 54.5%, and Re-ReST further improves the pass rate; GQA results also improve with the method.Table 3 summarizes improvements on code generation and visual programming.Source: §3.3 · p. 6, Table 3 · p. 6

3.4 Text-to-Image Generation

  • On VPGen evaluated with VPEval, Re-ReST further improves on self-training across all text-to-image dimensions.BLIP-2 evaluation results provide environmental feedback to the reflector.Source: Table 4 · p. 7, §3.4 · pp. 7–8

3.5 Analysis

  • A pretrained reflector improves self-training, while training it specifically for self-correction yields further agent-performance gains.The comparison is reported in Table 5.Source: Table 5 · p. 7, §3.5 · p. 8
  • Sampling more generations initially improves performance but plateaus, and it does not match Re-ReST in performance or solved training instances.The analysis varies the number of generations per instance from one to six; more generated data does not always improve performance.Source: Figure 3 · p. 7, §3.5 · p. 8
  • Self-consistency enables test-time reflection without ground-truth feedback; three agent samples plus three reflection samples improve over six self-consistency model samples.The method reflects on each sampled answer and aggregates answers by majority vote.Source: Table 6 · p. 7, §3.5 · p. 8
  • Integrating DPO with Re-ReST generally improves or matches supervised training on positive samples alone.The reflector’s incorrect-to-correct generations provide negative-positive pairs for preference optimization.Source: §3.5 · p. 8, Table 7 · p. 8

4 Related Work

Prior work studies language agents, self-training, and self-correction; Re-ReST bootstraps agents with their own reflective ability rather than distilling knowledge from stronger external models. The section distinguishes its training-time use of environmental feedback from reflection methods focused on inference.

Language Agents

  • Re-ReST outperforms self-training methods across various tasks, while prior language-agent finetuning often distills from stronger LLMs such as GPT-4 [Chen et al., 2023; Yin et al., 2024].The paper instead bootstraps agent performance using its own reflective ability.Source: §4 · p. 9
  • Language agents combine model reasoning and acting, with ReAct generating reasoning traces before actions and follow-up work exploring inference-time reflection and planning [Yao et al., 2022; Shinn et al., 2023; Yao et al., 2023; Yang et al., 2023].Agents can act through generated commands or external tool APIs.Source: §4 · p. 9

Self-Training for Language Models

Self-Reflection/Self-Correction for Language Models

  • Many self-correction methods focus on inference, whereas Re-ReST uses oracle environmental feedback during training and distills corrected generations into the agent for inference [Huang et al., 2024].The cited discussion notes that high-quality external feedback is essential for existing self-correction methods and may be unavailable at test time.Source: §4 · p. 9

5 Conclusion

Re-ReST uses a reflector to obtain high-quality self-training samples, and experiments report that it outperforms self-training methods across various tasks. The paper identifies improving reflection and agent-reflector training paradigms as future directions.

5 Conclusion

  • Re-ReST outperforms self-training methods across various tasks, with experiments supporting the efficiency and effectiveness of incorporating reflection.Source: §5 · p. 9
  • Re-ReST uses a reflector to efficiently obtain high-quality samples for language-agent self-training.Source: §5 · p. 9
  • Future work can improve the reflection mechanism and develop better training paradigms for the agent and reflector.Source: §5 · p. 9

Limitations

The approach assumes ground-truth feedback during training, which can be difficult to acquire in diverse real-world settings. The paper also notes potential bias amplification through self-training and calls for careful calibration before deployment.

Limitations

  • Accurate ground-truth feedback can be difficult to acquire in diverse real-world scenarios, constraining the approach beyond language-agent tasks.The paper says its study focuses on language-agent tasks rather than broader general language modeling.Source: Limitations · p. 10
  • Self-training may amplify biases encoded in LLMs, so the authors call for careful calibration before deployment.Source: Limitations · p. 10
Loading 2406.01495v3…