PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentRefine: Enhancing Agent Generalization through Refinement Tuning

Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, Weiran Xu

arXiv:2501.01702v2cs.AIcs.CLcs.RO

TL;DR

Existing agent-tuning data can perform well on held-in tasks yet generalize poorly to held-out environments. AgentRefine synthesizes diverse environments and trains models to refine actions from feedback; it reports stronger held-out performance, including a 13.3% lead over Agent-FLAN in Sciworld Success Rate.

  • Problem

    Existing agent-tuning methods perform well on held-in tasks but have poor generalizability to held-out sets, while models can repeat erroneous actions despite negative environment feedback.

    Source: §1 · p. 2

  • Method

    AgentRefine synthesizes diverse agent environments and tasks, then trains on trajectories in which a strong LLM refines erroneous actions using environment observations.

    Source: §1 · p. 2

  • Results

    AgentRefine leads Agent-FLAN by 13.3% in Sciworld Success Rate and shows significant advantages on held-out tasks over other agent works.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The authors conclude that self-refinement enables robustness and that diversity of agent environments and thoughts further enhances performance.

    Source: §11 · p. 10

    The authors identify potential misuse, including enhancing LLM capabilities for malicious agent tasks such as generating attack codes.

    Source: ETHICS STATEMENT · pp. 13–14

Abstract

from arXiv · show

Large Language Model (LLM) based agents have proved their ability to perform complex tasks like humans. However, there is still a large gap between open-sourced LLMs and commercial models like the GPT series. In this paper, we focus on improving the agent generalization capabilities of LLMs via instruction tuning. We first observe that the existing agent training corpus exhibits satisfactory results on held-in evaluation sets but fails to generalize to held-out sets. These agent-tuning works face severe formatting errors and are frequently stuck in the same mistake for a long while. We analyze that the poor generalization ability comes from overfitting to several manual agent environments and a lack of adaptation to new situations. They struggle with the wrong action steps and can not learn from the experience but just memorize existing observation-action relations. Inspired by the insight, we propose a novel AgentRefine framework for agent-tuning. The core idea is to enable the model to learn to correct its mistakes via observation in the trajectory. Specifically, we propose an agent synthesis framework to encompass a diverse array of environments and tasks and prompt a strong LLM to refine its error action according to the environment feedback. AgentRefine significantly outperforms state-of-the-art agent-tuning work in terms of generalization ability on diverse agent tasks. It also has better robustness facing perturbation and can generate diversified thought in inference. Our findings establish the correlation between agent generalization and self-refinement and provide a new paradigm for future research.

1 INTRODUCTION

Existing agent-tuning methods perform well on held-in tasks but generalize poorly to held-out environments. AgentRefine trains models to refine actions using environment feedback, and reports stronger generalization, perturbation robustness, and thought diversity.

1 INTRODUCTION

  • Existing methods perform well on held-in tasks but generalize poorly to held-out sets, a gap also illustrated by a clear held-in versus held-out performance drop.Agent-FLAN was trained using Alfworld for the held-in comparison.Source: §1 · p. 2, §2 · p. 2
  • AgentRefine synthesizes diverse environments and tasks, then prompts a strong LLM to refine erroneous actions using observations and verifier feedback.The authors call tuning on this self-refinement data Refinement Tuning.Source: §1 · p. 2
  • Across five agent evaluation tasks, AgentRefine significantly outperforms state-of-the-art agent-tuning work.The experiments use LLaMA3 and Mistral-v0.3.Source: §1 · p. 2
  • Refinement tuning exhibits greater robustness to environmental changes than normal-trajectory tuning under small perturbations such as altered action descriptions.The cited comparison concerns perturbations to agent environments.Source: §1 · p. 2
  • The authors report that diverse agent environments and thoughts contribute to refinement tuning.They also state that refinement tuning helps agents explore viable actions in bad situations.Source: §1 · p. 2

2 RETHINK THE GENERALIZATION OF AGENT-TUNING

  • Agent-FLAN fits the held-in environment but struggles with subtle perturbations and new tasks; after an error action, it can remain stuck in the same error mode.The authors characterize traditional approaches as memorizing correct trajectories rather than generalizing.Source: §2 · p. 3

3 METHODOLOGY

AgentRefine constructs multi-turn trajectories with errors and refinements across synthesized environments, then verifies the data. Refinement tuning converts these trajectories into supervised turns and adjusts the loss according to action correctness.

3.1 DATA CONSTRUCTION

  • The data pipeline generates persona-based scripts, simulates multi-turn interactions containing errors and refinements, and verifies scripts and trajectories before retaining data.Scripts specify environments, tasks, available actions, and validation rules; verification can prompt regeneration or continuation.Source: §3.1 · pp. 3–4

3.2 GENERATION SETUP

  • The authors use gpt-4o-2024-05-13 to generate scripts and trajectories, retaining trajectories that pass verification within four LLM calls.They use one-shot trajectory examples and three-shot script examples to guide format and diversity.Source: §3.2 · p. 4

3.3 REFINEMENT TUNING

  • Refinement tuning converts DM observations into user turns and player thoughts and actions into ReAct-formatted assistant turns, with the loss conditioned on action correctness.The supplied prose defines the correctness indicator as 1 for a correct action and 0 otherwise.Source: §3.3 · p. 4
  • The authors note that actions with near-infinite parameter spaces, such as code editing, answering, and searching, are not verified during script or trajectory generation.Source: §3.3 · p. 4

4 EXPERIMENTS

AgentRefine performs better on held-out tasks than other agent-tuning works, while experiments examine refinement tuning, robustness to perturbations, and diversity. Results also cover data scaling, Best-of-N performance, and the experimental setup across five agent tasks.

4.2 MAIN RESULTS

  • 13.3% is AgentRefine’s lead over Agent-FLAN in Sciworld Success Rate; AgentRefine also shows advantages on held-out tasks overall.It does not outperform methods on their held-in tasks, according to the reported comparison.Source: §4.2 · p. 5, Table 1 · p. 6
  • Masking refinement-trajectory tokens or training without refinement trajectories leads to substantial performance drops across tasks.Masking refinement tokens is associated with an approximately 43% drop in Sciworld; removing refinement trajectories produces drops of similar magnitude across tasks.Source: §4.2 · pp. 5–6, Table 2 · p. 6
  • Training on erroneous-turn tokens has adverse effects, including a nearly 75% drop in Sciworld.The authors report that continued learning from erroneous trajectories can drive performance lower and argue that masking erroneous turns is crucial for refinement tuning.Source: §4.2 · p. 6, Table 2 · p. 6
  • Performance increases significantly as AgentRefine training data scales from 4k to 64k.Figure 5 presents model performance as training data size increases.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3 ROBUSTNESS ANALYSIS

  • Under semantic-preserving action perturbations, AgentGym’s average Success Rate drops 25.6% and Agent-FLAN’s drops 30.4%, while AgentRefine’s average increases 3.7%.The reported standard deviation for the baselines is close to 20%; AgentRefine’s is 3.73%.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 DIVERSITY ANALYSIS

  • AgentRefine’s thought data are more widely distributed and numerous than Agent-FLAN’s and AgentGym’s, indicating greater thought diversity.The analysis sampled 8,100 thoughts and visualized their distribution with t-SNE.Source: §4.4 · p. 7, Figure 6 · p. 7
  • The measured similarity between AgentRefine environments and test environments is below 0.5, indicating some environmental difference.The comparison sampled instructions from AgentRefine and the test sets.Source: §4.4 · p. 7, Figure 7 · p. 7
  • AgentRefine’s Best-of-N improvement averages over 25%, and Best-of-N performance is better than greedy decoding for all three training datasets.The authors suggest AgentRefine’s larger gain may relate to its thought diversity and quality.Source: §4.4 · pp. 7–8, Table 4 · p. 8

5 SYNTHESIS FROM OPEN SOURCE MODEL

AgentRefine data synthesized with DeepSeek-v2.5 outperforms Agent-FLAN on held-out tasks in the reported comparison, including a 11.6% advantage in BabyAI Success Rate. The results also show a gap between data synthesized with DeepSeek and GPT-4o.

5 SYNTHESIS FROM OPEN SOURCE MODEL

  • 11.6% is AgentRefine’s lead over Agent-FLAN in BabyAI Success Rate when the AgentRefine data are synthesized with DeepSeek-v2.5.The comparison uses 4,000 training examples and reports advantages on held-out tasks.Source: §5 · p. 8, Table 5 · p. 8
  • AgentRefine data synthesized with DeepSeek-v2.5 performs better on held-out tasks than Agent-FLAN, while a noticeable gap remains between DeepSeek- and GPT-4o-synthesized data.The authors interpret the gap as indicating that more capable synthesis models yield higher-quality training data and greater performance gains.Source: §5 · p. 8, Table 5 · p. 8

6 GENERLIZATION IN REASONING TASK

AgentRefine outperforms other methods on HotpotQA’s exact match and F1 metrics. Case studies describe how it revises actions in Jericho and SciWorld, while a human comparison finds GPT-4’s turn judgments align with human judgments at reported rates of 94% and 82%.

6 GENERLIZATION IN REASONING TASK

  • AgentRefine outperforms other methods on HotpotQA’s EM and F1 metrics, extending the reported generalization results to a reasoning task.The evaluation uses HotpotQA with Wikipedia search as the environment.Source: §6 · p. 8, Figure 8 · p. 8

7 CASE STUDY

  • In Jericho, Agent-FLAN repeats an unsuccessful go-to-cell/check-valid-actions loop, whereas AgentRefine tries new methods when its actions do not achieve the goal.The case study describes AgentRefine as avoiding repeated incorrect actions.Source: §7 · pp. 8–9, Figure 9 · p. 9
  • In SciWorld, AgentRefine uses observations and short-term memory to revise a mistaken decision and reach the bedroom, unlike Agent-FLAN’s reliance on memorized Alfworld actions and locations.The example concerns finding a fork after searching for a thermometer.Source: §7 · p. 9, Figure 9 · p. 9

8 GPT-4 JUDGEMENT’S RELIABILITY

  • GPT-4 judgments align with human judgments on 94% of turns GPT-4 labels right and 82% of turns it labels wrong.The comparison is based on randomly sampled turns from 50 trajectories.Source: §8 · p. 9, Figure 10 · p. 9

9 GENERALIZATON BETWEEN GENERAL DATA AND AGENT DATA

General data such as ShareGPT continues to improve held-out task performance, while AgentRefine combines diverse agent data with refinement steps and reports stronger generalization. The conclusion also identifies misuse risks from enhanced agent capabilities.

9 GENERALIZATON BETWEEN GENERAL DATA AND AGENT DATA

  • General data such as ShareGPT continually improves held-out task performance.Both AgentTuning and Agent-FLAN reported benefits from general data, and this paper reports the same pattern.Source: §9 · p. 10

10 RELATED WORK

  • The related-work review says prior approaches remain limited by dataset overfitting, reasoning failures, and incorrect patterns, while synthetic refinement improves generalization.It distinguishes AgentRefine’s emphasis on refinement from AgentGen’s shared focus on synthesizing diverse environments.Source: §10 · p. 10
  • AgentRefine synthesizes queries and responses with verification, and trains step-level refinement from environment feedback rather than requiring instance-level pipeline guidance.The related-work discussion contrasts its step-level refinement with compulsory instance-level guidance and notes its joint query-response synthesis.Source: §10 · p. 10

11 CONCLUSION

  • The conclusion reports that AgentRefine outperforms agent-tuning work on diverse benchmarks and associates self-refinement with robustness.It also says diversity in agent environments and thoughts further enhances performance.Source: §11 · p. 10

ETHICS STATEMENT

  • The ethics statement flags potential misuse, including using enhanced LLM capabilities for malicious agent tasks such as generating attack code.The authors say they use security-processed Persona-Hub data and review generated tasks and environmental information for safety.Source: ETHICS STATEMENT · pp. 13–14

A TASKS STATISTIC

The appendix defines a weighted held-out task score and reviews task settings, prior agent-tuning approaches, data synthesis, training, dataset comparisons, and IND filtering.

A TASKS STATISTIC

  • The held-out task score weights BabyAI, SciWorld, PDDL, and Jericho scores by 112, 90, 60, and 20, respectively, over 282.The task-statistics table reports the test-data counts and domains for the five tasks.Source: A TASKS STATISTIC · p. 14, Table 6 · p. 14

B THE HISTORY OF AGENT-TUNING

  • Prior agent tuning often used fixed environments, improving similar held-in tasks but dropping sharply when tasks changed; later work added general data and diversified data or environments.The authors say these approaches still face single-dataset overfitting, reasoning stalls, and incorrect reasoning patterns, which synthetic data and refinement steps are intended to address.Source: B THE HISTORY OF AGENT-TUNING · p. 14

C SYNTHESIS DATA WITH PERSONA

  • Persona-Hub contains one billion filtered personas, and the passage reports that a cosine-similarity filter of 0.5 can yield one million diverse personas.It also reports that Persona-Hub-generated data diversity had not saturated at one million examples in the math-problem scaling experience.Source: C SYNTHESIS DATA WITH PERSONA · p. 14

D TRAINING HYPER PARAMETER

  • The default AgentRefine setting uses 32,000 training examples, a 5e-6 learning rate, batch size 64, and 10 training epochs.The reported checkpoint is selected by best average results; maximum sequence length is 8192 for 7/8B models and 4096 for 70B models.Source: D TRAINING HYPER PARAMETER · p. 14

E COMPARISON AMONG AGENT DATASETS

  • AgentRefine scales its data and includes refinement steps; its comparison with AgentGen centers on shared environment synthesis but greater emphasis on refinement.The comparison table covers trajectory counts, environment and trajectory sourcing, held-in tasks, and refinement availability.Source: E COMPARISON AMONG AGENT DATASETS · p. 15, Table 7 · p. 15

F IND FILTERING EXPERIMENTS

G REFLEXION EXPERIMENT

AgentRefine outperforms other methods when paired with Reflexion, including on Alfworld despite lacking Alfworld training data. The appendix also reports robustness, instruction-following, and perturbation analyses.

G REFLEXION EXPERIMENT

  • With Reflexion, AgentRefine outperforms other methods, especially on Alfworld despite having no Alfworld training data.The authors say this indicates AgentRefine can use Reflexion more effectively than the other methods.Source: G REFLEXION EXPERIMENT · p. 15, Table 9 · p. 15

H STANDARD DEVIATIONS

  • In most OOD tasks, AgentRefine’s average performance exceeds other methods by at least 2 standard deviations.The table reports averages and standard deviations using temperature 1.0 and 10 samples; underlining marks tasks whose training data came from the same environment.Source: H STANDARD DEVIATIONS · pp. 15–16, Table 10 · p. 16

I ROBUSTNESS ANALYSIS WITH DIFFERENT COMPONENTS

  • With 4,000 training examples, the success-score standard deviation is almost double the baseline, which the authors attribute to training-data quantity being the key robustness factor.The ablation table also varies data size and verification conditions.Source: I ROBUSTNESS ANALYSIS WITH DIFFERENT… · p. 16

J MODEL’S INSTRUCTION-FOLLOWING ABILITY

  • AgentRefine scores approximately 0.2 points above Agent-FLAN on instruction following, while ShareGPT raises both models’ scores by about 2 points.The study uses MT-bench, scored by gpt-4o-2024-05-13.Source: J MODEL’S INSTRUCTION-FOLLOWING ABILITY · p. 16

K PERTURBATION DETAILS

  • Alfworld perturbations test changed action wording or order, spacing changes, and removal of IND data, with the environment revised to match.The listed changes preserve action meaning for the wording perturbations.Source: K PERTURBATION DETAILS · p. 16

L SCRIPT GENERATION

The script-generation schema specifies the environment, player, goal, and available actions through structured fields, including task completion conditions and action constraints.

L SCRIPT GENERATION

  • The player fields include restrictions, while the schema’s Thought field is marked compulsory.The provided schema fragments identify player information and a required Thought string.Source: L SCRIPT GENERATION · p. 17
  • The environment schema includes an initial state and places or objects, whose information may be revealed when examined or when the player enters their receptacle.It distinguishes information shown to the player from information provided only to the environment’s DM.Source: L SCRIPT GENERATION · p. 17
  • The goal must be clear, achievable by one person, and have unique, concrete completion conditions.The schema separately lists the goal and the conditions required to complete it.Source: L SCRIPT GENERATION · p. 18
  • Available actions may include descriptions and special formats, with a verification code specifying the action’s regular expression.The special-format field is required when answering the question and editing code, according to the schema text.Source: L SCRIPT GENERATION · p. 18
  • Action parameters list all possible values, and the environment strictly checks player actions against those values.The schema notes that parameter values may include locations, objects, or items referenced in the environment or completion conditions.Source: L SCRIPT GENERATION · p. 18

M TRAJECTORY GENERATION

Trajectory generation uses alternating player and Dungeon Master turns, with required fields that capture actions, observations, and errors. The schema also records task progress and completion.

M TRAJECTORY GENERATION

  • Dungeon Master turns are even-numbered and include a thought, an observation for the player, and error logs.The Dungeon Master turn number begins at 0; the error logs track parameter, place, and logic errors.Source: M TRAJECTORY GENERATION · pp. 18–19
  • Player actions that fail to match available actions are logged as parameter or place errors, while returning to a prior situation can count as a logic error.The schema’s logic-error example is moving north and then south.Source: M TRAJECTORY GENERATION · p. 19
  • Each turn records progress rate and whether the task is finished, alongside its turn number and role.Progress rate is capped at 1.0 when the task is finished.Source: M TRAJECTORY GENERATION · p. 19
  • Player turns are odd-numbered and include the player’s thought and action, with the action required to match the script’s format and parameters.The player turn number begins at 1.Source: M TRAJECTORY GENERATION · p. 19

N ERROR TURN STATISTICS

AgentRefine contains error-refine sequences of varying lengths, including single-turn cases and sequences longer than three turns.

N ERROR TURN STATISTICS

  • About 16% of all turns belong to one-turn error-refine pairs, while AgentRefine also contains pairs longer than three turns.Figure 13 reports the error-turn statistics for AgentRefine’s 32,000 examples.Source: N ERROR TURN STATISTICS · p. 19, Figure 13 · p. 19

O TRAJECTORY VERIFICATION

Trajectory Verification checks generated trajectories for required formatting, action validity, error-refinement turns, and successful task completion. It retains turns only when they pass the checks.

O TRAJECTORY VERIFICATION

  • Verification rejects trajectories that fail JSON-format or required-key checks, and checks player actions against available actions when the Dungeon Master signals no error.A mismatching player action triggers rejection.Source: O TRAJECTORY VERIFICATION · p. 20
  • A trajectory’s final turn must have no error signal, show task success, and follow at least two error-refinement turns.The pipeline rejects the trajectory if any of these final-turn conditions is unmet.Source: O TRAJECTORY VERIFICATION · p. 20
  • Turns that pass verification are appended to the verified trajectory as the pipeline iterates through the generated trajectory.The algorithm returns the verified trajectory and a signal when a check fails.Source: O TRAJECTORY VERIFICATION · p. 20
Loading 2501.01702v2…