PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentTuning: Enabling Generalized Agent Abilities for LLMs

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, Jie Tang

arXiv:2310.12823v2cs.CLcs.AIcs.LG

TL;DR

Open LLMs perform well on many benchmarks but lag commercial models on complex agent tasks, while prior work has focused mainly on task-specific prompting and frameworks. AgentTuning combines a multi-task interaction-trajectory dataset with hybrid instruction tuning, producing AgentLM models whose agent performance generalizes while general abilities remain strong. AgentLM-70B matches GPT-3.5-turbo on agent tasks according to the paper's conclusion.

  • Problem

    Open LLMs lag commercial models on complex agent tasks, and existing research has not provided an end-to-end way to improve generalized agent abilities while maintaining general LLM capacities.

  • Method

    AgentTuning constructs 1,866 verified trajectories across six agent tasks and mixes them with general-domain instructions to fine-tune Llama 2 into AgentLM.

  • Results

    AgentLM performs strongly on unseen agent tasks while preserving general abilities, and AgentLM-70B matches GPT-3.5-turbo on agent tasks.

  • Takeaways & Limitations

    AgentTuning provides an open approach for generalized agent abilities, with AgentLM-70B offering an open model that matches GPT-3.5-turbo on agent tasks.

  • Takeaways & Limitations

    The instruction-construction process could risk test-data leakage, although the authors report finding no evidence of leakage.

Abstract

from arXiv · show

Open large language models (LLMs) with great performance in various tasks have significantly advanced the development of LLMs. However, they are far inferior to commercial models such as ChatGPT and GPT-4 when acting as agents to tackle complex tasks in the real world. These agent tasks employ LLMs as the central controller responsible for planning, memorization, and tool utilization, necessitating both fine-grained prompting methods and robust LLMs to achieve satisfactory performance. Though many prompting methods have been proposed to complete particular agent tasks, there is lack of research focusing on improving the agent capabilities of LLMs themselves without compromising their general abilities. In this work, we present AgentTuning, a simple and general method to enhance the agent abilities of LLMs while maintaining their general LLM capabilities. We construct AgentInstruct, a lightweight instruction-tuning dataset containing high-quality interaction trajectories. We employ a hybrid instruction-tuning strategy by combining AgentInstruct with open-source instructions from general domains. AgentTuning is used to instruction-tune the Llama 2 series, resulting in AgentLM. Our evaluations show that AgentTuning enables LLMs' agent capabilities without compromising general abilities. The AgentLM-70B is comparable to GPT-3.5-turbo on unseen agent tasks, demonstrating generalized agent capabilities. We open source the AgentInstruct and AgentLM-7B, 13B, and 70B models at https://github.com/THUDM/AgentTuning, serving open and powerful alternatives to commercial LLMs for agent tasks.

1 INTRODUCTION

AgentTuning addresses the gap between open and commercial LLMs on complex agent tasks by combining interaction-trajectory instruction tuning with general-domain data. AgentLM shows generalized agent performance while preserving broad language capabilities.

  • Motivation: Open LLMs substantially lag behind GPT-3.5 and GPT-4 on complex real-world agent tasks despite strong traditional NLP performance.Existing work largely targets prompting or task-specific frameworks rather than improving the LLM's underlying agent capabilities.
  • Approach: AgentTuning combines 1,866 verified interaction trajectories from six agent tasks with a hybrid strategy mixing agent and general-domain instructions.Each trajectory is produced through instruction construction, GPT-4-based interaction, and reward-based filtering.
  • Approach: AgentTuning fine-tunes Llama 2 models into AgentLM-7B, 13B, and 70B variants to improve agent-task performance.The method is intended to enhance agent capabilities while preserving generalization ability.
  • Results: AgentLM generalizes to unseen agent tasks, with AgentLM-70B comparable to GPT-3.5 while maintaining performance on MMLU, GSM8K, HumanEval, and MT-Bench.The reported result covers both agent capabilities and general NLP abilities.
  • Results: Mixing agent and general data matters because training solely on agent data decreases generalization performance, while agent tasks require broad planning and reasoning abilities.The analysis identifies general capabilities as crucial for generalizing across agent tasks.
  • Results: AgentTuning reduces basic errors including formatting mistakes, duplicated generation, and refusal to answer, supporting activation rather than overfitting of agent abilities.The authors interpret the error analysis as evidence that the underlying model already possesses relevant task capability.

2 THE AGENTTUNING APPROACH

AgentTuning improves generalized agent abilities by training on high-quality interaction trajectories from diverse tasks while mixing agent and general-domain instructions. The approach uses filtering and mixture training to support performance on unseen agent tasks without sacrificing general capabilities.

  • Trajectory Interaction: Each trajectory records a conversation history of user inputs and model responses, with a final reward reflecting task completion.The interaction process repeatedly combines model thoughts and actions with environmental feedback until completion or termination.
  • AgentInstruct Construction: AgentInstruct contains interaction trajectories for six diverse agent tasks, constructed through instruction construction, trajectory interaction, and trajectory filtering.The construction process uses existing training sets or task derivation and self-instruction for tasks without training data.
  • Trajectory Interaction: ReAct supplies a thought before each action, attaching detailed Chain-of-Thought rationales to collected interaction trajectories.These rationale traces expose the reasoning process used for agent decisions during training-data collection.
  • Trajectory Filtering: Filtered trajectories outperform unfiltered trajectories on held-in and held-out tasks, showing that data quality matters more than data quantity for agent tasks.The comparison is conducted by fine-tuning 7B models on filtered versus unfiltered trajectories.
  • Mixture Training: Mixture training combines AgentInstruct with general-domain data, and the selected agent-data ratio is η = 0.2.Training solely on agent tasks performs worse on unseen tasks than mixed training, while η is selected by scanning values from 0 to 1 in increments of 0.1.

3 EXPERIMENTS

The experiments evaluate AgentLM on held-in, held-out, and general tasks, finding stronger agent performance than Llama 2 while preserving general capabilities. Analyses attribute gains partly to fewer elementary errors and examine effects of task and instruction composition.

  • 3.1 EVALUATION SETUP: The evaluation covers diverse held-in and held-out agent tasks alongside knowledge, mathematics, coding, and human-preference tasks.Held-in and held-out tasks include multiple agent environments, while general evaluation uses MMLU, GSM8K, HumanEval, and MT-Bench.
  • 3.2 MAIN RESULTS: AgentLM significantly outperforms Llama 2 on held-in and held-out tasks while maintaining performance on general tasks.Held-out improvements reach up to 170%, and AgentLM-13B and AgentLM-70B surpass GPT-4 on several tasks.
  • 3.2 MAIN RESULTS: 70B AgentLM approaches GPT-4 overall on held-in tasks and performs close to GPT-3.5 on held-out tasks.The held-out improvement is reported as +176% for the 70B model, compared with +76% for the 7B model.
  • 3.2 MAIN RESULTS: AgentLM performs on par with Llama 2 across knowledge, mathematics, coding, and human preferences.These dimensions are represented by MMLU, GSM8K, HumanEval, and MT-Bench.
  • 3.3 ERROR ANALYSIS: AgentLM reduces elementary mistakes including invalid actions, repeated generations, formatting errors, and refusals to answer.The error analysis examines ALFWorld, WebShop, and Knowledge Graph, while a case study shows AgentLM adjusting after failure and producing a correct function call.
  • 3.4 ABLATION STUDY: General instructions are crucial for generalization, while individual-task training primarily benefits its corresponding task and may provide cross-task gains.Training solely on agent data improves held-in results but harms generalization across agent and general tasks; Mind2Web shows minimal cross-task enhancement.

4 RELATED WORK

Related work covers the emergence of LLM-based agents and instruction tuning. Prior studies combine reasoning with actions or target particular agent tasks, while instruction tuning studies general multi-task instruction following and zero-shot generalization.

  • LLM-as-Agent: LLM-agent research shifted from reinforcement learning and encoder models toward language-model agents that combine reasoning with actions.ReAct is identified as combining chain-of-thought reasoning with agent actions.
  • LLM-as-Agent: Many studies apply language models to specific agent tasks, including online shopping and web browsing.This reflects task-focused applications within the broader LLM-agent literature.
  • Instruction Tuning: Instruction tuning aligns language models to follow human instructions across multiple general tasks.FLAN and T0 are cited for strong zero-shot generalization after fine-tuning on multiple task datasets.

5 CONCLUSION

The paper addresses the gap between open and commercial LLMs on agent tasks with AgentTuning, combining AgentInstruct trajectories with general-domain instructions to tune AgentLM. AgentLM generalizes to unseen agent tasks while preserving general abilities, and AgentLM-70B matches GPT-3.5-turbo on agent tasks.

  • 5 CONCLUSION: AgentTuning combines 1,866 verified agent interaction trajectories in AgentInstruct with general-domain instructions to tune Llama 2 models.The resulting open models are AgentLM-7B, 13B, and 70B.
  • 5 CONCLUSION: AgentLM performs strongly on unseen agent tasks while preserving performance on MMLU, GSM8K, HumanEval, and MT-Bench.The conclusion presents this as evidence of generalized agent abilities alongside retained general capabilities.
  • 5 CONCLUSION: AgentLM-70B is reported as the first open LLM to match GPT-3.5-turbo on agent tasks.The paper open-sources AgentInstruct and the AgentLM model series.

A HYPER-PARAMETERS

The appendix lists AgentLM training hyperparameters and data-mixture settings. AgentLM uses AgentInstruct with sampling ratio η = 0.2 alongside ShareGPT data.

  • A HYPER-PARAMETERS: Table 6 lists the hyperparameters used for AgentLM training.
  • A HYPER-PARAMETERS: AgentLM results use ShareGPT and AgentInstruct, with AgentInstruct sampling ratio η = 0.2.Within ShareGPT, GPT-3.5 and GPT-4 sampling ratios are 0.8 and 0.2, respectively.

B DATA CONTAMINATION

The authors analyze possible contamination from task derivation and self-instruction using tokenized n-gram matching. They report no evidence of data leakage overall, while identifying higher contamination rates and six dirty ALFWorld examples requiring explanation.

  • Motivation and conclusion: The authors conducted systematic contamination analysis because task derivation and self-instruction could otherwise overestimate performance.They report finding no evidence of data leakage.
  • Analysis method: A token-based analysis matched training and test tokenized 10-grams, allowing at most 4 mismatched tokens.An evaluation example’s contamination rate is defined from contaminated tokens.
  • Overall findings: No dirty examples appeared in most tasks, supporting the absence of data leakage or contamination in the dataset.The summary reports these findings in Table 7.
  • Task-specific findings: Database and Operating System tasks had higher contamination rates, but no dirty examples and mostly clean examples.The authors attribute this pattern to coding-related keyword overlap rather than contamination.
  • Task-specific findings: ALFWorld contained 6 dirty examples, partly because short task descriptions make the 80% threshold easier to reach.The authors also found near-duplicate task descriptions differing by only one or two words.

C PROMPT FOR DATA CONSTRUCTION

The prompt-construction appendix specifies templates for generating diverse Database and Operating System tasks. Database prompts require structured MySQL tasks, while Operating System prompts define realistic Linux interactions, initialization, solutions, and evaluation procedures.

  • C.1.1 DATABASE: Database prompts ask Benchmarker-GPT to generate diverse database-related tasks in JSON without extra text.They encourage varied themes, entities, and randomly selected names.
  • C.1.1 DATABASE: Each Database task specifies a description, valid MySQL label, table name, table information, columns, types, and rows.Column types must be supported by MySQL, while spaces and special characters are allowed in column names.
  • C.1.2 OPERATING SYSTEM: Operating System prompts request realistic Linux tasks that encourage complex, multi-round interaction with the shell.The task description should use explicit files or directories rather than opaque references.
  • C.1.2 OPERATING SYSTEM: The Operating System execution process initializes an Ubuntu environment, runs reference code, and lets students interact before submitting answers.An example solution script is also required for evaluation.
  • C.1.2 OPERATING SYSTEM: Operating System evaluation supports exact_match, bash_script, integer_match, size_match, and string_match procedures.Integer, size, and string matching normalize outputs in task-specific ways.
  • C.1.2 OPERATING SYSTEM: The Operating System prompt separately asks the generator to produce a question satisfying the specified task constraints.The request appears after the format and execution instructions.
  • C.1.2 OPERATING SYSTEM: The prompt requires a concise Linux-related problem statement, optional solution hints, initialization code, an example script, and evaluation parameters.Executable-script tasks use bash_script evaluation rather than output matching.

C.2 TASK DERIVATION

The task-derivation prompts construct Database trajectories from questions, reference SQL, and model interactions, while requiring structured JSON thoughts and answers. They emphasize concise, logical reasoning and correctness against database outputs.

  • C.2.1 DATABASE (Thought Construction): The Database thought-construction section presents JSON examples containing intermediate thoughts, evidence, conversations, SQL, and committed answers.These examples illustrate how an agent should reason over database questions and results.
  • C.2.1 DATABASE (Thought Construction): The task asks the model to fill in agent thoughts using supplied evidence while reasoning step-by-step and remaining concise.The conversation log and response are both represented in JSON format.
  • C.2.1 DATABASE (Thought Construction): Another example filters movies released in 1945, sorts them by popularity descending, and returns the first five titles.Its demonstrated query uses WHERE, ORDER BY, and LIMIT clauses.
  • C.2.1 DATABASE (Thought Construction): Database prompts require responses in pure JSON with either a thought-and-SQL structure or a final commit structure.Malformed JSON is judged as an immediate failure.
  • C.2.1 DATABASE (Thought Construction): The prompt requires accurate answers that match the correct database result and allows a flexible commit field for database modifications.The agent receives raw MySQL output and must handle it itself.
  • C.2.1 DATABASE (Thought Construction): The response template explicitly separates the requested answer format from the surrounding database conversation.The section includes a dedicated instruction introducing the expected response.
  • C.2.1 DATABASE (Thought Construction): The examples instruct the agent to derive SQL through SELECT, JOIN, GROUP BY, COUNT, ORDER BY, and LIMIT operations.One example finds the professor advising the most students by grouping and ordering student counts.

D PROMPT FOR EVALUATION

The evaluation prompts define action-selection formats for household, web-shopping, restaurant, and knowledge-graph environments. They provide observations, available actions, structured outputs, and task-specific tool constraints for interactive evaluation.

  • D.1 ALFWORLD: ALFWorld prompts require choosing either THOUGHT or ACTION from the current household observation and task goal.Invalid actions are rejected, and repeated invalid outputs may require trying alternatives.
  • D.2 WEBSHOP: WebShop prompts require responding to each observation with search[keywords] or click[value] actions.Search keywords are freely chosen, but click values must come from the available action list.
  • D.3 MIND2WEB: The Mind2Web prompt represents webpage context with HTML, task descriptions, previous actions, and available answer choices.Some examples include noisy or incomplete webpage content while preserving the action-selection format.
  • D.3 MIND2WEB: Mind2Web examples frame the task as selecting the next action from webpage elements after prior actions and HTML observations.The required choice can be None of the above when no listed action is correct.
  • D.3 MIND2WEB: Mind2Web examples demonstrate clicking a date control to modify a rental reservation and selecting Pickup from a restaurant-type dropdown.These decisions are based on visible element labels and task requirements.
  • D.4 KNOWLEDGE GRAPH: Knowledge Graph evaluation provides get_relations to enumerate relations connected to an entity or prior query result.The argument must be an entity or variable rather than arbitrary text.
  • D.4 KNOWLEDGE GRAPH: Knowledge Graph evaluation uses get_neighbors to follow a selected relation and intersection to combine same-typed variable sets.get_neighbors may be used only after get_relations identifies viable relations.

D.5 OPERATING SYSTEM

The operating-system task requires the model to alternate reasoning with exactly one action per turn, using bash, finish, or answer. Additional tasks constrain SQL interaction and household operations through structured, turn-by-turn protocols.

  • Action protocol: Each turn requires one of three actions: bash, finish, or answer.The model must first think, then execute exactly one permitted action.
  • Action protocol: The operating-system interface formats bash, finish, and answer actions with explicit Think and Act fields.Answer actions place the response inside the required parentheses.
  • Action protocol: Bash actions must contain code without input operations, while truncated outputs must be handled by the agent.The environment may truncate long output, requiring the model to manage incomplete results.
  • Database interaction: SQL interaction permits one SQL statement per turn and requires SQL in a single markdown code block.The environment executes only the first SQL code block and returns its output for subsequent decisions.
  • Database interaction: The database task ends with a structured final answer after the agent completes its operations.For modification questions, the final answer field may contain any value after the operation is done.
  • Household interaction: Household tasks allow alternating step-by-step thoughts and environment actions, with each action following the preceding plan.The protocol emphasizes using reasoning and action together to solve the task.

D.8 HOTPOTQA

HOTPOTQA evaluates interleaved reasoning and information-retrieval actions, while other agent settings require planning and controlled computer or game interaction. These protocols specify permitted actions, observations, and completion formats.

  • HOTPOTQA protocol: HOTPOTQA alternates Thought, Action, and Observation steps to answer questions through sequential evidence retrieval.The permitted actions are Search, Lookup, and Finish.
  • HOTPOTQA protocol: Search retrieves an entity’s first Wikipedia paragraph, while Lookup finds the next sentence containing a keyword in the latest passage.Finish concludes the task with the answer.
  • HOTPOTQA example: The worked Milhouse example resolves the answer by searching the entity and then looking up the phrase “named after.”The final action is Finish[Richard Nixon].
  • Computer control: Computer-control tasks ask agents to create ordered plans whose steps correspond to executable instructions over an accessibility tree or webpage.The prompt supplies the current task, action history, available actions, and webpage HTML.
  • Tool planning: Tool-using planning tasks require each plan to identify an external tool and store retrieved evidence in sequential variables.The solver must then answer using plans and potentially noisy evidence.
  • Pet-fish game: The pet-fish game requires guessing a live enemy fish’s identity each round and returning a JSON move with a fish type and target position.Correct guesses reveal identities and damage the enemy team, while victory depends on having more fish alive.

D.12 MMLU

The MMLU section presents multiple-choice prompts alongside worked arithmetic examples that demonstrate step-by-step reasoning. The examples cover proportional change, wages, allocation, vehicle tires, missing laundry items, and totals.

  • Multiple-choice format: MMLU prompts ask the model to select the most suitable answer among choices A, B, C, and D.The template provides several question-answer examples before the target question.
  • Worked arithmetic: The cookout example computes 20 children in 2006 from 60 children in 2004, halving in 2005 and taking two-thirds thereafter.The worked calculation is 60/2=30, followed by 30/3*2=20.
  • Worked arithmetic: The wage example calculates an annual increase of $20,800 from weekly earnings of $800 versus $400 over 52 weeks.The comparison uses 20*40 and 16*25 to obtain the weekly difference.
  • Worked arithmetic: The treats example divides 80 total items among 20 students, yielding 4 treats per student.The total combines 20 cookies, 25 cupcakes, and 35 brownies.
  • Worked arithmetic: The vehicle example counts tires by separating 8 motorcycles from 16 cars and accounting for 4 cars with spare tires.The passage continues the resulting total-tire calculation from these categories.
  • Worked arithmetic: The orchard example totals 350 apples by adding 120 picked over 30 days to 230 remaining apples.The daily collection is 4 apples for each of 30 days.
Loading 2310.12823v2…