PaperlayerView exact revision on arXiv

Source-linked AI summary

FireAct: Toward Language Agent Fine-tuning

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, Shunyu Yao

arXiv:2310.05915v1cs.CLcs.AIcs.LG

TL;DR

Language agents commonly rely on prompting off-the-shelf LMs, but the scope and benefits of fine-tuning them for agentic use remain insufficiently studied. FireAct fine-tunes smaller LMs on diverse GPT-4-generated trajectories spanning tasks and prompting methods, using QA with Google search. Across the evaluated settings, fine-tuning consistently improves agents, with greater data diversity providing further gains, while the study remains limited to QA and Google search.

  • Problem

    Fine-tuning language models for agents has been sparsely studied, leaving its generality across models, prompting methods, and tasks insufficiently established.

  • Method

    FireAct fine-tunes smaller LMs on successful GPT-4-generated trajectories drawn from multiple tasks and prompting methods and unified in ReAct format.

  • Results

    Fine-tuning consistently improves agent performance across the evaluated settings, and diverse fine-tuning data provides further gains; for example, Llama-2-7B improves HotpotQA performance by 77%.

  • Takeaways & Limitations

    Fine-tuning is presented as better suited to known downstream tasks with effective prompting methods and sufficient data, while prompting remains useful for exploration and convenience.

  • Takeaways & Limitations

    The study is constrained to a single task type, QA, and a single tool, Google search.

Abstract

from arXiv · show

Recent efforts have augmented language models (LMs) with external tools or environments, leading to the development of language agents that can reason and act. However, most of these agents rely on few-shot prompting techniques with off-the-shelf LMs. In this paper, we investigate and argue for the overlooked direction of fine-tuning LMs to obtain language agents. Using a setup of question answering (QA) with a Google search API, we explore a variety of base LMs, prompting methods, fine-tuning data, and QA tasks, and find language agents are consistently improved after fine-tuning their backbone LMs. For example, fine-tuning Llama2-7B with 500 agent trajectories generated by GPT-4 leads to a 77% HotpotQA performance increase. Furthermore, we propose FireAct, a novel approach to fine-tuning LMs with trajectories from multiple tasks and prompting methods, and show having more diverse fine-tuning data can further improve agents. Along with other findings regarding scaling effects, robustness, generalization, efficiency and cost, our work establishes comprehensive benefits of fine-tuning LMs for agents, and provides an initial set of experimental designs, insights, as well as open questions toward language agent fine-tuning.

1 INTRODUCTION

Language-agent fine-tuning is understudied despite limitations of prompting off-the-shelf LMs. This work systematically evaluates fine-tuning and proposes FireAct, finding benefits from stronger and more diverse training data.

  • Most language agents prompt off-the-shelf LMs, which were not developed for agentic actions or self-evaluations.This limits performance and robustness and can require GPT-4, increasing cost and latency while raising controllability and reproducibility concerns.
  • Fine-tuning language models for agents has received limited attention compared with the separate fields of language agents and LM fine-tuning.Prior work mainly examined web navigation, API tool use, or limited scaling analyses.
  • FireAct fine-tunes LMs on successful agent trajectories generated across multiple tasks and prompting methods, unified in the ReAct format.The study uses open-domain QA with Google search and GPT-4-generated fine-tuning data.
  • 39.2 EM is achieved on HotpotQA by fine-tuning GPT-3.5 with 500 ReAct trajectories, versus 31.4 from few-shot ReAct prompting.Mixing ReAct and CoT trajectories further raises EM to 41.0.
  • More diverse fine-tuning data improves agents and reveals interactions among base models, prompting methods, tasks, scaling, robustness, generalization, efficiency, and cost.The experiments also report actionable practitioner insights and open questions for future language-agent fine-tuning.

2 RELATED WORK

Language agents use LMs to reason and act through external environments, while LM fine-tuning adapts models to downstream needs. Their intersection remains sparsely studied beyond early task-specific and scaling investigations.

  • Language agents: Language agents use language models to interact with the world through actions, observations, reasoning, reflection, planning, and other capabilities.Grounding spans games, APIs, webpages, and physical environments.
  • Language model fine-tuning: Language-model fine-tuning includes instruction datasets, adapted models, parameter-efficient methods, and data-selection principles.
  • Language agent fine-tuning: Prior language-agent fine-tuning studies mainly examine model-size scaling, tool retrieval, or task-specific ablations.The evidence remains limited across model families, prompting methods, and tasks.

3 FI R EAC T: FINE-TUNING LMS WITH DIVERSE REACT TRAJECTORIES

FireAct fine-tunes smaller LMs on diverse, successful agent trajectories generated by a strong LM and unified in ReAct format. During inference, the resulting agent can adapt its method and trajectory length to task complexity.

  • ReAct trajectories contain repeated thought-action-observation rounds that support information extraction, planning, progress tracking, tool interaction, and feedback.
  • A prior PaLM-62B result showed fine-tuning could outperform prompted PaLM-540B, but generalization across models, prompting methods, and tasks remained unknown.
  • FireAct distills diverse trajectories from a strong LM into a smaller LM by mixing multiple training tasks and prompting methods.Successful trajectories are unified in the ReAct format for fine-tuning.
  • CoT trajectories become one-round ReAct trajectories, making the method suitable for simple questions without tool use.Reflexion adds feedback and self-reflection to pivot strategies during longer ReAct trajectories.
  • FireAct inference can operate without few-shot prompting, implicitly select a suitable method, and use flexible trajectory lengths.The paper attributes stronger generalization and robustness to broader and more diverse learning support.

4 EXPERIMENTAL SETUP

The experiments evaluate language-agent fine-tuning on several QA tasks with a Google-search tool, multiple LM families, and LoRA or full-model fine-tuning. Studies increase from single-method, single-task training to multi-method and multi-task settings.

  • Tasks: Four QA datasets cover multi-step retrieval, multi-hop questions, implicit reasoning, and 57-domain multiple-choice evaluation.The datasets are HotpotQA, Bamboogle, StrategyQA, and MMLU.
  • Tool: SerpAPI supplies a Google search tool that returns a short answer from an answer box, answer snippet, highlighted words, or the first result snippet.
  • LMs: The study compares GPT models, Llama-2 models with 7B or 13B parameters, and CodeLlama models with 7B, 13B, or 34B parameters.GPT-4 generates fine-tuning data, while GPT-3.5 is fine-tuned and prompted.
  • Fine-tuning methods: LoRA is used for most fine-tuning experiments, with full-model fine-tuning included for some comparisons.
  • Experimental design: Experiments progress from one prompting method and task to multiple methods on one task, then multiple methods across multiple tasks.

5 SINGLE-TASK, SINGLE-METHOD FINE-TUNING

Single-task, single-method fine-tuning improves agent performance, inference efficiency, robustness, and some cross-task generalization over prompting. Results also vary with model type, scale, data size, and fine-tuning method.

  • Performance and efficiency: Fine-tuning consistently and significantly improves HotpotQA EM over prompting, including a 77% increase for Llama-2-7B and a 25% increase for GPT-3.5.Fine-tuned Llama-2-13B can outperform all GPT-3.5 prompting methods, while fine-tuned GPT-3.5 surpasses GPT-4 + IO but trails GPT-4 + CoT/ReAct.
  • Performance and efficiency: 70% lower inference time, from 9.0s to 2.7s per trial, accompanies lower inference cost for fine-tuned GPT-3.5 despite 8× higher per-inference pricing.Removing few-shot examples reduces the accumulated context required during iterative agent inference.
  • Robustness and generalization: Under noisy search observations, ReAct EM drops 33.8% with “None” responses and 28.0% with random responses, versus 14.2% and 5.1% for FireAct.The trajectories include examples of noisy search queries and successful reactions to them.
  • Robustness and generalization: 44.0 EM versus 40.8 EM shows fine-tuned GPT-3.5 generalizes better than prompted GPT-3.5 from HotpotQA to Bamboogle.HotpotQA fine-tuning slightly improves MMLU, but downgrades StrategyQA, motivating multi-task experiments for different question styles and answer formats.
  • Fine-tuning factors: 30.2 EM versus 26.2 EM shows full-model fine-tuning outperforming LoRA by 15.3% for Llama-2-7B, while LoRA is substantially more affordable.LoRA trains 5.4 examples per second on one RTX 4090, whereas full fine-tuning trains 19.7 examples per second using four A100 GPUs.
  • Fine-tuning factors: Non-trivial scores emerge for Llama models with 500 trajectories, and most models improve further with 1,000, whereas GPT-3.5 reaches around 35 EM with 100 samples.Llama-2-13B fine-tuned on 1,000 samples can match GPT-3.5 fine-tuned on 100 samples.
  • Fine-tuning factors: GPT-3.5 outperforms all Llama-based models in prompting and fine-tuning, while 13B (Code)Llama models generally outperform their 7B counterparts.CodeLlama-34B can underperform CodeLlama-13B when trained only on ReAct trajectories, indicating joint effects from model type, scale, data, and method.

6 MULTI-METHOD FINE-TUNING

The section studies fine-tuning with multiple prompting methods, showing greater flexibility but model-dependent outcomes. Combining methods can help agents select different strategies for different questions, although no mixture is universally best.

  • Data construction: 500 GPT-4-generated trajectories from ReAct, CoT, and Reflexion support multi-method fine-tuning on HotpotQA.The data include 500 ReAct trajectories, 187 successful CoT trajectories reformatted as single-round ReAct, and 47 long Reflexion trajectories.
  • Agent flexibility: Multi-method fine-tuning lets agents solve simple questions internally and recover from unproductive searches by changing strategy.Examples show CoT+ReAct avoiding an unnecessary search and Reflexion+ReAct changing a time constraint after repeated unhelpful queries.
  • Model dependence: The best method mixture depends on the base LM: ReAct+CoT helps GPT-3.5 and Llama-2 but hurts CodeLlama, while all three methods work best for CodeLlama-34B.The findings indicate non-trivial interactions between the backbone model and fine-tuning data.
  • Method selection: ReAct+CoT improves HotpotQA EM and shortens trajectories, whereas ReAct+Reflexion lowers EM and lengthens trajectories.These results suggest the two mixtures shift method selection in different directions.

7 MULTI-TASK FINE-TUNING

The multi-task experiments combine QA trajectories from HotpotQA, StrategyQA, and MMLU to test whether diverse task data transfers across benchmarks. Adding tasks largely preserves HotpotQA and Bamboogle performance, while combining tasks and methods improves performance across tasks but leaves MMLU below naive prompting.

  • Training data: 1,?
  • Training data: HotpotQA, StrategyQA, and MMLU provide mixed ReAct and CoT trajectories for GPT-3.5 multi-task fine-tuning.The training set contains 500 ReAct and 277 CoT HotpotQA samples, 388 ReAct and 380 CoT StrategyQA samples, and 456 ReAct and 469 CoT MMLU samples.
  • Multi-task transfer: Adding StrategyQA and MMLU data leaves HotpotQA and Bamboogle performance almost unchanged.The different question distributions and tool-use strategies make transfer difficult, but the added data does not harm those performances.
  • Multi-task transfer: Multi-task, multi-method fine-tuning increases performance across all tasks.This result reinforces the value of combining task diversity with prompting-method diversity.
  • MMLU behavior: All fine-tuned agents and CoT/ReAct prompting underperform naive IO prompting on MMLU.The authors suggest MMLU questions may be too easy to require reasoning and acting, or may invite answer-choice memorization.

8 DISCUSSION

The discussion frames fine-tuning as useful for exploiting accumulated agent experience, while prompting remains convenient for exploration. It also highlights model-choice trade-offs, unresolved decisions about tool use and reflection, and important scope boundaries.

  • When to fine-tune: Prompting offers flexibility for new agents and tasks, whereas fine-tuning is presented as more suitable for downstream exploitation.The distinction reflects exploration versus exploitation use cases.
  • Which LM to fine-tune: GPT-3.5 consistently outperforms the evaluated Llama-based models, but open-source LMs can catch up with enough data and suitable task-method mixtures.GPT-3.5 is also described as sample-efficient and costing around $10 per fine-tuning experiment in this setup.
  • Which LM to fine-tune: Practitioners must balance GPT-3.5 convenience and performance against open-source controllability and reproducibility.
  • Tools and reflection: Multi-method fine-tuning improves flexibility and robustness, but deciding when to use tools or reflection remains unresolved.Prompting-based agents may overuse tools or fail to recover when trajectories deviate from successful patterns.
  • Limitations and future directions: The study is constrained to QA tasks, Google search, three prompting methods, three QA tasks, and GPT-3.5 as its strongest fine-tuned backbone.The authors call for broader grounding setups, more complex agent contexts, larger multi-task studies, and more diverse benchmarks.

A.1 STANDARD ERROR OF EM SCORES

The appendix provides standard-error references for prompting and fine-tuning comparisons, including multi-task results.

  • Prompting results: Table 6 reports prompting results with standard errors.
  • Prompting versus fine-tuning: Table 7 reports prompting-versus-fine-tuning results with absolute and relative increases plus standard errors.
  • Multi-task results: Table 8 reports multi-task GPT-3.5 results with standard errors.

A.3 DATA MIX

Mixing agent-method trajectories aims to approach the best available method while changing trajectory distributions and preserving model-specific performance differences. The experiments compare mixed-method performance, parameter-efficient versus full fine-tuning, tokenizer choices, and observation masking.

  • Mixed agent methods: 32.4 and 52.0 EM define the lower and upper performance bounds for randomly selecting versus always selecting the best agent method.Mixed-method fine-tuning is evaluated against these theoretical boundaries.
  • Evaluation: The mixed-method investigation is reported on HotpotQA, while the broader comparison table covers different tasks and methods.Table 10 is specifically titled as the mixed agent methods investigation on HotpotQA.
  • Data distribution: Combining data changes the trajectory distribution toward the training distribution, with ReAct+CoT examples differing in average turns and EM.The cited comparison reports 2.7 turns and 41.0 EM versus 3.8 turns and 38.8 EM.
  • Fine-tuning method: LoRA and full-weight fine-tuning are compared in both single-task and multi-task settings for Llama-2-7B HotpotQA trajectories.The cited passages identify the comparison design but do not provide the corresponding scores.
  • Model processing: Tokenizer choice is evaluated for CodeLlama-7B and CodeLlama-13B on HotpotQA, while observation masking is assessed for Llama-2 models.Observation masking generally slightly improves CodeLlama-7B but has inconsistent effects for CodeLlama-13B.

A.8 TRAINING EPOCHS

Fine-tuning epochs affect HotpotQA performance nonlinearly: EM and F1 generally improve through four epochs, after which additional training yields diminishing returns. The section also situates the experiments across GPT, Llama-2, and CodeLlama model families.

  • Training epochs: EM and F1 generally improve as GPT-3.5 fine-tuning increases from 1 to 4 epochs.The improvement is described as increased exact-answer precision and overall answer quality.
  • Training epochs: Beyond 4 epochs, EM increases only slightly while F1 plateaus and sometimes dips, indicating diminishing returns from additional fine-tuning.The two metrics therefore do not continue improving uniformly after four epochs.
  • Base models: The study examines GPT, Llama-2, and CodeLlama families, including GPT-3.5, GPT-4, Llama-2 models from 7B to 70B, and CodeLlama models from 7B to 34B.CodeLlama is derived from Llama-2 and specialized for code generation.

B.2 SINGLE-METHOD SINGLE-TASK SETUP

The single-method, single-task setup fine-tunes language models on agent trajectories for question answering with interleaved reasoning, search actions, observations, and final answers. Examples span HotpotQA, MMLU-style multiple choice, and input-output formats.

  • Data and evaluation: The setup generates 500 ReAct trajectories from HotpotQA training samples and evaluates on 500 development examples using exact match and F1.GPT-4 generates the trajectories, with human-in-the-loop validation; the models are fine-tuned with model-specific tooling.
  • ReAct prompting: ReAct trajectories interleave Thought, search[question] or finish[answer] actions, and observations returned by Google search.Search snippets may omit the answer, requiring alternative searches.
  • HotpotQA examples: The examples demonstrate multi-step retrieval by first identifying the High Plains and then searching its elevation range before finishing with the answer.The trajectory returns 1,800 to 7,000 ft and receives reward 1.
  • Question examples: Other examples use internal reasoning or search to answer questions about Richard Nixon, Finnish rock groups, shared professions, and magazine chronology.The supplied examples include both ReAct-style and chain-of-thought-style answer formats.
  • Tasks and formats: The broader task set includes HotpotQA, Bamboogle, StrategyQA, and MMLU, while implementation examples include input-output and chain-of-thought prompts.These tasks provide established QA formats and exact-match evaluation.
Loading 2310.05915v1…