PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning

Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, Huajun Chen

arXiv:2401.05268v4cs.CLcs.AIcs.HCcs.LGcs.MA

TL;DR

Language-agent planning can depend on costly, hard-to-reproduce data and place multiple planning functions on one model. AUTOACT augments limited QA examples, synthesizes trajectories with tools, and differentiates specialized agents; experiments report performance that is better or parallel to strong baselines.

  • Problem

    Existing fine-tuning approaches can require substantial annotated QA data and closed-source-generated planning trajectories, while a single agent is tasked with multiple planning abilities.

    Source: §1 · pp. 1–2

  • Method

    AUTOACT expands limited task examples through self-instruct, synthesizes tool-based planning trajectories, and fine-tunes a META-AGENT into task-decomposition, tool-invocation, and reflection agents.

    Source: §1 · p. 2, §2.2 · p. 2, §2.3 · p. 4

  • Results

    Experiments with different LLMs report that AUTOACT achieves better or parallel performance than various strong baselines, with analysis supporting its division-of-labor strategy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AUTOACT demonstrates a route to open-source agent learning from scratch without large-scale annotated data or closed-source planning trajectories.

    Source: §4 · p. 5

    The paper focuses on complex QA, and self-instruct can be limited by the model’s ability to access internal knowledge.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Language agents have achieved considerable performance on various complex question-answering tasks by planning with external tools. Despite the incessant exploration in this field, existing language agent systems still struggle with costly, non-reproducible data reliance and face the challenge of compelling a single model for multiple functions. To this end, we introduce AutoAct, an automatic agent learning framework for QA that does not rely on large-scale annotated data and synthetic planning trajectories from closed-source models (e.g., GPT-4). Given limited data with a tool library, AutoAct first automatically synthesizes planning trajectories without any assistance from humans or strong closed-source models. Then, AutoAct leverages a division-of-labor strategy to automatically differentiate based on the target task information and synthesized trajectories, producing a sub-agent group to complete the task. We conduct comprehensive experiments with different LLMs, which demonstrates that AutoAct yields better or parallel performance compared to various strong baselines. Further analysis demonstrates the effectiveness of the division-of-labor strategy, with the trajectory quality generated by AutoAct generally outperforming that of others. Code will be available at https://github.com/zjunlp/AutoAct.

1 Introduction

Language agents use tools and planning for complex QA, but existing approaches can require costly, hard-to-reproduce data and ask one model to carry all planning functions. AUTOACT learns from limited user-provided examples, synthesizes trajectories, and differentiates specialized agents; experiments report better or parallel performance against strong baselines.

1 Introduction

  • Fine-tuning-based agents can require substantial annotated QA data and planning trajectories synthesized by closed-source models, which may be difficult to obtain in sensitive settings.The paper also identifies the pressure of asking a single agent to learn all planning abilities.Source: §1 · pp. 1–2
  • AUTOACT learns from limited user-provided examples, synthesizes planning trajectories without human or strong closed-source-model assistance, and differentiates into three specialized sub-agents.The sub-agents handle task decomposition, tool invocation, and self-reflection through parameter-efficient training.Source: §1 · p. 2
  • AUTOACT experiments with different LLMs show better or parallel performance than various strong baselines.The empirical analysis also supports the effectiveness of its division-of-labor strategy.Source: §1 · p. 2

2 AUTOACT

AUTOACT starts from limited task information and a tool library, augments its examples, then uses self-planning to synthesize training trajectories and differentiate a META-AGENT into collaborating specialists.

2.1 Critical Components of AUTOACT

  • The META-AGENT uses a task name, description, and a small set of question-answer examples, alongside a tool library, as the basis for learning.The task information is the only user-provided task knowledge; users may expand the tool library as needed.Source: §2.1 · p. 2

2.2 Starting from Scratch via Self-Instruct

  • Self-instruct begins with the supplied examples, generates new question-answer pairs through few-shot prompting, and filters format errors and duplicates before adding them to the database.The resulting database is much larger than the initial examples.Source: §2.2 · p. 2

2.3 Automatic Agent Learning via Self-Planning

  • The META-AGENT selects tools for the task, then synthesizes Thought-Action-Observation trajectories on the augmented database and retains only exactly correct answers for training.Tool selection uses the task information and the prepared tool library.Source: §2.3 · p. 3
  • Self-differentiation trains three agents: the PLAN-AGENT decomposes questions and selects tools, the TOOL-AGENT sets invocation parameters, and the REFLECT-AGENT reviews trajectory history.Each agent receives a specific LoRA through parameter-efficient fine-tuning.Source: §2.3 · pp. 3–4
  • During group planning, the PLAN-AGENT and TOOL-AGENT continue after tool observations until a prediction is reached, then the REFLECT-AGENT checks it and can trigger another planning round.The reflection decision determines whether the process ends or resumes with the reflection information.Source: §2.3 · p. 4

3 Experimental Setup

The experiments evaluate AUTOACT on two QA benchmarks using sampled, difficulty- or grade-stratified questions, with separate reward definitions for each task.

3 Experimental Setup

  • HotpotQA evaluation uses 300 randomly selected development questions across three levels and grades predictions with answer F1.ScienceQA evaluation samples 120 test questions at each of three grade levels and uses accuracy for its multiple-choice answers.Source: §3 · p. 4
  • The comparisons include prompt-based and trained agent baselines, with FIREACT and AUTOACT each using 200 training trajectories.Prompt-based baselines receive two examples, and non-Reflexion methods are tested twice to account for Reflexion's answer-correctness labels.Source: §3 · p. 4
  • Models are fine-tuned with LoRA in the Alpaca format, and training and inference run on eight V100 GPUs within 16 hours.Source: §3 · p. 4

4 Results

AUTOACT outperforms FIREACT on both evaluated QA tasks with Llama-70B and generally performs better than prompt-based agent-learning baselines. The results also favor multi-agent learning overall, while showing that finer-grained tool specialization is not necessarily beneficial.

Compare to Prompt-based Agent Learning Baselines

  • ↑3.77% on HotpotQA and ↑6.39% on ScienceQA for Llama-70B versus GPT-3.5-Turbo; the tested models also consistently outperform prompt-based baselines.The comparison uses GPT-3.5-Turbo's zero-shot plan performance.Source: §4 · p. 4, Table 1 · p. 5

Compare to Fine-tuning-based Agent Learning Baselines

  • ↑5.77% on HotpotQA and ↑6.67% on ScienceQA with Llama-70B versus FIREACT.AUTOACT decouples planning across sub-agents, while FIREACT assigns the planning task to one model.Source: §4 · pp. 4–5, Table 1 · p. 5

Single-agent Learning vs. Multi-agent Learning

  • Multi-agent architectures generally outperform single-agent ones, but Chameleon, a single-agent framework, beats BOLAA and even FIREACT on ScienceQA.The authors note that Chameleon's specialized prompts for tool parameters resemble multi-agent structure.Source: §4 · pp. 5–6

Approach Ablations

  • Removing multi-agent differentiation noticeably reduces performance; removing reflection has the least impact, and removing fine-tuning also degrades results.The -multi ablation is comparable to FIREACT; the authors say this indirectly suggests 70B-generated trajectories may be no worse than GPT-4-generated ones.Source: Table 2 · p. 5, §4 · p. 6

Case Study

  • AUTOACT's additional verification rounds can validate inner answers, but longer context can gradually pull it away from the original question.In the case study, AUTOACT also addresses a REACT failure through a more effective tool combination and more accurate tool invocations.Source: Figure 5 · p. 7

5 Analysis

The analysis finds that increasing self-instruct data beyond 200 yields little further performance change, while moderate division of labor and trajectory quality show distinct benefits and trade-offs.

Training Data Scale

  • Performance stabilizes with minimal variation once self-instruct data exceeds 200, and larger models produce higher-quality synthesized data.The authors speculate that limited access to internal knowledge and increasingly similar generated data constrain gains from scaling.Source: §5 · p. 7

Division-of-Labor Granularity

  • Tool-specific differentiation can underperform both AUTOACT and even a single agent, especially on harder problems requiring more planning and tool collaboration.The authors attribute AUTOACT's advantage to unified tool invocation helping agents learn tool interconnectedness and compensate for information gaps.Source: §5 · pp. 7–8

Human Evaluation

  • Human evaluation finds AUTOACT stronger in action types and parameters, while its extra planning rounds can help on hard problems but lengthen context and drift on simple ones.The evaluation also compares planning rounds, thought correctness, and overall coherence; AUTOACT tends to use more rounds than other methods.Source: §5 · p. 8

6 Related Work

Prior LLM-agent work addresses planning, tools, and multi-agent collaboration, but the paper identifies customization and generalist-skill demands as persistent challenges. It positions AUTOACT's fine-tuned, differentiated agents as a response to those challenges and to data-reliance burdens.

LLM-Powered Agents

  • LLM-agent research spans planning, external-tool use, and multi-agent collaboration, while prompt-based customization can yield unexpected performance and demands that each agent master many skills.The paper responds with fine-tuned agents assigned distinct duties that collaborate on tasks.Source: §6 · p. 8

Agent Fine-Tuning

  • Agent fine-tuning has been limited and prior approaches may require a generalist model, extensive annotated data, or trajectories annotated by closed-source models.AUTOACT instead has its META-AGENT synthesize trajectories and divide labor in a zero-shot manner without closed-source models.Source: §6 · p. 8

7 Conclusion and Future Work

AUTOACT is presented as an automatic QA agent-learning framework that avoids large-scale annotation and closed-source synthetic trajectories while dividing work among agents. The authors identify broader task coverage, richer self-instruct knowledge, and iterative trajectory improvement as future directions.

7 Conclusion and Future Work

  • AUTOACT avoids reliance on large-scale annotated data and closed-source synthetic trajectories while explicitly dividing workload among agents.The stated aim is to alleviate pressure on individual agents.Source: §7 · p. 8
  • Future work includes expanding to more realistic tasks, boosting knowledge through self-instruct, and iteratively improving synthetic trajectories.The paper points to web, household, and travel scenarios among possible broader task settings.Source: §7 · p. 9

Limitations

The paper identifies complex QA as its current focus, notes that self-instruct is limited by the model's access to internal knowledge, and leaves richer knowledge acquisition and broader applications for future work.

Tasks

  • The evaluated scope focuses on complex question-answering, while web, household, travel, and robotics scenarios remain beyond the current study.The authors plan further research applying AUTOACT to a wider range of tasks.Source: Limitations · p. 9

Boosting Knowledge via Self-Instruct

  • Self-instruct planning performance can be limited by the model's ability to access internal knowledge, leaving knowledge enrichment under limited-data constraints as an open research need.The paper describes lightweight self-differentiation in parameters and data as a benefit of the current setup.Source: Limitations · p. 9

Self-Improvement

  • The authors propose iterative self-improvement on self-synthesized data as a possible way to enhance AUTOACT's performance.They describe this as a future direction rather than a demonstrated result.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes the datasets and human-evaluation practices used in the research and concludes that the work raises no ethical concerns.

Ethics Statement

  • All experiments use publicly available datasets, avoiding privacy, confidentiality, and personal biological-information concerns.Source: Ethics Statement · p. 9
  • The human evaluation process is described as fair and transparent.Source: Ethics Statement · p. 9
  • The authors conclude that the research is free from ethical concerns.Source: Ethics Statement · p. 9

B Baselines and Training Setups

The experiments compare AutoAct with single-agent, multi-agent, and other baselines using Llama-2 and Mistral-7B backbones. The appendix describes a blinded human trajectory comparison and reports the scope of a planning-round comparison, but the supplied passages give no outcomes for either evaluation.

B Baselines and Training Setups

  • The comparison includes CoT, REACT, Chameleon, Reflexion, BOLAA, ReWOO, FIREACT, and GPT-3.5-Turbo, with Llama-2 and Mistral-7B as AutoAct backbones.FIREACT and AutoAct each use 200 training trajectories; prompt-based baselines receive two examples, and methods other than Reflexion are tested twice with the correct run selected.Source: B Baselines and Training Setups · p. 13
  • All models are fine-tuned with LoRA in the Alpaca format using FastChat and DeepSpeed.Training hyperparameters are referenced in Table 4, but are not specified in the supplied passage.Source: B Baselines and Training Setups · p. 13

C Detailed Process of Human Evaluation

  • Five NLP volunteers compare trajectories on planning rounds, thought logic, action types and parameters, and coherence, with method identities hidden and majority votes determining results.AutoAct reflection-related portions are removed and trajectory order is shuffled to reduce potential bias.Source: C Detailed Process of Human Evaluation · p. 13

D Average Planning Rounds

  • The planning-round comparison covers HotpotQA with Llama-2-70B-chat and excludes AutoAct reflection steps for fairness.The supplied text points to Table 5 for average rounds and Figure 6 for win rates, but gives no results.Source: D Average Planning Rounds · p. 13

E Task Information

The task-information materials describe HotpotQA and ScienceQA and provide examples of each. They also describe a tool library that can be expanded for users’ needs.

E Task Information

  • HotpotQA is a multi-hop question-answering task involving multi-step reasoning, while ScienceQA is multimodal and uses tools to convert image information into text.The supplied examples include a HotpotQA question and ScienceQA multiple-choice questions with image captions.Source: E Task Information · pp. 13–15
  • The related-work comparison table is described as distinguishing data and trajectory acquisition, planning, multi-agent structure, fine-tuning, generality, and reflection.Source: Table 3 · p. 14

F Tool Library

  • The tool library contains 15 commonly used tools for complex question answering and can be expanded for users’ specific needs.Source: F Tool Library · p. 15

H Database Cases

The database cases show question-answer examples for HotpotQA and ScienceQA. The accompanying prompts describe tool selection and trajectory generation, with examples of the three sub-agent roles.

H Database Cases

  • HotpotQA database cases include questions about the Pacific Ocean, the century of gravity’s discovery, the Wright brothers, Mars, and Pemberley.Source: H Database Cases · p. 15
  • ScienceQA cases show multiple-choice questions on weather phenomena, biodegradation, renewable energy, atmospheric circulation, and electrolysis.Source: H Database Cases · p. 15

Prompt for Trajectories Synthesis

I Training Data Example

  • The training-data example assigns thought generation to Plan-Agent, tool invocation to Tool-Agent, and answer reflection to Reflect-Agent.In the example, Reflect-Agent judges the answer wrong after reviewing the history.Source: I Training Data Example · pp. 18–19
Loading 2401.05268v4…