PaperlayerView exact revision on arXiv

Source-linked AI summary

KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang

arXiv:2403.03101v3cs.CLcs.AIcs.HCcs.LGcs.MA

TL;DR

Language agents can generate plans that violate established knowledge rules or commonsense, a problem the paper calls planning hallucination. KNOWAGENT adds external action knowledge and knowledgeable self-learning to guide action generation; experiments report performance comparable or superior to baselines across HotpotQA and ALFWorld.

  • Problem

    Language agents, particularly open-source models, can generate plans that violate established knowledge rules or commonsense, termed planning hallucinations.

    Source: §1 · pp. 1–2

  • Method

    KNOWAGENT uses a task-specific external action knowledge base to guide action generation and a knowledgeable self-learning phase to improve how models apply that knowledge.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across HotpotQA and ALFWorld, KNOWAGENT competes with or surpasses baselines; the 13b model gains ↑15.09% (average) and ↑49.01% (unseen) over ReAct on the two datasets.

    Source: §1 · p. 2, §4.2 · p. 5, §6 · p. 9

  • Takeaways & Limitations

    Analysis indicates that action knowledge mitigates planning hallucinations, while manually refined LLM-derived action knowledge can reduce human labor and enhance performance.

    Source: §1 · p. 2

    Experiments cover only commonsense QA and household datasets, and the study focuses on single agents rather than multi-agent systems.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges, especially when interacting with environments through generating executable actions. This inadequacy primarily stems from the lack of built-in action knowledge in language agents, which fails to effectively guide the planning trajectories during task solving and results in planning hallucination. To address this issue, we introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge. Specifically, KnowAgent employs an action knowledge base and a knowledgeable self-learning strategy to constrain the action path during planning, enabling more reasonable trajectory synthesis, and thereby enhancing the planning performance of language agents. Experimental results on HotpotQA and ALFWorld based on various backbone models demonstrate that KnowAgent can achieve comparable or superior performance to existing baselines. Further analysis indicates the effectiveness of KnowAgent in terms of planning hallucinations mitigation. Code is available in https://github.com/zjunlp/KnowAgent.

1 Introduction

KNOWAGENT addresses planning hallucinations—plans that violate established rules or commonsense—by incorporating external action knowledge into trajectory generation and knowledgeable self-learning. Experiments on HotpotQA and ALFWorld report performance comparable or superior to existing baselines.

1 Introduction

  • Planning hallucinations are plans that violate established knowledge rules or commonsense, including unnecessary or conflicting action sequences.Examples include seeking information without searching or picking an apple without verifying that the table and apple are present.Source: §1 · pp. 1–2
  • KNOWAGENT builds a task-specific action knowledge base, converts its knowledge into text to guide action generation, and iteratively improves its use of that knowledge through self-learning.Source: §1 · p. 2
  • Across HotpotQA and ALFWorld with various backbone models, KNOWAGENT achieves comparable or superior performance to existing baselines.Source: §1 · p. 2
  • Further analysis indicates that action knowledge mitigates planning hallucinations, and manually refined action knowledge from LLMs can reduce human labor and enhance performance.Source: §1 · p. 2

2 Background

The paper represents an agent’s planning trajectory as thoughts, executable actions, and environment observations. At each step, the agent generates a thought and action from its history, then adds the action’s feedback as the next observation.

2 Background

  • A planning trajectory is a Thought-Action-Observation triplet, and its history contains prior thoughts, actions, and observations.Source: §2 · p. 2
  • At each step, the agent generates its next thought from the history and determines its action from both the history and current thought.Source: §2 · p. 2
  • The action’s feedback becomes the next observation and is added to the trajectory history; A_i denotes a trajectory action identical to action a_i in the action set.Source: §2 · p. 2

3 KNOWAGENT

KNOWAGENT defines task-specific action knowledge, uses it to guide planning paths, and iteratively refines trajectories through knowledgeable self-learning. The section describes how these components are represented, applied, and used to improve the model.

3 KNOWAGENT

  • The framework defines action knowledge, uses it to generate planning paths, and refines those paths through knowledgeable self-learning.The overall process is summarized as three successive stages.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Definition of Action Knowledge

  • Action Knowledge combines a defined set of actions with rules governing permissible transitions, and task-specific knowledge bases guide action generation and decisions.The knowledge is initially constructed with GPT-4 from expert-provided task knowledge, then filtered and refined by experts before final rules are generated.Source: §3.1 · p. 3

3.2.1 Action Knowledge to Text

  • Task-specific action knowledge is converted into text, including rules that describe which action transitions are permitted.The HotpotQA Search rule allows transitions to Search, Retrieve, Lookup, or Finish.Source: §3.2.1 · p. 3

3.2.2 Path Generation

  • Planning paths are guided by action rules, while trajectories also include thoughts, executable actions, and environmental feedback.The paper distinguishes a path—the action sequence—from a trajectory containing the broader problem-solving output.Source: §3.2.2 · pp. 3–4
  • Planning prompts organize action-knowledge guidance into an overview, action-step definitions, path-generation principles, and demonstrations.These segments specify actions and constrain planning output while providing examples.Source: §3.2.2 · p. 4

3.3 Planning Path Refinement via Knowledgeable Self-Learning

  • Self-learning iteratively synthesizes and filters trajectories, fine-tunes the model, and repeats until performance improvement becomes small.Trajectories are filtered and merged using action knowledge before fine-tuning.Source: §3.3 · p. 4

4 Experiments

Experiments evaluate KNOWAGENT on HotpotQA and ALFWorld across Llama-2 and other backbone models, against prompting and fine-tuning baselines. Results cover benchmark performance, action-knowledge and iteration ablations, planning errors, efficiency, and limitations.

4.2 Main Results

  • ↑15.09% (average) and ↑49.01% (unseen) over ReAct are reported for the 13b model across HotpotQA and ALFWorld; KNOWAGENT also surpasses GPT-4 on ALFWorld with 13b and 70b models.Across both datasets, KNOWAGENT consistently outperforms prompt-based baselines on open-source models.Source: §4.2 · p. 5
  • Fewer than 300 self-synthesized correct trajectories per iteration yield results comparable to methods using data generated by more advanced models such as GPT-4.FireAct and NAT use 500 GPT-4/GPT-3.5-Turbo trajectories on HotpotQA; the paper reports that the strategy also applies to ALFWorld.Source: §4.2 · p. 6

4.3 Analysis

  • With action knowledge, Llama models outperform versions without it at every self-learning iteration, and the performance gap grows as iterations increase.The authors attribute this pattern to a cycle in which action knowledge supports higher-quality training trajectories, which in turn help models learn the knowledge.Source: §4.3 · p. 6, Figure 3 · p. 6
  • Increasing self-learning from one to two iterations substantially improves performance, while extending to four continues to help with progressively smaller gains.The method is reported as effective across Llama-2, Vicuna-7b, and Mistral-7b, with differences suggesting varying ability to use structured external knowledge.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
  • On HotpotQA, incorporating action knowledge significantly reduces erroneous actions and invalid action paths; with constraints, KNOWAGENT avoids ReAct’s faulty Lookup->Search sequence, where Lookup depends on the subsequent Search step.Table 3 reports unreasonable action rates on HotpotQA with Llama-2-13b; invalid actions do not meet the action rule, while misordered actions differ from the logical action sequence.Source: §4.3 · p. 7, Table 3 · p. 7
  • GPT-4-distilled action knowledge is concise and performs comparably to human-crafted knowledge on simpler HotpotQA tasks, but underperforms on complex tasks requiring longer action sequences.On ALFWorld, distilled knowledge closely mirrors human-crafted knowledge and shows little difference in effectiveness.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • KNOWAGENT struggles with complex queries and long-text summarization because of insufficient reasoning and memory capacity for long contexts.Reported errors include incorrect or misaligned answers, such as giving a yes/no response when a specific entity is required.Source: §4.3 · p. 8
  • For Llama-2-7b on ALFWorld, KNOWAGENT takes about 20 hours per cycle and 40 hours total, versus about 50 hours for FireAct.The reported FireAct timing follows incorporation of around 710 GPT-4-synthesized trajectories; both methods have similar memory requirements on 8 NVIDIA V100 32G GPUs.Source: §4.3 · p. 8

5 Related Work

Prior language-agent work uses prompts, fine-tuning, and specialized strategies, while knowledge-augmentation research draws on retrieval, generation, and external tools. KnowAgent adds action-knowledge rules to constrain trajectory generation.

5 Related Work

  • Training trajectories annotated by LLMs may violate action-knowledge rules, making trained agents’ action performance unstable.Related approaches include FSM-based reasoning, combining natural and formal language, and iterative self-improvement.Source: §5 · p. 8
  • Knowledge-augmented LLM research includes retrieval, generated knowledge, and external tools such as search engines, knowledge bases, and Wikipedia.Other work regulates multi-agent workflows, uses state-aware guidelines, or develops memory mechanisms for long-horizon tasks.Source: §5 · p. 8
  • KnowAgent incorporates action-knowledge rules to constrain trajectory generation and reduce unreasonable action logic.Language agents interact through natural-language action-observation pairs; prior work has used human-crafted or machine-generated prompts and fine-tuning.Source: §5 · pp. 8–9

6 Conclusion

KnowAgent incorporates external action knowledge into synthetic trajectories and uses knowledgeable self-learning to refine action generation. Across various models, it competes with or surpasses baselines.

6 Conclusion

  • KnowAgent’s experiments across various models show performance that competes with or surpasses other baselines.The conclusion describes integrating external action knowledge as beneficial for streamlining planning and improving performance.Source: §6 · p. 9
  • KnowAgent incorporates external action knowledge into synthetic trajectories to mitigate planning hallucinations.Source: §6 · p. 9
  • A knowledgeable self-learning phase continuously improves the framework by guiding the model’s action generation.Source: §6 · p. 9

Limitations

The stated limitations concern evaluated task scope, single-agent focus, and the manual effort required to create action knowledge bases.

Limitations

  • Experiments cover only commonsense QA and household datasets; applicability to medical, arithmetic, web-browsing, and embodied-agent tasks remains unexplored.Source: Limitations · p. 9
  • The study focuses on single agents; multi-agent planning through division of labor and collaboration is left for future study.The authors suggest this could help agents handle complex tasks and adapt to changing environments.Source: Limitations · p. 9
  • Action-knowledge-base creation remains manual, time-consuming, and labor-intensive, with manual adjustments still needed after GPT-4 distillation.Future work should automate the process to reduce manual effort and improve autonomous learning and versatility.Source: Limitations · p. 9

A Experimental Settings

Experiments use HotpotQA and ALFWorld, with task-specific sampling, evaluation sets, and metrics. The study compares single-agent baselines and evaluates several LLM backbones.

A Experimental Settings

  • HotpotQA training samples 500 instances; testing uses 100 examples at each of three difficulty levels and F1, while ALFWorld evaluates goal-conditioned success rates on seen and unseen tasks.ALFWorld training samples 85 instances per task category; its test set contains 140 seen and 134 unseen tasks.Source: A Experimental Settings · pp. 13–14
  • The baseline comparison is single-agent only and excludes multi-agent studies.Baselines include CoT, ReAct, Reflexion, FireAct, and NAT.Source: A Experimental Settings · p. 14
  • Backbones include Llama-2 7B, 13B, and 70B chat models, plus Vicuna-7B, Mistral-7B, GPT-3.5 Turbo, and GPT-4.Self-learning uses two training iterations; most experiments use V100 32G GPUs, with ALFWorld 13B and 70B runs on A800 80G GPUs.Source: A Experimental Settings · p. 14

B Datasets

HotpotQA uses four actions to retrieve and search for information, inspect found passages, and return an answer.

B Datasets

  • HotpotQA provides Retrieve, Search, Lookup, and Finish actions for finding information and completing the task.Retrieve accesses Wikipedia; Search uses Bing; Lookup returns a matching sentence from a found passage; Finish returns the answer.Source: B Datasets · p. 14

C Error Analysis

KNOWAGENT's relative performance against FireAct varies with model size and task difficulty, while longer-text handling is identified as a challenge. HotpotQA errors include inconsistency and summarization failures.

C Error Analysis

  • In HotpotQA, inconsistency errors divert answers from the question, while summarization errors involve failing to extract needed trajectory information.The paper suggests improving long-text processing and reasoning abilities.Source: C Error Analysis · p. 14
  • The paper includes a hyperparameter table.Table 5 is titled “Hyper-parameters used in our paper.”Source: Table 5 · p. 15
  • KNOWAGENT performs better than FireAct on 7b models; at 13b, FireAct does better on easy tasks, while KNOWAGENT does better on medium and hard tasks.At 70b, KNOWAGENT's advantage appears on easy and medium tests but decreases on hard tests.Source: C Error Analysis · pp. 14–15
  • Longer-text handling challenges in KNOWAGENT's setup contribute to the observed model-scaling performance trend.For 70b, the method's advantage decreases on hard tests.Source: C Error Analysis · p. 15

D Prompt

The appendix includes a path-generation figure, a HotpotQA prompt, and task-specific ALFWorld prompt tables.

D.3 Prompt for ALFWorld

Loading 2403.03101v3…