PaperlayerView exact revision on arXiv

Source-linked AI summary

Ask-before-Plan: Proactive Language Agents for Real-World Planning

Xuan Zhang, Yang Deng, Zifeng Ren, See-Kiong Ng, Tat-Seng Chua

arXiv:2406.12639v2cs.CLcs.AI

TL;DR

Language agents can struggle with unclear instructions, and existing clarification approaches do not account for environment observations or how clarification affects subsequent planning. The paper introduces CEP, a three-agent framework, and the Ask-before-Plan dataset; its evaluations and analyses substantiate the framework’s effectiveness across diverse settings.

  • Problem

    Language agents can struggle with unclear instructions, while existing clarification approaches overlook environment observations and the effect of disambiguation on subsequent planning.

    Source: §1 · pp. 1–2

  • Method

    CEP uses clarification, execution, and planning agents to address proactive agent planning.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Extensive evaluations and analyses across diverse settings substantiate the effectiveness of CEP.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Ask-before-Plan provides a dataset specifically for studying Proactive Agent Planning.

    Source: §6 · p. 9

    CEP has not been verified on other benchmarks; larger models were not tested, and agent cooperation was not thoroughly explored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

The evolution of large language models (LLMs) has enhanced the planning capabilities of language agents in diverse real-world scenarios. Despite these advancements, the potential of LLM-powered agents to comprehend ambiguous user instructions for reasoning and decision-making is still under exploration. In this work, we introduce a new task, Proactive Agent Planning, which requires language agents to predict clarification needs based on user-agent conversation and agent-environment interaction, invoke external tools to collect valid information, and generate a plan to fulfill the user's demands. To study this practical problem, we establish a new benchmark dataset, Ask-before-Plan. To tackle the deficiency of LLMs in proactive planning, we propose a novel multi-agent framework, Clarification-Execution-Planning (\texttt{CEP}), which consists of three agents specialized in clarification, execution, and planning. We introduce the trajectory tuning scheme for the clarification agent and static execution agent, as well as the memory recollection mechanism for the dynamic execution agent. Extensive evaluations and comprehensive analyses conducted on the Ask-before-Plan dataset validate the effectiveness of our proposed framework.

1 Introduction

Language agents can fail on ambiguous real-world instructions, while prior clarification approaches do not fully account for environmental observations and the downstream planning effects of disambiguation. The paper formulates Proactive Agent Planning and introduces CEP and the Ask-before-Plan dataset to study it.

1 Introduction

  • Prior approaches leave two gaps in agent planning: they rely on query understanding without environmental observations and neglect how disambiguation affects subsequent planning.The paper frames these gaps as motivation for the new task.Source: §1 · pp. 1–2
  • Proactive Agent Planning requires agents to predict clarification needs from conversation and environmental observations, use tools to gather information, and produce a plan conditioned on interaction history.The benchmark is built on TravelPlanner for real-world travel planning.Source: §1 · p. 2
  • CEP assigns clarification, environment-interacting execution, and plan generation to three specialized agents.The Clarification agent asks questions to uncover intentions, Execution gathers information for clarification, and Planning aggregates the clarification process into a final plan.Source: §1 · p. 2
  • Trajectory Tuning trains the clarification and execution agents, while memory recollection targets execution-agent memory utility in long-context reasoning.The paper notes that repeated self-reflection can increase inference time and add noise to context.Source: §1 · p. 2
  • Ask-before-Plan is introduced as the first dataset for studying Proactive Agent Planning, and the authors report evaluations across diverse settings that validate CEP's effectiveness.The supplied introduction gives no numerical result for those evaluations.Source: §1 · p. 2

2 Related Work

Related work spans language agents and clarification-question systems, but prior studies often assume instructions are clear or focus on clarification without integrating the broader planning task.

2 Related Work

  • Language-agent studies cover domains such as web, games, and medicine, and typically assume user instructions are clear enough to execute.Real-world instructions may instead be succinct and ambiguous.Source: §2 · p. 2
  • Clarification research includes question ranking, end-to-end generation, LLM-generated questions, and questions informed by environmental reward feedback.The cited approaches treat clarification as a problem with need-prediction and question-generation phases.Source: §2 · p. 2
  • The paper distinguishes its task from work on implicit intentions because that work does not clarify needs from predefined preferences or cover planning and task-solving as a whole.The passage specifically names Qian et al. (2024) in this contrast.Source: §2 · pp. 2–3

3 Ask-before-Plan Dataset

Ask-before-Plan extends travel planning with uncertain instructions and proactive clarification, while defining the task across clarification, tool use, and plan generation.

3 Ask-before-Plan Dataset

  • Ask-before-Plan integrates clarification, planning, and environment interaction by adding uncertain instructions to TravelPlanner scenarios.Existing benchmarks either focus on clarification alone or assume instructions are clear and explicit.Source: §3 · p. 3

3.1 Annotation & Quality Control

  • Dataset construction has two phases: forming indefinite details and generating proactive conversations.The conversation-generation pipeline uses GPT-3.5 for simulated agent and user turns, followed by human quality control; it achieved an 87.2% human-examination pass rate.Source: §3.1 · p. 3
  • Under-specified instructions are made by removing details, while over-specified ones receive infeasible details; each instruction has 0 to 3 indefinite details.The details are ordered by dependencies, with a priority assigned to each.Source: §3.1 · p. 3

3.2 Dataset Statistics

  • The dataset contains 1,000 training samples and 1,000 testing samples, with indefinite details drawn from 11 candidate types.The passage reports 10% with no indefinite details and says 30% include 1, 2, or 3 indefinite details each.Source: §3.2 · p. 3

3.3 Problem Definition

  • Clarification need prediction uses conversation history, environment interactions, and unclarified details to decide whether to ask a question about vague or infeasible responses.Details are addressed according to topological priority, with details sharing a priority handled randomly.Source: §3.3 · p. 4
  • Tool learning generates valid function calls and observations from the ongoing conversation, while planning produces a JSON plan with predefined fields after dialogue clarification.The supplied tool-learning passages define a correct call by accurate function names and valid parameters.Source: §3.3 · p. 4

4 Method

CEP divides proactive agent planning among clarification, execution, and planning agents. Trajectory tuning trains the clarification and static execution agents, while memory recollection supports dynamic tool interaction.

4 Method

  • CEP assigns clarification, environment interaction, and plan generation to three specialized agents, with static or dynamic tool interaction options.Static execution generates the whole valid tool-call chain in one step; dynamic execution generates one action at a time using prior interaction history.Source: §4 · pp. 4–5

4.1 Trajectory Tuning

  • Trajectory tuning fine-tunes the clarification agent on conversation and interaction history, and the static execution agent on conversation-to-tool-chain examples.Training uses turn-wise trajectory sequences in an autoregressive objective.Source: §4.1 · p. 5

4.2 Memory Recollection

  • Memory recollection reuses reflective feedback across turns during dynamic tool interaction, where invalid tool calls generate rationales stored for later use.Because turns can share clarified details, the mechanism accumulates prior feedback to help prevent repeated mistakes and reduce inference time.Source: §4.2 · p. 5

4.3 Overall Framework

  • The execution agent first produces the interaction chain for the current conversation; clarification then iteratively asks about unresolved details before planning generates a JSON plan.Planning conditions on the conversation and interaction history once indefinite details have been recovered.Source: §4.3 · pp. 5–6

5 Experiment

Across clarification, tool learning, and planning evaluations, CEP performs strongly, while the integrated results show a slight clarification drop from incorrectly predicted tool use and the value of clarification for valid plans. The analyses also identify difficulties with dynamic tool interaction and planning constraints.

5.1 Experimental Setups

  • The evaluation tests clarification, tool learning, and planning separately using ground-truth outputs from earlier subtasks, then evaluates the integrated framework separately.The setup specifies subtask-specific baselines and metrics, and notes that ReAct and Reflexion are not tested for Mistral-7B because it cannot generate valid JSON plans.Source: §5.2 · p. 6, §5.1 · p. 6

5.2.1 Evaluation on Clarification

  • CEP with Mistral-7B or LLaMA-3-8B outperforms other clarification methods, and LLaMA-3-8B performs better than Mistral-7B.Direct GPT-3.5 with both observations exceeds Proactive and ProCoT on question quality and macro clarification accuracy, but remains below the environment-only method overall.Source: Table 2 · p. 6, §5.2.1 · p. 7

5.2.2 Evaluation on Tool Learning

  • Dynamic tool-learning agents generally perform worse than static agents, while brute-force leads the dynamic setting.In the static setting, LLaMA-3-8B even surpasses brute force; dynamic agents do not consistently generate correct actions throughout an interaction.Source: §5.2.2 · p. 7, Table 3 · p. 7

5.2.3 Evaluation on Planning

  • Only GPT-3.5 with direct prompting produces a passed planning result, succeeding on one of 1,000 test samples.Greedy Search reaches a 100% delivery rate and meets most constraints, but does not produce a valid plan; Mistral ReAct and Reflexion enter a fail-retry loop.Source: §5.2.3 · p. 7, Table 4 · p. 8

5.2 Overall Evaluation

  • Table 5’s integrated evaluation uses execution interactions generated by CEP (LLaMA-3-8B) for subsequent clarification and final planning.The clarification stage also uses environment interactions from the execution agent and the current conversation history.Source: §5.2.4 · pp. 7–8, Table 5 · p. 8

5.2.4 Evaluation on the Integral Framework

  • Without clarification, the integrated planning agent fails to formulate valid plans; integrated clarification performance also drops slightly when tool use is incorrectly predicted.The clarification result is compared with the ground-truth-supervised independent version.Source: §5.2.4 · p. 8, Table 5 · p. 8

5.3 Detailed Analysis

  • CEP (LLaMA-3-8B) leads other methods in both clarification clearness and vagueness judgment.Proactive and environment-only methods tend to ask for clarification, while ProCoT, Direct, and ICL tend not to clarify the conversation.Source: §5.3 · p. 8, Figure 4 · p. 8
  • CEP integral underperforms Greedy Search on most planning constraints, but clarification supports most commonsense constraints and all hard constraints.Minimum nights stay is an exception; lengthy clarification dialogues may add noise to planning for that mathematically demanding constraint.Source: §5.3 · pp. 8–9, Table 6 · p. 8

6 Conclusion

The paper frames Proactive Agent Planning as a practical problem for agents handling unclear user instructions, and introduces CEP and Ask-before-Plan to study it. Evaluations and analyses across diverse settings support the framework’s effectiveness.

6 Conclusion

  • Proactive Agent Planning addresses challenges LLM-powered agents face when handling unclear user instructions.The task is introduced as a practical problem.Source: §6 · p. 9
  • CEP combines clarification, execution, and planning agents, with trajectory tuning and memory recollection for challenges in tool use and long-context interactions.Trajectory tuning is used for clarification and static execution; memory recollection is used for dynamic execution.Source: §6 · p. 9
  • Ask-before-Plan is the first dataset specifically constructed for Proactive Agent Planning, and evaluations across diverse settings support CEP’s effectiveness.Source: §6 · p. 9

Limitations

The paper’s evaluation uses static user simulation, and CEP’s effectiveness has not been verified on other benchmarks. The authors also report limits on model scale and their exploration of agent cooperation.

Limitations

  • Clarification evaluation uses static rather than real-time user simulation to focus on question generation without interference from incorrect simulated users.The authors suggest extending the task to dynamic real-time user simulation in future work.Source: Limitations · p. 9
  • CEP’s effectiveness has not been verified on other benchmarks because no other suitable datasets are available for the newly introduced task.Source: Limitations · p. 9
  • Limited computing resources prevented experiments with larger-size models.Source: Limitations · p. 9
  • The work does not thoroughly explore cooperation and division of labor among LLM-based agents.Source: Limitations · p. 9

Ethical Consideration

The passage states that Ask-before-Plan was built from the publicly available TravelPlanner dataset, with permission from its authors.

Ethical Consideration

A Dataset Specifications

The dataset adapts TravelPlanner for proactive information seeking, pairing unclear travel instructions with generated user-agent conversations and a modified tool environment.

A Dataset Specifications

  • Ask-before-Plan adapts TravelPlanner by specifying destination cities and arrival days, using departure dates, and combining room type and house rules as accommodation.The environment is modified to support proactive information seeking.Source: §A.1 · p. 12, A Dataset Specifications · p. 12

A.1 Indefinite Detail Formation

  • Each instruction receives zero to three indefinite details, reordered by dependency before the rewritten instruction is generated.Details include unavailable preferences and infeasible budgets; Figure 5 presents their dependency graph.Source: §A.1 · p. 12, Figure 5 · p. 12

A.2 Proactive Conversation Generation

  • GPT-3.5 generates proactive conversations using prompts that include XML and Python-type structures, with keyword checks and proofreading of a random 20% of samples.The prompts guide role-playing dialogue generation.Source: §A.2 · p. 12

A.3 Environment Settings

  • The environment removes NotebookWrite and CitySearch, adds BudgetEstimator, and updates accommodation and restaurant search parameters to reflect dataset preferences.BudgetEstimator calculates the minimum trip budget from the cheapest accommodation and transportation options.Source: §A.3 · pp. 12–13

B Details of Experimental Setups

The experimental setup defines baselines and task-specific metrics, then specifies model training, inference, and interaction constraints for evaluation.

B.1 Baselines

  • Baselines cover clarification strategies, tool-use methods, and planning approaches, including direct prompting and in-context examples for clarification.Execution includes brute-force, Direct, ToolLLM, ReAct, and Reflexion; planning includes Greedy Search, Direct, and CoT.Source: §B.1 · p. 14

B.2 Evaluation Metrics

  • Micro averages conversational metrics per turn and macro per sample, while planning metrics follow TravelPlanner’s constraint-pass definitions.Clarification evaluates prediction accuracy and clear/vague recall; execution evaluates tool-call matching and pass rates; planning evaluates delivery and constraint pass rates.Source: §B.2 · p. 15

B.3 Implementation Details

  • The authors fine-tune selected models with 4-bit LoRA for five epochs and a maximum context length of 4096, using one NVIDIA A5000 24GB.Dynamic execution and planning exclude LLaMA-3-8B because of its limited context window; dynamic strategies allow up to 30 interaction steps and three action retries.Source: §B.3 · p. 16

C Additional Study on Dynamic Execution Agents

C.1 Analysis of Time Complexity

Memory recollection reuses rationales across turns, reducing rationale inference complexity; runtime and execution-step comparisons examine its effects alongside ReAct and Reflexion.

C.1 Analysis of Time Complexity

  • Figure 6 compares average runtime and execution steps: Reflexion takes longer than ReAct for both models, while memory recollection alleviates that runtime increase.Mistral-7B takes longer on average than GPT-3.5; execution steps decrease from ReAct to memory recollection for Mistral-7B but increase for GPT-3.5.Source: Figure 6 · p. 16, §C.1 · p. 16
  • Rationale inference complexity falls from linear to constant time when memory recollection reuses prior rationales across turns.The analysis contrasts cumulative rationale generation in Reflexion with retaining the maximum rationale set across turns in memory recollection.Source: §C.1 · p. 16

C.2 Analysis of Execution Status

Execution-status distributions differ by model: reflective strategies reduce some error types, but their success and finish outcomes diverge between Mistral-7B and GPT-3.5.

C.2 Analysis of Execution Status

  • GPT-3.5’s performance improvements appear as higher Success and Finish rates and lower error rates.The figure omits execution statuses with proportions below 0.1%.Source: §C.2 · pp. 16–17, Figure 7 · p. 17
  • Figure 7 shows Reflexion eliminates Invalid API, Retry Limit Exceeded, and Invalid Parameters errors; Mistral-7B nevertheless has lower Success and Finish rates under Reflexion and memory recollection.For Mistral-7B, API Limit Exceeded and Null Action exceptions rise.Source: §C.2 · p. 16, Figure 7 · p. 17

D Case Study

This section examines how clarification of missing or infeasible travel details shapes subsequent plans and describes the planning subtask, in which agents generate structured itineraries from conversation and interaction data. It also documents prompts used across annotation and agent subtasks, including dynamic planning actions and handling errors or failed attempts.

D Case Study

  • Without clarification, a missing destination confines the first-day itinerary to Nashville, while unaddressed unavailable Houston accommodation preferences leave the plan incomplete.The comparisons examine plans generated with and without clarification for a missing destination and unfeasible accommodation detail.Source: D Case Study · p. 17, Table 8 · p. 18, Table 9 · p. 18

E.1 Data Annotation

  • The data-annotation appendix presents prompts for original-instruction revision and examples of missing and unfeasible details in proactive conversation generation.These materials appear in Tables 10–13.Source: §E.1 · p. 17

E.2 Clarification Subtask

  • Clarification prompts use execution-stage interaction history and include conversation-only, direct-strategy, and evaluation prompts.The appendix provides the relevant tool description and prompts for conversation-only methods, including Proactive and ProCoT, as well as direct clarification and GPT-4 evaluation.Source: §E.2 · p. 17

E.3 Execution Subtask

  • The execution appendix presents tool descriptions, static and dynamic prompts, and error prompts for dynamic interaction.These are shown in Snippets 4–7.Source: §E.3 · p. 17

E.4 Planning Subtask

  • The planning appendix presents prompts for static planning, dynamic planning, and errors in dynamic interaction.These are shown in Snippets 8–10.Source: §E.4 · p. 17
  • Error prompts address unparseable JSON, unsupported multi-day subplans, and invalid actions, while Reflexion prompts provide suggestions after a failed subplan attempt.Valid actions are CostEnquiry(subplan) and Finish(final_json_plan).Source: §E.4 · pp. 27–28
  • The planner must produce a JSON travel itinerary using only provided information, with specified fields, flight and cost labels, and conventions for city changes and unnecessary details.The instructions also require diverse attractions and meals and prohibit dollar signs and comments in the plan.Source: §E.4 · p. 28
  • Dynamic planning alternates Thought, Action, and Observation, using CostEnquiry for a complete one-day subplan and Finish to submit the final JSON plan.Each action calls one function once; the final response must use Finish(final_json_plan).Source: §E.4 · p. 28

Related papers

Loading 2406.12639v2…