Source-linked AI summary
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su
TL;DR
The paper asks whether LLM-powered language agents can plan in realistic, complex settings beyond the constrained scenarios of earlier AI systems. It introduces TravelPlanner, a travel-planning benchmark with rich data, tools, and multi-constraint queries, and finds that current agents perform poorly, with GPT-4 achieving only a 0.6% final pass rate. The benchmark is intended as a challenging testbed for improving complex planning agents.
Problem
The paper investigates whether language agents can perform planning in realistic, complex settings involving long horizons, multiple constraints, and tool-mediated information gathering.
Method
TravelPlanner provides around four million data entries accessed through six tools and 1,225 curated travel queries with reference plans and varied constraints.
Results
0.6% final pass rate was achieved by the most advanced evaluated language agents, while agents generally failed to consider all constraints and produce feasible plans.
Takeaways & Limitations
TravelPlanner offers a challenging benchmark for developing more sophisticated strategies for multi-constraint planning and tool use.
Takeaways & Limitations
The benchmark’s commonsense criteria reflect the authors’ consensus, and some environment data comes from publicly available Internet sources.
Abstract
from arXiv · showhide
Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.
1. Introduction
TravelPlanner tests whether language agents can plan realistic, multi-day travel under interdependent explicit and commonsense constraints. Its benchmark exposes severe limitations in current agents while offering a challenging testbed for future planning research.
- Motivation: Travel planning combines long-horizon decisions, explicit user requirements, implicit commonsense constraints, and proactive tool use.The task requires coordinating places, lodging, transportation, dining, and information gathered from a partially observable environment.
- Benchmark: TravelPlanner provides around four million data entries, six tools, and 1,225 curated queries with reference plans.Each query imposes a different combination of constraints.
- Findings: GPT-4 satisfies all constraints on only 0.6% of tasks, while all other evaluated LLMs complete none.This result indicates that current state-of-the-art language agents cannot reliably handle TravelPlanner’s complex planning tasks.
- Findings: ReAct and Reflexion are insufficient for TravelPlanner’s multi-constraint tasks because agents often convert reasoning into incorrect actions and lose track of global constraints.The authors argue that more sophisticated planning strategies are needed to approach human-level planning.
- Findings: Common failure modes include tool-use argument errors, dead loops, and hallucinations.These analyses identify concrete operational problems beyond final-plan correctness.
- Implications: TravelPlanner is a challenging testbed for future agents, and agents can generate plans in 1–2 minutes versus 12 minutes for human annotation.The authors present the benchmark’s research value alongside a potential time-saving advantage.
2. Related Work
Prior evaluations commonly isolate arithmetic reasoning, tool use, or web navigation rather than testing integrated planning in complex real-world settings. TravelPlanner addresses this gap by evaluating language agents’ tool use and multi-constraint planning in travel scenarios.
- LLM-Based Agents: Language-agent research has expanded from LLM reasoning into systems with memory, tool use, and planning modules.These agents decompose tasks and produce solutions through sequences of reasoned actions.
- Planning: Planning involves decomposing tasks, searching for solutions, and making final decisions across domains including robotics and transportation scheduling.The paper frames planning as a central capability associated with human-level intelligence.
- Evaluation Gap: Previous studies typically focus separately on arithmetic reasoning, tool use, or web navigation, leaving complex real-world planning less directly assessed.TravelPlanner evaluates flexible travel planning through diverse tools while satisfying user and commonsense constraints.
3. TravelPlanner
TravelPlanner is a benchmark for evaluating language agents’ tool use and complex planning under environmental, commonsense, and personalized constraints. It combines a static sandbox, diverse travel queries, validated reference plans, and multidimensional evaluation metrics.
- Benchmark scope: TravelPlanner evaluates language agents in tool use and complex planning while satisfying environmental, commonsense, and hard constraints.Hard constraints represent personalized user needs, while commonsense and environment constraints test realistic planning behavior and adaptability.
- Dataset construction: The benchmark contains 1,225 queries divided into nine groups by travel duration and number of hard constraints, with 45 training, 180 validation, and 1,000 test queries.Training queries include human-annotated plans, and the dataset construction varies trip duration and constraint count to create different complexity levels.
- Environment and tools: TravelPlanner uses a static, closed sandbox with shared databases and a NotebookWrite tool for recording information during planning.The setup avoids variability from dynamic data while testing working-memory management and limiting context accumulation.
- Reference plans: Human annotation and quality control produced 1,225 validated query-plan pairs, requiring each reference plan to satisfy all constraints in the evaluation script.Reviewers checked every query and plan, and budgets were recalibrated using costs from human-annotated plans.
- Evaluation: Evaluation extracts transportation, restaurants, attractions, and accommodations into structured plans before automatically testing delivery and constraint satisfaction.Delivery Rate counts plans completed within 30 steps, while commonsense and hard constraints are assessed using micro and macro pass rates.
4. Experiments
TravelPlanner evaluates language models and planning strategies under complex tool-use and multi-constraint conditions. Results show severe failures in delivering valid plans, with performance worsening when agents must both collect information and plan.
- Baselines: The evaluation compares LLMs under ReAct-based information collection and tests Direct, ZS-CoT, ReAct, and Reflexion as planning strategies.Greedy search is included as a cost-optimization baseline.
- Main Results: GPT-4-Turbo with ReAct achieves only 0.6% in final pass rate, while none of the other LLMs pass any task in two-stage mode.
- Main Results: Across all metrics, every model scores lower in two-stage mode than sole-planning mode, with the largest gap exceeding 30%.The difference exposes difficulty in jointly collecting information and planning.
- Main Results: Agents often achieve higher micro scores than macro scores, indicating that they satisfy some constraints while overlooking others.This reflects difficulty considering multiple constraints holistically.
- Main Results: TravelPlanner remains challenging despite strong performance of existing agents on traditional tasks, providing a benchmark for developing more capable language agents.
5. In-Depth Analysis
The analysis identifies failures in tool use, constraint satisfaction, information collection, global planning, and self-correction that prevent agents from completing TravelPlanner plans.
- Tool-Use Errors: GPT-4-Turbo agents produce 37.3% of errors from invalid actions and 6.0% from repetitive action loops, persisting despite invalid-action or null-result feedback.These loops indicate failure to dynamically adjust plans using environment feedback.
- Constraint Analysis: Agents consistently achieve below 10% pass rates across hard-constraint difficulty levels, with performance deteriorating as more constraints are introduced.This pattern highlights persistent difficulty with multi-constraint planning.
- Information Collection: Agents often fail to complete information collection, causing made-up information or omitted details and lower “Within Sandbox” and “Complete Information” pass rates.The discrepancy from reference tool use becomes more pronounced as travel duration increases.
- Global Planning: Global constraints such as “Minimum Nights Stay” and “Budget” require anticipating future implications, which current autoregressive models handle poorly.The analysis points to backtracking and heuristic forward-looking planning as potential strategies.
- Case Studies: Incorrect date inputs can yield null results because the sandbox data is based on 2022, eventually causing agents to stop planning.The cases indicate that agents cannot self-correct initial incorrect assumptions.
- Case Studies: Agents confuse information details, such as reusing one flight number for departure and return flights, producing plans misaligned with sandbox data.The analysis associates this behavior with difficulty handling mass information, described as “Lost in the Middle.”
6. Conclusion
TravelPlanner is introduced as a real-world benchmark for evaluating multi-constraint planning and tool use in language agents. The benchmark is highly challenging: agents achieve only a 0.6% final pass rate and fail to consider all constraints in feasible plans.
- Conclusion: TravelPlanner evaluates language agents’ multi-constraint planning and tool-use abilities in real-world scenarios.The benchmark is designed to assess whether agents can deliver plans under realistic complexity.
- Conclusion: 0.6% is the final pass rate achieved by the most advanced language agent frameworks on the benchmark.Further analysis shows that agents cannot take all constraints into consideration when delivering feasible plans.
- Conclusion: TravelPlanner is intended as a benchmark for developing language agents in increasingly complex scenarios.The authors position its logic and applicability as components of progressive agent development.
7. Impact Statements
The impact statement positions TravelPlanner as a future-research benchmark while acknowledging data provenance and disagreement about commonsense evaluation.
- Impact Statements: TravelPlanner is intended to provide an effective benchmark for complex planning research.The authors plan to release evaluation scripts and encourage feedback-informed methods while prohibiting cheating on validation and test sets.
- Impact Statements: Some environment data comes from publicly available Internet sources and does not represent the authors’ viewpoints.This statement separates the benchmark’s data content from the authors’ positions.
- Impact Statements: The commonsense evaluation criteria reflect the authors’ consensus, and the authors encourage further discussion to enrich this dimension.The statement acknowledges that commonsense may be defined differently by different people.
Appendices
The supplementary material is organized into appendices covering benchmark details, experiment details, and case presentations.
- Appendices: Appendix A covers benchmark details.
- Appendices: Appendix B covers experiment details.
- Appendices: Appendix C covers case presentations.
A.1. Dataset Distribution
TravelPlanner documents its dataset distribution across nine query groups and provides detailed dataset and tool descriptions in Tables A.1 and A.2.
- Table A.1 reports the detailed distribution of queries across the training, validation, and test sets.
- Table A.2 describes the tools and reports the number of data entries in each database.
A.3. Environment Database Construction
The environment constructs fixed travel databases by adapting public flight, restaurant, accommodation, distance, and attraction data for benchmark-specific constraints and consistency.
- FlightSearch uses selected fields from a Kaggle flight dataset covering March 1–April 1, 2022, and adds synthetic prices based on distance.Prices are generated by multiplying distance by a random factor ranging from 0.2 to 0.5.
- DistanceMatrix uses Google Distance Matrix API results to provide driving distances, travel times, and prices for self-driving and taxi modes.The search results are stored in a database to create a fixed evaluation dataset.
- RestaurantSearch adapts Kaggle restaurant data by retaining restaurant names and average costs, randomly assigning cities, and categorizing cuisines.
- AttractionSearch uses Google Places data, retaining identifying and location fields before storing results in a standardized database.
- AccommodationSearch adapts Airbnb data with lodging attributes and randomly assigns cities and room rules to satisfy benchmark constraints.The assigned rules include restrictions on parties, smoking, children under 10, pets, and visitors.
B. Experiment Details
The experiments compare greedy search, four planning strategies, and five long-context LLMs within a tool-based TravelPlanner environment.
- Experiment Details: Greedy search is included as a rule-based baseline that optimizes travel-plan cost.
- Planning Strategies: The study evaluates Direct, ZS-CoT, ReAct, and Reflexion strategies for complex, constraint-rich travel planning.ToT and GoT are excluded because their search-space exploration is prohibitively costly for TravelPlanner.
- Results: GPT-4-Turbo results with Reflexion on the validation set are reported in Table B.3.
- ReAct: ReAct interleaves Thought, Action, and Observation steps, stores retrieved information in Notebook, and submits it to a Planner tool.The framework uses travel-specific search tools and prohibits nested tool use.
- Direct: The Direct strategy instructs the model to generate detailed plans from supplied information while adhering to the requested format and commonsense.The examples specify outputs such as flight numbers, restaurant names, and accommodation names.
- ReAct and Reflexion: ReAct and Reflexion can use CostEnquiry for complete one-day subplans and Finish for the final complete plan.
C.2. Additional Case Study
Additional case studies present travel queries and figures covering two-stage, direct, and Reflexion planning strategies in sole-planning mode.
- Additional Case Study: The case studies include 3-day, 5-day, and week-long travel queries with varying group sizes, destinations, budgets, and constraints.
- Two-stage Mode: Figures C.1–C.3 document Cases 4–6 using Two-stage Mode.
- Additional Case Study: One trajectory shows flight-search actions followed by a Planner call after alternative transportation could not be searched under the nested-tool restriction.
- Direct strategy in sole-planning mode: Figures C.4–C.6 document Cases 7–9 using the Direct strategy in sole-planning mode.
- Reflexion strategy in sole-planning mode: Figure C.7 documents Case 10 using the Reflexion strategy in sole-planning mode.