PaperlayerView exact revision on arXiv

Source-linked AI summary

UTP-Bench: Uncertainty-aware Travel Planning Benchmark

Etcharla Revanth Rao, Priyanshu Karmakar, Shubhojit Mallick, Manish Gupta, Shreya Ghosh, Abhik Jana

arXiv:2609.02421v1cs.AIcs.CL

TL;DR

Existing travel benchmarks largely evaluate deterministic feasibility, leaving robustness to transit delays, crowd fluctuations, and stochastic disruptions underrepresented. UTP-Bench introduces a large-scale uncertainty-aware benchmark with empirical uncertainty signals, traveler risk profiles, and robustness metrics. Experiments show that current LLMs remain substantially below human-authored plans, especially in temporal buffering, delay-aware transportation scheduling, and crowd-sensitive planning.

  • Problem

    Existing benchmarks assume deterministic environments and static constraint satisfaction, limiting evaluation of itinerary robustness under real-world uncertainty.

  • Method

    UTP-Bench combines travel data, empirical delay and crowd signals, risk-aware traveler profiles, and BAS, TDAS, and CATS for uncertainty-aware itinerary evaluation.

  • Results

    Current LLMs perform substantially below human-authored plans across settings and struggle with temporal uncertainty, transportation variability, traveler risk preferences, and long-horizon dependencies.

  • Takeaways & Limitations

    UTP-Bench advances travel-planning evaluation from deterministic feasibility toward robustness under realistic uncertainty.

  • Takeaways & Limitations

    UTP-Bench is static, relies on historical rather than real-time signals, and is geographically concentrated in India.

Abstract

from arXiv · show

Large Language Models (LLMs) have recently demonstrated strong capabilities in automated travel itinerary generation. However, real- world travel planning is inherently uncertain: transportation delays, crowd fluctuations, and unexpected stochastic delays frequently inval- idate otherwise feasible schedules. Existing benchmarks like TravelPlanner and TripCraft assume deterministic environments, evaluating only static constraint satisfaction and ignoring whether generated plans remain robust when such uncertainties arise. To address this limitation, we introduce UTP-Bench1 , a large-scale benchmark for uncertainty-aware travel planning. The dataset integrates real-world travel data spanning 504 cities of India, including attractions, restau- rants, accommodations, and multi-modal trans- portation networks. To model realistic disrup- tions, UTP-Bench incorporates empirical delay distributions and crowd-density patterns col- lected from major cities, enabling evaluation of travel plans under stochastic conditions. We further propose three evaluation metrics, namely Buffer Adequacy Score (BAS), Crowd- Aware Timing Score (CATS), and Transport Delay Absorption Score (TDAS), which quan- tify the ability of generated itineraries to main- tain robustness against transit delays and crowd variability. Experiments with state-of-the-art LLMs like GPT-5, Qwen3, Mistral and Phi-4 re- veal substantial gaps between model-generated and human-authored plans, particularly in tem- poral buffering, delay-aware transportation scheduling, and crowd-sensitive planning.

1 Introduction

Existing travel-planning benchmarks largely assume deterministic conditions and static feasibility, overlooking robustness to delays, congestion, and traveler risk tolerance. UTP-Bench addresses this gap with uncertainty-aware data, profiles, and metrics for evaluating robust itineraries.

  • Research gap: Existing benchmarks assume fixed schedules and conditions, so static constraint satisfaction does not measure itinerary robustness under realistic disruptions.Travel delays, congestion, and unexpected disruptions can make otherwise feasible itineraries impractical.
  • Benchmark: UTP-Bench integrates attractions, restaurants, accommodations, and multimodal transportation data with empirical transit delays and crowd-density patterns.The benchmark supports evaluation of travel plans under realistic uncertainty.
  • Evaluation metrics: BAS, TDAS, and CATS evaluate temporal buffers, transportation-delay resilience, and avoidance of peak crowd periods, respectively.These metrics move evaluation beyond binary feasibility checks.
  • Traveler profiles: Risk-Tolerant, Risk-Optimized, and Risk-Averse profiles represent different tolerances to scheduling uncertainty and influence itinerary evaluation.The profiles capture variation in acceptable buffer time and itinerary density.

2 Related Work

Prior work shows that LLMs can support structured planning, but reliable open-ended planning remains difficult under dependencies, cascading violations, and uncertainty. UTP-Bench is positioned as a benchmark that explicitly evaluates stochastic robustness in travel planning.

  • LLM planning: LLM planning research combines language-model reasoning with grounded few-shot methods and classical search techniques such as Monte Carlo Tree Search.These approaches target scheduling, commonsense reasoning, and heuristic-guided decision-making.
  • Open challenges: LLMs still struggle with subgoal dependencies, cascading constraint violations, and uncertainty in open-ended planning domains.Travel planning inherits these reliability challenges while requiring reasoning over interconnected constraints.
  • UTP-Bench: UTP-Bench explicitly models stochastic robustness in LLM-based travel planning under empirical uncertainty distributions.The benchmark extends evaluation beyond deterministic travel-planning assumptions.

3 UTP-Bench Dataset Curation

UTP-Bench combines structured travel queries, geographically broad Indian travel data, uncertainty signals, constraints, personas, and quality-controlled human itineraries. Its curation process supports robustness evaluation against transport delays and time-dependent crowd patterns.

  • Dataset scope: UTP-Bench contains 1,000 queries covering 3-, 5-, and 7-day itineraries, each paired with a human-annotated gold-standard itinerary.The benchmark also includes model-generated itineraries produced under uncertainty-aware prompting settings.
  • Data sources: The dataset covers 504 cities across 33 Indian states and union territories, with transportation, attractions, restaurants, accommodations, and related metadata.Records are collected from real-world sources and normalized for consistency.
  • Uncertainty modeling: Historical transit delays and crowd-density patterns are added as uncertainty signals for evaluating robustness beyond conventional constraint satisfaction.Signals include delays for flights, trains, buses, and cabs and crowd information from 20 major Indian cities.
  • Constraints and personas: Uncertainty constraints require buffers for empirically observed transit delays and adjustments for crowd-dependent activity durations.Risk-Tolerant travelers accept tighter schedules, Risk-Averse travelers prefer larger recovery margins, and Risk-Optimized travelers lie between them.
  • Annotation and quality control: Fourteen trained annotators refined gold itineraries using contextual delay, crowd, and visit-duration information, followed by expert review and automated structural checks.Each itinerary also includes a rationale explaining planning decisions and uncertainty-buffer choices.
  • Uncertainty distributions: Train delays average 40.8–46.7 minutes, flight delays average 40.59 minutes, and road travel has the highest average delay at 52.2 minutes.Mean delays exceed medians across modes, while attractions are more crowded during daytime and afternoons and restaurants show stronger evening concentration.

4 Evaluation Metrics

UTP-Bench evaluates travel plans with conventional feasibility metrics alongside BAS, CATS, and TDAS, three uncertainty-aware measures of robustness under stochastic disruptions. These metrics assess buffer alignment, crowd-sensitive timing, and transport-delay absorption.

  • UTP-Bench combines five metrics adopted from TripCraft with BAS, CATS, and TDAS for evaluating robustness under stochastic travel disruptions.The adopted metrics cover temporal, spatial, persona-specific, and ordering properties, while the three new metrics target uncertainty-aware planning.
  • Buffer Adequacy Score (BAS): BAS measures whether allocated POI-transition buffers fall within an uncertainty-adjusted acceptable range shaped by crowd congestion and traveler risk preferences.The expected range uses crowd congestion and risk-profile scaling parameters; deviations are penalized using the nearer boundary to avoid over-penalization.
  • Crowd-Aware Timing Score (CATS): CATS evaluates whether POI visits overlap favorable low-crowd periods, producing normalized scores where higher values indicate better crowd-aware timing.For each visit, crowd intensity is computed from overlap with predefined crowd patterns, and the resulting POI score is normalized to [0, 1].
  • Transport Delay Absorption Score (TDAS): TDAS evaluates whether transport buffers can absorb realistic delays without propagating disruptions to subsequent itinerary activities.Expected delays are estimated from historical data across transport modes, and buffer deviations are penalized for both under-allocation and over-allocation.
  • Transport Delay Absorption Score (TDAS): Higher TDAS values indicate transport buffers that are better aligned with expected delays.The score is intended to distinguish fragile schedules from schedules with inefficiently excessive buffers.

5 Experimental Results and Analysis

Experiments show that uncertainty-aware metrics expose distinct weaknesses in LLM-generated itineraries, especially for transportation delays, temporal buffering, and longer planning horizons. Agentic decomposition and deterministic planning baselines do not eliminate these robustness and constraint-satisfaction failures.

  • Key observations: CATS scores reach 47–64%, exceeding BAS and TDAS and indicating that avoiding crowded periods is easier than modeling temporal or transportation uncertainty.Phi-4-mini-Instruct reaches 63.94% for Risktol at 7-day, while stronger crowd-aware behavior does not imply stronger overall robustness.
  • Key observations: TDAS scores remain uniformly low at 0.37–11.85%, substantially below human performance.GPT-5 performs best at 11.85% for Riskav, while Qwen3 declines from 7.58% to 1.57% for Risktol as the horizon increases.
  • Key observations: Uncertainty-aware metrics reveal model-specific failures: GPT-5 and Qwen3 struggle with buffering, Qwen3 degrades over longer horizons, and all models struggle with Riskopt.Phi-4’s strong crowd-aware behavior masks weak temporal robustness.
  • Key observations: Performance declines with itinerary length, and human-annotated plans also fall from 91% to 63% between 3-day and 7-day itineraries.The results indicate that long-horizon planning is challenging for humans as well as models, although models remain substantially below human levels.
  • Agentic baseline: ATLAS does not consistently outperform uncertainty-aware prompting and performs lower on delivery rate, constraint preservation, BAS, and TDAS for longer itineraries.Its performance is competitive on selected crowd-awareness measures such as CATS, but agentic decomposition alone does not fully address UTP-Bench’s challenges.
  • Performance Without Uncertainty: Without explicit uncertainty modeling, qualitative metrics remain relatively stable while constraint satisfaction degrades sharply as itinerary length increases.Macro-CPR falls from 29% to 4.6%, macro-HCPR from 10.66% to 0%, and Final Pass Rate from 2.5% to 0% across 3-, 5-, and 7-day plans.

6 Conclusion

UTP-Bench is presented as a high-fidelity benchmark for evaluating uncertainty-aware travel itinerary planning beyond deterministic constraint satisfaction. The conclusion highlights its empirical uncertainty signals and metrics while identifying static data, geographic, language, and scope limitations.

  • UTP-Bench bridges deterministic planning assumptions and stochastic real-world travel by testing itinerary robustness under uncertainty.
  • The benchmark combines fine-grained attraction categories, traveler risk profiles, transportation-delay data, and crowd-dynamics signals for spatio-temporal reasoning.
  • BAS, CATS, and TDAS evaluate itinerary robustness beyond conventional constraint-satisfaction metrics.
  • Limitations: UTP-Bench is static and relies on historical statistics rather than real-time transportation and traffic signals.The paper identifies live API integration as a future extension.
  • Limitations: The benchmark is geographically concentrated on India and designed primarily in English, limiting current geographic and multilingual coverage.The paper notes that broader regional coverage and regional-language support remain future directions.
  • Conclusion: The study establishes a benchmark and evaluation framework rather than proposing a novel planning architecture.Future work may evaluate agentic and tool-augmented planning paradigms under realistic travel uncertainty.

A Data Sourcing Details

UTP-Bench combines India-specific data from travel, transportation, event, crowd, and pricing sources to construct realistic itinerary components. The dataset covers attractions, restaurants, accommodations, multimodal transport, and events across India.

  • Restaurants: Restaurant records used TripAdvisor attributes, Numbeo city-specific price indices, and BestTime crowd and peak-hour data.TripAdvisor price tiers were converted into estimated absolute prices using city-level indices.
  • Attractions: Attraction data came from TripAdvisor, with visit durations estimated from TripCraft category averages when predefined durations were unavailable.Crowd patterns for attractions were collected through BestTime across 20 major Indian cities.
  • Accommodations: Accommodation listings came from TripAdvisor, while Numbeo baselines calibrated nightly prices and defined each day’s itinerary start and end points.Listings included property type, star rating, amenities, and coordinates.
  • Transportation: Transportation data covered 68,068 train records, 34,925 bus records, and 29,346 domestic flight records sourced from Tripozo.Historical train, bus, and flight delay information supplied empirical parameters for TDAS modeling.
  • Events: Event data scraped from District by Zomato yielded 783 concerts, sports, arts, theatre, music, and film events across India.The event collection broadens the benchmark’s itinerary content beyond fixed attractions and services.

A.8 Road Delay and Traffic Data

UTP-Bench models city-to-city road delays using TomTom Traffic API data from 4,052 routes across India. Each route records estimated delay, historical delay, and free-flow traffic information.

  • Road Delay Coverage: TomTom Traffic API data covered 4,052 city-pair routes across India for road-delay modeling.The route data provides the empirical basis for representing road uncertainty in the benchmark.
  • Recorded Traffic Variables: Each route entry includes estimated delay, historical delay, and free-flow traffic measurements.These fields characterize current, historical, and uncongested travel conditions.

A.9 Crowd Density and Best-Time Data

UTP-Bench combines venue crowd patterns, cost data, travel distances, routing times, and transit-stop proximity to support crowd-sensitive and spatially grounded itinerary planning. The section also references category-wise attraction visiting durations in Table 7.

  • Crowd Data: BestTime supplies hour-by-hour crowd intensity classifications for attractions and restaurants across 20 major Indian cities.Five tiers—Low, Below Average, Average, Above Average, and High—provide congestion factors used in BAS and risk values.
  • Cost Data: Numbeo data from 20 major Indian cities provides meal, local transport, and accommodation cost baselines for budget calibration.These baselines also convert TripAdvisor price tiers into absolute restaurant and accommodation estimates.
  • Attraction Duration: Table 7 reports attraction visiting durations in hours by category, allowing attractions with multiple categories to receive category-based duration information.The table notes that an attraction can belong to one or more categories.
  • Routing Data: TomTom Routing API computed 253,513 city-pair distance and travel-time entries across the 504-city benchmark.These entries support spatial and temporal reasoning between cities.
  • Transit Accessibility: OpenStreetMap distances identified 10,025 nearest public-transit stops for accommodations, restaurants, and attractions.The resulting mappings enable transit-aware itinerary generation across urban and semi-urban locations.

B Evaluation Metrics (Details)

The paper evaluates travel itineraries with metrics covering temporal realism, crowd awareness, transport robustness, and overall schedule resilience. These metrics are computed independently for human-annotated and LLM-generated plans.

  • Four complementary metrics evaluate temporal realism, crowd awareness, transport robustness, and global schedule resilience.

B.1 Buffer Adequacy Score (BAS)

This section defines inputs and procedures for evaluating itinerary buffers, crowd-aware timing, and transport-delay absorption. The metrics compare planned schedules with visit-duration, crowd, and historical transport information.

  • B.1 Buffer Adequacy Score (BAS): BAS evaluates whether each POI buffer matches visit duration, crowd congestion, and the traveler’s risk profile.
  • B.1 Buffer Adequacy Score (BAS): BAS values near 1 indicate buffers aligned with expected planning behavior, whereas values near 0 indicate buffers that are too short or too long.
  • B.2 Crowd-Aware Timing Score (CATS): CATS scores POI timing against crowd windows, quiet hours, and peak hours using place-specific crowd data.
  • B.3 Transport Delay Absorption Score (TDAS): TDAS evaluates whether transport buffers absorb historical delays across flights, trains, buses, and cabs without disrupting later activities.
  • B.3 Transport Delay Absorption Score (TDAS): Transport evaluation uses planned and historical durations, mode-specific delay information, road-delay data, and risk-profile buffer ranges.

D Case Studies

The case studies illustrate both robust and flawed itinerary behavior, including well-calibrated buffers, delay-aware transport, inconsistent timings, missing uncertainty information, excessive buffers, and temporal overlaps.

  • BAS cases: A good BAS case uses well-calibrated POI buffers consistent with a risk-averse profile.
  • TDAS cases: A good TDAS case includes a delay-aware train buffer within the risk-averse acceptable range.
  • Failure cases: Other cases show inconsistent filling timings, missing buffer or crowd information, and excessive crowd buffers that inflate visit duration.
  • Failure cases: A separate case demonstrates temporal overlap between consecutive activities, indicating schedule infeasibility.
  • Annotation context: The annotation process used structured skeletons and helper scripts, with graduate student annotators voluntarily contributing without separate monetary compensation.

E.2 Annotator Demographics

The benchmark’s annotators were a young, India-based graduate student cohort with varied regional backgrounds and substantial formal English education. Annotation guidelines supported the construction of realistic travel plans.

  • Demographics: Most annotators had 13–18 years of formal English education, indicating substantial language-training experience.
  • Demographics: Annotators were recruited from India, consistent with the benchmark’s India-centric scope, and represented diverse geographic regions.
  • Annotation process: Annotation guidelines covered travel-plan constraints, personas, and practical scheduling considerations.
Loading 2609.02421v1…