Source-linked AI summary
DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents
Yansong Ning, Rui Liu, Jun Wang, Kai Chen, Wei Li, Jun Fang, Kan Zheng, Naiqiang Tan, Hao Liu
TL;DR
Existing travel-planning agents rely on crafted prompts and fixed workflows, while autonomous training is challenged by changing tool outputs and open-ended rewards. DeepTravel combines a travel sandbox, hierarchical verification, and replay-augmented reinforcement learning; it reports 82% online itinerary accuracy and offline comparisons favoring Qwen3-32B over frontier reasoning models and prevailing RL algorithms.
Problem
Existing travel-planning studies rely on hand-crafted prompts and fixed workflows, while dynamic travel data and open-ended itineraries without explicit ground truth complicate autonomous agent training and reward verification.
Source: §1 · p. 2Method
DeepTravel trains an autonomous travel-planning agent in a cached travel sandbox with trajectory- and turn-level reward verification, then uses reply-augmented reinforcement learning to replay failures.
Source: §1 · p. 2Results
82% accuracy in travel itinerary generation was achieved in three-month online tests, and offline results report Qwen3-32B outperforming frontier reasoning LLMs and prevailing RL algorithms.
Source: §1 · p. 2Takeaways & Limitations
DeepTravel is presented as an end-to-end agentic RL framework for autonomous travel planning, with online deployment and offline tests validating its effectiveness.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Travel planning (TP) agent has recently worked as an emerging building block to interact with external tools/resources for travel itinerary generation, ensuring an enjoyable user experience. Despite its benefits, existing studies rely on hand-craft prompt and fixed agent workflow, hindering more flexible and autonomous TP agents. This paper proposes DeepTravel, an end-to-end agentic reinforcement learning framework for building an autonomous travel planning agent, capable of autonomously planning, executing tools, and reflecting on tool responses to explore, verify, and refine intermediate actions in multi-step reasoning. To achieve this, we first construct a robust travel sandbox by caching transportation, accommodation and POI data, facilitating TP agent training without being constrained by real-world APIs limitations (e.g., inconsistent outputs). Moreover, we develop a hierarchical reward modeling system, where a trajectory-level verifier first checks spatiotemporal feasibility and filters unsatisfied travel itinerary, and then the turn-level verifier further validate itinerary's detail consistency with tool responses, enabling efficient and precise reward service. Finally, we propose the reply-augmented reinforcement learning method that enables TP agent to periodically replay from a failure experience buffer, emerging notable agentic capacity. We deploy the trained TP agent in the DiDi Enterprise Solutions application. A three-month online test shows that it achieves 82% accuracy in travel itinerary generation. Comprehensive offline evaluations further demonstrate that DeepTravel enables small-sized LLMs (e.g., Qwen3-32B) to significantly outperform frontier LLMs (e.g., OpenAI o1/o3 and DeepSeek-R1) and existing TP agent frameworks.
1 Introduction
Existing travel-planning agents rely on crafted prompts and fixed workflows, while autonomous agents face dynamic tool outputs and open-ended itinerary rewards. DeepTravel trains an agent to plan, call tools, and reflect, and reports online and offline performance.
1 Introduction
- Existing TP approaches use carefully designed prompts or fixed pipelines, requiring labor-intensive design and struggling to adapt to new queries or recover from tool failures.TravelPlanner and TripTailor use task-specific prompts, while PTS and RETAIL use structured workflows.Source: §1 · pp. 1–2
- Autonomous TP training is challenged by changing travel data that makes repeated queries inconsistent and by open-ended itineraries without explicit ground truth.Personalized preferences and budgets make itinerary outcomes difficult to verify.Source: §1 · p. 2
- DeepTravel caches transportation, accommodation, and POI data across timestamps in a sandbox, enabling repeated training trials despite API rate limits and inconsistent outputs.The sandbox simulates dynamic tool interactions using data from multiple real-world APIs.Source: §1 · p. 2
- Hierarchical reward modeling checks itinerary spatiotemporal feasibility at the trajectory level and detail consistency with tool responses at the turn level, while reply-augmented RL replays failures to refine reasoning.The replay method follows a sequential cold-start and RL process.Source: §1 · p. 2
- 82% accuracy in travel itinerary generation was achieved in three-month online tests; offline results report Qwen3-32B outperforming frontier reasoning LLMs and prevailing RL algorithms.The agent was deployed in the DiDi Enterprise Solutions application.Source: §1 · p. 2
2 Preliminary
The preliminary section defines user queries and itineraries, then formulates agentic travel planning as multi-turn tool interaction that produces an itinerary.
2 Preliminary
- A user query is natural-language input describing spatiotemporal travel intention and personalized preferences.The example requests a three-day air trip from Shanghai to Beijing.Source: §2 · p. 2
- A travel itinerary is a structured plan covering accommodation, transportation, and a detailed daily plan with activity and POI suggestions.Source: §2 · p. 2
- Agentic travel planning maps a query and prior turns into the next thought and action, with the final action containing the itinerary.Each prior turn comprises the agent's thought, action, and environment observation; Figure 1(b) illustrates tool use and reflection across turns.Source: §2 · pp. 2–3
3 DeepTravel
DeepTravel combines a sandbox for simulated tool interactions, hierarchical itinerary verification, and replay-augmented reinforcement learning to train autonomous travel-planning agents.
3.2.1 Unified Toolkit Interface Construction.
- Six categories of real-world tools are standardized into one executable interface for travel-planning agents.The interface standardizes heterogeneous APIs into a consistent invocation format.Source: §3.2.1 · p. 3
3.2.2 Mock Data Caching.
- A 24-hour on-demand cache refreshes travel data while preserving identical tool responses for replay during training.Novel queries retrieve fresh API data; caching also addresses API request limits and inconsistent retrieval.Source: §3.2.3 · p. 4, §3.2.2 · p. 4
3.3.1 Trajectory-Level Verifier.
- The trajectory-level verifier screens complete itineraries for event order, geographic plausibility, and user requirements within the available timeframe.Only potentially valid itineraries proceed to finer-grained verification.Source: §3.3.1 · p. 4
3.3.2 Turn-Level Verifier.
- The turn-level verifier checks itinerary details against tool responses, and the joint reward is 1 only when every turn passes.A trajectory-level violation sets the reward to 0 immediately; human-generated rubrics guide the DeepSeek-R1-based verifier.Source: §3.3.2 · p. 4, §3.3.3 · p. 4
3.4.1 Reasoning Format Cold Start with SFT.
- Cold-start training distills multi-turn tool-use traces, filters incorrect or improperly formatted trajectories, and trains on verified outputs conditioned on the prompt and user query.Environmental observation tokens are masked from the loss calculation.Source: §3.4.1 · p. 4
3.4.2 Reinforcement Learning with Experience Replay.
- During RL, queries whose sampled trajectories all fail verification enter an experience buffer for replay after later training steps.The replay strategy is intended to let an improved policy handle previously failed hard samples.Source: §3.4.2 · pp. 4–5
- RL filters groups with reward standard deviation at or below 0.1 and masks tool-response tokens so policy gradients cover agent-generated tokens.The objective uses group-reward advantages and a KL-divergence term.Source: §3.4.2 · p. 5
4 System Deployment
DeepTravel-32B was adapted for DiDi Enterprise Solutions and deployed with an asynchronous process for refining the model from service failures.
Backbone selection and adaptation.
- After DeepSeek-R1 showed domain-performance and latency drawbacks, the team trained Qwen3-32B with the DeepTravel framework and deployed DeepTravel-32B.The model was trained with supervised fine-tuning and large-scale reinforcement learning before production deployment.Source: §4 · p. 5
Asynchronous agent update.
- A professional annotation team and product managers identify service failures, which prompt system-prompt changes or dataset updates and retraining before product validation.The validated updated models are then pushed to the online environment.Source: §4 · p. 5
Online serving strategy.
- 25 seconds is the average production latency for a complete itinerary, about 90 seconds less than direct DeepSeek-R1 use.About 19 seconds go to deep thinking and 6 seconds to answer summarization and order-page rendering.Source: §4 · p. 5
5 Experiments
The experiments use curated training data, online production queries, and synthesized offline tests, with itinerary validity assessed by LLM-as-a-Judge and human evaluation. They report strong offline and online performance, then examine component contributions, training behavior, and user experience.
5.2.1 Metric and Evaluation Protocol.
- LLM-as-a-Judge has 82% raw agreement with human evaluation; accounting for 7% human annotation error raises estimated judge accuracy to approximately 89%.The alignment analysis uses 200 samples.Source: Table 3 · p. 6, §5.2.1 · p. 6
5.3 Online Test (RQ1)
- DeepTravel-32B achieves a 82.58% Final Pass Rate online; intent understanding, completeness, and clarity each score above 92%.Personalized requirements score 76.62%, while factual hallucination scores 84.42%.Source: Table 4 · p. 6, §5.3 · p. 7
5.4 Offline Test (RQ2)
- Offline without constraints, DeepTravel-32B reaches 69.34% Final Pass Rate, 28.9% above K2; DeepTravel-8B reaches 54.25%, 0.1% above K2.Across other settings, DeepTravel-32B consistently outperforms frontier reasoning LLMs; replay-augmented RL also performs especially well on hard queries.Source: Table 5 · p. 7, §5.4 · p. 7
5.2.2 Baselines.
- The comparison includes reasoning LLMs, prompting- and multi-agent travel-planning frameworks, and PPO, GRPO, and DAPO RL baselines.Offline tests include 500 queries with personalization constraints and 500 without, while online testing collected 6,224 production queries over three months.Source: §5.2.2 · pp. 6–7, §5.2 · p. 6
5.5 Ablation Study (RQ3)
- Ablations show that removing experience replay, cold-start, or either verifier lowers performance; the turn-level verifier matters more, especially on hard problems.The trajectory-level verifier also improves training efficiency by avoiding fine-grained turn-level checks when an itinerary fails its feasibility verification.Source: Table 6 · p. 8, §5.5 · p. 8
5.6 In-Depth Analysis (RQ4)
- Cold-start initializes a more effective policy, and replay of failed samples yields later gains; model entropy remains non-decreasing during RL training.The analysis tracks validation reward, entropy, interaction turns, and tool-call accuracy.Source: Figure 4 · p. 8, §5.6 · pp. 8–9
- Real-world API tool-call accuracy is unstable and lower than in the sandbox, with no clear reward improvement; the sandbox caches about one million mock data.The agent executed 12,883 tool calls in one training run.Source: Table 7 · p. 8, §5.6 · p. 8
5.7 Real-World User Study (RQ5)
- Both cold-start and RL improve user satisfaction, while RL reduces factual hallucinations from rates reaching 50% to below 20%.Cold-start improves fundamental itinerary capabilities but alone is insufficient for advanced personalized preferences.Source: Figure 5 · p. 9, §5.7 · p. 9
6 Related Work
Prior travel-planning agents use handcrafted prompts or fixed workflows, while agentic RL has been applied in other domains but remains unexplored for travel planning.
6.1 LLMs as Travel Planning Agent.
- Prompt-based and fixed-workflow travel agents can support tool use and itinerary generation, but are labor-intensive and generalize poorly to new queries or changing tools and resources.Examples include TravelPlanner, TripTailor, ChinaTravel, TravelAgent, PTS, and RETAIL.Source: §6.1 · p. 9
6.2 Agentic Reinforcement Learning for LLMs.
- Agentic RL trains autonomous agents through tool interaction in a sandbox and reward feedback, but its application to travel planning remains unexplored.Prior examples address math, deep research, and web tasks.Source: §6.2 · p. 9
7 Conclusion and Future Work
DeepTravel combines sandboxed agentic RL and hierarchical verification for autonomous travel planning, and the authors report strong performance in online and offline evaluations.
7 Conclusion and Future Work
- DeepTravel combines a robust sandbox, trajectory- and turn-level itinerary verification, and replay of failed cases in agentic RL, and its deployed agent outperforms frontier reasoning LLMs and travel-planning frameworks in reported evaluations.The authors identify more flexible automatic reward modeling and extension to other domains as future work.Source: §7 · p. 9
A.1 Training Details
The appendix describes training resources and tracked diagnostics for DeepTravel.
A.1 Training Details
- Training uses Megatron-LM for SFT and verl for RL, with DeepSeek-R1 serving as verifier backbone.An 8B model uses one node with 8 H800 GPUs; a 32B model uses 32 H800 GPUs across four nodes.Source: §A.1 · p. 10
- SFT takes about 2–3 hours for 1,000 samples, while RL takes about 30 hours per 100 steps for 500 samples.Source: §A.1 · p. 10
- Training monitors entropy, gradient norm, response length, reward, turn count, tool-call accuracy, verifier success, sample keep rate, and loss-mask ratio.Low entropy can indicate poor exploration; high gradient norms indicate instability, and low tool-call accuracy or verifier success can signal execution or evaluation errors.Source: §A.1 · pp. 10–11
A.2 Human Annotation
The human annotation pipeline assesses itinerary quality across seven dimensions, covering user requirements, completeness, feasibility, affordability, clarity, personalization, and factuality.
A.2 Human Annotation
- The seven dimensions cover user-intention understanding, itinerary completeness and feasibility, affordability, clarity, personalization, and factual accuracy.Completeness includes essential travel components and temporal coherence; feasibility checks practical executability and logical routing.Source: §A.2 · p. 11
- The assessment checks whether recommendations capture user parameters and preferences, provide coherent and executable travel segments, and avoid nonexistent services or logically inconsistent outputs.Affordability includes bookability, accurate prices, and compliance with organizational travel policies; clarity includes efficient reasoning and transparent justifications.Source: §A.2 · p. 11
A.3 Case Study
The case study illustrates DeepTravel-32B handling a real-world travel query through multi-turn tool use and reflection. Its example workflow grounds recommendations in tool responses and revises plans when constraints are unmet.
A.3 Case Study
- The authors state that ethical requirements prevent release of the training data, and provide a real-world query and response instead.The supplied example is illustrative rather than a release of the training dataset.Source: §A.3 · p. 11
- The example is presented to help readers understand how the agent autonomously plans, executes tools, and reflects on tool responses during multi-turn reasoning.The case study draws on selected real-world user cases from the deployed DiDi Enterprise Solution App.Source: §A.3 · p. 11
- DeepTravel-32B grounds a travel plan through intent and constraint parsing, tool planning, schema-aligned calls, evidence-linked interpretation, reflection, and finalization.When constraints are unmet, such as transfer-related arrival risk, it revises departure windows or station choices and calls tools again.Source: §A.3 · p. 11
- The illustrative query asks for travel from Suzhou to Wuhan Conference Center on June 26, arrival before 3:00 PM, and lodging near the riverside.The response includes train and hotel recommendations, including a hotel described as 233 meters from the river beach and river-view rooms at another property.Source: §A.3 · p. 12