PaperlayerView exact revision on arXiv

Source-linked AI summary

PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents

Huayi Lai, Shichao Song, Qingchen Yu, Simin Niu, Mengwei Wang, Hanyu Wang, Xun Liang

arXiv:2609.34930v1cs.AI

TL;DR

Existing personalized tool-use benchmarks largely assess isolated calls or reactive execution, leaving the full planning lifecycle unclear. PDEU-Bench evaluates plan definition, execution, and update in personalized tasks; evaluations reveal a gap between local tool execution and lifecycle-level planning.

  • Problem

    Existing benchmarks largely assess isolated personalized calls or reactive execution, leaving unclear whether agents can formulate, execute, and update explicit plans while preserving preferences.

    Source: Introduction · pp. 1–2

  • Method

    PDEU-Bench evaluates personalized plan definition, execution, and feedback-driven update in multi-round tool-calling interactions.

    Source: Introduction · p. 2

  • Results

    Evaluations of 15 LLMs found a gap between plan execution and lifecycle-level planning, with personalized plan definition and feedback-driven updating as key bottlenecks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Six evaluated personalization and memory-augmentation methods did not improve all three planning stages simultaneously.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Large language model (LLM) agents are evolving from tool-calling systems that execute isolated instructions into task-oriented agents that pursue user goals through sustained, multi-step interactions. However, existing benchmarks for personalized tool use largely assess isolated calls or reactive execution, leaving unclear whether agents can formulate, execute, and revise an explicit plan while preserving user preferences throughout long-term interaction. To address this gap, we introduce \textbf{PDEU-Bench} (\textbf{P}ersonalized plan \textbf{D}efinition, plan \textbf{E}xecution, and plan \textbf{U}pdate \textbf{Bench}mark), a benchmark for evaluating the complete planning lifecycle of personalized tool-using agents. PDEU-Bench comprises 214 long-horizon interaction tasks spanning 12 everyday domains and 94 tools, with stage-specific assessments of preference adherence and plan quality. Extensive evaluations of 15 representative open-source and closed-source LLMs reveal a pronounced gap between local tool execution and dynamic planning: LLMs can often instantiate preferences in individual calls, yet struggle to construct coherent plan definition and plan update. We further evaluate mainstream personalization and memory-augmentation methods. Although these methods improve particular stages, none of the evaluated methods reliably propagates user preferences throughout the complete lifecycle, and their gains frequently fail to transfer to subsequent execution. Fine-grained error analysis further reveals that preference omissions and conflicts persist throughout the planning lifecycle, highlighting the need for future research to parameterize LLMs with preference-aware information retrieval and memory capabilities. We provide the relevant code and data in the appendix to support future research.

Abstract

PDEU-Bench evaluates personalized tool-using agents across plan definition, execution, and feedback-driven update, addressing benchmarks that focus on isolated calls or reactive execution. Its evaluations reveal a gap between local tool execution and lifecycle-level planning.

  • Plan definition tests whether an agent combines an underspecified request with personalized context to create a structured, executable, preference-aligned plan.Source: Introduction · p. 2
  • Plan execution tests tool selection and whether task constraints and preferences are accurately instantiated as tool-call arguments.Source: Introduction · p. 2
  • Plan update tests whether an agent revises affected later steps after tool outputs while preserving valid goals, dependencies, and preferences.Source: Introduction · p. 2
  • PDEU-Bench covers 214 personalized scenarios across 12 everyday domains and 94 tools, with separate assessments of plan definition, execution, and update.It integrates user goals, personalized contexts, tool interfaces, and executable environments into task trajectories.Source: Introduction · p. 2
  • 15 evaluated LLMs showed a gap between plan execution and lifecycle-level planning, with personalized plan definition and feedback-driven updating identified as key bottlenecks.The results highlight that strong individual tool calls do not necessarily translate into coherent planning across stages.Source: Introduction · p. 2
  • Six evaluated personalization and memory-augmentation methods did not improve all three stages simultaneously, exposing difficulty maintaining preferences across the lifecycle.Source: Introduction · p. 2

Related Work

Prior personalized tool-use benchmarks generally evaluate execution or isolated planning, while PDEU-Bench evaluates the full planning lifecycle. Existing multi-turn settings extend interaction but remain centered on task completion rather than plan definition.

Benchmarks for LLM Personalized Tool Use

Stage 2:Preference and Task Construction

The supplied preference examples describe a user as relaxed and slow-paced, healthy, and educated to the master’s level.

Stage 2:Preference and Task Construction

Task Construction

The supplied task example is renting a car for the user.

Task Construction

Difficulty Judge

The material lists human quality control, task difficulty, and task evaluators.

Difficulty Judge

  • The listed topics are human quality control, easy tasks, hard tasks, and task evaluators.Source: Difficulty Judge · p. 3

O C E A N

The example asks how to get to the library and offers walking as a possible answer.

O C E A N

  • The example query asks, “How to go to library?”Source: O C E A N · p. 3
  • The response says walking may be better and includes the sequence 1.0, 4.5, 3.5, 2.2, 3.1.Source: O C E A N · p. 3

PDEU Benchmark

PDEU-Bench combines personalized tool schemas, interaction histories, and multi-step tasks in a logged, three-stage planning protocol. It evaluates preference adherence and plan quality with stage-specific criteria.

Tool Construction

  • Generated API schemas use preference-sensitive parameters and are manually checked for functional correctness and preference coverage by two annotators.Schemas failing either check are revised and rechecked.Source: Tool Construction · p. 4

Preference and Task Construction

  • The benchmark’s generation pipeline builds simulated interaction histories around coarse user profiles and preferences expressed in individual interactions.Source: Preference and Task Construction · p. 4

Preference Construction

  • Preferences span ten everyday domains and are conveyed implicitly through 15–25 two-turn episodic interactions per user; 83.0% of instances were retained unchanged and 17.0% after minor revision.Three professional annotators reviewed consistency among profiles, preferences, and interactions; none were discarded.Source: Preference Construction · p. 4

Task Construction

  • Tasks are based on profiles, direct preferences, and available tools, with preferences omitted from user requests; 214 retained tasks comprise 68 easy and 146 hard instances.Three annotators assessed preference relevance and task reasonableness; tasks lacking both were discarded, while others were revised and re-evaluated.Source: Task Construction · p. 4

Trajectory Acquisition

  • Each task uses a versioned plan: a schema-validated initial plan precedes tool use, and every action–observation cycle, including failures, requires a plan update.Inputs, tool contracts, and sandbox states are fixed; logs are automatically checked for temporal and structural consistency.Source: Trajectory Acquisition · p. 4

Trajectory Evaluation

  • Evaluation uses binary rubrics for preference adherence and plan quality in definition and update, while execution checks preference-aware tool arguments only.A task succeeds only when all applicable preference and planning requirements hold at every relevant stage and transition.Source: Trajectory Evaluation · p. 4

Evaluation Matrix

  • Preference accuracy is computed separately for definition, execution, and update, while plan quality is assessed only for definition and update using six criteria at each stage.The overall Quality Score averages definition and update quality accuracy; Joint Accuracy measures tasks passing both preference and plan-rationality checks.Source: Evaluation Matrix · pp. 4–5

Quality Control

  • The benchmark contains 214 tasks across six categories, with source interactions spanning nine user-interest themes; human-involved stages follow predefined criteria and instructions.Majority voting across different parameters and model families was used to stabilize consensus labels.Source: Quality Control · p. 5

Experiments

Across 15 models, local preference grounding is stronger than coherent plan definition and update. Personalization methods yield stage-specific gains that do not reliably carry through execution, while preference and plan-quality failures persist across the lifecycle.

Baseline Models

  • The baseline evaluation covers 15 mainstream LLMs from diverse model families and capability tiers, using default thinking intensity for models with multiple modes.Experiments use API calls at temperature 0 for reproducibility.Source: Baseline Models · p. 6, Main Result · p. 6

Experimental Methods

  • Seven personalization settings compare reminders, retrieval, and memory augmentation against an unaugmented baseline on gemini-3.1-flash-lite and deepseek-v4-flash.Settings include RAG, PAG, MemoryBank, PersonaAgent, and Agentic Memory; all saved trajectories use the same fine-grained binary rubrics.Source: Experimental Methods · p. 6

Main Result

  • Across 15 models, average preference score is 49.27% versus 29.08% average quality score; gpt-5.6-terra leads both metrics but reaches only 66.32% preference score.Overall, models remain substantially limited in defining and updating plans.Source: Main Result · p. 6

Baseline Performance Analysis

  • Plan update is the main bottleneck: preference, quality, and joint accuracies are 18.80%, 10.08%, and 7.02%, versus 31.86% definition joint and 77.52% execution preference accuracy.Definition preference and quality accuracies are 51.51% and 48.07%, but preferences, objectives, dependencies, and feasibility constraints are rarely integrated simultaneously.Source: Baseline Performance Analysis · p. 6

Baseline Error Analysis

  • Among failed instances, update preference omission reaches 79.9–81.3% and conflict 89.3–91.0%; execution preference errors affect 22.2–38.3%.At definition, omission affects 43.9–75.0% and conflict 10.7–18.1%; update goal forgetting reaches 86.0–88.2% and logic errors 92.4–93.5%.Source: Table 4 · p. 7, Baseline Error Analysis · p. 7

Impact of Task Difficulty

  • Deepseek-v4-pro and mimo-v2.5-pro show closely aligned performance on easy and hard subsets across all three stages and Traj., with no consistent easy-task advantage.The observed deficiencies are reported as largely independent of task difficulty.Source: Impact of Task Difficulty · p. 7

Analysis of Improved Methods

  • Every tested method improves Definition joint accuracy and several improve Update, but execution preference accuracy falls for both models under all methods.The baseline imbalance is reversed rather than resolved: stronger formulation and revision accompany less reliable preference grounding in tool arguments.Source: Analysis of Improved Methods · p. 7

Error Analysis

  • Failures are grouped into preference omission, preference conflict, goal forgetting, and logic errors; execution applies only the two preference categories.Categories are non-mutually-exclusive, so multiple types can be assigned to one failed trajectory.Source: Error Analysis · p. 7

Conclusion

PDEU-Bench evaluates personalized tool-using agents across plan definition, execution, and update. Experiments reveal brittle initial plans, an update bottleneck, and stage-specific gains that do not reliably transfer across the lifecycle.

Conclusion

  • PDEU-Bench evaluates personalized tool-using agents across Plan Definition, tool-grounded Plan Execution, and feedback-driven Plan Update.Its 214 long-horizon tasks span 12 everyday domains and 94 tools, with an explicit, versioned plan as the central evaluation object.Source: Conclusion · p. 7
  • Evaluations of 15 mainstream LLMs reveal a gap between local preference grounding and lifecycle planning, with brittle initial plans and Plan Update as the principal bottleneck.Six personalization and memory-augmentation methods produce stage-specific gains that do not reliably transfer across the lifecycle.Source: Conclusion · p. 7
  • Error analysis identifies persistent preference omission, preference conflict, goal forgetting, and logical inconsistency across the evaluated planning lifecycle.The benchmark is presented as a reproducible testbed for developing agents that incorporate environmental feedback while maintaining preferences, objectives, and dependencies.Source: Conclusion · p. 7

Related papers

Loading 2609.34930v1…