Source-linked AI summary
IntentDelta: A Human-Constructed Benchmark for Evolving User Intent
Jieyu Zhou, Zijian Wang, Yun Wang
TL;DR
Agents need to adapt when users revise requirements and priorities during complex tasks, but existing benchmarks have limited human-grounded evidence of such changes. IntentDelta is a human-constructed, cross-verified benchmark that pairs evolving utterances with cumulative intentions and reference actions. Across eleven agents, intention recovery and action scores remain low, and the paper reports difficulties preserving requirements and executing the resulting intentions.
Problem
Existing benchmarks provide limited human-grounded evidence about how users revise competing requirements and limited turn-level human reference actions for comparing agent outcomes.
Source: §1 · p. 1Method
IntentDelta pairs human-constructed user updates with cumulative intention annotations and reference actions across WebShop and TravelPlanner.
Source: §1 · p. 1, §4.2 · p. 5Results
33.1% Intention Recovery in TravelPlanner and 44.7% in WebShop accompany Action Scores of 31.6% and 41.0%, respectively, across eleven agents.
Source: §1 · p. 1Takeaways & Limitations
The results expose difficulties preserving earlier requirements, understanding trade-offs, and executing resulting constraints; higher intention recovery does not ensure successful execution.
Source: §6 · p. 8, §7 · p. 8The two-domain human-authored trajectories may not represent natural user interactions or generalize to other domains, and Hard Success depends on a selected human reference compromise.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Language-model agents must adapt as users revise their requirements and priorities during complex tasks. We introduce IntentDelta, a human-constructed multi-turn benchmark with 638 trajectories and 3,766 turns across WebShop and TravelPlanner. Each turn pairs a user utterance with a cumulative intention state and human reference actions. The trajectories are cross-verified and cover feasibility- and preference-driven changes in constraints and priorities. We evaluate eleven agents on intention recovery and action quality. Average Intention Recovery is 33.1\% in TravelPlanner and 44.7\% in WebShop, while average Action Score is 31.6% and 41.0%, respectively. Agents struggle to preserve earlier requirements and distinguish soft preferences from must-have constraints. Correct intention recovery also does not guarantee successful execution, highlighting challenges in translating evolving user intentions into actions.
1 Introduction
Users revise requirements as they encounter task results, so agents must retain valid constraints while dropping superseded ones. IntentDelta addresses gaps in existing benchmarks with human-constructed evolving-intention trajectories and evaluates both intention recovery and action quality.
1 Introduction
- Complex tasks require agents to preserve requirements that remain valid while discarding those superseded by later user feedback.Users may add, relax, replace, or reprioritize constraints after observing infeasible product combinations or itinerary trade-offs.Source: §1 · p. 1
- Existing benchmarks provide limited evidence of plausible human revisions, decision-making about competing requirements, and turn-level human reference actions.Many rely on synthesized updates or simulated interactions.Source: §1 · p. 1
- IntentDelta pairs natural-language updates with cumulative active-constraint and priority annotations and human reference actions at each turn.Its human-constructed episodes cover feasibility- and preference-driven additions, relaxations, overrides, and reprioritizations, with separate evaluation of intention recovery and action quality.Source: §1 · p. 1
- 33.1% Intention Recovery in TravelPlanner and 44.7% in WebShop accompany Action Scores of 31.6% and 41.0%, respectively, across eleven agents.The results indicate substantial weaknesses in maintaining evolving intentions and executing actions that satisfy them.Source: §1 · p. 1
2 Related Work
Prior work studies evolving requirements, multi-turn interaction, and preference refinement, but these settings do not consistently test grounded task execution as user specifications change. IntentDelta evaluates both maintained intention states and downstream task behavior.
2 Related Work
- Existing benchmarks typically assess completion under fixed requirements, multi-turn instruction following, simulated interactions, or clarification of underspecified tasks.Multi-turn interaction alone does not establish whether agents maintain requirements as task results prompt revisions.Source: §2 · p. 2
- Dialogue-state tracking and conversational recommendation study evolving requirements, but primarily evaluate state recovery, response generation, or recommendation efficiency.IntentDelta instead grounds changes in task results and separately evaluates maintained intentions and downstream behavior.Source: §2 · pp. 2–3
3 IntentDelta Benchmark
IntentDelta evaluates whether agents maintain and act on user intentions as constraints and priorities evolve. It models the interaction, represents revisions and trade-offs, and measures intention recovery alongside action execution.
3 IntentDelta Benchmark
- The benchmark evaluates agents on maintaining and acting on user intentions as constraints and priorities evolve.It defines agent and user decision processes and metrics for intention recovery and action execution.Source: §3 · p. 3
3.1 Task Formulation
- The task formulation treats user intention as hidden from an agent that observes both the environment and user utterances.A controlled evaluation replays the same fixed utterance sequence across agents, while each agent interacts independently with the environment.Source: §3.1 · p. 3
3.2 Modeling Evolving User Intention
- At each turn, intention comprises active constraints with values, scopes, and priorities, which may change after infeasibility or preference revision.Constraints are ranked must-have, preferred, or optional; updates can add or remove requirements, revise values, or change priorities.Source: §3.2 · pp. 3–4
3.3 Evaluation Metrics
- Evaluation separately measures recovery of active constraints and whether actions satisfy them under environmental constraints.Action Score combines must-have satisfaction with weighted lower-priority preferences, and is conditioned on Hard Success.Source: §3.3 · p. 4
4 Benchmark Construction
IntentDelta uses WebShop and TravelPlanner to construct and validate human-authored trajectories of changing intentions. Its dataset includes frequent compound revisions and reprioritizations across both domains.
4.1 Domain
- WebShop covers product-level requirements, while TravelPlanner involves interdependent constraints across itinerary components.The benchmark reuses the environments and initial user requests from both tasks.Source: §4.1 · pp. 4–5
4.2 Human Construction
- Authors manually constructed 638 trajectories, creating a cumulative intention, user utterance, and human reference execution at each turn.Each trajectory was cross-verified by another author, and external annotators achieved 97% agreement on a 20% sample.Source: §4.2 · p. 5
4.3 Dataset Characteristics
- Reprioritization comprises approximately 46% of change events in both domains; compound changes occur in 66.5% of changed WebShop turns and 81.5% of TravelPlanner turns.Feasibility-driven changes account for 45.1% of changed turns in WebShop and 26.8% in TravelPlanner; Figure 2 presents trajectory-length distributions.Source: §4.3 · p. 5, Figure 2 · p. 5
5 Experiments
IntentDelta evaluates whether agents can recover evolving user intentions and execute them across TravelPlanner and WebShop. Across eleven models, both intention recovery and action quality remain limited, with failures tied to missed or misprioritized requirements, forgetting carried-over constraints, and execution errors.
5.1 Setup
- The evaluation uses eleven models in a shared ReAct framework, replaying the same annotated user-utterance sequences across 304 TravelPlanner trajectories and 334 WebShop trajectories.Agents predict the complete active intention alongside their actions; gold annotations and human outcomes are withheld, and judge assessments achieved 95% agreement with human evaluations on 100 sampled outputs.Source: §5.1 · p. 6
5.2 Main Results
- Average Intention Recovery is 33.1% in TravelPlanner and 44.7% in WebShop, while average Action Score is 31.6% and 41.0%, respectively.Average Hard Success is 35.8% in TravelPlanner and 50.2% in WebShop.Source: §5.2 · p. 6, Table 3 · p. 6
- Claude Sonnet 5 leads TravelPlanner intention recovery at 39.1%, but Grok 4.6 leads Hard Success at 60.3% and Action Score at 57.1%; in WebShop, Claude Opus 4.8 leads recovery at 52.0%, while GPT-5.6 Sol leads Hard Success at 58.6% and Action Score at 49.0%.Rankings remain robust under tested alternative soft-constraint weightings (Kendall’s τ ≥ 0.964 in both domains).Source: §5.2 · p. 7
5.3 Research Challenge Analysis
- When multiple constraints change in one turn, Intention Recovery falls from 52.5% to 28.7% in TravelPlanner and from 52.4% to 32.3% in WebShop.The decline occurs across all eleven models in both domains; missed constraints, incorrect content, and especially incorrect priority limit recovery.Source: §5.3 · p. 7
- For unchanged carried-over constraints, recovery falls from 58.7% to 27.6% in TravelPlanner and from 63.1% to 32.7% in WebShop as constraint age increases from one turn to at least four.The decline holds across all eleven models in both domains.Source: §5.3 · p. 7, Figure 3 · p. 7
- Agents misclassify preferred constraints as must-have in 91.2% of TravelPlanner cases and 82.1% of WebShop cases, and optional constraints in 93.1% and 80.5%, respectively.Priority Accuracy is 58.9% in TravelPlanner and 62.8% in WebShop; priority misunderstandings can lead to incorrect execution trade-offs.Source: §5.3 · p. 7, Figure 4 · p. 7
- Action Score is higher with full than incomplete intention recovery: 42.9% versus 30.4% in TravelPlanner and 58.8% versus 39.1% in WebShop.Correct recovery still does not guarantee correct action: among failed fully recovered turns, 79.4% of TravelPlanner cases violate budget and 60.4% of WebShop cases violate a product attribute.Source: §5.3 · pp. 7–8, Figure 5 · p. 7
6 Discussion
The results expose challenges in revising intentions while preserving earlier requirements, understanding trade-offs, and executing the resulting constraints. The discussion connects these challenges to clarification and suggests considering feedback, dialogue history, and environment state together.
6 Discussion
- Future approaches could jointly consider user feedback, dialogue history, and environment state to determine which requirements changed and which remain active.The paper connects this challenge to preference trade-offs and memory updates (Wu et al., 2024).Source: §6 · p. 8
- The benchmark evaluates both maintained intention states and downstream behavior as constraints and priorities evolve.Its results identify intention revision, preservation of earlier requirements, trade-offs, and execution as distinct challenges.Source: §6 · p. 8
- When intention is uncertain, targeted clarification questions could ask about constraint content, scope, priority, and changes across turns.These are the finer-grained information targets the benchmark specifies for evolving intentions.Source: §6 · p. 8
7 Conclusion
IntentDelta provides a human-constructed benchmark for evaluating evolving intentions and their execution in WebShop and TravelPlanner. Across eleven models, results show incomplete requirement recovery, priority errors, and execution failures even when intention recovery is higher.
7 Conclusion
- IntentDelta evaluates agents on evolving constraints and priorities in WebShop and TravelPlanner using human-constructed trajectories.The trajectories pair natural-language updates with cumulative intention annotations and human reference actions at each turn.Source: §7 · p. 8
- Across eleven models, agents show incomplete recovery of active requirements, frequent priority errors, and execution failures despite higher intention recovery.The benchmark separately evaluates intention recovery and action quality.Source: §7 · p. 8
- The findings motivate methods for maintaining current requirements, updating priorities, and verifying actions against environment evidence.Source: §7 · p. 8
8 Limitations
IntentDelta’s human-authored trajectories cover two task environments, and its evaluation simplifies both reference trade-offs and preference priorities.
8 Limitations
- Human-authored trajectories in WebShop and TravelPlanner may not fully represent natural user interactions or generalize to other domains.Source: §8 · p. 8
- For infeasible tasks, Hard Success compares against one human reference compromise, so its assessment depends on that reference and misses alternative reasonable trade-offs.Source: §8 · p. 8
- Three discrete priority tiers simplify preferences that may instead be contextual or continuous.Source: §8 · p. 8
A Human Annotation
This section describes the construction of cumulative intentions and reference actions, along with a fixed-user evaluation protocol for recovering active requirements and making evidence-supported choices. It also outlines execution templates that pair evolving-intent tracking with environment actions in WebShop and TravelPlanner.
A Human Annotation
- The annotation interface links each utterance to a cumulative intention state and human reference action, with active constraints assigned Must-have, Preferred, or Optional tiers.Annotators review environment evidence, assess joint feasibility of must-haves, and may record unavoidable trade-offs; reference actions need not satisfy every active requirement.Source: A Human Annotation · p. 10
B.1 Execution Protocol
- Evaluation replays the same ordered user-update sequence for every agent, which infers cumulative intentions and acts without access to gold intentions, priorities, feasibility labels, or reference actions.User updates are constructed from outcomes along a human reference trajectory; evaluation does not generate new responses to each agent’s actions.Source: §B.1 · p. 13
B.2 Shared Intention Representation
- The shared schema represents each active requirement by its field, value, and must-have, preferred, or optional priority, preserving relevant traveler, location, and time context.Optional requirements remain in the predicted intention until withdrawn or superseded, even when they may be unsatisfied in the selected outcome.Source: §B.2 · pp. 13–14
B.3 Shared Action Selection Policy
- Agents must submit the best concrete supported solution when requirements conflict, prioritize must-haves, and disclose unmet conditions without changing their inferred intention to make the action appear compliant.They should continue researching alternatives rather than treating the first failed exact match as infeasibility.Source: §B.3 · p. 14
B.4 WebShop Execution Template
- The WebShop template requests one action at a time alongside the agent’s current intention understanding, using the utterance history, within-turn trace, and current page observation as context.The template directs agents to infer active requirements from the latest utterance while using earlier utterances as trajectory context.Source: §B.4 · p. 14
- The WebShop runtime context includes the current and prior user utterances, internal-step traces, and current-page evidence.Source: §B.4 · p. 15
B.5 TravelPlanner Execution Template
- TravelPlanner prompts the agent to infer cumulative intent and choose one action, using search tools, a notebook workflow, and final itinerary submission.Source: §B.5 · p. 15
- TravelPlanner rules preserve person-specific needs and evolving requirements, then require a complete, grounded itinerary with named lodging, transport, meals, and cost accounting.The prompt requires stable traveler IDs and separate intent items; it also prohibits inventing tool results and treating unpriced components as free.Source: §B.5 · pp. 15–16
C Evaluation Details and Prompts
The evaluation details and prompts define separate assessments of intention recovery and action quality using shared human-annotation baselines and frozen judging criteria. They specify deterministic score aggregation, shared intention-matching rules, and domain-specific evidence requirements for TravelPlanner and WebShop.
C.1 Evaluation Procedure
- Each model is evaluated once per trajectory, and turn-level scores are averaged over scored turns; Table 3’s Average is the arithmetic mean across eleven models.Source: §C.1 · p. 16
- Action judgments independently assess outcomes against frozen criteria, using evidence from selected products and options in WebShop or itineraries and environment records in TravelPlanner.A stated intention or claimed attribute alone does not establish that an action satisfies a requirement.Source: §C.1 · p. 16
- Action scoring treats each in-scope gold constraint field as one unit; every component of a conjunction must be supported, and only satisfied verdicts earn credit.Intention atomization does not create additional action-scoring units; violated and unknown verdicts receive no credit.Source: §C.1 · p. 16
- WebShop checks require a valid final product and recorded, catalog-supported option selections, while TravelPlanner checks verify booked-night minimum stays and party capacity.TravelPlanner accommodation feasibility is checked separately from individual user requirements; minimum stays use booked nights, not assumed extra nights.Source: §C.1 · p. 17
- Unknown satisfaction earns no credit, and intention and action metrics use separate coverage counts and denominators; evaluator, API, and data errors must be distinguished from agent failures.When human feasibility or reference outcomes are absent, the scorer uses specified fallback assumptions; missing gold intention is an evaluation error.Source: §C.1 · p. 17
C.2 TravelPlanner Baseline Preparation
- TravelPlanner preparation builds reusable gold atoms and criteria; criteria restate the gold, resolve scope from dialogue, and report conflicts rather than changing gold requirements.Accessibility or mobility needs and within-city travel feasibility are marked out of scope because candidate records cannot audit them.Source: §C.2 · p. 17
C.3 WebShop Baseline Preparation
- WebShop preparation uses the current human intention snapshot as the authority for active fields, while dialogue may clarify but cannot add, remove, or relax constraints.The shared baseline is independent of tested models, and gold atoms are split for intention matching rather than constraint criteria.Source: §C.3 · p. 18
C.4 Shared Intention Judge
- The shared intention judge matches predicted atoms one-to-one with frozen gold atoms, checking semantic value and applicability scope; priority correctness comes from the agent’s original tiers.Changes are measured against the agent’s previous prediction: rephrasing is unchanged, a different value is changed, and a previously absent requirement is new.Source: §C.4 · pp. 18–19
- The intention judge labels a matching predicted requirement’s priority as new, changed, or unchanged, without treating a value change alone as a priority change.At the first turn every atom is new; missing priority remains missing, and priority cannot guide matching to a Gold atom.Source: §C.4 · p. 19
C.5 TravelPlanner Action Judge
- TravelPlanner judgments apply frozen baseline criteria to the final plan and use candidate records—not agent wording—for entity attributes and constraint evidence.Absent entities cannot satisfy attribute-dependent constraints; budget is priced by code rather than judged by the model.Source: §C.5 · p. 19
- TravelPlanner judges apply only explicit final revisions to the itinerary, then assess the revised plan as a whole against frozen criteria without reinterpreting requirements.The structured response records the resolved plan, revisions, per-constraint verdicts and evidence, disclosures, and annotation issues.Source: §C.5 · p. 19
- Both action judges record explicit unmet-requirement disclosures separately, and disclosures do not change whether a constraint is satisfied.TravelPlanner excludes vague trade-off language; WebShop records exact quotes from the selected action rationale.Source: §C.5 · p. 20, §C.6 · p. 20
C.6 WebShop Action Judge
- WebShop products fail requested constraints without supporting evidence, and judgments assess the selected ASIN and options rather than merely available variants.The judge inspects full descriptions, bullets, and attributes; agent claims and plausible properties are not evidence.Source: §C.6 · p. 20
- WebShop treats each frozen constraint as one scoring unit, requires evidence for every AND component or one OR alternative, and does not infer attribute values or units.Negative ingredient requirements need evidence of absence, and price judgments use the selected variant’s price and the criterion’s exact comparator.Source: §C.6 · p. 20
D Additional Error Analysis
Table 4 examines action failures in turns where intention recovery is incomplete or complete, focusing on constraint violations and invalid bookings.
D Additional Error Analysis
- Table 4 reports constraint violations and invalid bookings among failed turns, conditioned on whether the agent fully recovered the user’s intention.The supplied passages describe the table’s scope but do not provide its cell values.Source: Table 4 · p. 21, D Additional Error Analysis · p. 21
E Sensitivity to Soft-Constraint Weights
The paper tests whether alternative soft-constraint scoring schemes change model rankings, and reports supplementary diagnostics for prediction precision and obsolete retention.
E Sensitivity to Soft-Constraint Weights
- Action Score is recomputed under six alternatives to the 2:1 preferred-to-optional weighting, while the Hard Success gate remains unchanged.Alternatives include three linear weightings, a lexicographic scheme, a minimum of reference-relative satisfaction ratios, and a squared 2:1 ratio.Source: E Sensitivity to Soft-Constraint Weights · p. 21
- Kendall’s τ is at least 0.964 in both domains; no model shifts more than one rank, and the top three models and their ordering remain unchanged.Average Action Score changes by at most 0.5 percentage points under linear weightings and 3.5 percentage points under nonlinear variants.Source: Table 5 · p. 21, E Sensitivity to Soft-Constraint Weights · p. 21
F Prediction Precision and Obsolete Retention
- Table 6 reports supplementary intention diagnostics averaged across eleven models, including Precision and Obsolete Retention.Precision measures the fraction of predicted constraint atoms matched to active gold constraints; Obsolete Retention tracks previously correct requirements that persist after an override, relaxation, or removal.Source: Table 6 · p. 21, F Prediction Precision and Obsolete… · p. 21
- Obsolete Retention uses exact matches to prior predictions and may miss paraphrased cases.Precision also warrants caution because unmatched predictions may be valid fixed task parameters omitted from the gold intention or reflect different constraint granularity.Source: F Prediction Precision and Obsolete… · p. 21