Source-linked AI summary
AcCoRD: Evaluating User-Agent Collaboration Under Realistic User Preference Dynamics
Tejas Srinivasan, Shikib Mehri, Nandita Shankar Naik, Anirban Das, William M. Campbell, Jesse Thomason
TL;DR
Existing benchmarks usually treat user preferences as fully specified or merely underspecified, missing preferences that change, emerge, or require negotiation during interaction. AcCoRD introduces a two-domain benchmark covering four preference dynamics and evaluates five LLMs with ReAct and uncertainty-guided prompting. Current agents achieve less than 30% Perfect Outcome Rates even at their strongest, while prompting alone provides no consistent benefit and triggered preferences remain especially difficult.
Problem
Most agent benchmarks assume goals and preferences are fully specified upfront, although real preferences may be withheld, incompletely formed, or unknown until interaction surfaces them.
Method
AcCoRD evaluates five LLMs across four controlled preference dynamics in online shopping and travel-planning scenarios, comparing ReAct with uncertainty-guided UncReAct prompting.
Results
Perfect Outcome Rates remain below 30% even for the strongest frontier models; triggered preferences are hardest, and prompting-based uncertainty reasoning provides no consistent benefit.
Takeaways & Limitations
Reliable user-agent collaboration requires uncertainty modeling capabilities beyond what prompting alone can elicit.
Takeaways & Limitations
AcCoRD covers only two task domains and uses a controllable user simulator whose findings may not generalize directly to real-world user satisfaction.
Abstract
from arXiv · showhide
User preferences in user-agent collaboration are rarely static and fully-specified upfront: preferences are formed, revealed, adjusted, and relaxed during interaction. Existing benchmarks for evaluating user-agent collaboration focus almost exclusively on resolving underspecified preferences, thereby failing to capture the richer dynamics of real-world interaction. We introduce AcCoRD, a user-agent collaboration benchmark requiring agents to handle diverse user preference dynamics in two domains: online shopping and travel planning. We evaluate five frontier LLMs under two prompting strategies: vanilla ReAct, and an uncertainty-guided variant that prompts models to identify and resolve ambiguity about user preferences. Our results reveal that frontier models can handle underspecification but struggle to satisfy preferences that emerge or evolve mid-interaction and require more sophisticated uncertainty modeling. Further, prompting alone fails to elicit the required uncertainty recognition. We release AcCoRD as a resource for developing agents that can navigate the full complexity of real-world user preferences.
1 Introduction
ACCORD targets collaboration settings where preferences may be missing, flexible, infeasible, or emerge during interaction. It benchmarks these dynamics across shopping and travel while showing that current agents still struggle, especially when preferences evolve.
- 1 Introduction: ACCORD evaluates four preference dynamics: hard underspecification, soft underspecification, unachievable preferences, and triggered preferences.These require elicitation, flexibility handling, negotiation, and recognition of preferences that emerge during interaction.
- 1 Introduction: The benchmark covers online shopping and travel planning through 100 collaboration scenarios in each domain.Agents interact with LLM-based user simulators while satisfying controlled mixes of preference dynamics.
- 1 Introduction: Five LLMs are evaluated with ReAct and UncReAct, which adds an explicit decision about whether to elicit or act.UncReAct augments the standard rational-then-act baseline with structured uncertainty reasoning.
- 1 Introduction: Even Claude-Sonnet-4.5 satisfies all user preferences less than 30% of the time.The results identify unachievable and triggered preferences as universally hardest dynamics.
- 1 Introduction: Prompting models to reason about preference uncertainty does not improve collaboration outcomes or meaningfully change behavior.The findings point to a gap between prompting-based elicitation and the uncertainty modeling required for real collaboration.
2 Related Work
Prior agent benchmarks typically assume that goals and preferences are fully specified upfront, while dialogue systems study interactive preference elicitation. Related collaboration work begins addressing sustained, user-centered interaction through training and prompting.
- Interactive Agent Benchmarks: Interactive agent benchmarks span shopping, computer use, software engineering, mobile use, and office software systems.Despite this breadth, these benchmarks commonly assume that user preferences are fully specified upfront.
- Preference Elicitation Systems: Preference elicitation systems model interaction as sequentially asking informative questions, updating beliefs, and recommending accordingly.LLMs support open-ended clarification and maximally informative query selection in this line of work.
- Human-LLM Collaboration: RLHF-trained LLMs are optimized primarily for single-turn instruction following, limiting their fit for sustained multi-turn collaboration.This motivates methods that optimize task success and user-centric objectives through simulated-user reinforcement learning.
- Human-LLM Collaboration: Existing collaboration approaches use multiturn-aware rewards, step-level critics, simulated environments, and prompting scaffolds to guide clarification.These methods target active collaboration, collaborative refinement, and productivity, proactivity, or personalization.
3 User-Agent Collaboration
The paper models user-agent collaboration as sequential interaction in which agents pursue goals while handling preferences that may be hidden, flexible, infeasible, or formed during the episode. It evaluates outcomes by both timely task completion and preference satisfaction, exposing limitations in existing benchmarks that assume preferences are fully specified upfront.
- Collaboration model: User-agent collaboration is modeled as a partially observable process where an agent assists a user with a goal in an environment while satisfying fine-grained outcome preferences.The goal specifies the high-level objective, while preferences characterize desirable outcomes such as price or color constraints.
- Collaboration model: At each timestep, the agent combines partial environment observations with user messages and interaction history to choose either an environment action or a response.The agent generates actions conditioned on the current observation and prior observations and actions; environment actions update the environment, while responses elicit the user’s next message.
- Collaboration model: Episodes terminate when the environment reaches a terminal state or the agent exceeds the maximum step limit, after which the outcome is mapped from the termination condition.A timeout produces a null outcome, while a terminal environment state is mapped to an outcome in the outcome space.
- Evaluation: Evaluation separates whether an outcome is produced from how many user preferences it satisfies, then combines these dimensions into a task score and a stricter perfect-outcome criterion.Task Completion measures timely outcome production; Outcome Score measures preference satisfaction conditioned on goal accomplishment; Task Score multiplies the two, while Perfect Outcome requires all preferences to be satisfied.
- Benchmark scope: Existing agent benchmarks generally assume goals and relevant preferences are fully specified in the initial instruction, omitting much of real human-agent interaction complexity.Some newer benchmarks support preference elicitation, but most focus only on underspecified preferences rather than broader preference dynamics.
- Benchmark scope: AcCoRD evaluates four preference dynamics: hard underspecification, soft underspecification, unachievable preferences, and triggered preferences.These dynamics require elicitation, flexibility management, negotiation over infeasible constraints, or proactive exposure of information that causes new preferences to form.
4 Benchmark Design
ACCORD evaluates user-agent collaboration across four preference dynamics in online shopping and travel-booking tasks. Its benchmarks use scenario generation, dynamic user simulation, and domain-specific outcome scoring to represent preferences that are elicited, flexible, adjusted, or revealed during interaction.
- Benchmark scope: ACCORD covers hard underspecification, soft underspecification, unachievable preferences, and triggered preferences across shopping and travel-booking domains.The suite includes ACCORD-Shop and ACCORD-Travel.
- ACCORD-Shop: ACCORD-Shop requires agents to elicit product attributes, customizations, and maximum budget before purchasing from WebShop.Scenarios are generated by sampling target products and constructing realistic preference conditions, including unavailable distractor values.
- Preference dynamics: Soft preferences receive partial reward, while triggered preferences emerge when relevant information is presented and unachievable preferences use fallback constraints.The travel construction assigns 0.3 ≤r(ω, pi) ≤0.7 to unsatisfied flexible preferences.
- Outcome scoring: Travel outcomes are scored deterministically per domain and averaged across bookings, whereas shopping preferences are evaluated by an LLM judge that handles surface-form variation.PERFECTOUTCOME rewards an agent only when all bookings satisfy all preference criteria.
- User simulator: The user simulator embeds goals, preferences, and revelation rules in scenario-specific prompts, with consistency checking used to reduce inconsistent responses.A secondary LLM call can regenerate a response up to five attempts when it detects inconsistency.
5 Experiments
The experiments evaluate five instruction-tuned LLMs under ReAct and uncertainty-augmented UncReAct prompting across the two ACCORD benchmarks. Results are averaged across scenarios and analyzed with multiple metrics, including preference-dynamic breakdowns, while keeping the user simulator fixed across agent conditions.
- Models and prompting: Five instruction-tuned LLMs are evaluated: Llama-3.1-70B-Instruct, DeepSeek-V3.2, GPT-4.1, GPT-5.1, and Claude-Sonnet-4.5.GPT-5.1 is evaluated with high reasoning effort.
- Models and prompting: ReAct prompts agents to produce a rationale followed by an action, while UncReAct adds structured uncertainty reasoning before that action.The added step is intended to make agents identify and resolve preference ambiguity.
- Evaluation protocol: Each scenario uses a single rollout with up to 50 environment steps and 20 dialogue turns, with models sampled at temperature 1.0.These limits apply to both ACCORD-Shop and ACCORD-Travel.
- Evaluation protocol: The user simulator is held fixed across agent conditions after GPT-5.1 with consistency checking is selected for its comparable performance and lower per-token cost.Simulator unit-test pass rates are reported for Claude-Sonnet-4.5 and GPT-5.1.
- Metrics and scoring: Outcome scoring uses an LLM judge for ACCORD-Shop and deterministic structured predicates for ACCORD-Travel.Travel scoring requires no LLM judge.
- Metrics and scoring: Main results are averaged across 100 scenarios with 3 rollouts per scenario, with four performance metrics and per-dynamic outcome-score breakdowns reported.The breakdowns analyze which preference dynamics are most challenging.
6 Results and Analysis
Across ACCORD-Shop and ACCORD-Travel, models complete many tasks but leave substantial preference satisfaction unrealized, especially for unachievable and triggered preferences. UncReAct prompting generally fails to improve outcomes or materially change interaction behavior.
- Claude 4.5 Sonnet is the strongest model across both benchmarks, followed by GPT-5.1, GPT-4.1, DeepSeek-V3.2, and Llama-3.1-70B.
- Outcome Scores range from 0.6–0.8 and Perfect Outcome Rates remain under 30%, despite generally high task completion.Perfect Outcome Rate is considerably lower on ACCORD-Travel than ACCORD-Shop because scenarios contain more preferences requiring simultaneous satisfaction.
- UncReAct does not improve preference elicitation or satisfaction: weaker models over-ask and exhaust dialogue budgets, while GPT and Claude show marginal or negative Task Score changes.
- Unachievable and triggered preferences receive lower satisfaction than hard and soft underspecification, whose scores are 0.7–0.9 and 0.8–0.95, respectively.Unachievable preferences require recognizing infeasibility and negotiating a fallback, while triggered preferences emerge only after relevant environmental information is exposed.
- UncReAct generally leaves environment actions and user utterances unchanged, with frontier models averaging fewer than 2 Shop utterances and fewer than 4 Travel utterances under either method.Table 4 reports preference scores alongside mean and median environment actions and utterances for completed trajectories.
7 Conclusions
The paper presents ACCORD to evaluate preference dynamics beyond upfront underspecification across two task environments. Its evaluation finds persistent failures on evolving preferences and no consistent prompting benefit, while identifying limits to generalization and user realism.
- ACCORD introduces four controlled user preference dynamics across online shopping and travel-planning environments to expose failures missed by existing benchmarks.
- Across five models, Perfect Outcome Rates remain below 30% even for the strongest frontier models, and triggered preferences are universally hardest.
- Prompting-based uncertainty reasoning provides no consistent benefit, motivating investigation of training-based approaches for human-agent collaboration.
- ACCORD currently covers only two task domains, and its simulator may not capture noisier real users or all preference dynamics.Gains on ACCORD may not directly translate to real-world user satisfaction.
B User Simulator Details
The user simulator optionally applies a secondary consistency check to candidate responses after generation.
- A secondary LLM checker evaluates each candidate response against the simulator’s instructions and conversation history for scenario consistency.Examples include rejecting hard-preference violations and premature revelation of triggered preferences.
B.2 Unit Test Validation
The simulator is quantitatively validated with eight unit tests applied to rollout sub-trajectories, and GPT-5.1 with consistency checking is selected for the experiments.
- Eight unit tests evaluate simulator utterances on sub-trajectories extracted from agent rollouts.Table 5 lists the tests and their evaluation prompts.
- Claude-Sonnet-4.5 and GPT-5.1 perform comparably on simulator validation, so GPT-5.1 with consistency checking is used for cost efficiency.
C LLM Evaluation Prompt
Table 7 provides the prompt used to evaluate whether a WebShop purchase satisfies a user-preference criterion.
- Table 7 contains the Claude-Sonnet-4.5 prompt for evaluating whether a WebShop purchase satisfies a user-preference criterion.
D Compute for Experiments
The experiments used different compute setups across models, while accompanying tables document simulator unit tests and pass rates.
- All models except LLaMA were evaluated through their APIs on CPUs, while LLaMA used an A100 GPU.
- All evaluations took 8–20 hours.
- Table 5 lists unit-test prompts for the ACCORD-Shop and ACCORD-Travel user simulators.
- Table 6 reports unit-test pass rates for Claude-Sonnet-4.5 and GPT-5.1 on both benchmarks.