PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

Haoming Li, Defu Cao, Xu Hu, Yingtao Luo, Yiyang Wang, Xianghe Xu, Jiashu He, Wanting Jiang, Yifan Zhang, Yunhan Zhao, Siyang Wu, Chen Yang, Haixin Wang, Zheyuan Deng, Taoran Li, Jingwen Zhang, Zhejian Peng, Shushu Liang, Yue Zhao, Bingzhe Li, Feng Chen, Yifan Wang, Jintao Huang, Yijun Wang, Yuexing Hao, Xiaomin Li

arXiv:2610.22990v1cs.HCcs.AIcs.CYcs.LG

TL;DR

Fixed task specifications do not show whether computer-use agents serve users whose requests omit requirements, add constraints, or seek confirmation before consequential actions. PersonaCUABench holds tasks fixed while varying who asks, then finds lower task completion on persona requests across nine backbones. Its results show why evaluation should account for user-expressed requirements and constraints as well as completion.

  • Problem

    Fixed-specification benchmarks do not reveal whether agents serve users who express goals incompletely, add constraints, or request confirmation before consequential actions.

    Source: §1 · p. 1

  • Method

    PersonaCUABench holds tasks fixed while a persona agent rewrites each request and answers questions, so the CUA works from dialogue without access to the profile or original instruction.

    Source: §1 · p. 2

  • Results

    Across nine backbones, persona-phrased requests lower average task completion from 0.689 to 0.441, with losses including engaged-but-missed requirements and user-requested stops scored as failures.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    CUAs should be evaluated on whether they meet the requirements and constraints users express, alongside task completion and user satisfaction.

    Source: §1 · p. 2, §7 · p. 9

    The benchmark uses constructed personas and LLM-simulated users, so performance differences should not be generalized to real populations and simulated satisfaction cannot replace real-user feedback.

    Source: Ethical Statement · p. 10

Abstract

from arXiv · show

Computer-use agents (CUAs) are commonly evaluated against fixed task specifications, but users may omit requirements, add constraints, or require approval before consequential actions. Completing the benchmark instruction does not guarantee serving the person who asked. We introduce PersonaCUABench, which holds 74 tasks across 11 domains fixed and varies only who asks. A persona agent, playing one of 44 personas in four groups, rewrites each instruction in that user's own words and answers the CUA's questions, while the CUA sees only the dialogue, never the profile or the original instruction. Across nine backbones (3,330 runs), mean task completion falls by 36% relative to the original requests. The losses have two sources: in 59.9% of cells that lose points, the agent engages with what the user asked for and still gets it wrong, and 11.2% are stops the user requested that the checker scores as failures. Even when a persona request is fully completed, the user gives the top satisfaction rating in only 62.7% of cells, against 84.1% for the original requests. CUAs should therefore be evaluated not only on task completion but also on whether they meet the requirements and constraints users express. Code: https://github.com/MatrAIx-ai/PersonaCUABench. Project page: https://matraix.ai/research/persona-cua.html

1 Introduction

PersonaCUABench tests whether computer-use agents serve the person asking, not merely complete a fixed specification, by holding tasks constant while varying who asks. Across nine backbones, persona-phrased requests lower task completion, with losses including missed requirements and user-requested stops that checkers score as failures.

1 Introduction

  • Fixed task specifications cannot reveal whether an agent serves users who omit requirements, add constraints, or ask for confirmation before consequential actions.A cautious user may be satisfied when an agent stops at a draft, even if a checker expects a sent action.Source: §1 · p. 1
  • PersonaCUABench holds tasks fixed while varying who asks, isolating how the requester affects whether an agent serves the user.The CUA receives only the persona agent’s dialogue, not the profile or original instruction.Source: §1 · p. 2
  • The benchmark pairs existing, independently checkable tasks with persona-written requests and reports task completion, user satisfaction, and requirement-level diagnoses.Each persona agent writes the opening request in character and answers the CUA’s clarifying questions.Source: §1 · p. 2
  • Task completion falls from 0.689 to 0.441 across nine backbones on persona-phrased requests, with every backbone losing between 28% and 45%.The diagnosis also identifies engaged-but-missed requirements and user-requested stops counted as failures among the sources of loss.Source: §1 · pp. 2–3

2 Serving, Not Just Solving

PersonaCUABench frames computer-use evaluation around serving the person who asks, tracking who asks, how the interaction unfolds, and how outcomes are scored. It pairs checkable task requirements with persona-shaped requests, dialogue-only CUA interaction, and measures of completion, satisfaction, and requirement handling.

2.1 Who Asks: Tasks, Users, and Requests

  • Each task is represented by its original instruction, set of requirements, and evaluation procedure, with tasks drawn from existing benchmarks whose outcomes can be checked automatically.The original instruction describes the fully satisfying outcome; the persona profile shapes how the user expresses that goal.Source: §2.1 · p. 3
  • A persona agent rewrites the original instruction into the user’s opening request, while the CUA receives only that request and later dialogue.The profile and original instruction remain private to the persona agent; the CUA can ask clarification questions.Source: §2.1 · p. 3, §2.2 · p. 3

2.2 Interaction

  • The CUA interacts from the dialogue history through tool calls and observations, and the episode ends when it calls finish or reaches its turn budget.The rollout record includes the dialogue, tool calls and observations, final answer, tabs left open, and final site state for action tasks.Source: §2.2 · p. 3

2.3 Scoring: Raised, Engaged, Met

  • Rollouts are scored for task completion and persona-rated satisfaction, while each requirement is labeled by whether it was raised, engaged with, and met.Completion is the share of checkable requirements met; the labels distinguish requirements never raised, not engaged with, or engaged with but missed.Source: §2.3 · pp. 3–4

3 PersonaCUABench

PersonaCUABench pairs fixed computer-use tasks with different personas and evaluates how requests and outcomes vary by who asks. Its protocol keeps task outcomes independently checkable while the CUA sees only the dialogue.

3.1 Task Sources and Curation

  • The benchmark retains 74 reproducible tasks—38 information and 36 action tasks—from Odysseys and REAL, spanning 11 domains and life and work scenarios.Information tasks run on live websites; action tasks use deterministic replicas, with outcomes checked without a human.Source: §3.1 · p. 4

3.2 Persona Design and Assignment

  • Four persona groups are instantiated across 11 domains, yielding 44 personas described by 37 attributes selected from the MatrAIx schema.Each persona has 32 shared general attributes and five domain-specific attributes.Source: §3.2 · p. 4

3 PersonaCUABench

  • Each task is paired with its domain’s four personas and the original request, allowing the same task to be compared across requesters.The protocol’s task set and personas are built independently and paired by domain.Source: §3 · p. 4, §3.2 · p. 5

3.3 Request Construction

  • For each task, the persona agent rewrites the original instruction as a profile-consistent request that may state, soften, omit, or add requirements.A requirement omitted from the opening request may still surface in answers to the CUA’s clarifying questions.Source: §3.3 · p. 5

4 Experiments

Across nine backbones, persona requests lower task completion relative to original requests, with losses across models and scenarios. The experiments hold the task set, user model, environment, and scoring fixed while varying the backbone and persona condition.

4.1 Setup

  • The controlled comparison varies the execution backbone and persona condition while holding tasks, user model, environment, and scoring fixed.Nine backbones are run on the same 370 cells, covering each task under four domain personas and the original request.Source: §4.1 · pp. 5–6

4.2 Overall Results

  • Task completion averages 0.689 for original requests versus 0.504 for expert, 0.402 for senior, 0.414 for young, and 0.445 for practical persona requests.Every model’s persona-group completion is below its original-request result.Source: §4.2 · p. 6
  • Compared with original requests, all nine models have lower task completion across both work and life scenarios, and at least eight of nine lose in every domain.Across models, absolute drop grows with original-request score (r = −0.60), while relative drop does not (r = −0.14).Source: §4.2 · p. 6

5 Analysis

Task completion and user satisfaction diverge: users may be dissatisfied with fully successful tasks, while failed tasks may appear complete. Analysis attributes most loss cells to agents missing requirements they engaged with, and interventions do not recover the loss.

5.1 Task Completion vs. User Satisfaction

  • Among fully successful persona cells, 62.7% receive the top satisfaction rating, compared with 84.1% for fully successful original requests.At delivery, mean satisfaction is 4.04 for original requests and 3.10–3.59 across persona groups.Source: §5.1 · p. 6, Figure 3 · p. 7
  • Of 800 failed persona action cells, 56% are judged done at delivery, falling to 27% after site-state verification.Final replies are inconsistent with outcomes in 45% of information loss cells and 26% of action loss cells; the findings show divergence in both directions.Source: §5.1 · p. 7

5.2 Where the Drop Comes From

  • 59.9% of loss cells involve agents engaging with raised requirements yet still missing at least one; 17.1% concern requirements never raised, and 2.1% raised requirements not engaged with.A further 20.9% end before the final step; 11.2% of all loss cells are user-requested stops that the checker scores as failures.Source: §5.2 · p. 8
  • Unraised requirements account for 22.9% of senior losses versus 6.2% of expert losses, while user-requested stops account for 20.2% of expert losses.The user-requested-stop share is 16.7% for practical users and 3.5% for young users.Source: §5.2 · p. 8

5.3 Interventions

  • Showing GPT-6 Astra the profile does not recover the loss, and forcing a question moves persona-request completion from 0.434 to 0.421.Forced questions lower original-request action-task completion from 0.583 to 0.472; only 24% of 609 main-experiment questions target an omitted requirement.Source: §5.3 · p. 8

6 Related Work

Related work evaluates computer-use agents on fixed tasks, personalizes agents with stored user information, or studies simulated users and clarification. PersonaCUABench differs by varying who asks the same task and tracing whether requirements are raised, engaged with, and met.

Computer-use agent benchmarks

  • Computer-use benchmarks cover fixed-instruction web tasks, long-horizon tasks, and desktop environments, but none poses the same task for different users.OSWorld 2.0 adds a simulated user; MyPCBench adds persona data while keeping the instruction fixed.Source: §6 · pp. 8–9

Personalized agents

  • Personalized-agent benchmarks condition on profiles, histories, or preferences, while this work records whether each requirement was raised, engaged with, and met.The authors report that most loss cells arise after the agent has the information.Source: §6 · p. 9

LLM user simulation

  • LLM user-simulation benchmarks study interactive agents and simulated customers; PersonaCUABench’s persona agents state requests and rate outcomes in the first person.The protocol exposes both the start and end of the service.Source: §6 · p. 9

7 Conclusion

PersonaCUABench holds computer-use tasks fixed while varying who asks, and persona requests lower task completion across nine backbones. Completion and satisfaction can diverge, so evaluation should account for users’ expressed requirements and constraints.

7 Conclusion

  • Across nine backbones, persona requests lower task completion from 0.689 to 0.441, and neither revealing profiles nor requiring clarification recovers the loss on GPT-6 Astra.The conclusion identifies carrying user requests through long action sequences as the bottleneck.Source: §7 · p. 9

Limitations

PersonaCUABench evaluates web-based information and action tasks across 11 domains, but excludes several application types and persistent interactions.

Limitations

  • The benchmark covers browser tasks, not desktop applications, standalone chat, coding workflows, or persistent user memory across interactions.Its tasks each begin with one request and focus on one task; future work could add repeated interactions and more backbones.Source: Limitations · p. 9

Ethical Statement

The paper cautions against generalizing persona-based results to real populations and distinguishes simulated satisfaction from feedback by real users. The construction appendix details materials used to build PersonaCUABench.

Ethical Statement

  • Persona construction may encode stereotypes and miss diversity within demographic groups, so performance differences should not be generalized to real populations.LLM-simulated satisfaction ratings cannot replace feedback from real users; action tasks use sandboxed replicas to avoid real transactions or communications.Source: Ethical Statement · p. 10

A PersonaCUABench Construction

A.1 Task Curation

The appendix describes task curation, persona design and prompting, persona fidelity, and the experimental harness used to run and score episodes.

A.1 Task Curation

  • The selected tasks exclude blocked or unreachable sites, pure navigation tasks, duplicates, and a video-watching task; the action tasks use ten REAL websites.The resulting set contains 18 easy and 20 medium information tasks, plus 35 medium and one hard action task, according to the source benchmarks’ labels.Source: §A.1 · p. 14
  • The 107 REAL-task requirements were derived from checker code and task manifests, then manually reviewed for intent analysis.Domains group tasks by website, spanning 40 life and 34 work tasks across 11 domains.Source: §A.1 · p. 14

A.2 Persona Attributes

  • Each persona combines 32 general attributes with five domain-specific attributes drawn from the MatrAIx schema.The general attributes include five that define persona groups; expertise is reflected in technology savviness, seniority, and domain-specific skills.Source: §A.2 · pp. 14–15, Table 6 · p. 15, Table 7 · p. 15

A.3 Persona Agent Prompt

  • The persona agent receives a profile and original instruction to write an opening request, then answers clarification questions in character.For original requests, the instruction is passed verbatim and questions are answered without a profile.Source: §A.3 · p. 15

02 Situation and Request

  • The prompt asks personas to write naturally from their situation and goal, using the expert checklist only to know what they want rather than copying its wording or procedure.It directs them to mention websites or technical steps only when their background makes those relevant, and to leave unsaid details for possible clarification.Source: §02 · p. 16

03 Persona Expression

  • Persona expression is guided by profile-consistent language, message length, circumstances, and response preferences, without explicitly listing or labeling profile traits.When asked, personas should answer truthfully according to the checklist and keep replies short and natural.Source: §03 · p. 16, §05 · p. 16

A.4 Persona Fidelity

  • On average, 15.4% of profile dimensions are clearly expressed, while 66.8% of cells clearly express at least one dimension.Contradictions occur in 11.1% of cells and excluding them leaves the mean Δ nearly unchanged, from −0.248 to −0.249; accessibility and device are rated not expressed in 95% and 82% of judgments, respectively.Source: §A.4 · p. 17, Table 8 · p. 17, Figure 6 · p. 17

B.1 Harness

  • All backbones use the same Chrome harness and nine tools, with page-text observations; screenshots are stored for auditing but are not shown to agents.The harness refuses questions beyond Q=5 and ends an episode when the agent calls finish.Source: §B.1 · p. 18

B.2 Evaluation Details

The appendix specifies how task completion, user intent, and satisfaction are measured, then reports benchmark costs, human-judge agreement, and additional results on the completion drop. The drop persists across scenarios and domains, while its size varies by persona.

B.2 Evaluation Details

  • Task completion is judged requirement by requirement from the final answer, action log, and open tabs, or by a programmatic checker of the final site state.Odysseys tasks use rubric judges; REAL tasks use state checkers.Source: §B.2 · p. 18
  • Raised records whether the user expressed a requirement, engaged records whether the agent took it up, and met records whether it was satisfied.Together, these judgments locate whether a requirement was not raised, raised but not engaged, or engaged but not met; engagement does not establish correct understanding or execution.Source: §B.2 · p. 18, Table 9 · p. 18
  • The persona agent rates its own satisfaction from 1 to 5 after the episode, and action-task ratings are repeated after site-state verification.An external judge also rates the rollout against the profile, original instruction, dialogue, actions, answer, and open tabs.Source: §B.2 · pp. 18–19

B.3 API Cost

  • At list prices without cache discounts, eight priced backbones cost $4,737 total, or $1.60 per cell on average; per-cell costs range from $0.26 to $3.26.Persona-agent and judge costs are estimated at about $202 overall, excluding opening-request generation; Muse Glimmer 30B has no list price.Source: §B.3 · p. 19, Table 10 · p. 19

B.4 Human Annotation

  • On 22 tasks, human annotators agree with the automated judge on whether requirements were met in 86.2% and 88.2% of cases.For satisfaction, annotators are within one point of the persona agent’s delivery rating on 77.7% and 79.9% of outputs; after verification, agreement is 64.6%.Source: §B.4 · p. 20, Table 11 · p. 20, §B.5 · p. 20

C.1 Overall Results

  • Across models, task-completion drops grow with original-request scores (r = −0.60), while relative drops are unrelated to them (r = −0.14).Excluding profile-contradicting cells changes mean Δ from −0.248 to −0.249; every interval remains below zero.Source: §C.1 · p. 21, Table 13 · p. 21

C.2 Generality of the Drop across Domains and Users

  • Task completion falls for all nine models across work and life scenarios and all persona groups, and for at least eight models in every domain.Work and life losses are −0.274 and −0.226, respectively.Source: §C.2 · p. 21, Table 14 · p. 22
  • Expert personas lose the least (−0.185), while senior personas lose the most (−0.287); expert requests also achieve the highest task completion under all nine models.Expert requests score 0.100 to 0.129 above the lowest group; satisfaction is highest for experts under eight models and lowest for seniors under seven.Source: §C.2 · pp. 21–22

C.3 Task Completion vs. User Satisfaction

Persona requests were fully successful less often and received top satisfaction ratings less often than original requests. The analysis traces completion losses to missed requirements and checker-scored stops, examines clarification behavior, and tests profile visibility and forced questioning.

C.3 Task Completion vs. User Satisfaction

  • The external judge barely separated successful from failed persona action cells, with AUC 0.56, while the user’s own rating reached AUC 0.62 at delivery and 0.76 after verification.Information-answer ratings can be verified only by reading cited pages, so the study did not collect after-verification ratings for those tasks.Source: §C.3 · p. 22, Table 17 · p. 26
  • Only 22.5%–27.4% of persona cells were fully successful, against 47.4% for original requests; top delivery ratings among successful cells were 62.7% versus 84.1%.Among successful persona cells, 25.9% received a rating of 3 or lower, against 9.5% of successful original-request cells.Source: §C.3 · p. 26

C.4 Where the Drop Comes From

  • In 59.9% of loss cells, agents engaged with a raised requirement but missed it; 11.2% were user-requested stops scored as failures.Another 17.1% lost points only on requirements users never raised, so 28.3% of loss cells reflected changed or unstated requests rather than failure on what the user asked.Source: §C.4 · pp. 26–27

C.5 When Agents Proactively Question

  • Only GPT-6 Astra and Claude Opus 5.5 asked clarification questions in more than 5% of information cells with omitted requirements; across models, within-task effects were positive for four and negative for five.Of questions asked, 24% targeted omitted requirements, 53% concerned requirements already stated, and 23% were unrelated; omitted requirements were eventually met in 10 of 33 information cases and 14 of 20 action cases when targeted.Source: §C.5 · p. 28

C.6 Interventions

  • Showing GPT-6 Astra the profile changed task completion by −0.027 on information tasks and +0.028 on action tasks, with intervals excluding zero only for the information-task decline.Effects varied by persona: expert satisfaction rose by 0.34 while completion changed by −0.028; senior completion fell by 0.070 and satisfaction did not rise.Source: §C.6 · p. 28, Table 23 · p. 29
  • Forcing GPT-6 Astra to ask raised questions to every persona cell but did not recover completion, which moved from 0.434 to 0.421.The agent asked 2.4 questions and took 9.5 more steps per cell; forced questions also lowered completion on original action requests from 0.583 to 0.472.Source: §C.6 · p. 28, Table 24 · p. 29

D Case Studies

The case studies show that benchmark scores can diverge from what users requested, in both directions. They illustrate checker failures for appropriate stops, checker passes despite unfinished tasks, and partial credit for weak information synthesis.

D Case Studies

  • A checker can fail an agent that stopped where the user asked, or pass one that left part of the original task undone.The expert information case received 5/7 because the final synthesis and song rationale did not meaningfully integrate evidence the agent had opened.Source: D Case Studies · p. 30, Figure 12 · p. 31
  • In the young-user case, the checker awarded 1/1 despite the agent not sending the requested group-chat link, because it checked search criteria only.The user had agreed to forward the link, and the agent’s final answer said it had not sent it.Source: D Case Studies · p. 30, Figure 13 · p. 31

Related papers

Loading 2610.22990v1…