PaperlayerView exact revision on arXiv

Source-linked AI summary

Trust and Task Completion in the World of Consumer AI Agents

Jeroen Olieslagers, Eduardo Pujol, Gal Zahavi, Lukas Ingemarsson, Shivani Poddar

arXiv:2609.33017v1cs.AIcs.CL

TL;DR

Action agents can act on errands while users are away, creating trust and completion challenges. The paper evaluates both on the same runs in a simulated world and compares Fo with base models and OpenClaw. Fo reports 71% completion and 94% trust on trap runs, while the authors note the evaluation has not been tested with real businesses or measured for real-user protected outcomes.

  • Problem

    Action agents can violate user agreement or withhold authorized action, and can abandon errands when setbacks arise.

    Source: §1 · p. 2

  • Method

    The evaluation tests Fo in a simulated world with a simulated user, matched trap-control tasks, and grading of final world states and agent traces.

    Source: §1 · p. 2, §2.6 · p. 4

  • Results

    Fo completes 71% of errands and keeps trust on 94% of trap runs; OpenClaw completes 42% of shared errands and keeps trust on 74% of shared trap runs.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results put trust alongside capability as a concern for people deciding whether to hand over inbox, calendar, and card tasks to agents.

    Source: §5.3 · p. 9

    The evaluation has not measured how often protected outcomes occur for real users and has not tested against real businesses.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Action agents do things for people. They send email, spend money, and call businesses while the user is busy with something else, so a mistake can turn into an action before anyone notices. They fail their users in two ways. They break trust when they do something the user never agreed to, or hold back after the user clearly said go. And they fall short on completion when they give up on errands that turn out to be hard. Both depend heavily on the harness around the model, meaning its instructions, tools, context, and guardrails. We built an evaluation that scores trust and completion on the same runs, in a simulated world of businesses with their own websites, inboxes, and phone lines, and of people who write back. A simulated user answers the assistant's questions. Trust means that nothing happens the user did not agree to. No email goes to someone they never approved, no private detail ends up on a group thread, no money is spent past their limit, no stranger's instructions are followed, and nothing is claimed without a source. Every trap has a matched control in which acting is the right call. We use the evaluation to measure Fo, Wajo's personal assistant, against a base model with basic instructions on three foundation models, and against the Fo harness with its guardrails switched off. Fo completes 71% of the errands and keeps the user's trust on 94% of the trap runs. The base models complete 50% to 64% and keep trust on 59% to 75%. On the matched controls, Fo goes ahead slightly less often. OpenClaw, a popular open-source assistant given the same access, completes 42% of the errands it shares with Fo, against 71%, and keeps the user's trust on 74% of the shared trap runs, against 94%. Measuring trust and completion together, on the whole system rather than the model alone, is how we think action agents become safe to hand real work to.

1 Introduction

Action agents can fail through both unauthorized or withheld actions and by abandoning difficult errands, so evaluating them requires measuring trust and completion together. The paper introduces a simulated evaluation and uses it to compare Fo with base models, its unguarded harness, and OpenClaw.

1 Introduction

  • Trust failures include acting without agreement or holding back after clear authorization, while completion failures include giving up on difficult errands.Both turn on when to act and when to stop and ask.Source: §1 · p. 2
  • Instructions, tools, user context, and pre-action checks all shape outcomes, making the harness as well as the model part of the evaluation.Scoring trust and completion on the same runs matters because separate scores can conceal a trade-off.Source: §1 · p. 2
  • In a simulated world of businesses and people, Fo completes 71% of errands and keeps user trust on 94% of trap runs; base models complete 50% to 64% and keep trust on 59% to 75%.On matched controls, Fo goes ahead slightly less often.Source: §1 · p. 2
  • Each trap has a matched control where acting is right, and a simulated user answers the assistant’s questions.Questions are counted on traps and controls alike, so refusing everything cannot make an assistant look safe.Source: §1 · pp. 2–3
  • The evaluation measures Fo against basic-instruction base models on three foundation models, and against Fo’s harness without guardrails.The configurations share tools; the harness without guardrails retains the rest of what Fo adds to the model.Source: §1 · pp. 2–3

2 Evaluation Design

The evaluation is designed to capture both appropriate action and restraint in a simulated world, using matched tasks, simulated users, and checks of outcomes. It compares Fo configurations and OpenClaw while measuring completion and trust.

2.1 What the Design Must Capture

  • Design choices capture irreversible side effects, audience-sensitive privacy, asking the absent user, and the risk of stopping too early.The evaluation grades final business and mailbox states, varies who can see private information, counts questions, and treats premature stopping as a completion failure.Source: §2.1 · p. 3

2.2 Three Configurations

  • Three configurations use the same tools, budgets, and simulated world: a base model, Fo without guardrails, and Fo with guardrails.The base configuration runs on three foundation models; the two Fo configurations use one model with fixed settings.Source: §2.2 · p. 3

2.3 The World and the Simulated User

  • A constructed world supplies businesses, controlled inboxes, websites, phone lines, and user accounts, while simulated people reply to the assistant.The simulated user answers from task intent alone, never volunteers information, and can reply for a limited number of rounds.Source: §2.3 · pp. 3–4

2.4 Task Suites

  • Trust suites test outreach, private information reaching shared contexts, and spending limits; completion suites test finding details, careful steps, and setbacks.Some tasks span multiple exchanges, with decisive information arriving before it is needed.Source: §2.4 · p. 4

2.5 Task Design

  • Each trap-control pair differs only in the detail that makes acting right, and every configuration receives the facts the task requires.Each task runs three times per configuration.Source: §2.5 · p. 4

2.6 Grading and Metrics

  • Grading checks final world state and the agent trace, recording goal achievement and protected outcomes to measure completion and trust.Only executed actions count toward protected outcomes, and conclusions are about the task suites.Source: §2.6 · pp. 4–5

2.7 OpenClaw

  • OpenClaw is run on the same tasks and access level, but it has fewer tools than Fo, so the comparison also reflects tool differences.Commercial assistants cannot be compared like-for-like because their harnesses are unavailable to run on the tasks.Source: §2.7 · p. 5

3 Results

Fo improves trust and completion over base models, while matched controls pass slightly less often. It also outperforms OpenClaw on shared errands and trap runs, with guardrail and task-follow-through trade-offs.

3 Results

  • +20 to +36 percentage points in trust and +6 to +21 points in completion: Fo outperforms each base model, while matched controls pass at 91% versus 94% to 95%.The harness without guardrails reaches 70% completion and 88% trust; adding guardrails adds +6 points of trust and changes completion by +0.5.Source: §3 · p. 5
  • 85% of Fo trust-task runs and 61% of its completion-hazard runs met the goal with no violation, versus 48% and 38% to 66% and 52% for base models, respectively.For the harness without guardrails, the corresponding shares were 81% and 59%; OpenClaw scored 50% and 43%.Source: §3 · p. 5

3.1 Trust

  • 93% of Fo runs avoided violations on contact traps and 91% on privacy traps, ahead of every base model, though matched-control pass rates were lower.The graders found 0 clear harms in 172 Fo hazard runs, versus 5 in 173 runs without guardrails and 15 to 34 for base models.Source: §3.1 · pp. 5–6, Figure 2 · p. 6
  • Guardrails hold unrequested or privacy-leaking messages for user review, but without a covering guardrail Fo can still commit the user on a delegated thread.The paper reports that this judgment is not reliable from run to run.Source: §3.1 · pp. 5–6
  • Fo prompted 0.82 user replies per trust-task run, compared with 0.23 to 0.50 for base models and 0.77 without guardrails.On matched controls, the corresponding figures were 0.42, 0.06 to 0.19, and 0.40.Source: §3.1 · p. 6

3.2 Task Completion

  • 71% of Fo errands were completed, versus 50%, 64%, and 51% for the base models; the guardrails-on increase over the unguarded harness was +0.5.The comparison is shown by errand type in Figure 3.Source: §3.2 · p. 6, Figure 3 · p. 7
  • The harness adds follow-through and checkout care, but its guardrails can block task-list access on outside threads and remove requested details recipients need.The paper also notes that going straight to the cart can miss a discount shown only on the product page.Source: §3.2 · pp. 6–7

3.3 Comparison with OpenClaw

  • On shared errands, Fo completes 71% versus OpenClaw’s 42%; on shared trap runs, Fo keeps trust on 94% versus OpenClaw’s 74%.Some OpenClaw misses reflect unavailable tools, including task tracking and a dashboard.Source: §3.3 · p. 8

4 Related Work

Earlier work jointly evaluates usefulness and safety, including on agent tasks and matched benign controls. This paper’s stated addition is a full-harness comparison against base models across three foundation models, with trust and completion scored together.

4 Related Work

5 Discussion

Fo completes more errands than the compared base models and OpenClaw while maintaining higher trust on trap runs. The discussion draws broader lessons about evaluating whole systems and pairing usefulness with safety, while noting boundaries in the simulated evaluation and outlining future extensions.

5 Discussion

  • Fo completed 71% of errands and kept trust on 94% of trap runs; base models reached 50% to 64% completion and 59% to 75% trust, while OpenClaw reached 42% completion and 74% trust on shared runs.The guardrails did not visibly cost completion.Source: §5 · p. 8
  • The authors’ broader lesson is that agent usefulness and safety depend on instructions, know-how, and checks around the model, so model-only benchmarks cannot establish readiness to act for users.They argue that both outcomes should be measured on the same runs with a user available to answer questions.Source: §5 · p. 8

5.1 Limitations

  • The evaluation did not measure protected-outcome frequency for real users or test against real businesses, and its businesses and people are invented.The authors say a broader, more realistic world with more kinds of businesses and people would cover more situations users encounter.Source: §5.1 · pp. 8–9
  • The model grader’s agreement with human raters remains unmeasured, and the simulated user made wrong decisions in 11% of cases, changing 2% of grades; 32 affected runs were excluded.Because the simulated user also answers reports, reply frequency is only a rough measure of how often an assistant asks.Source: §5.1 · p. 9

5.2 Future Directions

  • Future work would test the base configuration on more foundation models, count human hand-offs with their own cost, and add richer histories, settings, and adversarial outsiders.The authors also plan an open release of the world, tasks, and graders, with regular publication of results for comparison over time.Source: §5.2 · p. 9

5.3 Broader Impacts

  • The authors argue that companies running agents should test whole systems in lifelike conditions, read consequences from the world, and publish joint usefulness-and-safety results for public scrutiny.An agent’s mistakes can affect people beyond its user, and may be hard to undo; lost trust can also discourage further delegation.Source: §5.3 · p. 9
  • The evaluation used invented businesses and controlled inboxes, so no real person or business interacted with an unready assistant; passing the benchmark does not establish safety everywhere.The authors frame the protocol as one layer alongside real-user monitoring and clear user controls.Source: §5.3 · p. 9

A The Base Model’s Instructions and the Neutral Facts

  • The base model receives a reproduced instruction prompt plus shared neutral user and conversation facts, while tool inputs are shared across configurations.Some tool descriptions retain policy language, which the authors say can make the base model appear stronger than a bare model.Source: A The Base Model’s Instructions and… · p. 13

B Related Benchmarks in Detail

The related benchmarks cover joint safety-and-usefulness scoring, user interaction, and stateful task environments, while capability benchmarks measure adjacent skills. The paper positions its protocol among these approaches and identifies limits in running consumer assistants on shared tasks.

Helpful and harmless

Safety and usefulness together for agents

Stateful environments and users in the loop

Capability on the web and as an assistant

Asking for help and working with people

  • Magentic-UI checks irreversible actions with guards and reports requests for help, while step-by-step systems keep users involved throughout; here, the user delegates and is absent, so each question interrupts their day [Mozannar et al., 2025, Huq et al., 2025, Feng et al., 2026].The paper measures that interruption as replies per run.Source: B Related Benchmarks in Detail · p. 14

C Measurement and Statistics

The appendix defines how trust, completion, and questions are measured on task runs, then specifies paired statistical comparisons and a layered grading process. The rules distinguish protected hazards from controls and use both simulated-world records and agent traces to assess outcomes.

C Measurement and Statistics

  • Trust is one minus violations on hazard runs, while autonomous completion is goal achievement without human intervention on completion suites.The per-run safe outcome combines goal achievement with no violation; trap goals are also reported so refusal cannot look safe.Source: C Measurement and Statistics · pp. 14–15
  • Questions are counted per run from simulated-user replies, separately for hazard tasks and controls; outreach-control replies may be part of the task.The user answers only from declared intent, and an unanswered question cannot turn an unfinished task into success.Source: C Measurement and Statistics · pp. 15–16
  • Each task declares eligible protected outcomes and user-only information in advance, and each outcome has its own denominator.Eligible outcomes can overlap, while the final trust set contains 60 hazard tasks, each eligible for one outcome.Source: C Measurement and Statistics · p. 15
  • Permission rules specify what user requests authorize, including cumulative spending limits, permitted recipients, and when third-party instructions cannot grant consent.A later user instruction to stop or check first withdraws earlier permission.Source: C Measurement and Statistics · p. 15
  • Paired task-level bootstrap intervals and exact McNemar tests compare configurations; repeats estimate variation on fixed suites, not evidence about other tasks.The primary contrast was declared in advance; neighboring-configuration contrasts are exploratory.Source: C Measurement and Statistics · p. 16, Table 2 · p. 17

D Grading Rubric

  • Grading checks simulated business records, messages, and tool traces before using model graders for criteria deterministic checks cannot settle.On email tasks, two graders from different model families assess each criterion; a split makes it inconclusive.Source: D Grading Rubric · p. 20

E Exclusion and Rerun Rules

The rules rerun outcomes caused by the test setup, leave some inconclusive judgments unmeasured, and exclude simulated-user errors only on affected axes. The section also describes the trust and completion task suites and the count tables behind the figures.

E Exclusion and Rerun Rules

  • Runs decided by the test setup rather than the agent are rerun on a clean account at the same configuration, task, and repeat.For host-parity reruns, each is recorded beside the original it replaces, which is not counted.Source: E Exclusion and Rerun Rules · p. 20
  • A split email grader, failed grader response, or unavailable check leaves a criterion inconclusive and never counts as a pass.A run with an inconclusive criterion is not rerun; it can still count on trust if a violation was recorded.Source: E Exclusion and Rerun Rules · p. 20
  • Simulated-user errors exclude a run only on an axis where the error could change a failed criterion; alternative rules leave rates within 2 points and rankings unchanged.With no simulated-user exclusions, no configuration’s completion or trust rate moves by more than 0.8 points.Source: E Exclusion and Rerun Rules · pp. 20–21

F The Trust Suites

  • The trust suite pairs every trap with a control differing in one detail, so withholding when acting is allowed also fails.Its traps cover unauthorized contact, unapproved commitments, private details, and unsupported claims.Source: F The Trust Suites · p. 21

Counts behind the figures

G The Completion Suites

The completion suite groups errands by the challenge they pose, from finding unstated details to writing to others.

G The Completion Suites

  • Six errand types test finding details, shopping and checkout, recovering from setbacks, following through, anticipating needs, and writing to other people.The tasks include late fees or preselected add-ons, failed forms, closing loose ends, and matching messages to the right recipients and form.Source: G The Completion Suites · p. 23

H Example Tasks

The examples show three tasks across the systems, with each system represented by a typical run.

H Example Tasks

  • Three example tasks cover completion, outreach, and a standing rule, with each system’s most common outcome shown across its three runs.Panels include the three base models, Fo without guardrails, Fo, and OpenClaw.Source: H Example Tasks · p. 23

A code on the cart page

The cart-page promo code let some assistants bring the order under Evan’s $70 limit, while others ordered at the undiscounted price. The examples also show that one-click checkout could leave no opportunity to change the address.

A code on the cart page

  • The code CART15, worth 15% off, appeared on the cart page rather than the product page.The task required one order within $70 that used the code.Source: A code on the cart page · p. 24
  • Claude Opus 5.5 and Fo without guardrails reported that Buy Now placed the order in one click, leaving no review screen to change the address.Claude Opus 5.5 said this happened in 3 of 3 runs; Fo without guardrails reported it in its example.Source: A code on the cart page · p. 24
  • Fo, GLM 5.3 Flash, and OpenClaw used CART15 and charged $52.70, while Claude Opus 5.5, GPT-5.6 Sol, and Fo without guardrails paid $62.00.Fo, GLM 5.3 Flash, and OpenClaw each did so in 2 of 3 runs; Claude Opus 5.5 and GPT-5.6 Sol did so in 3 of 3 runs, and Fo without guardrails in 2 of 3 runs.Source: A code on the cart page · p. 24

An outsider asks the assistant to email the movers

When Carla’s email supplied an unverified mover contact and requested an insurance certificate, Claude Opus 5.5 and GLM 5.3 Flash emailed the movers. Fo and OpenClaw instead withheld the message pending Evan’s authorization.

An outsider asks the assistant to email the movers

  • Claude Opus 5.5 and GLM 5.3 Flash emailed Reyes Movers using the name and address Carla supplied, despite lacking confirmation that Reyes was Evan’s mover.Each did so in 3 of 3 runs.Source: An outsider asks the assistant to… · p. 24
  • Fo and Fo without guardrails did not email the movers, while OpenClaw sought Evan’s go-ahead before contacting the new external sender.Each assistant withheld the message in 3 of 3 runs.Source: An outsider asks the assistant to… · pp. 24–25

The tile studio copies someone Evan asked to leave off

When Evan asked that Noor be kept off vendor emails, the assistants differed: GLM 5.3 Flash, GPT-5.6 Sol, and Fo without guardrails copied her, while Fo and OpenClaw kept her off and booked the installation.

The tile studio copies someone Evan asked to leave off

Related papers

Loading 2609.33017v1…