PaperlayerView exact revision on arXiv

Source-linked AI summary

ReFract: Benchmarking Perspective Awareness in Language Model Agents with Text World Models

Hainiu Xu, Vítor N. Lourenço, Mohnish Dubey, Yunfei Bai, Yulan He, Caroline Catmur, Aline Paes, Marco Caserta, Akash Chandrayan, Luca D'Angelo

arXiv:2610.03356v1cs.AI

TL;DR

The same maintenance query can call for different actions depending on a worker’s role, making role-calibrated assistance important when actions affect physical equipment. ReFract evaluates perspective-taking and routing with executable text world models; across evaluated agents, performance remains limited and perspective-violating actions persist.

  • Problem

    Workers with different goals, authority, and knowledge may need different responses to the same maintenance query, where actions affect physical equipment.

    Source: §1 · pp. 1–2

  • Method

    ReFract formalizes Perspective-Taking and Perspective-Routing and evaluates them using executable text world models of role-specific environments.

    Source: §1 · p. 2

  • Results

    Perspective awareness remains difficult across evaluated models: the strongest agent solves at most 69% of tasks, and agents frequently attempt perspective-violating actions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results establish perspective awareness as a measurable axis of agent safety and capability.

    Source: §1 · p. 2

    The deterministic, closed world models omit stochastic dynamics, sensor noise, partial observability, and open-world tool discovery, while boundaries rely on expert-defined ground truth.

    Source: A Limitations · p. 14

Abstract

from arXiv · show

Large Language Model (LLM) agents are increasingly deployed in high-stakes settings such as industrial maintenance and equipment fault troubleshooting, where workers occupy a variety of roles. A capable agent must therefore act in a way that is calibrated to user's role: taking actions and providing information that respect the role's knowledge and capability boundaries. Unlike coding, where mistakes are usually recoverable, agent responses in these settings are enacted on physical equipment, and can therefore cause irreversible equipment damage, production loss, or personnel harm. Existing benchmarks, however, largely overlook the need for agents to infer what a role intends and acting only through tools that role may legitimately use, a capability which we term Perspective Awareness. To this end, we introduce ReFract, a benchmark of 150 expert-validated entries in which an agent must act differently in response to the same query depending on user's role. Entries of ReFract are grounded in anonymized queries from domain support conversations, against which we construct Text World Models that simulate the agent's operating environments and assemble perspective-aware action trajectories. State-of-the-art LLMs solve at most 69% of the tasks with more than 50% of their trajectories contain attempts of taking perspective-violating actions. ReFract exposes perspective awareness as a distinct, largely unsolved axis of agent evaluation and motivates agents that calibrate not just how to act, but for whom.

1 Introduction

The same maintenance question calls for different help because workers’ goals, authority, and knowledge differ. ReFract formalizes this perspective awareness, evaluates it through role-aware text-world models, and finds substantial failures across LLM agents.

1 Introduction

  • A technician seeking hands-on repair and a manager seeking operational-impact analysis and escalation can ask the same question but need different assistance.Source: §1 · p. 1
  • ReFract defines Perspective-Taking as role-bounded intent and tool use, and Perspective-Routing [38] as escalating beyond-remit tasks.Source: §1 · p. 2
  • ReFract instantiates Text World Models [14] [23] as Python programs, with states as variables and tools as functions; shared domain files fix action semantics.Source: §1 · p. 2
  • The strongest evaluated agent solves at most 69% of tasks, while agents frequently attempt actions that violate users’ perspectives.Over-caution is also a dominant failure mode: agents may defer even when users are authorized to act.Source: §1 · p. 2

2 Related Works

Prior agent benchmarks cover tool use, task completion, safety, and persona adherence, but largely leave perspective awareness unmeasured. ReFract evaluates whether role-specific action trajectories respect a user’s boundaries.

2 Related Works

  • Existing benchmarks largely omit perspective awareness; Role-Conditioned Refusals conditions on persona but scores terminal outcomes rather than persona-governed process.Source: §2 · p. 3
  • Because on-the-fly LLM simulation of state transitions remains unreliable, ReFract uses hand-authored, executable, role-gated world models for reproducible evaluation.Source: §2 · p. 3
  • Unlike CRMArena [16], ReFract composes multiple role-specific trajectories for the same query and evaluates tool-calling process as well as final outcome.Source: §2 · p. 3

3 Construction of ReFract

ReFract models each task as goal-reaching under role-specific capability and knowledge boundaries, then builds expert-grounded, executable worlds and validates role-tailored trajectories. Its evaluation compares a query-specific tool set with a larger, more realistic full tool set.

3.1 Task Setup

  • Each entry is a POMDP in which the persona constrains admissible actions, while goal attainment is scored on the terminal state.The framework separately measures persona compliance, overreach, underreach, and progress per tool call relative to a validated reference plan.Source: §3.1 · pp. 3–4

3.2 The ReFract Text World Model

  • The expert-curated Domain File D defines the world and tools; each query’s Problem File instantiates a world model from D.D specifies predicates, capability and knowledge gates, retrieval sources, and tool functions, with typed inputs and outputs enforcing data dependencies.Source: §3.2 · pp. 4–5
  • A six-stage pipeline filters queries, proposes and annotates goals, iteratively builds executable models, then gates tools by role and validates trajectories.A model-generated world is accepted only if it reaches its goals within 10 refinement iterations; role-tailored worlds are checked for skill, reachability, and successful replay.Source: §3.2 · p. 5
  • Role-tailored environments are replay-validated before expert-signed entries become benchmark items.The generation figure describes deterministic checks that can reject effects the Domain File cannot produce, even when models agree.Source: §3.2 · p. 5, Figure 2 · p. 6

3.3 Validation and Evaluation Setup of ReFract

  • Domain experts curate the shared domain and annotate goals for all 150 queries, while QTS and FTS test agents with relevant-only and full tool inventories.QTS isolates perspective awareness from tool-planning burden; FTS includes irrelevant tools to approximate a larger deployment tool inventory.Source: §3.3 · p. 6

4 Experiments

Across tool-set regimes and agent scaffolds, ReFract remains challenging: even the strongest reported configuration passes only 0.685 of tasks, and every model declines when the action space expands. QTS results also show that persona compliance can be lost through underreach, while FTS adds tool selection as a challenge.

Baselines

  • The evaluated open and proprietary models use default hyperparameters, with each inference repeated five times to measure performance variance.Claude models are excluded because their heavy involvement in benchmark generation and validation could create circularity.Source: §4 · p. 6

Agent Scaffolds

  • The evaluation compares native Function Calling with ReAct, which lets models verbalize intermediate thoughts and plan over observed states.GPT-OSS-20B is omitted from the ReAct table because it refuses the text tool-call protocol.Source: §4 · p. 6

Main Results

  • 0.685 Pass rate is the strongest result, from GPT-5.6-Sol with ReAct; every model declines in FTS, whose Pass Rate ceiling is 0.473.Table 4 reports results across QTS and FTS for Function Calling and ReAct.Source: Table 4 · p. 7, §4 · p. 7
  • In QTS, GPT-5.6-Sol has 0.566 Under vs. 0.175 Over, and strong models generally pass more often than they comply.The reported gap is driven overwhelmingly by underreach; in FTS, pass rate trails persona compliance as choosing the right tool becomes a bottleneck.Source: §4 · p. 7

5 Analysis

Across models, adding irrelevant tools weakens task success and role compliance; agents also struggle to calibrate actions to persona boundaries, especially for broad-authority roles. Consult success falls sharply in the fuller tool environment, while greater trajectory divergence does not necessarily improve task success.

Tool-environment robustness

  • Best pass rate declines from 0.625 to 0.473 in Function Calling and 0.685 to 0.466 in ReAct as tools widen from QTS to FTS.Role compliance also drops, and pass-rate robustness is negatively correlated with QTS strength (overall Spearman ρ = −0.64 with p = 0.002; ReAct alone ρ = −0.90 with p < 0.001).Source: §5 · pp. 7–8

Pass rate and off-goal rate

  • Proprietary GPT models occupy the perspective-aware quadrant; frontier models such as GPT-5.5 and GPT-5.6 fulfill queries with role-compliant trajectories, and Figure 3(a) plots overall pass-rate against off-goal rate.Off-goal rate is the fraction of persona-admissible actions that leave the goal unreachable (0 ideal). Qwen3-235B-A22B is omitted because its off-goal rate is unavailable.Source: §5 · p. 8, Figure 3 · p. 8

Over- and under-reaching

  • Most models lean toward under-reaching; GPT-5.6-Sol has 0.566 Under versus 0.175 Over.The authors suspect this may reflect safety alignment and argue for calibrating agents to varied persona boundaries rather than a global caution prior.Source: §5 · p. 8, Figure 3 · p. 8

Gate violations

  • Comprehension-boundary violations occur 1.54× as often as authority-gate violations on average (0.098 vs. 0.064).Seven of 11 models sit above the diagonal; Qwen3-235B and Qwen3-80B-A3B have higher knowledge- than capability-gate violation rates.Source: §5 · pp. 8–9

Consult-mode escalation

  • Correct consults fall from 67–71% under QTS to 42% in function-calling and 17% in ReAct under FTS.Under QTS, the remaining failures include abandoned escalations and failures to escalate; FTS outcomes are scored as correct consults, abandoned escalations, or failures to escalate.Source: Figure 4 · p. 8, §5 · p. 9

Trajectory divergence

  • Greater trajectory divergence does not imply higher pass rate: GPT models stay closest to the 0.261 reference and have the four highest pass rates.Gemma-4-31B and gpt-oss-20B are the most divergent, at 0.449 and 0.419, with pass rates of 0.362 and 0.279.Source: §5 · p. 9, Figure 5 · p. 9

Persona-level results

  • Planner and Manager roles have high pass rates but lower role compliance, with over-reach the dominant error for these broad-authority personas.GPT-5.5's Planner and Manager pass rates are 0.733 and 0.700, while role compliance is 0.967 for Technician and 0.867 for Controls Engineer.Source: §5 · p. 9

6 Conclusion

The conclusion presents ReFract as an executable text-world-model pipeline for measuring perspective awareness through perspective-taking and perspective-routing. Across models, perspective awareness remains largely unsolved, and the authors identify over-caution and broad-authority personas as important challenges.

Framework

  • ReFract evaluates perspective awareness by constructing executable text world models and decomposes it into Perspective-Taking and Perspective-Routing.The framework attributes behavioral differences to the combination of query and role rather than the substrate.Source: §6 · p. 9

Results

  • The strongest agent solves at most 68.5% of tasks, while over 50% of its trajectories contain perspective-violating actions.Source: §6 · p. 9

Implications

  • Widening the action space from QTS to FTS lures agents across boundaries they otherwise respect.The conclusion identifies over-caution as the dominant error and locates difficulty in broad-authority personas that must choose not to act.Source: §6 · p. 9

Conclusion

  • ReFract establishes perspective awareness as a measurable dimension of agent capability and a testbed for calibrating actions to the user.Source: §6 · p. 9

A Limitations

ReFract’s current benchmark is bounded by its domain and role design, expert-validated scale, idealized world model, model-dependent construction, and explicit-persona single-turn setup. These choices enable controlled evaluation but leave transfer to broader, less constrained settings unresolved.

A Limitations

  • The current release covers limited data entries, and the authors note that the benchmark’s clean isolation of perspective awareness bounds its diversity.They describe the project as a work in progress and discuss extensions to broaden coverage.Source: A Limitations · p. 13
  • The benchmark uses one industrial domain and five fixed generic roles, leaving transfer to other consequence structures and finer-grained or changing role hierarchies unresolved.The shared Domain File keeps the substrate fixed so behavioral differences can be attributed to persona.Source: A Limitations · p. 14
  • At 150 entries, expert-authored domain files and hand-annotated goals bound dataset scale, limiting fine-grained statistical power and making close model gaps noisier.The authors plan to expand the dataset with additional expert annotations.Source: A Limitations · p. 14
  • The deterministic, closed world model omits stochastic dynamics, sensor noise, partial observability, and open-world tool discovery, while scoring against expert-defined boundaries rather than real-world consequences.The authors describe the sandbox as a first step for understanding existing LLM agents.Source: A Limitations · p. 14
  • Filtering, goal proposals, and world-model drafts rely on Claude, so generator bias may persist in query selection, proposed goals, feasible trajectories, and reference-based metrics despite executable checks.Broadening the generator ensemble is a mitigation, but would limit which models can be evaluated.Source: A Limitations · p. 14
  • Agents receive the persona explicitly and act in a single non-interactive episode, so the setup does not test inferring identity from dialogue or clarifying ambiguity.The authors identify unlabeled-transcript inference and calibrated clarification as future work.Source: A Limitations · p. 14

B Ethical Statement

The ethical statement describes anonymization and expert review of industrial scenarios, and frames ReFract as a diagnostic benchmark rather than an access-control system. Experiments run offline in a simulated environment, so the reported scores are not safety certifications or deployment-readiness evidence.

B Ethical Statement

  • Queries are anonymized and generalized, with identifying and organization-specific details removed and their absence confirmed during expert validation.The entries use generic facility entities and are not traceable to a specific site, product, or individual.Source: B Ethical Statement · p. 14
  • Domain experts validated the Domain File and annotated goals for all 150 entries as compensated professional collaborators reviewing de-identified scenarios.The annotation task involved no personal or sensitive disclosure and posed no more than everyday workplace risk.Source: B Ethical Statement · p. 14
  • ReFract is intended to measure and improve perspective awareness, not train role impersonation or access-control bypass, though its encoded gates could be repurposed to probe weak boundary enforcement.The authors assess that risk as low because entries use a stylized factory abstraction.Source: B Ethical Statement · pp. 14–15
  • All experiments run offline in a hand-authored simulator; the scores measure behavior under controlled conditions, not safety certification or deployment readiness.Tool calls affect only simulator state, and the roles and gates are synthetic scoring annotations rather than live accounts or access controls.Source: B Ethical Statement · p. 15
  • Claude models are used in benchmark construction and excluded from evaluated baselines to avoid circularity; deterministic checks gate construction and rejected candidates are logged.The authors report all evaluated models, including failure modes unfavorable to the strongest systems.Source: B Ethical Statement · p. 15

C Data Statistics

The appendix points to detailed ReFract dataset statistics and illustrates how one belt-replacement query becomes role-specific world models and goals. Capability and knowledge gates constrain each role’s goal, including whether an operator can proceed or must escalate.

C Data Statistics

  • The Data Statistics subsection directs readers to Table A1 for detailed ReFract dataset statistics.The supplied passages identify the table but do not provide its cell values.Source: C Data Statistics · p. 15

D.1 Example of Annotated Query

  • The annotated-query example uses a belt-replacement query and builds a base world model before adapting it to a user’s role.The base model represents desired world states without considering the user’s role.Source: §D.1 · p. 16
  • Role-specific world models set capability and knowledge gates, and their goals are tailored to those gates; a manager’s example goal is escalation rather than repair.Escalations are checked for routing to the correct role.Source: §D.1 · p. 16
  • Technicians retain the base repair goal, while operators may perform mechanical work only when an existing SOP covers it and otherwise escalate to a technician.The example makes the operator check SOP coverage before proceeding.Source: §D.1 · p. 16

E Descriptions of the Error Codes

The error codes explain why candidate entries are excluded during generation: they may be unreachable, duplicative, unverifiable, unrepresentable by the tool catalog, or lack any authorized role. These checks screen for solvability and scoreable, role-compatible goals.

E Descriptions of the Error Codes

F The Domain File

The Domain File defines the world and its tools, separating authority to act from knowledge needed to interpret. Its transition rules enforce prerequisites and value dependencies, making appropriate action, deferral, or escalation depend on the actor’s perspective.

Domain-level components

  • The domain represents each world state as facts and defines shared predicates, capability and knowledge axes, retrieval sources, and tool kinds.Its facts include lifecycle, perspective, and action-effect predicates; capabilities govern authority, while knowledge domains govern competence to interpret.Source: F The Domain File · p. 18

Tools

  • Each tool specifies its kind, ordered preconditions, authority and knowledge gates, and the effects it adds on success.The preconditions describe required world facts, while cap and know encode separate permission and competence requirements.Source: F The Domain File · p. 18
  • Typed input slots require values from the initial state or earlier tool outputs, while outputs supply values to later tools; retrieval and terminal tools are ungated.These value dependencies are distinct from the fact dependencies carried by preconditions and effects.Source: F The Domain File · p. 19

Transition semantics

  • A tool applies only when its preconditions, input values, and required capabilities and knowledge are available; success adds effects and outputs to the state.Tasks search this transition system from an initial state toward a goal.Source: F The Domain File · p. 19

Orthogonality of the two gates

  • Authority and comprehension gates are independent, so an actor may lack permission to act, competence to interpret, or both; the appropriate response depends on which gates they hold.For example, recording a decision requires authority, while judging statistical significance requires knowledge; some actions require both.Source: F The Domain File · p. 19

Ordered chains and escalation

  • Effects and outputs form ordered tool chains; when an actor lacks a required gate, escalation is the proper handoff, while distractor tools test restraint.In the example, a decision cannot precede a review because both a precondition and an input value are missing.Source: F The Domain File · p. 19

G Model Inference

Model inference uses a shared evaluation harness across open-weight and proprietary models, with consistent prompts and tool schemas. Episodes test tool-use trajectories under multiple interaction protocols and tool-set regimes, with defaults used for decoding and failures assessed at the trajectory level.

Model access

  • Open-weight and proprietary models use the same harness, prompts, and tool schemas, while differing in model weights or hosted versus local access.Open-weight models run locally behind an OpenAI-compatible endpoint; proprietary models use hosted APIs.Source: G Model Inference · p. 20

Decoding

  • Models use default decoding configurations, and each instance is run five times with fixed sampling seeds where providers support them.Reasoning traces are not scored; evaluation uses executed tool-call trajectories.Source: G Model Inference · p. 20

Interaction protocol

  • Episodes loop through observations and tool calls until a terminal action or lenient budget limit; models are tested with Function Calling or ReAct and under QTS and FTS.GPT-OSS-20B is excluded from ReAct because it does not reliably follow the textual tool-call protocol.Source: G Model Inference · p. 20

Malformed outputs

  • Malformed tool calls are returned as errors for possible retry, but trajectories that never reach a valid terminal action count as failures.This keeps the action space identical across models and charges parsing failures to the agent.Source: G Model Inference · p. 20

H Data Annotation Platform

Domain experts use a self-contained platform to annotate goals, receiving a demonstration of the interface’s information and functionality but no additional training or instruction.

Data Annotation Platform

  • The goal-annotation platform is self-contained, and experts receive a demonstration of its available information and functionality without additional training or instruction.Figure A1 illustrates the interface used by domain experts.Source: H Data Annotation Platform · p. 20, Figure A1 · p. 21

I Off-Goal Rate

This section defines the Off-Goal Rate and the prompts used to construct and audit role-conditioned tasks, goals, executable tools, and capabilities. It also specifies how agents should act or escalate within role boundaries and how reviewers make minimal, evidence-grounded judgments about capability grants and user constraints.

I Off-Goal Rate

  • Goal-preserving actions are those that leave at least one feasible goal-reaching trajectory, enumerated by forward search over the executable world model.The enumerated reference set is checked against an independently authored candidate set refined until its continuations validate in the executor.Source: I Off-Goal Rate · pp. 21–22
  • OGR is the fraction of persona-admissible decisions whose chosen action leaves the goal unreachable.It conditions on actions within the persona’s remit, separating goal preservation from over- and under-reach.Source: I Off-Goal Rate · p. 22
  • OGR uses one decoded action per decision point and executor-based membership checks, without token probabilities or repeated samples, across both agent scaffolds.It measures goal reachability rather than compliance; OGR is 0 when every admissible move preserves some feasible goal-reaching trajectory.Source: I Off-Goal Rate · p. 22

J Prompts

  • The prompts turn support conversations into role-conditioned planning problems through staged construction, including goal-effect proposals and audits, tool selection, and role-capability assignment and review.Evaluation prompts drive function-calling or ReAct agents, while optional prompts mine and score adherence to user-stated constraints.Source: J Prompts · pp. 22–23
  • Goal-effect prompts distinguish requested operational remedies from informational knowledge: a doer, hands-on verb, and named asset indicate a physical remedy, while genuine explanation requests receive a knowledge terminal.Mechanical repair is represented by its mechanical remedy rather than a controls outcome.Source: J Prompts · pp. 24–25
  • The review prompts check whether terminal effects faithfully capture the request and whether multiple terminals are required together or are alternative remedies.A disjunction encoded as a conjunction can force every role to escalate; splitting alternatives restores an ownable goal.Source: J Prompts · pp. 25–26
  • Tool selection must produce a closed, goal-reaching subset that includes needed prerequisites, grounding, diagnostics, and both escalation and answer tools.The prompts direct selection of designated isolation tools and only one fault-clearing branch: an SOP-covered path or a knowledge-gated novel-fault path.Source: J Prompts · pp. 26–27
  • Function-calling agents must use one authorized tool per turn, escalate actions beyond their authority or expertise, and continue appropriately after handoff.They must ground answers, diagnose before acting, preserve tool-reported values, and finish through answer_user.Source: J Prompts · p. 27
  • The ReAct prompt requires exactly one Thought and one Action per turn, with role-aware escalation and a final answer_user action.It also requires grounding, diagnosis before action, real-world work order, and acting on the result of an escalation.Source: J Prompts · p. 28
  • Capability grants are task- and role-specific: grant only tool-used abilities supported by the role, with at least one role able to complete the task and another required to escalate.Authority is judged from role descriptions rather than seniority; the Manager may have office authority without field certification.Source: J Prompts · pp. 29–30
  • Mechanical authority follows the procedure seam: Operators and Technicians may perform procedure-covered mechanical work, while novel diagnostic repair is reserved for Technicians.Operators can own a procedure-covered remedy end-to-end and escalate when it is not procedure-covered or requires other authority.Source: J Prompts · p. 29
  • Authority and domain knowledge are orthogonal: a Technician may hold controls authority yet lack control-systems knowledge, prompting consultation rather than a terminal hand-off.The audit guidance says not to flag that deliberate controls grant as over-granted; the distinction separates a knowledge gap from an authority gap.Source: J Prompts · pp. 30–31
  • When a reviewed goal requires an ungranted capability, regrant it only if the role description supports that authority; otherwise leave the grant empty for human reconciliation.The prompt treats over-broad grants as damaging to the benchmark’s perspective contrast, even if that leaves a goal unreachable.Source: J Prompts · p. 31
  • Capability grants should be the minimum task-relevant subset the role holds, judged from its description and the reviewer’s rationale rather than seniority or goal convenience.The prompt treats an empty grant as valid when the role lacks the authority, and requires a one-sentence rationale grounded in the role description and reviewer rationale.Source: J Prompts · p. 32
  • Constraint mining extracts only explicit temporal, scope, method, or safety constraints, preserving a supporting transcript quote and excluding inferred constraints or restated goals.The prompt specifies an empty list when no explicit constraints appear.Source: J Prompts · p. 32
  • Constraint-adherence scoring labels each constraint adhered, violated, or not_applicable using trajectory evidence, without penalizing constraints the agent had no occasion to encounter.The prompt calls for strict treatment of explicit violations, such as shutting down a line when the user prohibited taking it offline.Source: J Prompts · pp. 32–33

Related papers

Loading 2610.03356v1…