PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs Trust Appropriately? Evaluating Calibrated Reliance on Imperfect Partners

Mengdi Chu, Jintao Huang

arXiv:2610.34107v1cs.AIcs.CLcs.HC

TL;DR

It remains unclear whether LLMs translate evidence about partner performance into rational downstream reliance decisions, despite the costs of both accepting unreliable work and exhaustive verification. The paper introduces a benchmark of 324 scenarios across 23 collaboration categories and evaluates seven models. Models frequently exhibit excessive protection, verification with zero expected value of information, missed learning through cooperation, and misdirected assistance, while GPT-5.6 Sol selects reference-optimal actions at all three stages in 314 of 324 scenarios (96.9%).

  • Problem

    It remains unclear whether LLMs translate evidence about partner performance into rational downstream reliance decisions, despite the costs of both accepting unreliable work and exhaustive verification.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces a benchmark of 324 scenarios across 23 collaboration categories and evaluates seven models.

    Source: §1 · p. 2

  • Results

    Models frequently exhibit excessive protection, verification with zero expected value of information, missed learning through cooperation, and misdirected assistance, while GPT-5.6 Sol selects reference-optimal actions at all three stages in 314 of 324 scenarios (96.9%).

    Source: §5 · p. 6, §7 · p. 10

  • Takeaways & Limitations

    Calibrated trust requires coordinating empirical evidence with operational costs and downstream task horizons, not merely reacting with caution or inspection.

    Source: §7 · p. 10

    The benchmark uses a deliberately three-decision horizon, whereas many real-world human–AI and multi-agent collaborations involve longer or indefinite interactions.

    Source: D. Limitations · p. 21

Abstract

from arXiv · show

Large language models increasingly collaborate with humans and peer AI agents whose contributions may be incomplete, inconsistent, or incorrect. Effective collaboration requires calibrated trust, whereby decisions to rely on, verify, or assist a partner adapt rationally as empirical evidence accumulates. We investigate whether models translate partner evidence into appropriate reliance policies. We introduce a benchmark of 324 scenarios across 23 collaboration categories. Each scenario involves three sequential decisions with explicit transition dynamics, feedback emissions, and cost schedules, permitting exact analytical evaluation of choices against their expected consequences over the remaining interaction. Evaluating seven models alongside 36 controlled-history probes, we dissociate evidence sensitivity from normatively appropriate action. While the strongest frontier model approaches near-optimal decisions, other models exhibit recurring pathologies: procuring excessive safeguards, purchasing verification tests with zero expected value of information, prioritizing immediate savings over cooperative learning, and misallocating assistance to suboptimal channels. Supplying explicit posterior distributions resolves certain errors while introducing new failure modes. Calibrated trust in autonomous agents critically demands coupling empirical evidence with forward-looking decision utility.

1. Introduction

The paper asks whether LLMs convert evidence about imperfect partners into normatively calibrated reliance, balancing the risks of accepting unreliable work against the costs of exhaustive verification. It introduces a sequential benchmark with an analytical reference policy and finds both near-optimal performance in one model and recurring reliance errors in others.

1. Introduction

  • Effective cooperation requires deciding whose advice to follow, what to delegate, and when to verify, because unreliable work can propagate errors while exhaustive checks undermine delegation.The paper frames this as the dual hazard of excessive reliance and disuse.Source: §1 · p. 1
  • The central question is whether models translate partner evidence into normatively calibrated reliance, rather than merely reacting to partner characteristics.Prior work leaves open whether evidence-driven belief updates yield rational downstream decisions.Source: §1 · pp. 1–2
  • Actions can shape both later evidence and partner performance, while controlled-history pairs test whether models adapt reliance to changing evidence or maintain it when warranted.The benchmark includes 36 pairs of controlled histories.Source: §1 · p. 2
  • The benchmark covers 324 scenarios across 23 collaboration categories, with three sequential decisions and fully specified transitions, feedback, and costs.These specifications permit computation of a Bayes-optimal reference policy and expected cost-to-go for each available action.Source: §1 · p. 2
  • Across seven models, GPT-5.6 Sol approaches near-optimal reliance, while others show excessive protection, zero-value verification, missed cooperative learning, and misdirected assistance.The supplied passage also notes that explicit posterior distributions resolve some errors while introducing novel failure modes.Source: §1 · pp. 2–3

2. Background and Related Work

Prior work studies trust, advice taking, verification, and source credibility, including LLMs as trusting agents. This paper places those questions in sequential cooperation, where reliance, checking, and assistance have costs and shape later interactions.

2. Background and Related Work

  • Trust research distinguishes perceived partner qualities from the decision to depend on a partner, which automation research frames in relation to context and goals.Advice-taking work examines accepting valid advice, rejecting errors, and how verification effort affects reliance.Source: §2 · p. 3
  • Studies of LLMs as trusting agents examine partner-related biases, how human attributes affect trust, and costly verification against a memoryless baseline.These works consider trust formation, collapse, and repair.Source: §2 · p. 3
  • Research on source cues studies motivated communication, human-versus-AI labels, credibility, and framing, while this paper extends source evaluation to sequential cooperation with tangible costs.In the studied setting, relying, checking, and assisting shape downstream interactions.Source: §2 · p. 3

3. Benchmark Design

The benchmark evaluates calibrated reliance as choosing actions that minimize cumulative expected cost under fully specified scenario rules. It combines sequential interaction, analytical reference decisions, broad task coverage, and controlled-history probes.

3. Benchmark Design

  • Calibrated reliance is operationalized as choosing actions that minimize cumulative expected cost under each scenario’s fully specified rules.The criterion connects an agent’s evidentiary state with the normative validity of its actions.Source: §3 · p. 3

3.1. Interaction and reference decisions

  • Each scenario is a three-stage interaction in which actions such as delegation, verification, assistance, or takeover reveal feedback, and partner performance can vary or change over time.The initial prompt supplies the setting, partner rules, candidate actions, and costs; many environments use two contribution channels with distinct performance rules.Source: §3.1 · pp. 3–4
  • The reference policy compares actions by immediate fees and outcome costs plus expected downstream costs under optimal continuation.The specified decision tree integrates latent partner states and future feedback; the example shows that diagnostic evidence can matter across stages while observations from another domain may provide no evidence.Source: §3.1 · p. 4

3.2. Task construction and coverage

  • The benchmark contains 324 scenarios across 23 categories and 74 blueprints, with each blueprint varying environmental parameters, cost schedules, and interaction processes.The scenarios are grounded in empirical coordination dilemmas and specify latent partner states, observation emissions, and action costs.Source: §3.2 · p. 4
  • Across 300 core scenarios, five interaction processes vary feedback and partner-state transitions; 24 specialized scenarios cover contracts, disclosure, communication, and commitment.Workflows differ in whether success requires both channels or either channel, and whether feedback isolates contributions or aggregates them, so the same reliability estimates can imply different optimal actions.Source: §3.2 · pp. 4–5
  • Thirty-six controlled-history pairs test whether models switch reliance when optimal actions differ and remain stable when those actions are identical.The pairs span all 23 categories; optimal action sets are disjoint in 31 pairs and identical in five.Source: §3.2 · p. 5

4. Experimental Setup

Seven models are evaluated on isolated scenario trajectories and controlled histories, with a separate condition supplying exact Bayesian state distributions for specialized scenarios. The evaluation measures optimal-action accuracy, regret, and targeted reliance behaviors.

4.1. Models and interaction protocol

  • Seven models receive identical instructions and actions without external tools, and scenario histories accumulate while feedback draws are fixed across models.One trajectory is evaluated per primary scenario and one response per controlled-history input; exact Bayesian state distributions are additionally supplied on 24 specialized scenarios.Source: §4.1 · p. 5

4.2. Evaluation measures

  • Decision accuracy counts reference-optimal actions, averages reached-stage accuracy equally across 74 blueprints, and also reports all-stage accuracy across 324 scenarios.Unparseable responses count as incorrect decisions.Source: §4.2 · p. 5
  • Normalized regret measures excess expected cost under optimal continuation relative to the action-value spread, averaged over reached decisions with non-zero spread.The regret quantity is defined at decision nodes and normalized to aggregate across heterogeneous cost scales.Source: §4.2 · pp. 5–6
  • Controlled-history evaluation measures pairs with both decisions optimal and agreement with the expected switch-or-stability relation.Analyses also examine over-reliance, superfluous verification, and performance by category, interaction process, and stage.Source: §4.2 · p. 6
  • Targeted analyses test whether verification benefits exceed fees, whether immediate cost minimization conflicts with sequential optimality, and whether assistance targets the highest-value contribution.Aggregate metrics use bootstrap intervals over blueprints, while diagnostic proportions use Wilson score intervals.Source: §4.2 · p. 6

5. Results and Analysis

Across the benchmark, one model achieves near-optimal three-stage reliance, while other models make costly errors in evidence response, verification, cooperative learning, assistance, and use of explicit reliability information.

5. Results and Analysis

  • 314/324 scenarios (96.9%) receive reference-optimal actions at all three stages from Sol, showing sustained calibrated reliance is achievable in these environments.Blueprint-macro decision accuracy ranges from 80.9% to 99.0%, and all-stage accuracy ranges from 56.5% to 96.9%.Source: §5 · p. 6, Table 2 · p. 6

5.1. Responding to partner evidence

  • 100% relation agreement by Gemini, Sonnet, and DeepSeek across 36 pairs coexists with paired accuracy of 97.2%, 86.1%, and 94.4%, while contract choices show costly overprotection.Gemini and Sonnet choose more protective contracts after success and failure than the expected-cost-minimizing choices warrant.Source: §5.1 · pp. 6–7, Figure 2 · p. 7

5.2. Deciding when to verify

  • Six models over-test; among 108 unneeded tests, 107 target unchanged channels and 61 provide zero gross benefit before fees.Gemini makes 26/32 unnecessary tests (81.3%) and Sonnet 29/32 (90.6%).Source: §5.2 · p. 7
  • Haiku skips 8/42 worthwhile tests and Qwen 5/42; initial local-replacement decisions err at 28.2%, versus 10.1% and 4.2% later.The pooled first-stage error rate is 28.2% (95% CI 22.0–34.6).Source: §5.2 · p. 8

5.3. Learning about partners through cooperation

  • Sol chooses the exploratory action in all 15 learning-over-savings cases, while Haiku does so in none; route isolation yields cumulative cost 8.681 versus 9.282 for hedging.Sonnet also chooses the myopic cost minimizer in 14 of the 15 cases.Source: §5.3 · pp. 8–9
  • Sonnet chooses myopic actions in all seven ordinary-cooperation cases but buys optimal audits in all seven matched audit cases, revealing a workflow-dependent learning contrast.The matched workflows share production payoffs and prior state probabilities, differing in feedback timing and available actions.Source: §5.3 · p. 9, Figure 4 · p. 10

5.4. Allocating assistance for future cooperation

  • In assembly, assisting P yields expected cost 12.385 versus 14.313 for Q despite Q’s larger nominal gain; models often target Q.Sonnet makes this wrong-target choice in eight of its nine first-stage errors.Source: §5.4 · p. 9, Figure 4 · p. 10

5.5. Using explicit reliability information

  • Supplying explicit partner-state distributions repairs some errors but introduces new ones; Haiku repairs six of eight errors and adds five mistakes, netting one additional optimal decision.The net gain is across 46 paired decisions for Haiku.Source: §5.5 · p. 9, Figure 4 · p. 10

6. Discussion

Calibrated reliance requires separating responsiveness to partner evidence from choosing actions that improve downstream collaboration. Work allocation also shapes what can be learned, while decision support can repair errors and introduce new ones.

6. Discussion

  • Evidence-sensitive caution is not sufficient: safeguards are justified only when their benefits exceed their costs.Assistance is warranted only when it yields downstream collaborative payoff.Source: §6 · p. 10
  • Work allocation provides information as well as production: isolating contributions sharpens future delegation, while joint outcomes make credit assignment harder.Structuring informative workflows is a distinct trust-calibration demand from willingness to verify.Source: §6 · p. 10
  • Supplied reliability distributions resolve some errors but also introduce new mistakes, motivating tests of scaffolding that links actions to costs, feedback, and downstream consequences.Interventions should be evaluated for both errors corrected and competencies preserved.Source: §6 · p. 10

7. Conclusion

The conclusion argues that calibrated trust requires coordinating empirical evidence with operational costs and downstream task horizons. The paper reports AI assistance and makes its benchmark materials and code available.

7. Conclusion

  • Calibrated trust requires forward-looking coordination between empirical evidence, operational costs, and downstream task horizons.Across seven models, evidence responsiveness often coexists with excessive protection, zero-value verification, missed cooperative learning, and misdirected assistance.Source: §7 · p. 10

AI use statement

  • Generative AI tools assisted with literature discovery, code development, data analysis, and manuscript preparation.The authors retain responsibility for study design, analysis, interpretation, and the final manuscript.Source: AI use statement · p. 11

Reproducibility statement

  • The benchmark, evaluation environments, prompts, feedback draws, and scoring code are publicly available.Appendices A and B describe task construction, reference calculations, model configurations, and scoring.Source: Reproducibility statement · p. 11

A. Task Inventory and Construction Details

The benchmark combines broad activity coverage with structural and dynamic variation, and its reference calculations specify decision outcomes across sequential scenarios.

A. Task Inventory and Construction Details

  • The inventory covers 23 collaboration categories; each category groups related activities, while each blueprint specifies how contributions combine and which actions are available.The 74 blueprints contribute four or five scenarios each, totaling 324.Source: A. Task Inventory and Construction… · p. 13, Table 4 · p. 13
  • Of 324 scenarios, 300 use five interaction processes and 24 use specialized environments for contracts, disclosure, communication, and other cooperation forms.The 31 groups summarize static outcome relations; performance is also broken down by blueprint, collaboration category, and interaction process.Source: A. Task Inventory and Construction… · pp. 13–14
  • Task diversity varies contribution structure and interaction dynamics, including parallel routing versus sequential assembly and channel-specific versus aggregated feedback.Partner transitions or assistance withdrawal may also occur downstream, changing what agents can infer and how they should allocate work.Source: A. Task Inventory and Construction… · p. 13
  • The 74 blueprints are grouped into 31 static outcome-relation groups, which ignore action labels, cost magnitudes, action fees, and temporal processes.Complete scenarios additionally specify feedback, state changes, and sequential choices.Source: A. Task Inventory and Construction… · p. 14
  • The reference environment contains 10,652 decision nodes, 43,086 node-action options, and 75,772 feedback branches; 277 scenarios have positive adaptation value and 47 have zero adaptation value.These scenario groups were determined from reference policies before model evaluation.Source: A. Task Inventory and Construction… · p. 14

A.1. From cooperation research to benchmark tasks

The benchmark translates foundational cooperation research into executable decision environments by specifying contribution channels, partner states, feedback, and action consequences.

A.1. From cooperation research to benchmark tasks

  • Task environments operationalize contribution channels, latent partner states, observational feedback, and action-consequence topologies.These design choices turn theoretical paradigms from organizational psychology, behavioral economics, and cognitive science into executable decision settings.Source: §A.1 · p. 14
  • The numerical parameters and reference policies belong to the constructed environments.Source: §A.1 · p. 14
  • Four foundational research paradigms are connected to benchmark tasks and trust-calibration questions in Table 5.Each paradigm is translated into an executable environment by operationalizing contribution channels, latent partner states, observational feedback functions, and action-consequence topologies.Source: Table 5 · p. 14, §A.1 · p. 14

B. Task and Evaluation Details

The appendices specify the benchmark’s sequential tasks, evaluation and scoring procedures, and controlled-history tests. They also report decision outcomes, including mismatches between evidence-responsive choices and normatively optimal actions.

B.1. A worked example

  • Each worked example asks the model to choose among delegation and backup across three jobs, with hidden partner states, feedback, transition rules, and costs.For the example, delegation at the first decision has expected total cost 6.63 versus 7.54 for backup.Source: §B.1 · p. 15

B.2. Prompts and evaluation protocol

  • Scenarios run in fresh contexts with visible history replayed; legal actions advance the environment, while empty, ambiguous, or unparseable responses end the trajectory as errors.The prompts request JSON actions and prohibit tools or external materials.Source: §B.2 · p. 15

B.3. Scoring and uncertainty

  • An action is correct when its expected cost matches the optimum, with ties credited; controlled-history paired accuracy requires both optimal actions, while relation agreement checks the expected switch or stability.Main-metric confidence intervals bootstrap blueprints with 10,000 resamples; diagnostic proportions use 95% Wilson intervals.Source: §B.3 · p. 16

B.4. Reliability-support condition

  • Reliability support supplies the exact posterior over hidden partner states from task rules and observed history, but gives neither action values nor a recommendation.The comparison uses 24 specialized scenarios and only shared histories with valid responses in both runs.Source: §B.4 · p. 16

B.5. Controlled histories requiring a changed or stable choice

  • Controlled-history pairs vary evidence while holding prior trajectories fixed, testing whether choices switch when optimal actions differ and remain stable when they do not.The contract pair requires a switch after pass versus failure; the worked-example pair requires delegation after either Y outcome because Y is independent of X.Source: §B.5 · pp. 16–17

C.1. Decision outcomes and activity categories

  • Table 6 reports reached stages, correct actions, errors, and unreached stages; Figure 5 groups accuracy by 23 collaboration categories, pooling reached stages.Category scores pool reached stages, whereas the main table averages accuracy across blueprints.Source: §C.1 · p. 17, Table 6 · p. 17

C.2. Controlled-history decisions

  • 20 model–pair observations show relation agreement without paired correctness; all six models other than Sol make this mismatch in the contract pair.After failure the optimal choice is the performance contract, and after success it is the default; Gemini and Sonnet instead choose the guaranteed contract after failure and the performance contract after success.Source: §C.2 · p. 18

C.3. Information acquisition before local replacement

The analyses examine when tests, ordinary cooperation, audits, and assistance improve decisions. They identify unnecessary testing, the value of learning from cooperation, misdirected support, and a targeted improvement under reliability support.

C.3. Information acquisition before local replacement

  • Of 74 initial decisions, testing is suboptimal in 32 scenarios and optimal in 42; 61 of 108 unnecessary test choices have zero expected benefit.In 17 scenarios, both tests have zero benefit, and all 17 are among the 32 skip-optimal scenarios.Source: §C.3 · pp. 18–19, Table 8 · p. 18, Table 9 · p. 19

C.4. Learning through cooperation and explicit audits

  • In the route-planning example, using the record adviser has immediate cost 1.860 and total cost 8.680600, versus 1.837 and 9.282070 for reserving both routes.Across 15 scenarios selected because myopic and optimal actions differ, every incorrect first action is among the immediate-cost-minimizing actions.Source: §C.4 · p. 19
  • Seven blueprints also have delayed-feedback counterparts where an explicit audit is optimal; the audit adds 0.4 to the execution fee and supplies an immediate check.Ordinary execution in that workflow provides no first-job quality feedback.Source: §C.4 · p. 19, Table 10 · p. 20

C.5. Assistance allocation

  • In the assembly example, supporting P costs 12.38525 under optimal continuation, versus 14.31325 for Q, although Q yields the larger local reliability gain.Sonnet nevertheless supports Q; the 74 improvement-or-assessment scenarios also distinguish wrong support targets from other decision errors.Source: §C.5 · pp. 19–20, Table 11 · p. 20

C.6. Reliability-support comparisons

  • Qwen’s scenarios with three optimal decisions rise from 12 to 20 under reliability support; among 39 paired valid decisions, two improve and none deteriorate.The comparison covers 24 specialized scenarios.Source: §C.6 · p. 20

D. Limitations

The empirical findings are bounded by the benchmark’s short, controlled evaluation and use of commercial model APIs. The authors identify longer interactions, stochastic behavior, and endpoint changes as areas where further evaluation could matter.

Sequential horizon length and extended interactions

  • Three-stage scenarios keep the decision process analytically tractable, but do not assess longer interactions involving cumulative reputation, partner drift, or renegotiated responsibilities.The authors identify balancing exploration, learning, and safeguard procurement over extended horizons as future work.Source: D. Limitations · p. 21

Single-trajectory evaluation and deterministic feedback draws

  • Each scenario uses one trajectory with fixed feedback draws, so the evaluation does not capture output variance across repeated trials or sampling temperatures.The authors suggest multi-trajectory rollouts and temperature-sensitivity analyses for future work.Source: D. Limitations · p. 21

Commercial API opacity and model evolution

  • Commercial APIs may undergo unannounced backend updates, prompt transformations, and behavioral drift, complicating reproducibility over time.The public release includes frozen records of the workflow, prompts, draws, outputs, and scoring pipeline for independent verification and rescoring.Source: D. Limitations · p. 21
Loading 2610.34107v1…