Source-linked AI summary
Trust Hinders Bidirectional Human-AI Alignment
Michaela Benk, Kacper Sokol
TL;DR
AI-assisted decision making is often framed as calibrating trust, but this framing is limited when decision problems are ill-defined and goals or states are unknowable. The paper develops a broader account of uncertainty strategies, problem characteristics, and higher-order decisions. It concludes that human–AI alignment in these settings involves jointly uncovering, contesting, and negotiating uncertainty.
Problem
Method
Results
Takeaways & Limitations
The paper situates trust calibration within a broader process in which humans and AI tools jointly uncover, contest, and negotiate uncertainties across decision-making levels.
Source: §7 · p. 4, §5 · p. 4The paper critiques trust calibration as a general-purpose paradigm but does not argue that trust is irrelevant; its concern is use beyond scope without checking assumptions.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Trust calibration remains the dominant framing of AI-assisted decision making: the goal is to close the gap between perceived and actual trustworthiness of AI systems so that users rely on them appropriately. We argue that this approach, while useful for decision problems that are well-defined and clearly scoped, overly restricts what it means to make a decision with AI support. To overcome this limitation, we develop three dimensions of AI-assisted decision making. First, trusting AI is only one of several possible strategies for managing uncertainty. Second, trust calibration is often misapplied when AI models large-world decision problems, i.e., ill-defined situations where goals and states are only partially known or unknowable. Third, in such settings human-AI alignment requires addressing higher-order decision processes. We conclude by reflecting on the implications of our observations on the design of process-oriented human-AI coupled systems that jointly identify and negotiate uncertainty throughout the decision-making task.
1 Introduction
The hiking vignette shows that AI-assisted decisions involve uncertainty about both which option to choose and how to decide. The paper argues that trust calibration alone narrows this process and proposes examining uncertainty, problem characteristics, and higher-order decisions.
1 Introduction
- Fog, conflicting reasons, and an approaching storm leave the hiker uncertain not only about which path to take but whether to wait, seek advice, or turn back.An AI recommendation for the shorter path is correct, yet following it leads to injury in the storm.Source: §1 · p. 1
- The vignette asks whether decision quality depends on a trustworthy recommendation, following it, or the resulting injury.The paper argues that quality instead depends on alignment across multiple levels of uncertainty and on choosing an appropriate strategy.Source: §1 · p. 1
- The paper argues that trust-focused assistance hinders bidirectional human–AI alignment and develops dimensions covering uncertainty strategies, decision-problem characteristics, and higher-order processes.Trust calibration is one of several approaches to uncertainty management, useful for well-defined small-world problems but limited in large-world settings.Source: §1 · pp. 1–2
2 Preliminaries and Definitions
The preliminaries distinguish small-world problems, where relevant states and consequences are known or knowable, from large-world uncertainty, where the situation may be ill-defined. They also describe trust calibration and how it is commonly assessed.
2 Preliminaries and Definitions
- The paper builds its three dimensions on the concepts of small- and large-world problems and trust calibration.The section introduces these concepts as foundations for the paper’s framework.Source: §2 · p. 2
2.1 Flavours of Uncertainty in Small- and Large-world Decision Problems
- Risk has known states and consequences, ambiguity has unknown but knowable probabilities, and large-world uncertainty concerns ill-defined situations whose states or consequences may be unknowable.Expected utility maximisation applies only to small-world problems.Source: §2.1 · p. 2
2.2 Origins, Definitions and Measurements of Trust Calibration
- Trust is considered calibrated or warranted when it is caused by system trustworthiness, and studies commonly assess it by manipulating trustworthiness and measuring users’ reliance.Agreement with and following AI predictions have been linked to both trust calibration and appropriate reliance.Source: §2.2 · p. 2
3 Trusting AI Is One of Many Strategies for Managing Uncertainty
Decision making includes both choosing an option and choosing how to make that choice. The paper places trust calibration within this higher-order process, where delegating to AI is one strategy among several.
3 Trusting AI Is One of Many Strategies for Managing Uncertainty
- First-order decisions choose an option, while second-order decisions choose the strategy, rule, or advice source used to make that choice.For example, the hiker chooses both a path and a way of deciding which path to take.Source: §3 · p. 3
- Rules, heuristics, routines, and delegation can reduce the burden of a later choice by restricting what the decision maker considers.These strategies are often chosen before the first-order decision arises.Source: §3 · p. 3
- Deciding whether to act on AI advice is delegation, so trust calibration evaluates whether the delegate merits that choice rather than covering the whole decision process.The paper argues that treating this higher-order concern as the primary decision problem obscures real-world decision processes.Source: §3 · p. 3
4 Trust Calibration Is Misapplied to Large-world Decision Problems
The paper argues that trust calibration’s assumptions fit small-world problems but are often carried into large-world decisions. In those settings, goals, decision frames, and benchmarks may be unsettled or unknowable.
4 Trust Calibration Is Misapplied to Large-world Decision Problems
- Trust calibration has been extended from automation to domains including medicine, law, finance, and language-model-supported decisions without examining whether its assumptions fit those contexts.The paper notes that the suitability of these assumptions has not yet been examined or validated in the new contexts.Source: §4 · p. 3
- Trust calibration requires an oracle to judge output correctness, but large-world decisions often lack knowable ground truth for questions such as which path is fastest or safest.In the vignette, the shortest path is knowable, while these other questions are ill-specified or unanswerable.Source: §4 · p. 3
- An AI may change a user’s beliefs, preferences, or goals, shifting the benchmark for trustworthiness as preferences emerge during decision making.Empirical studies tend to circumvent this issue by imposing artificial stability.Source: §4 · p. 3
- Trust calibration assumes a fixed decision frame, reducing the process to whether to trust and potentially masking misalignment in goals, strategies, or alternatives.A trustworthy recommendation followed by injury does not reveal whether the choices and strategies considered were appropriate.Source: §4 · p. 3
- Outcome-based reliance measures obscure the difficulty of structuring a problem and reflecting on its possible states, while studies may reformulate large-world problems as small-world variants.Once a problem’s structure and states are given, the decision itself is nearly trivial.Source: §4 · p. 3
5 Human–AI Alignment in Large-world Settings Requires Addressing Higher-order Decision Processes
The paper proposes a hierarchy of decision processes to locate where AI assistance can contribute beyond choosing whether to accept a recommendation. This broader view makes room for humans and AI to jointly structure decisions, select strategies, and address uncertainty.
5 Human–AI Alignment in Large-world Settings Requires Addressing Higher-order Decision Processes
- Delegating to a decision support tool presupposes that the problem and AI assistance's role are already fixed, making meaningful interaction redundant.The proposed hierarchy distinguishes levels of assistance to address this concern.Source: §5 · p. 4
- The hierarchy distinguishes first-order choice, second-order strategy selection, and third-order decision structure.These levels ask what to choose, which strategy is appropriate, and which goals, alternatives, and criteria matter.Source: §5 · p. 4
- When a decision support tool offers one output for acceptance or rejection, the AI makes the choice and the human supervises it.This interaction follows the classical supervisory-control paradigm.Source: §5 · p. 4
- AI can also help structure decisions, identify appropriate strategies, and assess whether delegation is suitable.Locating uncertainty at different levels opens opportunities for humans and AI to negotiate its types and objects.Source: §5 · p. 4
6 Limitations and Counterarguments
The authors acknowledge that process-oriented evaluation is difficult and limit their critique to using trust calibration as a general-purpose paradigm. They argue that formal trust-calibration approaches can depend on small-world assumptions, while some accounts treat non-utility-theoretic studies as primarily descriptive.
6 Limitations and Counterarguments
- Outcome-based evaluation's psychological pull impedes a shift toward judging decision quality by process.The authors note that calling a choice successful after an injury can seem absurd even if the process was sound.Source: §6 · p. 4
- Guo et al.'s formalization assumes known utilities, Bayesian updating, and an optimal benchmark, which the authors identify as small-world assumptions [7].They argue the framework works when the decision problem is already settled.Source: §6 · p. 4
- Although Hullman et al. find AI-assisted decision problems are often underspecified, their proposed rigor criterion anchors evaluation to utility theory [8].The authors argue this risks excluding settings of particular interest to their account.Source: §6 · p. 4
- The critique targets trust calibration's use as a general-purpose paradigm, not the relevance of trust itself.The authors argue that calibration is used beyond its scope without checking whether its assumptions hold.Source: §6 · p. 4
7 Conclusion
The paper argues that trust calibration is useful for well-specified, small-world problems but limiting when applied to large-world settings. It situates calibration within a process where humans and AI jointly uncover, contest, and negotiate uncertainty, a concern the authors connect to AI's shift toward critical-thinking and reasoning support [17] [18].
7 Conclusion
- Trust calibration is valuable for well-specified small-world problems but becomes limiting in large-world settings.The paper describes large-world settings as increasingly characteristic of practical AI use.Source: §7 · p. 4
- The authors place calibration within a process where humans and AI jointly uncover, contest, and negotiate decision-shaping uncertainties.They do not treat calibrating human trust in a decision support tool as alignment by itself.Source: §7 · p. 4
- The authors say these considerations are especially important given the shift toward AI for critical thinking and reasoning support [17] [18].Source: §7 · pp. 4–5
A Overview of Second-order Strategies
The appendix table overviews second-order strategies for managing uncertainty, drawing on Sunstein and Ullmann-Margalit. It identifies relying on an AI recommendation as delegating.
A Overview of Second-order Strategies
- Table 1 overviews second-order strategies people use to manage uncertainty.Source: A Overview of Second-order Strategies · p. 7, Table 1 · p. 7
- The table treats relying on an AI recommendation as delegating [19].Source: Table 1 · p. 7