Source-linked AI summary
From Observation to Intervention: Identifiability of Latent User Bias in Human-AI Interaction
Lucas Biechy, Yuxiao Li, Zhonghao He, Tianyi Qiu
TL;DR
The paper asks when observed behavior can reveal a user’s preferences if cognitive bias distorts experienced evidence. It models the interaction as a bandit and analyzes passive observation and intervention. It shows that passive action traces can be observationally equivalent across biased and unbiased users, while interventions can break that limitation under the paper’s controlled assumptions.
Problem
Method
Results
Takeaways & Limitations
Within the model, active interventions can turn otherwise uninformative action traces into signal, but the results are not a deployment-ready test of human cognition.
Source: §7 · p. 9The confirmation-trap result is conditional on the theorem’s specified prior and competing-arm assumptions.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Alignment methods typically treat user behavior as a reliable proxy for underlying preferences. However, this assumption breaks down when users hold belief-dependent cognitive biases, risking irreversible ``confirmation traps'' and harmful feedback loops. In this paper, we formalize when an AI system can distinguish true user preferences from behavior shaped by cognitive bias. Modeling this interaction as a multi-armed bandit, we first prove a non-identifiability theorem: under passive observation, a static binary biased user produces action trajectory distributions strictly identical to an unbiased user in a transformed environment, forcing any action-only estimator to perform no better than random guessing. We then show that active preference elicitation breaks this observational equivalence. Specifically, we prove that an Observer applying a finite sequence of adversarial zero-reward interventions of length $m_\delta$ makes the post-intervention action distributions arbitrarily separable. Our theoretical results suggest that robust alignment requires actively intervening to tell informed preferences from biases.
1 Introduction
The paper asks whether behavior reveals user preferences when cognitive bias distorts experienced evidence, and shows that passive observation can fail through both persistent regret and non-identifiability. It argues that interventions can restore identifiability and presents this as a formal, controlled result rather than a deployment-ready test of human cognition.
1 Introduction
- The paper studies whether action trajectories alone can distinguish a cognitively biased user from an unbiased user, since actions may reflect a permanent, belief-dependent filter.Prior work often treats demonstrator suboptimality as transient and diminishing with more observation.Source: §1 · p. 1
- Belief-dependent confirmation bias can trap Thompson Sampling on a suboptimal arm with positive probability, turning logarithmic regret into linear regret.The result is a worst-case guarantee under the specified distortion model.Source: §1 · pp. 1–2
- Biased and unbiased users in transformed environments can generate identical action trajectories, leaving action-only estimators no better than random choice.The equivalence persists at every horizon under static channel bias.Source: §1 · p. 2
- A finite, computable block of zero-reward interventions can separate user types, including under protocols that do not override spontaneous choices.The intervention budget is denoted mδ.Source: §1 · p. 2
- A formal setup decouples Bayesian updating from cognitive distortion to isolate cognitive bias from reasoning failure.This is presented as one of the paper’s contributions.Source: §1 · p. 2
- The paper reports simulations of remediation protocols and LLM experiments probing whether Qwen3-4B exhibits the theoretical results.The simulations cover coercive and non-coercive intervention setups.Source: §1 · p. 2
2 Related Work
The related work distinguishes persistent belief-dependent cognitive bias from transient demonstrator suboptimality and static distortions that can still permit sublinear regret. It situates the paper’s action-only non-identifiability result alongside work on equivalence classes and active intervention.
Behavioral proxies for preference and learning from imperfect demonstrators
- The paper contrasts preference-learning pipelines and scalable oversight with approaches that treat rating bias as a data-quality issue or demonstrator suboptimality as temporary.It asks the prior question of whether the bias model is identifiable from behavior at all.Source: §2 · pp. 2–3
- Prior preference inference often studies learning or exploring agents, where demonstrator suboptimality is treated as transient and fades with more observation.The paper instead asks whether a bias model itself is identifiable from behavior.Source: §2 · p. 3
Confirmation bias and sycophancy
- Confirmation bias can asymmetrically favor positive feedback, while sycophantic models may validate initial assumptions and prevent disconfirming feedback.The cited work connects that dynamic to self-reinforcing confirmation traps [37, 38, 39].Source: §2 · p. 3
Bandit algorithms and distorted perceptions
Non-identifiability, active intervention, and behavioral feedback loops
3 Problem Setup
The setup models a multi-armed bandit user whose latent type determines whether rewards are processed objectively or through perceptual distortion. The observer is restricted to action histories and its own interventions, while the paper considers belief-dependent confirmation bias and a memoryless static-bias baseline.
MAB Environment & Perceptual Distortion Framework
- The model is a K-armed bandit with unknown arm means and a latent user type that determines whether ground-truth rewards or subjectively corrupted rewards are observed.The objective type processes ground-truth signals directly; the biased type’s distortion kernel depends on prior expectations.Source: §3 · p. 3
User Policy & Observer Agent (Aobs)
- The observer evaluates action likelihoods by treating subjective signals as objective, and the setup defines the selected arm’s prior expected return as a belief-weighted mean.The expected return is E˜bt−1[µat].Source: §3 · p. 3
Behaviorally Grounded Bias Instantiations
- The belief-dependent bias framework includes attenuation of negative prediction errors and persistent suppression of negative outcomes near certainty.The latter is described for Bernoulli outcomes and distortion parameters C > 0 and p > 1.Source: §3 · p. 4
- The static binary bias baseline is memoryless and independent of prior beliefs, with u and v specifying perceived-positive probabilities after positive and negative outcomes.The parameterization is given as Pbias(˜r = 1 | r = 1) = u and Pbias(˜r = 1 | r = 0) = v.Source: §3 · p. 4
User Policy & Observer Agent (Aobs)
- The user selects actions through Thompson Sampling, while the observer sees action history and its own interventions rather than ground-truth or subjective rewards.The setup notes that directly feeding subjective feedback to the observer risks sycophantic reinforcement.Source: §3 · p. 4
4 Theoretical Analysis: Passive Limitations
The section shows that belief-dependent cognitive bias can trap Thompson Sampling in a suboptimal action, while static bias can make a biased user indistinguishable from an unbiased user in a transformed environment.
Scope
- The analysis does not fully characterize the asymptotic separability of dynamic bias kernels.The non-identifiability result applies to static channels.Source: §4 · p. 4
Confirmation Trap
- Under negative prediction-error attenuation and persistent confirmation, Thompson Sampling can lock onto a strictly suboptimal arm with positive probability and incur linear regret.Without distortion, Thompson Sampling with the same priors achieves classical logarithmic regret [20].Source: §4 · pp. 4–5
Action-Only Non-Identifiability
- A static binary bias can produce action trajectories identical to those of an unbiased agent in a transformed environment, for every horizon.The models share a prior and Thompson Sampling policy, so passive action observation cannot distinguish them.Source: §4 · p. 5
- No action-only estimator can beat random guessing when the biased and unbiased models are observationally equivalent.Passive observation also leaves posterior odds unchanged.Source: §4 · p. 5
5 Active Interventions and Identification
The section uses zero-reward interventions to separate biased from unbiased users after passive observation fails. A finite intervention block yields a decision rule with bounded error and a computable horizon.
Intervention Mechanism
- Forcing objective failures makes an unbiased user learn the failure rate and abandon the arm, while a biased user filters disconfirming signals and continues playing it.The intervention creates the behavior divergence used for identification.Source: §5 · p. 5
Adversarial Zero-Reward Interventions
- A controlled zero-reward block forces m consecutive selections of arm k with interventions at each round.Forcing an objective failure incurs cost c > 0.Source: §5 · p. 6
Active Bias Identification
- A finite block of zero-reward interventions makes post-intervention action distributions separable, allowing a single-action rule to identify bias with error at most δ at cost c mδ.The theorem requires the stated bound on the initial state and a block length m ≥ mδ.Source: §5 · p. 6
Computable Intervention Horizon
- The sufficient intervention horizon mδ is computable in closed form from tail bounds and thresholds determined by the observer’s prior.The construction chooses m so both L0 and L1 are at most ξ.Source: §5 · p. 6
Non-Coercive Interventions
- In the non-coercive extension, the Observer intervenes only after spontaneous arm selection; the unbiased user abandons the zeroing arm while the biased user continues selecting it.Source: §5 · p. 6
6 Experiments
Across Thompson-sampling Bernoulli bandits, experiments test persistent regret under bias and whether passive observation or interventions identify user type. Passive histories stay at chance, while active policies separate types within dozens of rounds.
6.1 Experimental Setup
- Experiments use K-armed Bernoulli bandits with independent Beta priors under Thompson Sampling, testing boundary-tight distortion and static binary bias.The simulations compare coercive zero-reward blocks with a non-coercive identification policy; theoretical quantities are evaluated by exact numerical integration.Source: §6 · p. 6, §6.1 · p. 7
6.2 The Confirmation Trap Is Real and Suboptimal
- By T=1,000, biased Thompson Sampling incurs 17× its unbiased twin’s regret; its regret rate plateaus near 0.3, above the 0.129 theoretical lower bound.The lower bound counts trajectories trapped on one specified suboptimal arm, while other suboptimal lock-ins also add regret.Source: §6.2 · p. 7, Figure 3 · p. 7
6.3 Actions Alone Cannot Reveal User Bias
- Action-only classifiers achieve 0.500 ± 0.003 accuracy, and total variation remains ≤0.016 across tested configurations, matching the finite-sample noise floor.The action-only observer’s posterior remains at its prior across horizons T ∈ [25, 800].Source: §6.3 · p. 8, Figure 4 · p. 8
6.4 Active Interventions Restore Identifiability
- 16 coercive and 27 non-coercive rounds reach TV ≥0.9, versus m_δ = 260; non-coercive execution reduces expected unbiased-user forced failures by 40%.Within the certified parameter boundary, separation times remain 16–27 rounds; identification also succeeds empirically up to two orders of magnitude beyond theoretical bounds.Source: §6.4 · pp. 8–9, Figure 5 · p. 8
7 Discussion
The discussion identifies passive non-identifiability as a structural failure for alignment from behavior, while emphasizing that the formal results do not yet provide a deployment-ready test of human cognition.
Scope of the formal claims
- The confirmation-trap and exact non-identifiability results use different bias abstractions, so they do not yet provide one unified characterization of confirmation bias.The identification guarantees hold relative to the assumed model class; human choices may also involve time-varying bias, habits, risk preferences, or changing goals.Source: §7 · p. 9
From separability to useful intervention
- Even the empirically effective 16–27 intervention rounds are too disruptive for real interactions, and identifying bias does not itself correct it or improve welfare.Practical protocols should adapt intervention strength to uncertainty, stop when information gain no longer justifies user burden, and measure downstream welfare.Source: §7 · p. 9
Empirical and human validity
- Simulations confirm analytical bounds under set assumptions but do not validate these behavioral dynamics in human users.The paper also reports an LLM detection calibration bottleneck and calls for studies with behavioral scientists to validate kernels and stopping rules across heterogeneous users.Source: §7 · p. 9
Intervention design and ethics
- Non-coercive protocols reduce forced failures but leave consent, transparency, and false-positive harms unresolved; fabricated outcomes should be prohibited.The discussion calls for consented, low-stakes evaluations with verifiable counter-evidence and randomized controls.Source: §7 · p. 9
Population information and future directions
- With unrestricted environment distributions, biased populations can remain observationally equivalent to unbiased populations, despite possible recovery of the environment from batch observations.The discussion proposes adaptive policies that optimize information gained per unit of user burden as a bridge toward deployable elicitation.Source: §7 · p. 9
A.1 Proof of 4.1
The proof shows that static binary bias can create a confirmation trap: subjective evidence increasingly favors one arm while exploration vanishes, yielding strictly positive probability of linear expected pseudo-regret.
A.1 Proof of 4.1
- The proof bounds contradictory perceived failures using the probability of an objective failure and the bias channel’s failure probability.Under Assumption 2, the first term is zero, while the second is bounded by C(1 − mn)^p.Source: §A.1 · p. 13
- On the trap history, the probability of selecting arm k next approaches one as the probability of exploring another arm vanishes.Subjective evidence increasingly favors arm k.Source: §A.1 · p. 15
- A confirmation trap occurs with strictly positive probability and produces linear expected pseudo-regret.On the trap event, the agent selects arm k at every time step.Source: §A.1 · p. 15
A.2 Proof of 4.3
The appendix proves passive action histories cannot distinguish the biased user from an unbiased user in a rescaled environment. It also establishes finite-horizon intervention separation and a condition for reward-only sequential identification.
A.2 Proof of 4.3
- The biased and unbiased models have identical action-history distributions at every horizon, so action-only estimators cannot consistently distinguish them or beat random choice.Observing additional actions cannot update posterior odds between these observationally equivalent models.Source: §A.2 · pp. 16–17
A.3 Proof of 5.2
- The intervention proof uses a tail bound whose polynomial Gamma-function ratio is outweighed by exponential decay in (1 − x)^m.The resulting bound tends to zero as the intervention length grows.Source: §A.3 · p. 18
B Reward-only sequential identification
- Reward-only sequential identification makes action-path distributions arbitrarily well separated when confirming-path escape probabilities are summable.The proof uses a sequential union bound over perceived failures and Thompson Sampling deviations under M1.Source: B Reward-only sequential identification · p. 19
C Additional Experimental Results
The appendix collects supplementary simulations, parameter sweeps, diagnostic tests, and empirical grounding experiments. Figure 6 compares action-selection frequencies for a biased agent and an unbiased agent in a confounded environment.
C Additional Experimental Results
- The appendix includes supplementary simulation figures, parameter sweeps, diagnostic tests, and empirical grounding experiments evaluating Large Language Models.These materials are referenced in the main text and organized in Sections C.1 and C.2.Source: C Additional Experimental Results · p. 19
- Figure 6 compares per-arm selection frequencies for biased M1 in environment µ with unbiased M0 in confounded environment ν.The plotted setup uses (u, v) = (0.75, 0.15), 600 seeds per model, and T = 100.Source: Figure 6 · p. 20
- Figure 6 reports that the two agents’ action marginals overlap across all time steps.The figure includes 95% confidence bands.Source: Figure 6 · p. 20
C.1.1 Extended Non-Identifiability Sweeps and Action Overlay
Extended evaluations report action-trace similarity across environments and methods. Action frequencies closely match over time, while distribution distances and classifiers remain near their sampling-noise and chance levels.
C.1.1 Extended Non-Identifiability Sweeps and Action Overlay
- Across four arms, empirical action frequencies match throughout T = 100, with a maximum roundwise pull-frequency gap of 0.045 (≤2.44 standard errors).Figure 6 visualizes temporal arm-selection probabilities for M1 and M0 under confounded rewards.Source: §C.1.1 · p. 20
- Across six environments, estimated TV distances remain negligible at TV ≤0.0161, near the finite-sample noise level TV0.Table 2 summarizes classification and distribution-distance metrics at T = 500.Source: §C.1.1 · p. 20, Table 2 · p. 20
- Classifier and Bayes-optimal accuracies remain at random chance (0.500 ± 0.005), and no configuration rejects distributional identity after Holm–Bonferroni correction.Two unadjusted permutation tests fall below α = 0.05, but none remain significant after correction for six tests.Source: §C.1.1 · p. 20, Table 2 · p. 20
C.1.2 Individual Belief Trajectories and Sensitivity Analysis
Across simulations and Qwen3-4B experiments, active interventions reveal bias-related signals that passive action histories do not. The appendix also examines belief trajectories, intervention costs, and the scope of these identification results.
C.1.2 Individual Belief Trajectories and Sensitivity Analysis
- 1,824 of 4,000 runs enter the confirmation trap; their subjective expectations approach certainty despite the diagnostic arm’s true mean of 0.5.Of these runs, 1,643 satisfy event E_k; agents that leave the arm update downward before leaving and then retain estimates below 0.5.Source: §C.1.2 · p. 21, Figure 7 · p. 21
- Within the certified distortion region, empirical separation takes about 27 rounds non-coercively and 17 rounds under coercive blocks; empirical performance extends beyond the analytical boundary but degrades past a threshold.The mδ bound is conservative: both execution modes reach target error rates in fewer rounds, while larger C raises truncation thresholds below which arbitrary identification confidence cannot be guaranteed.Source: §C.1.2 · p. 21, Figure 8 · p. 22
C.2 Grounding in Real Language Models
- The neural grounding experiments use Qwen3-4B on textual interaction histories, while the theoretical guarantees assume an optimal observer estimating marginal action likelihoods by closed-form Beta updating.Observer evaluation uses reasoning features, a 4,096-token budget, and temperature zero.Source: §C.2 · p. 22
C.2.1 Active Identification Budget and Slack Sweeps
- At δ = 0.1, non-coercive testing averages 9.6 forced failures versus 16.1 for coercive execution, saving 6.5 per unbiased user.At δ = 0.5, the corresponding averages are 4.8 and 5.2; non-coercive identification is not reached within the 600-round budget at C = 4.0.Source: Table 3 · p. 22, §C.2.1 · p. 23
C.2.2 Does a Real LLM Face the Passive Identification Wall?
- 0.488–0.507 balanced accuracy across horizons T ∈ [25, 400] leaves Qwen3-4B near random chance when classifying users from passive action histories.The reported performance mirrors logistic regression and the theoretical non-identifiability result.Source: §C.2.2 · p. 23, Figure 9 · p. 23
C.2.3 Active Identification and Cognitive Bias in LLMs
- 0.92 AUROC under coercive interventions at T = 50, versus 0.77 under non-coercive execution; hard-classification accuracy plateaus at 0.68 and 0.55, respectively, at m = 256.Score separation reaches 0.60; the results indicate that continuous confidence scores carry more discriminative signal than hard classifications.Source: §C.2.3 · p. 23, Figure 10 · p. 24
- Under objective failures, Qwen3-4B’s success reporting rises with prior confidence, reaching 35% at ˜µ = 0.83; under positive outcomes, reporting integrity remains 100%.Induced pre-outcome confidence tracks target priors at Pearson r > 0.99, and the fitted distortion parameters are ˆγ = 0.15, ˆC = 36, and ˆp = 1.17.Source: §C.2.3 · pp. 23–24, Figure 11 · p. 24