PaperlayerView exact revision on arXiv

Source-linked AI summary

The Belief Update Gate: Separating Inertia from Learning in Human-AI Interaction

Shreyan Biswas, Alexander Erlei, Ujwal Gadiraju

arXiv:2608.20828v1cs.HC

TL;DR

Pooled belief-updating slopes may obscure whether elicited reports move at all versus how they move when they do. The paper reanalyzes repeated human–AI decisions with a measurement-aware belief update gate, finding substantial visible non-movement and stronger conditional updating, while cautioning that reports do not identify latent learning by themselves.

  • Problem

    Pooled updating analyses do not separately establish whether feedback changes an elicited belief report and how much it changes conditional on movement.

  • Method

    The paper reanalyzes repeated human–AI decision-making data using a belief update gate and complementary hurdle-style analyses separating movement occurrence from conditional update direction and magnitude.

  • Results

    Visible non-movement is frequent, and separating it from moving reports substantially reduces the descriptive appearance of pooled conservative updating.

  • Takeaways & Limitations

    Calibration analyses should distinguish visible non-movement in elicited reports from updating conditional on movement rather than treating reports as one continuous updating process.

  • Takeaways & Limitations

    The reanalysis cannot determine whether flat reports reflect latent non-updating, coarse reporting, satisficing, or insufficient incentive to report small changes.

Abstract

from arXiv · show

Repeated human-AI interaction is often analyzed through pooled belief-updating slopes: users observe AI successes and failures, revise reported beliefs in the feedback-consistent direction, but appear conservative on average. We show that such averages can obscure an important distinction between whether an elicited belief report changes at all and how it changes conditional on movement. We refer to this measurement-aware decomposition as the belief update gate. Reanalyzing a multi-task human-AI decision-making dataset with 240 participants, 7,200 trials, and three task domains, we find substantial non-movement in reported beliefs: 67.3% of trial-level belief changes are exactly zero, and 76.4% are smaller than five percentage points. Separating non-moving from moving reports changes the descriptive interpretation of pooled conservatism: the within-trajectory slope rises from 0.494 overall to 0.949 among rows with nonzero movement. Since this latter estimate conditions on observed movement, we interpret it as a descriptive decomposition rather than as evidence of a near-Bayesian latent learning process. Complementary hurdle style analyses (i.e., modeling zero vs. non-zero changes before predicting update magnitude) show that the absolute discrepancy between feedback and entering belief predicts whether a report changes, while the signed feedback discrepancy predicts the direction and magnitude of change among reports that move. Importantly, observed non-movement does not distinguish genuine latent belief inertia from small unexpressed updates, rounding, or other reporting processes. These findings show that calibration analyses of repeated human--AI interaction should distinguish visible non-movement in elicited belief reports from updating conditional on movement rather than treating reported beliefs as a single continuous updating process.

1 Introduction

Repeated human–AI interaction makes belief reports consequential, but pooled updating slopes can combine unchanged reports with stronger conditional adjustments. This paper introduces the belief update gate to separate visible movement from movement conditional on occurring.

  • Average updating slopes can conceal whether feedback produces any measurable change in reported beliefs.The pooled estimate combines the probability of movement with the magnitude of movement among reports that change.
  • The reanalysis uses 240 participants, 7,200 trial-level observations, and three task domains from a repeated human–AI decision-making experiment.Participants observed AI recommendations, decided whether to rely on them, received correctness feedback, and reported beliefs about future correctness.
  • Separating non-moving from moving reports changes the descriptive interpretation of average conservative updating.The paper characterizes pooled patterns as combinations of visible non-movement and conditional updating rather than uniformly weak adjustment.
  • The belief update gate separates whether an elicited belief report moves from how it changes conditional on movement.The framework applies a two-part perspective to repeated reported-belief dynamics in human–AI interaction.
  • The study also examines whether local evidence, alternative movement thresholds, and trajectory-level differences structure reported-belief movement.Supporting analyses assess threshold robustness, predictor patterns, and heterogeneity across reported-belief trajectories.

2 Background and Literature Review

Research on human–AI interaction increasingly treats reliance and belief formation as dynamic and heterogeneous across repeated encounters. Building on hurdle models of belief adjustment, this paper focuses on where heterogeneity enters the reported belief–reliance process.

  • Human–AI workflows shape whether users notice evidence, interpret it as diagnostic, and express belief change.Instructions, feedback policies, incentives, and workflow structure influence reported belief formation.
  • Repeated human–AI interaction makes reliance dynamic because users update expectations after sequences of recommendations, successes, and failures.Users adapt future-oriented beliefs about when AI is likely to help or mislead them.
  • Prior work links expectation management, confidence displays, limited feedback, interaction history, and model accuracy to trust and reliance patterns.These studies also document heuristics, imperfect calibration, and heterogeneous responses in repeated interaction.
  • The paper asks whether repeated evidence becomes measurable movement in elicited reports, how reports move after movement, and how these dynamics relate to delegation.The focus is the reported-belief layer of the belief–reliance process.
  • Double hurdle models distinguish whether a stated belief changes from the extent of adjustment conditional on change.This framework has related the two components to accumulated evidence, inattention, complexity, and deviations from Bayesian updating.

3 Method

The study reanalyzes repeated belief reports across three AI-assisted tasks and models both benchmark-relative updating and the occurrence of report movement. Its measurement-aware design treats visible non-movement as distinct from latent belief dynamics.

  • The dataset contains 240 participants, 720 participant-task blocks, and 7,200 trial observations across grammar, travel, and visual question answering.Each participant completed 10 trials per task and reported a 0–100 belief after observing AI correctness feedback.
  • The analysis compares observed belief updates with a Beta–Bernoulli Bayesian benchmark using participant-task priors and cumulative AI successes and failures.The canonical benchmark uses the original hybrid S=10 specification, with estimated or fallback effective prior sample sizes.
  • 3.6 Update-Gate Regression Analysis: The study recomputes within-trajectory slopes across nested subsets, including rows with nonzero belief movement.This tests whether pooled conservative updating is associated with visible non-movement.
  • 3.6 Update-Gate Regression Analysis: Hurdle-style regressions first predict whether a belief report crosses a movement threshold and then predict signed movement conditional on crossing it.The first component uses logistic models, while the conditional component uses linear regressions for signed belief movement.
  • 3.7 Predictors of Belief Report Movement: Trial-level movement models examine feedback discrepancy, feedback valence, lagged beliefs, confidence, correctness, task domain, and trial order.Participant-task models additionally test whether inertial trajectories concentrate among particular users, tasks, or starting conditions.
  • Delegation comparisons are descriptive because delegation is treated as downstream reliance behavior rather than a causal explanation for the update gate.Design-matched recovery simulations assess which trajectory-model distinctions are recoverable under the short 10-trial panels.

4 Results

Results show that pooled conservative updating combines substantial visible non-movement with stronger conditional movement among reports that change. Predictor audits and trajectory analyses further reveal local movement correlates and heterogeneous descriptive motifs, while recovery limits definitive model diagnoses.

  • Visible non-movement: 67.3% of belief changes are exactly zero, and 76.4% are smaller than five percentage points across 7,200 trials.This large mass of zero and near-zero reports contributes to the pooled average slope.
  • Visible non-movement: 35.5% of classified participant-task trajectories select no_update as the modal primary-generative model winner.The count is 233 of 656 classified trajectories.
  • Slope decomposition: The within-trajectory slope rises from 0.494 overall to 0.949 among rows with nonzero belief changes.Intermediate restrictions produce slopes of 0.682 after removing no_update trajectories and 0.719 when requiring at least two nonzero updates; the 0.949 estimate conditions on observed movement and is diagnostic rather than causal.
  • Slope decomposition: The pooled conservative-Bayesian estimate combines the frequency of report movement with the magnitude of movement when it occurs.This decomposition motivates separating visible movement from conditional movement.
  • Predictors of movement: An absolute feedback gap predicts whether a report moves, while the signed feedback gap predicts update magnitude among moving reports.For movement, the odds ratio is 3.79; among 2,352 nonzero-movement rows, the signed-gap model has R^2 = .430.
  • Predictors of movement: Trajectory-level audits find mostly null participant-task predictors, but local trial predictors and temporal structure are associated with visible movement.Larger feedback discrepancies increase movement odds, higher lagged belief lowers them, and visible update shares are front-loaded across trials.
  • Trajectory-level model comparison: No_update, good_bad_news, threshold, sticky, and coarse-reporting models capture distinct descriptive trajectory motifs.Design-matched recovery yields BIC self-recovery of 60% for no_update, 48% for threshold_bayes, and 40% for good_bad_news, with 4–16% for remaining families; labels are therefore not definitive diagnoses.

5 Discussion

The discussion reframes pooled conservative updating as a combination of visible non-movement and stronger conditional movement, while emphasizing that elicited reports do not directly identify latent beliefs. It also outlines implications for calibration research and limitations of the descriptive reanalysis.

  • Pooled conservatism combines many non-moving reports with stronger feedback-aligned movement among reports that change.The decomposition cautions against interpreting a low average slope as uniformly weak updating.
  • The belief update gate separates whether a reported belief moves from how strongly it moves conditional on movement.Hurdle-style analyses associate feedback discrepancy with movement occurrence and signed discrepancy with conditional direction and magnitude.
  • Visible non-movement is not reducible to poor performance, low AI literacy, confidence, or a simple lack of reasons to update.Movement is most consistently associated with local evidence structure, including feedback discrepancy, success or failure, and trial position.
  • Calibration failures can arise from no report movement, insufficiently sized movement, or movement that fails to change downstream reliance.These are distinct failure modes that should not be collapsed into one updating slope.
  • The trajectory model labels are descriptive motifs rather than definitive participant types or uniquely identified cognitive mechanisms.Short ten-trial panels and model-selection results limit fine-grained mechanistic interpretation.
  • Observed non-movement cannot distinguish latent inertia from coarse reporting, rounding, thresholds, satisficing, or insufficient incentives.The reanalysis was not designed to identify latent belief dynamics separately from reporting behavior.
  • The analyses are descriptive rather than causal, and belief movement is not jointly modeled with reliance behavior.Future work should manipulate feedback and reporting conditions and directly connect belief reports to delegation choices.

6 Conclusions

The conclusion argues that pooled belief-updating slopes are incomplete because reported beliefs often do not visibly move. It introduces the belief update gate as a framework for separating measurable movement from conditional updating and coarsened reporting.

  • Repeated human–AI calibration analyses should first test whether feedback produces measurable report movement, then estimate movement conditional on change.
  • Flat reports may reflect cognitive inertia, coarse slider use, rounded probabilities, expression thresholds, or other measurement features.Treating elicited beliefs as continuously moving latent states can obscure visible non-movement, conditional learning, and coarsened reporting.

A Summary of Key Results

This section presents the paper’s key-results overview and describes the model-fitting and selection procedures used to compare participant-task trajectories. BIC is emphasized while alternative diagnostics qualify winner-count interpretations.

  • Model Fitting and Selection: Each candidate model is fit separately to each participant-task trajectory using squared-error minimization for grid-based models.Conditional update-rule slopes use closed-form least squares, with changepoint grids where needed and predictions clipped to the valid probability range.
  • A Summary of Key Results: Table 2 summarizes the key results supporting the update-gate account.
  • Model Fitting and Selection: BIC is the primary model-selection criterion because it conservatively penalizes complexity in short panels.Winner counts are interpreted alongside AICc, recovery, absolute-fit, and source-panel sensitivity analyses.

C Original Task Interface

The original experiment used three task interfaces—grammar error detection, travel planning, and visual question answering—with participants completing them in randomized order.

  • Original Task Interface: The three experimental tasks were grammar error detection, travel planning, and visual question answering.
  • Original Task Interface: Participants completed the tasks in a randomized order.

D Model Definitions and Robustness

The appendix defines candidate belief-update models that vary how evidence is weighted, accumulated, discounted, distorted, or translated into reported movement. These models describe observed reports, with conditional update-rule diagnostics explicitly excluded as primary trajectory-generating accounts.

  • Model Definitions: λ = 1 recovers the standard Bayesian benchmark, λ < 1 indicates global evidence underweighting, and λ = 0 removes evidence accumulation.
  • Model Definitions: Candidate models include prediction-error learning, confirmatory misperception, anchoring, sublinear sample-size distortion, discounted Bayes, and divisible weighted Bayes.The slate also includes linear models with partial-step, asymmetric, recency, trial-1 spike, and changepoint specifications.
  • Model Definitions: Anchoring parameter a < 1 produces conservative movement toward the Bayesian posterior, whereas a > 1 permits amplified movement.
  • Model Definitions: When γ = 1, the sublinear sample-size model recovers the standard Bayesian benchmark; γ < 1 makes evidence accumulate sublinearly.
  • Robustness: Conditional update-rule diagnostics describe how observed reports move given the current report but are not treated as primary trajectory-generating accounts.

D.5 Source-Panel Robustness

Source-panel robustness changes the number of classifiable trajectories because prior-strength identification depends on the counterfactual-prior quality rule. Across canonical trajectories with at least two nonzero updates, good_bad_news is the leading AICc winner.

  • D.5 Source-Panel Robustness: The hybrid_S10 source-panel row reproduces the canonical all-candidate counts, while alternative prior-quality rules change the number of classifiable trajectories.The variation arises because n0 identification depends on the counterfactual-prior quality rule.
  • D.5 Source-Panel Robustness: 22.6% of canonical hybrid_S10 trajectories selected good_bad_news as the leading AICc winner, ahead of partial_step at 10.3%.The comparison covers 380 trajectories with at least two nonzero updates.
  • D.5 Source-Panel Robustness: The two-stage diagnostic predicts thresholded report movement first, then signed belief movement among rows that moved, with participant-clustered standard errors.Controls include lagged belief, lagged self-confidence, task domain, trial ordinal, and updater-family variables.

D.8 Absolute-Fit Diagnostics

Absolute-fit diagnostics compare fitted and observed trajectory signatures, including zero-update rates, mean final shifts, and mean absolute updates. The update-gate figure separates predictors of visible movement from predictors of movement among reports that move.

  • D.8 Absolute-Fit Diagnostics: Larger absolute feedback gaps increase the odds of visible report movement, whereas signed feedback gaps predict movement size and direction among moved reports.All annotated effects are significant at p < .001.
  • D.8 Absolute-Fit Diagnostics: The observed zero-update rate was 0.667, while standard_bayes predicted 0.049 and no_update predicted 1.000.
  • D.8 Absolute-Fit Diagnostics: The observed mean final shift was −0.049; changepoint predicted −0.049, asymmetric −0.047, partial_step −0.044, and good_bad_news −0.045.
  • D.8 Absolute-Fit Diagnostics: The observed mean absolute update was 0.040, matched by recency, which predicted 0.040.
  • D.8 Absolute-Fit Diagnostics: The trial-level audit outcome is whether the reported belief visibly moves, complementing the trajectory-level no_update classification outcome.

D.10 Cross-Task Consistency of no_update Classification

No_update classification shows cross-task consistency, but not a simple one-step carryover pattern. The diagnostic evaluates participant-level distributions and current-task classification as a function of no_update classifications in other tasks.

  • D.10 Cross-Task Consistency of no_update Classification: Table 9 summarizes participant-level distributions and current-task no_update rates by the number of no_update classifications in the other two task blocks.
  • D.10 Cross-Task Consistency of no_update Classification: An additional no_update classification in a participant’s other two task blocks predicts current-task no_update classification with odds ratio 4.46.The participant-task logistic model controls for task domain, task position, opening belief, mean confidence, and final user accuracy.
  • D.10 Cross-Task Consistency of no_update Classification: The immediately previous task shows no positive no_update carryover effect, with odds ratio .74 and p = .235.
  • D.10 Cross-Task Consistency of no_update Classification: The appendix concludes that no_update classification has some cross-task consistency but is neither universal nor a simple one-step carryover pattern.
  • D.10 Cross-Task Consistency of no_update Classification: The appendix reports participant-bootstrap confidence intervals for the slope decomposition by resampling participants before recomputing within-participant-task slopes.
Loading 2608.20828v1…