PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Once, Reflect Twice: How Feedback Shapes Human-AI Collaboration

Joshua Holstein, Philipp Spitzer, Sebastian Schäfer, Maximilian Förster, Gerhard Satzger

arXiv:2610.19991v1cs.HC

TL;DR

Research on human-AI collaboration has mainly examined reflection during decisions, leaving feedback on the outcomes of users’ own overrides underexplored. This study tests override feedback in a loan-decision experiment and finds that feedback affects reflection and reliance, while unidirectional feedback risks reducing subsequent reflection-in-action and overall team performance does not significantly change.

  • Problem

    Research has mainly studied reflection during decisions, leaving the role of feedback on outcomes of users’ own overrides underexplored.

    Source: §1 · p. 2, §2 · p. 3

  • Method

    A behavioral experiment compared successful, unsuccessful, balanced, and no override-feedback conditions during three rounds of AI-assisted loan decisions.

    Source: §4 · p. 7

  • Results

    Feedback on past overrides induced reflection-on-action, while unidirectional feedback was associated with lower subsequent reflection-in-action and opposing reliance shifts without significant overall team-performance changes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Balanced feedback avoided the reported risks of unidirectional feedback, preserving concurrent and retrospective reflection.

    Source: §7 · p. 20

    Retrospective reflection was measured with error-focused items, which may understate reflection in the successful-override condition.

    Source: §6.4 · p. 19

Abstract

from arXiv · show

As artificial intelligence (AI) increasingly supports human decision-making in high-stakes domains, designing effective human-AI collaboration becomes critical. Successful collaboration requires humans to reflect on decision-making processes and adjust their strategies accordingly. Current research typically investigates interventions, such as explainable AI, that target reflection during the decision-making process (reflection-in-action). Less attention has been given to retrospective examination of decisions (reflection-on-action). This work explores reflection-on-action by studying the impact of feedback on reliance behavior and human-AI team performance. Specifically, we investigate feedback interventions targeting overrides of AI advice through a behavioral study (N=244). We find that such feedback influences reflection-on-action, while unidirectional feedback risks reducing subsequent reflection-in-action. Furthermore, we identify opposing shifts in reliance behavior that leave overall human-AI team performance unchanged but show context-dependent effects. Balanced feedback did not show these risks, preserving both concurrent and retrospective reflection, offering insights for designing for appropriate reliance on AI.

1 Introduction

In high-stakes human–AI decisions, appropriate reliance requires reflection, yet research has focused mainly on reflection during decisions rather than retrospective reflection on outcomes. This study tests whether feedback on past overrides affects reflection, reliance, and team performance.

1 Introduction

  • Appropriate reliance means effectively complementing one’s own judgment with AI advice, avoiding both blind reliance and dismissal of useful advice.The introduction frames reflection as helping people recognize systematic errors in their own and the AI’s decisions and learn from them.Source: §1 · pp. 1–2
  • Reflection-on-action can use decision outcomes unavailable at decision time, but feedback on whether overriding AI advice produced better outcomes had been overlooked.The authors identify this as a gap and ask how such feedback affects human reflection, subsequent reliance, and human-AI team performance.Source: §1 · p. 2
  • A behavioral experiment tested feedback after two of three loan-decision rounds, comparing successful-override, unsuccessful-override, balanced, and no-feedback conditions.The interventions supplied systematic information about outcomes of participants’ past overrides.Source: §1 · p. 2
  • Feedback on past overrides induced reflection-on-action, while unidirectional feedback was associated with lower subsequent reflection-in-action, significantly after unsuccessful overrides.Successful-override feedback was associated with marginally lower adherence to AI advice; unsuccessful-override feedback was associated with more adherence to incorrect AI advice, without significant overall performance changes.Source: §1 · p. 2

2 Foundations and Related Work

Human-AI collaboration research links reflection to reliance calibration, but predominantly studies interventions that operate during decisions. Retrospective feedback on the outcomes of users’ own overrides remains underexplored.

2 Foundations and Related Work

  • Appropriate reliance means following AI when it is correct and overriding it when it is incorrect; overreliance and underreliance describe systematic departures from this pattern.Prior work links inappropriate reliance to mismatches between people’s confidence in themselves and their perception of AI competence.Source: §2 · p. 3
  • Reflection-in-action occurs during decisions, whereas reflection-on-action retrospectively examines completed decisions to adjust approaches for future decisions.Both forms are presented as mechanisms for reflective practice in human-AI collaboration.Source: §2 · p. 3
  • Prior reflection mechanisms include explanations, confidence or uncertainty displays, and cognitive forcing interventions, with effectiveness affected by presentation and individual or task context.The literature also considers deliberative AI, model updates, and confidence-calibration interventions.Source: §2 · p. 3
  • Most prior mechanisms operate alongside AI advice before a decision; existing retrospective studies address confidence calibration or AI updates, not outcomes of users’ own overrides.The authors position feedback on past overrides as an underexplored reflection-on-action mechanism.Source: §2 · p. 3

3 Theoretical Development

The theoretical development treats feedback on past overrides as a way to support retrospective reflection and proposes that feedback type may shape reliance and team performance.

3 Theoretical Development

  • The authors assume feedback on past overrides can support reflection-on-action by revealing whether and why past decisions were successful.The relevant reflective learning concerns when and why people successfully or unsuccessfully override AI advice.Source: §3 · p. 5
  • Hypothesis 1 predicts that feedback revealing when and why AI advice diverged from human decisions will increase reflection-on-action.The argument treats feedback as information for learning from past outcomes.Source: §3 · p. 5
  • Hypothesis 2 predicts greater AI reliance after unsuccessful-override feedback and lower AI reliance after successful-override feedback.The proposed directions reflect learning about missed opportunities versus appropriate skepticism.Source: §3 · p. 5
  • Hypothesis 3 predicts higher team performance under feedback on overrides, with balanced feedback producing the greatest gains.The rationale combines expected benefits from learning about unsuccessful and successful overrides.Source: §3 · pp. 5–6

4 Experimental Design

The study used a mixed-methods behavioral experiment on loan decisions to compare feedback about past overrides with no feedback. Participants made decisions with AI advice over three rounds, with feedback conditions varying which override outcomes they reviewed.

Experimental Design

  • The study recruited 244 participants for a convergent mixed-methods experiment with quantitative priority and open-ended responses used to explain and extend quantitative results.The treatment groups received feedback on successful overrides, unsuccessful overrides, both, or neither.Source: §4 · pp. 6, 8

Task and Dataset

  • Participants evaluated loan applications using the Home Credit Default Risk dataset and eight selected demographic, financial, and credit-history features.Participants received AI recommendations with ranked lists of the three features contributing most to each recommendation, without numerical values.Source: §4 · pp. 6–7

AI System and Instance Selection

  • The AI was a Random Forest classifier trained on a 70% data split, and the experiment sampled 18 decisions per participant from 48 curated instances.The curated set included correct and incorrect approvals and rejections, with overall AI advice accuracy of 62.5%.Source: §4 · pp. 6–7

Treatments

  • The between-subjects experiment compared successful, unsuccessful, and balanced override feedback with a no-feedback control across three rounds.Feedback appeared after rounds one and two and showed prior cases where participants disagreed with AI, along with the outcome; only overrides, not adherence cases, were shown.Source: §4 · p. 7

Study Procedure

  • The three rounds each contained six randomly selected decisions, followed by a post-task questionnaire; the staged feedback was intended to simulate delays before outcomes become available.Participants first received task instructions, completed a cognitive reflection test and baseline efficacy measures, and practiced with an example.Source: §4 · pp. 7–8

Measurement

  • Reliance was measured as the share of decisions adhering to AI advice, while team performance was final-decision accuracy against ground truth.Reflection-in-action was measured through retrospective reports of System 2 thinking, and reflection-on-action through perceived learning from errors; both were assessed after the task.Source: §4 · p. 8

Qualitative Analysis

  • Open-ended responses were analyzed using hybrid codebook thematic analysis, with a stratified subset independently coded and agreement assessed using multi-valued Krippendorff’s alpha.The two authors independently coded 108 responses; remaining responses were coded with LLM assistance and verified by one author.Source: §4 · p. 8

5 Results

Feedback on past overrides shaped reflection and reliance, but overall team-performance differences remained small as context-dependent shifts partially canceled. Balanced feedback increased retrospective reflection while maintaining reflection during decision-making.

5.1.1 Reflection-on-Action

  • ∆= 0.471 for balanced feedback and ∆= 0.468 for unsuccessful feedback in perceived learning from errors, both significant versus control; successful feedback was not significant (p = 0.136).Balanced and unsuccessful feedback each had adjusted means of µ = 4.68, compared with µ = 4.21 for control.Source: Figure 3 · p. 9, §5.1.1 · pp. 9–10

5.1.2 Reflection-in-Action

  • Unsuccessful feedback significantly reduced reflection-in-action (∆= −0.297), successful feedback showed a marginal reduction (∆= −0.262), and balanced feedback showed no significant change.Balanced feedback increased retrospective reflection while preserving reflection-in-action, unlike the unidirectional conditions.Source: Figure 4 · p. 10, §5.1.2 · p. 10

5.2.2 Reliance on AI Advice

  • 27.5% of decisions were overrides (3.3 cases per participant); adjusted adherence was highest after unsuccessful feedback (µ = 0.776) and lowest after successful feedback (µ = 0.656).The ordering was unsuccessful, balanced (µ = 0.764), control (µ = 0.724), then successful feedback; the two unidirectional conditions differed by a medium effect size (d = 0.59).Source: Figure 5 · p. 11, §5.2.2 · pp. 11–12, §5.2.1 · p. 11

5.2.4 Context-Dependent Effects

  • Opposing shifts for correct and incorrect AI advice partially canceled; balanced feedback had the highest adjusted performance (µ = 0.692), marginally above control (µ = 0.613).Unidirectional feedback was associated with overriding correct advice after successful feedback and adhering to incorrect advice after unsuccessful feedback.Source: Figure 6 · p. 12, §5.2.3 · p. 12, §5.2.4 · p. 13

5.2.3 Human-AI Team Performance

  • Adjusted team performance was highest with balanced feedback (µ = 0.692) and lowest with unsuccessful feedback (µ = 0.588), but tests found no support for H3.The treatment main effect was marginal, and none of the treatment-versus-control contrasts reached statistical significance.Source: Figure 6 · p. 12, §5.2.3 · p. 12

5.2.5 Mechanisms Underlying the Reliance Shifts

  • Changes in perceived AI efficacy were consistent with increased reliance after unsuccessful feedback, while successful feedback reduced reliance despite a mediator path that pointed toward increased reliance.No condition shifted perceived self-efficacy, and the mediation models do not establish that belief changes preceded reliance outcomes.Source: §5.2.5 · pp. 13–14

5.3 Qualitative Findings: Decision Cues and Effects

  • Open-ended responses suggest that participants engaged more with feedback showing their own errors and described reflection extending to understanding the AI and revising decision strategies.Participants reported engaging least with individual instances after successful overrides and most after unsuccessful overrides.Source: §5.3 · pp. 14–15

5.4 Integrating Quantitative and Qualitative Findings

  • Quantitative and qualitative findings agreed on reflection and behavior and outcomes, but diverged on perceptions.Source: §5.4 · p. 15

6 Discussion

The discussion finds that feedback on AI overrides can prompt retrospective reflection, but unidirectional feedback may narrow later reflection and shift reliance without improving overall team performance. It highlights biased feedback samples, measurement constraints, and the bounded study setting as important qualifications.

6.1 Feedback on Past Overrides Raises Reflection-on-Action but Can Narrow Reflection-in-Action

  • Feedback on past overrides induced reflection-on-action, while unidirectional feedback was associated with lower subsequent reflection-in-action, significantly after unsuccessful overrides.Balanced feedback may preserve the psychological tension that supports sustained reflection; participants also described learning about the AI, task, and decision strategy.Source: §6 · p. 15, §6.1 · pp. 16–17

6.2 Unidirectional Feedback Shifts Reliance Through Different Mechanisms

  • Adherence was marginally lower after successful-override feedback (p = 0.064) and descriptively higher after unsuccessful-override feedback (p = 0.118); the conditions differed by a medium effect (d = 0.59).The behavioral shifts were opposite, but their mechanisms were asymmetric.Source: §6.2 · p. 17
  • Unsuccessful-override feedback increased adherence through perceived AI efficacy, but participants updated from a biased sample in which the AI had been right.Successful-override feedback reduced adherence despite a belief-mediated path pushing adherence upward; the direct path remains unexplained, and case-level measures are needed to test local revisions of the AI’s error boundary.Source: §6.2 · pp. 17–18

6.3 More Reflection Does Not Mean Better Reliance Decisions

  • Overall team performance did not improve; context-dependent effects included lower performance in specified advice-correctness conditions, while balanced feedback had the highest adjusted performance, marginally above control (p = 0.068).Opposing effects may mask each other globally; the authors caution that increased reflection-on-action alone may not improve reliance decisions.Source: §6.3 · p. 18
  • Participants often reported lessons about the task rather than strategies for when to follow or override AI advice, and some did not attribute feedback’s influence to it.The authors suggest that feedback could make the reliance decision itself its object and recommend reporting effects separately by AI-advice correctness.Source: §6.3 · pp. 18–19

6.4 Limitations and Future Research

  • Reflection-in-action was measured retrospectively, while the error-focused reflection-on-action measure may understate learning from successful overrides.Process-tracing could test concurrent processing directly; a measure covering both valences could distinguish engagement from an item-wording artifact.Source: §6.4 · p. 19
  • Feedback volume depended on participants’ override rates, so the study could not vary feedback amount independently of the behavior being studied.More feedback did not predict more reported learning from errors, but the design cannot isolate amount as a cause.Source: §6.4 · p. 19
  • Findings are bounded by a 62.5%-accurate AI, novice participants, brief collaboration, SHAP feature-importance explanations, and a simplified loan task.The authors call for field studies with domain experts and longitudinal studies to examine real-world and longer-term collaboration.Source: §6.4 · pp. 19–20

7 Conclusion

The study examines feedback on past overrides as a way to support reflection-on-action in human-AI collaboration. Its findings show that feedback selection matters and that balanced feedback avoided the risks observed with unidirectional feedback in this study.

7 Conclusion

  • Feedback that raises reflection after a decision can lower reflection during subsequent decisions.The conclusion describes the two forms of reflection as interacting.Source: §7 · p. 20
  • Feedback restricted to one type of override can miscalibrate reliance, while balanced feedback avoided these pitfalls in this study.The authors identify designing feedback that makes the reliance decision itself the object of reflection as future work.Source: §7 · p. 20
  • Participants drew lessons mostly about the task and rarely about when to rely on the AI, so more reflection did not by itself produce better reliance decisions.Source: §7 · p. 20

Generative AI Disclosure Use

The disclosure describes generative AI tools used for visualization code, participant-facing wording, qualitative coding, and manuscript revision. Authors reviewed and verified the tools’ output and take responsibility for the article.

Generative AI Disclosure Use

A Statistical Reporting

The appendix tables report ANCOVA results for learning from errors, System 2 thinking, AI reliance, and human-AI team performance. Their notes specify sample size, tests, covariates, and adjustment conventions.

A Statistical Reporting

  • N = 244; learning-from-errors ANCOVA had an overall treatment effect of F(3,237) = 2.85, p = 0.038.Table 5 reports ANCOVA results for learning from errors (reflection-on-action); its note specifies one-sided Sidak-adjusted tests for treatment increases.Source: A Statistical Reporting · p. 26, Table 5 · p. 26
  • F(3,237) = 3.39, p = 0.019; the System 2 thinking ANCOVA note lists cognitive reflection degree, AI efficacy, and self-efficacy as covariates.Table 6 reports ANCOVA results for System 2 thinking (reflection-in-action).Source: A Statistical Reporting · p. 26, Table 6 · p. 26
  • F(3,234) = 4.09, p = 0.007; the AI-reliance ANCOVA note includes baseline reliance and feedback exposure among its covariates.Table 7 reports ANCOVA results for AI reliance in Round 3.Source: A Statistical Reporting · p. 27, Table 7 · p. 27
  • F(3,234) = 2.15, p = 0.095; the team-performance ANCOVA note includes baseline performance and feedback exposure among its covariates.Table 8 reports ANCOVA results for human-AI team performance in Round 3.Source: A Statistical Reporting · p. 27, Table 8 · p. 27

B Validation of Reflection Measures

The appendix examines whether the self-reported reflection measures are distinct and relate differently to objective anchors. Its findings support a treatment-dependent association between the measures and different patterns of association with decision-page time and cognitive reflection.

B Validation of Reflection Measures

  • The appendix evaluates whether perceived learning from errors and System 2 thinking are empirically distinct and independently anchored beyond self-report.Source: B Validation of Reflection Measures · p. 28
  • In control, the association between System 2 thinking and perceived learning from errors is small and nonsignificant; it is substantially larger under unsuccessful feedback.The authors characterize the larger association as treatment-induced, not evidence of baseline construct overlap.Source: B Validation of Reflection Measures · p. 28, Table 9 · p. 28
  • The cognitive reflection test and decision-page time both predict System 2 thinking and are unrelated to perceived learning from errors.Time on feedback pages shows a small, nonsignificant association with perceived learning from errors (r = 0.142, p = 0.055); those timings were available only in feedback conditions.Source: B Validation of Reflection Measures · p. 28, Table 10 · p. 28
  • System 1 thinking does not differ across conditions (F(3, 240) = 0.118, p = 0.949).Source: B Validation of Reflection Measures · p. 28

C Longitudinal Analysis Across Rounds

The appendix examines whether feedback effects on reliance differ across rounds. Successful-feedback effects are already evident in round 2, but the interaction analysis does not establish whether treatment effects change or remain constant across rounds.

C Longitudinal Analysis Across Rounds

  • The unsuccessful-feedback increase in adherence appears only by round 3, while the unsuccessful-condition interaction term only approaches significance (b = 0.459, SE = 0.243, p = 0.058).The model establishes neither that treatment effects are constant across rounds nor that they change.Source: C Longitudinal Analysis Across Rounds · p. 29
  • The Treatment × Round interaction was not significant (χ2(3) = 5.45, p = 0.142); round 2 planned contrasts found lower AI adherence under successful feedback (∆= −0.078, p = 0.021, one-sided, d = 0.42).The unsuccessful and balanced conditions did not differ from control in round 2.Source: C Longitudinal Analysis Across Rounds · p. 29, Table 11 · p. 29
  • The adjusted mixed-effects logistic regression uses rounds 2 and 3, with control and round 2 as reference levels.It includes treatment, round, their interaction, specified covariates, and participant random intercepts.Source: C Longitudinal Analysis Across Rounds · p. 29, Table 11 · p. 29
  • An unadjusted three-round model likewise found no significant Treatment × Round interaction (χ2(6) = 9.88, p = 0.130).The authors report no treatment main effect from either interaction model; every model adjusts for round 1 reliance, which differed across conditions (F(3, 240) = 3.09, p = 0.028).Source: C Longitudinal Analysis Across Rounds · p. 29

D Robustness to Feedback Frequency

Within the three feedback conditions, feedback volume did not predict the four outcomes, and reliance differences between feedback conditions remained when volume was included. In the balanced condition, realized exposure leaned descriptively toward successful overrides, but the skew was not significant.

D Robustness to Feedback Frequency

  • Feedback count predicted none of the four outcomes (all p > 0.11), and reliance differences between feedback conditions were unchanged after adjustment for count.The comparison was among the three feedback conditions (n = 183), excluding control; the models with and without count were comparable to each other, but not to models including control.Source: D Robustness to Feedback Frequency · p. 30, Table 12 · p. 30
  • More feedback was not associated with more reported learning from errors, supporting the authors’ interpretation that reflection-on-action effects relate to receiving feedback, not its amount.For reflection-in-action, feedback-condition differences were small in this subset and weakened further with count.Source: D Robustness to Feedback Frequency · p. 30
  • Balanced-condition participants saw an average of 1.64 (SD = 1.02) successful and 1.28 (SD = 1.31) unsuccessful override cases across both waves; the skew was nonsignificant (t(60) = 1.65, p = 0.103).The condition was balanced by design, not by realized exposure, because the cases participants saw depended on their own override outcomes.Source: D Robustness to Feedback Frequency · p. 30

E Mediation Analyses

The mediation analyses examine whether changes in perceived AI efficacy or self-efficacy relate to reliance after feedback. The appendix also reports the scope and organization of the qualitative coding table.

E Mediation Analyses

  • The appendix describes delta scores as post-task minus pre-task perceived AI and self-efficacy, with each feedback condition contrasted against control (n = 122).The mediator model estimates path a; the outcome model estimates direct path c′ and mediator path b for round 3 reliance.Source: E Mediation Analyses · p. 31
  • Perceived AI efficacy developed more favorably than control across all feedback conditions, while perceived self-efficacy changes were small and not distinguishable from zero.The positive AI-efficacy shift was largest under unsuccessful feedback; the balanced-condition path was not distinguishable from zero (a = 0.046, SE = 0.032).Source: E Mediation Analyses · p. 31
  • Participants whose perceived AI efficacy improved relied on the AI more, and the positive efficacy path combined with a negative direct path under successful feedback to yield competitive mediation.The self-efficacy paths did not yield a significant mediated effect.Source: E Mediation Analyses · p. 31, Table 13 · p. 31, Table 14 · p. 31

F Qualitative Reporting

  • Table 15 reports theme and code prevalence across feedback conditions, with cells marking participants whose open responses carried a theme or code at least once.It documents the codebook, and aggregates the nine task-detail codes.Source: Table 15 · p. 32

G Experimental Design

The appendix includes an overview of questionnaire items and measures.

G Experimental Design

  • The appendix provides an overview of all questionnaire items and measures.Source: Table 16 · p. 33

Related papers

Loading 2610.19991v1…