Source-linked AI summary
Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
Krzysztof Z. Gajos, Lena Mamykina
TL;DR
The paper asks whether people engage deeply enough with AI decision support to learn incidentally. Across three nutrition-decision experiments, it compares recommendation-and-explanation designs with explanation-only support. AI assistance improved immediate decisions across designs, but learning occurred only when participants received explanations without recommendations and had to decide themselves.
Problem
People may over-rely on AI recommendations, raising the question of whether AI assistance supports the cognitive engagement needed for incidental learning.
Method
Three experiments compared nutrition decisions under AI recommendation-and-explanation, Update, and explanation-only designs against feedback baselines.
Results
All tested AI interaction designs improved immediate decisions, but incidental learning occurred in the explanation-only condition and not in the recommendation-and-explanation or Update conditions.
Takeaways & Limitations
Leaving decisions to users while providing useful explanatory information can promote deeper engagement, higher-quality decisions, and learning.
Takeaways & Limitations
The simulated AI was always correct, and the study used low-risk decisions with participants lacking specialized nutrition expertise.
Abstract
from arXiv · showhide
When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process. However, such incidental learning can only occur when people cognitively engage with the information they receive and process this information thoughtfully. How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning? To answer these questions, we conducted three experiments in which individuals were asked to make nutritional decisions and received simulated AI recommendations and explanations. In the first experiment, we found that when people were presented with both a recommendation and an explanation before making their choice, they made better decisions than they did when they received no such help, but they did not learn. In the second experiment, participants first made their own choice, and only then saw a recommendation and an explanation from AI; this condition also resulted in improved decisions, but no learning. However, in our third experiment, participants were presented with just an AI explanation but no recommendation and had to arrive at their own decision. This condition led to both more accurate decisions and learning gains. We hypothesize that learning gains in this condition were due to deeper engagement with explanations needed to arrive at the decisions. This work provides some of the most direct evidence to date that it may not be sufficient to include explanations together with AI-generated recommendation to ensure that people engage carefully with the AI-provided information. This work also presents one technique that enables incidental learning and, by implication, can help people process AI recommendations and explanations more carefully.
1 INTRODUCTION
The paper examines whether people cognitively engage with AI assistance deeply enough to learn incidentally. Across three experiments, it tests interaction designs intended to improve decisions while supporting learning.
- AI assistance can improve immediate decisions, but people may process AI recommendations superficially rather than critically.
- The study tests whether presenting recommendations and explanations in different interaction designs affects incidental learning.
- Three nutrition-decision experiments compared AI assistance with minimal-feedback and explanation-feedback baselines.
- Recommendation-plus-explanation improved decisions without increasing learning, and the Update design also failed to produce learning.
- The explanation-only design produced both immediate decision benefits and incidental learning.
- The authors hypothesize that explanation-only support promotes deeper engagement because people must synthesize information to reach their own decisions.
2 RELATED WORK
Prior work links cognitive engagement and deep processing to learning, while AI decision aids can invite superficial reliance. This study adapts those ideas to incidental learning during brief decision tasks.
- People supported by AI often make better decisions, but overreliance may indicate superficial processing of AI recommendations.
- Deep processing involves meaningful analysis and linking information to existing knowledge structures.
- Cognitive engagement can depend on task structure, with more active or constructive activities generally associated with greater learning.
- The study examines learning gains from different types of AI-generated information rather than directly measuring cognitive engagement.
- Incidental learning occurs as a byproduct of activities such as problem solving or advice seeking, but requires deep engagement with information.
- Prior evidence that people learn from AI recommendations and explanations came from chess, where the task itself may encourage cognitive engagement.
- Navigation-aid research likewise shows that interface design can affect both immediate performance and learning about routes and environments.
- When solutions are presented as suggestions, people may process information more superficially than when they must synthesize it themselves.
3 EXPERIMENT 1: AI PROVIDES RECOMMENDATIONS AND EXPLANATIONS
Experiment 1 tests whether an explainable AI system offering a recommendation and explanation improves immediate nutrition decisions without producing incidental learning.
- Experiment 1 evaluates whether AI recommendations accompanied by explanations improve immediate task performance but fail to produce learning.
3.1 Tasks and Conditions
The experiments used repeated nutrition-concept questions and compared AI-support conditions with two feedback baselines. Contrastive explanations supplied information relevant to choosing between two meals.
- Tasks: Participants compared pairs of meals to identify which contained more of a specified macronutrient, using nutrition concepts such as avocado fat content.
- Tasks: Each nutrition concept appeared in a pre-test, intervention, and post-test sequence.
- Tasks: The pre-test measured existing knowledge, the intervention measured immediate task performance, and the post-test assessed learning from the intervention.
- Tasks: Concepts were randomly assigned to conditions, and question order was randomized across 8 concepts and 24 total questions.
- Conditions: Experiment 1 showed a recommendation and explanation before the participant made a decision, while Experiment 2 showed them after an initial decision and allowed revision.
- Conditions: Experiment 3 showed only an AI explanation, requiring participants to infer the supported decision themselves.
- Conditions: The Minimal feedback baseline provided only correctness feedback after participants answered.
- Conditions: The Explanation feedback baseline added a brief explanation to correctness feedback after each answer.
3.2 Procedures
Participants were recruited through LabintheWild.org and Amazon Mechanical Turk, with procedures tailored to each platform. LabintheWild participants could share the study or explore other studies, while MTurk participants received payment and a verification code.
- Participants were recruited through LabintheWild.org and Amazon Mechanical Turk.LabintheWild used unpaid participation with promised access to individual and aggregate results; MTurk participants were paid $1.
- LabintheWild participants were motivated by promised study results and social comparison opportunities.They could also share the study on social media or explore other LabintheWild studies.
- MTurk participants were paid $1, targeting approximately $10 per hour.The median completion time was six minutes, and participants received a verification code for MTurk.
3.3 Approvals
The experiment series received institutional ethics approval from Harvard University’s Internal Review Board.
- The experiments were reviewed and approved by Harvard University’s Internal Review Board under protocol IRB15-2398.
3.4 Design and Analysis
The study used a within-subjects design comparing three feedback conditions and measured both immediate performance improvement and later learning. Analyses used normalized-change measures, non-parametric significance tests, and outlier checks.
- The within-subjects experiment compared Minimal feedback, Explanation feedback, and AI recommendation and explanation.
- Two dependent measures captured immediate benefit and learning separately for each condition.
- Immediate benefit was normalized change from average pre-test accuracy to average intervention accuracy.
- Learning was measured analogously as normalized change from average pre-test accuracy to average post-test accuracy.
- The study targeted Cohen’s d≥0.2, used Wilcoxon signed-rank tests, and checked whether extreme trial times affected results.The minimum planned sample was 199 participants; effect sizes were computed as r=Z/√n.
3.5 Results
The experiment included 251 participants and found that AI recommendation plus explanation produced greater immediate benefit than both feedback baselines. Explanation feedback, however, produced the largest learning gain.
- 251 people participated in the experiment.Participant demographics were summarized in Table 1.
- M=0.341 immediate benefit for AI recommendation and explanation exceeded Explanation feedback (M=0.158) and Minimal feedback (M=0.167).Both comparisons were statistically significant, with Z=3.55 and Z=3.56, respectively, and r=0.22.
- M=0.325 learning gain for Explanation feedback exceeded AI recommendation and explanation (M=0.206) and Minimal feedback (M=0.189).The comparisons were significant, while AI recommendation and explanation did not differ significantly from Minimal feedback.
4 EXPERIMENT 2: PARTICIPANTS MAKE INITIAL DECISIONS BEFORE SEEING AI RECOMMENDATIONS AND EXPLANATIONS
Experiment 2 tested an Update design in which participants made an initial decision before seeing an AI recommendation and explanation. The design improved immediate task performance but did not produce greater learning than the baseline conditions.
- Motivation: The experiment examined whether making an initial decision before receiving AI assistance would support both improved performance and incidental learning.The Update design was motivated by prior findings of improved accuracy and reduced overreliance compared with showing recommendations and explanations before the decision.
- Design: The Update design showed participants an AI recommendation and explanation only after they made an initial decision, with the option to revise it.Participants first decided independently and then could change their answer in response to the Meal Assistant.
- Immediate benefit: M=0.391 immediate benefit in the Update condition, exceeding Minimal feedback (M=0.110) and Explanation feedback (M=0.178).The improvement was attributed to participants changing their answers; final answers were more correct than initial answers (M=0.129).
- Learning: M=0.121 learning in the Update condition, not significantly different from Minimal feedback (M=0.157).Explanation feedback produced significantly more learning (M=0.354) than both Update and Minimal feedback.
5 EXPERIMENT 3: AI PROVIDES EXPLANATIONS ONLY
Experiment 3 tested whether explanations without explicit recommendations would promote deeper processing during nutrition decisions. This design improved immediate performance and learning, and its learning conclusions replicated in an independent sample, although effects differed by cognitive motivation.
- Design: The AI explanation-only condition provided an explanation without identifying the correct answer, requiring participants to process the information and decide themselves.For example, participants learned that milk is a significant carbohydrate source but had to use that information to select an answer.
- Immediate benefit: M=0.422 immediate benefit in the AI explanation-only condition, exceeding Minimal feedback (M=0.158) and Explanation feedback (M=0.144).The experiment hypothesized that providing necessary information without an explicit recommendation would support both task performance and incidental learning.
- Learning: M=0.342 learning in the AI explanation-only condition, exceeding Minimal feedback (M=0.138).Learning did not significantly differ between AI explanation only (M=0.342) and Explanation feedback (M=0.320).
- Replication: All significant differences in the initial experiment remained significant with comparable effect sizes in the independent replication.The initial experiment included 221 participants and the replication included 270 participants.
- Intervention-generated inequalities: In the AI explanation-only condition, High NFC participants learned more (M=0.388) than Low NFC participants (M=0.255).No statistically significant learning differences between NFC groups appeared in either baseline condition, indicating that cognitive motivation moderated learning in the explanation-only condition.
6 DISCUSSION AND CONCLUSION
The discussion argues that AI assistance can improve immediate decisions without producing incidental learning unless the interaction design requires people to construct decisions from explanations. It also identifies moderators, assumptions, and practical trade-offs that constrain how broadly these findings should be applied.
- All tested human-AI interaction designs provided significant immediate benefits, helping participants make better nutrition decisions than without AI assistance.
- Incidental learning occurred with AI explanations alone, but not when participants received both an AI recommendation and explanation.The explanation-only design required participants to infer which decision the explanation supported, whereas recommendations allowed them to act without deeply processing the explanation.
- The authors attribute the learning difference to cognitive engagement: arriving at a decision required synthesizing the explanation, while receiving a solution did not.
- The findings are bounded by the use of an always-correct simulated AI, low-risk nutrition decisions, nonexpert participants, and immediate rather than long-term learning measures.The authors also note that contrastive explanations were used and that explanation-only support may be less effective with non-contrastive explanations.
- The proposed design emphasizes synthesizing information while leaving the final decision to users, but deeper engagement may trade off against efficiency, usability, and cognitive burden.The authors call for further research on these trade-offs, higher-stakes domains, and uncertainty in AI accuracy.