Source-linked AI summary
Supporting or Replacing Students' Reasoning? Cognitive Offloading in AI-Assisted Proof Learning
Eason Chen, Sophia Judicke, Kayla Beigh, Xinyi Tang, Isabel Wang, Nina Yuan, Zimo Xiao, Chuangji Li, Shizhuo Li, Reed Luttmer, Shreya Singh, Maria Yampolsky, Naman Parikh, Yvonne Zhao, Meiyi Chen, Scarlett Huang, Anishka Mohanty, Gregory Johnson, John Mackey, Jionghao Lin, Ken Koedinger
TL;DR
This study asks whether AI-supported work in proof-based mathematics translates into independent performance. Using staggered access and later usage analyses, it found higher homework scores but no significant exam difference with earlier access, while greater chatbot use was associated with lower subsequent exam scores.
Problem
In proof writing, success with assistance does not necessarily show that students can construct and explain similar work independently.
Source: §1 · p. 1Method
The study randomly assigned students to earlier or later access to a chatbot and homework-feedback interface, then analyzed later self-selected use and exam scores.
Source: §1 · p. 2Results
Students assigned earlier access earned higher homework scores but had no significant advantage on the next exam; greater chatbot use was associated with lower subsequent exam scores.
Source: §6 · p. 10, §7 · p. 13Takeaways & Limitations
The authors recommend against open-ended chatbots as tutors and suggest exploring course-embedded tools that give feedback on students’ own proof attempts.
Source: §7 · p. 13Self-selected use leaves possible unmeasured differences that may contribute to usage–exam associations, and the study assessed one proof-oriented course rather than transfer or delayed retention.
Source: §6.5 · p. 12
Abstract
from arXiv · showhide
We studied 153 students in an undergraduate proof-based mathematics course with staggered-access to an AI support system. Students were randomly assigned to receive an AI chatbot and a homework-feedback interface earlier or later in the course. During the period when only the early-access group had access, their homework scores were 2.56 points higher, but exam performance did not differ significantly between groups, suggesting that improved AI-supported homework performance did not translate into detectable gains on a subsequent exam. After all students received access, every additional 100 messages sent to the AI chatbot was associated with a 3.02-point lower score on the subsequent exam. A mediation analysis showed a significant indirect association linking lower baseline self-efficacy to greater chatbot use and, in turn, lower exam scores: a one-point decrease in self-efficacy corresponded to 0.62 fewer exam points through chatbot use. These findings raise concerns that students may use the chatbot to offload their cognitive reasoning. Use of the homework-feedback interface, which required students to submit their proof attempts before receiving feedback, was not negatively associated with exam performance. We recommend against using a chatbot for AI tutoring and suggest exploring a tutor embedded in the course activities.
1. Introduction
AI assistance may improve students’ coursework without ensuring they can reason independently, a distinction especially consequential in proof writing. This study examines access, interface use, and subsequent independent exam performance, finding higher homework scores with earlier access but no significant exam difference and lower exam scores associated with greater chatbot use.
1. Introduction
- The chatbot accepts questions without a proof attempt, whereas the feedback interface requires students to submit a written proof before receiving critique.Students could request direct answers despite chatbot instructions to avoid final answers and complete proof steps.Source: §1 · pp. 1–2
- The study randomly assigned students to earlier or later access, then examined usage and Midterm 3 after both groups received access.Students completed the next exam independently, without AI.Source: §1 · p. 2
- The study asks whether earlier access affects homework and independent exam scores, and how student characteristics and interface use relate to later exam performance.It also examines indirect associations between baseline self-efficacy and exam scores through prior performance and subsequent interface use.Source: §1 · p. 2
- Earlier access raised homework scores, but the groups did not differ significantly on the subsequent exam completed without AI.The findings suggest AI may help students complete homework without strengthening independent proof construction.Source: §1 · p. 2
- Students with lower self-efficacy used AI more, while greater chatbot use was associated with lower subsequent exam scores.The mediation analysis linked lower baseline self-efficacy to greater chatbot use and, in turn, lower exam scores.Source: §1 · p. 2
2. Related Work
The related work frames AI support as a trade-off between assistance and learners’ own reasoning, and motivates interpreting usage in course context. It also highlights dialogue analysis as a way to understand how students use chatbots and how use relates to learning outcomes.
2.1 Learning Analytics for Instructional Decision-Making
- Learning analytics should connect activity measures to theory, course context, instructional decisions, and evaluation of those decisions.For proof work, repeated requests can help identify where students need help, but educators should review evidence in context and prioritize support over punishment.Source: §2.1 · pp. 2, 4
2.2 AI Support, Help-Seeking, and Independent Reasoning
- AI support can help with coursework, but excessive assistance may replace cognitive activity needed for learning; feedback on submitted work requires an attempt first.Prior mathematics studies report gains on supported work under different safeguards and interface designs, while only Bastani et al. assessed later performance without AI.Source: §2.2 · p. 4
- Prior studies motivate examining independent proof construction, since reading a plausible proof differs from constructing one for oneself.The related work notes a programming-course study where both AI groups improved exercise scores but neither showed significant pre–post gains in knowledge or code comprehension.Source: §2.2 · p. 4
- Help-seeking varies with learner knowledge, motivation, and system design, so struggling students’ greater help use may contribute to negative usage–outcome associations.The study therefore models usage first, then later performance while accounting for prior achievement and self-efficacy.Source: §2.2 · p. 4
2.3 Chatbot Dialogue Analysis and Learning Outcomes
- Dialogue analysis can distinguish reflective or evaluative chatbot activity from answer-seeking and relate interaction patterns to assignment and AI-free exam outcomes.The study uses dialogue examples to interpret associations among self-efficacy, logged chatbot use, and later exam performance.Source: §2.3 · p. 4
- LLM coding of educational discourse can be unreliable for nuanced constructs, making task-specific validation important.The study distinguishes human-coder agreement from classifier agreement with human consensus.Source: §2.3 · p. 4
3. Course and the AI Support System
The course AI support system offered two distinct ways to seek help: an open-ended chatbot and feedback on a submitted proof. Both used language models, but differed in whether students had to attempt the proof first.
3. Course and the AI Support System
- The homework-feedback interface critiques a submitted proof with overall feedback and comments tied to specific proof passages.Instructors supplied the prompt and a reference solution as context for feedback.Source: §3 · p. 5
- The chatbot accepted math questions without requiring an attempt and allowed repeated exchanges, despite instructions to avoid final answers and complete proof steps.Students could ask for hints or direct answers and press for more explicit help.Source: §3 · p. 5
- Both interfaces called the OpenAI API without fine-tuning; the chatbot used a fixed GPT-4o snapshot, while feedback also offered an o1-preview snapshot.A separate model labeled chatbot sessions after the course; students did not interact with that classifier.Source: §3 · p. 5
- The system recorded timestamps, requests, sessions, and messages; chatbot messages could request different forms of help.Students had to submit a proof before requesting homework feedback.Source: §3 · p. 5
4. Methods
The study combines randomized timing of access with later observational analyses of self-selected interface use and exam performance. It draws on course records, system logs, a baseline survey, and intent labels, using regression and mediation analyses.
4. Methods
- The study conducted three analyses: access timing and performance, later interface use and Midterm 3, and indirect associations involving self-efficacy.Grades came from course records, usage from system logs, self-efficacy from a survey, and intent labels from coders and a classifier.Source: §4 · p. 5
4.1 Participants and Study Design
- Of 308 enrolled students, 155 consented and records were available for 153; individuals were randomly assigned to early or later access.The two interfaces became available to the later-access group after Midterm 2, and the final exam was excluded because data collection ended at Midterm 3 due to budget constraints.Source: §4.1 · p. 6
4.2 AI Use for Homework and Study
- Students could use the chatbot for math questions and were encouraged, but not required, to submit homework drafts for feedback.Course policy prohibited AI during exams but did not restrict external resources outside exams.Source: §4.2 · p. 6
4.3 Measures and Data Sources
- Homework and exam scores measured performance; logs counted chatbot messages and feedback requests between Midterms 2 and 3.The final labeling dataset included 659 sessions from 106 users; two coders labeled 80 sessions with agreement on 73, κ = .780.Source: §4.3 · p. 6
4.4 Resources for Reusing the Analysis
- The authors made analysis code, annotation prompts, model settings, and aggregate results available in a supplementary repository.They also prepared prompts, coding definitions, scripts, and aggregate results for researchers to inspect and adapt to approved conversation datasets.Source: §4.4 · p. 7
4.5 Statistical Analyses
- Access and usage–exam models used HC3 standard errors, while mediation used OLS path estimates and bootstrap confidence intervals.Usage models included self-efficacy and Midterm 2; the chatbot and feedback-request counts jointly predicted Midterm 3 in the focal model.Source: §4.5 · p. 7
5. Results
Earlier access raised homework scores but did not produce a detectable advantage on the next exam; after access expanded, greater chatbot use was associated with lower subsequent scores. The indirect-association analysis and interaction examples further connect lower self-efficacy and delegated reasoning with concerns about how students use AI support.
5.1 RQ1: Homework and Exam Scores with Earlier Access
- 2.56 points higher on homework, on average, for students assigned earlier access during the period when access differed.The estimate adjusted for Homework 1–4 average and assignment.Source: §5.1 · p. 8
- The earlier- and later-access groups did not differ significantly on Midterm 2, which students completed without AI.The adjusted early-access estimate was B = −2.47 (95% CI [−5.60,0.66], p = .122; N = 152).Source: §5.1 · p. 8
5.2 RQ2: Student Characteristics, AI Use, and Exam Scores
- At the same Midterm 2 score, students one self-efficacy point higher had predicted chatbot-message counts 39% lower and feedback-request counts 31% lower.The associations between Midterm 2 scores and predicted use were not statistically significant.Source: §5.2 · p. 8
- Each additional 100 chatbot messages was associated with a 3.02-point lower Midterm 3 score, while homework-feedback requests were not reliably associated with scores.The chatbot association remained negative after accounting for early access and after reducing the influence of very high usage counts.Source: §5.2 · p. 8
- The negative chatbot-use association persisted when models accounted for earlier access or used log(1+x) counts.The log-count coefficient is on a different scale and should not be compared numerically with the per-message estimate.Source: §5.2 · p. 8
5.3 RQ3: Indirect Associations between Self-Efficacy and Exam Scores
- The indirect association from baseline self-efficacy through chatbot use to Midterm 3 was B = .622 exam points per self-efficacy point.Its 95% bootstrap CI was [.121,1.560] and Holm-adjusted p = .021; paths involving homework feedback and the serial route through Midterm 2 and chatbot use had intervals including zero.Source: §5.3 · p. 9, Figure 5 · p. 9
5.4 Illustrative Student–AI Chatbot Interactions
- In another exchange, a student who wanted to understand the solution asked the chatbot to carry out the next step and did not attempt the invited verification.The exchange illustrates how a stated learning goal can coexist with delegating the reasoning needed to reach it.Source: §5.4 · p. 10
- In one example, the chatbot incorrectly called an omission in a student’s counting approach overcounting.Choosing three distinct non-Jack ranks excludes valid hands with repeated remaining ranks; it does not count the same hand more than once.Source: §5.4 · p. 10
6. Discussion
Earlier access improved homework scores without a detected advantage on the next independent exam. During later self-selected use, greater chatbot use was associated with lower subsequent exam scores, while the discussion considers possible explanations, instructional implications, and limits on interpretation.
6.1 Homework Performance and Independent Problem Solving
- Students assigned earlier access earned higher homework scores, but no significant Midterm 2 advantage appeared on the exam completed without AI.The exam used similar problem types but different questions, so the homework result did not establish better independent performance on new proof problems.Source: §6.1 · pp. 10–11
6.2 Interpreting the Association between Chatbot Use and Exam Scores
- Students with lower self-efficacy used both interfaces more often, while greater chatbot use was associated with lower subsequent exam scores.The authors interpret cognitive offloading as a likely explanation, but note that mental effort and use of AI-generated reasoning were not directly measured.Source: §6.2 · p. 11, §6.5 · p. 12
6.3 Student Reasoning in Chatbot Interactions
- The dialogue examples distinguish requests for further solution steps from feedback on a student's own attempt, while showing that inaccurate feedback can misdirect attention.A request's purpose also matters: reviewing an earlier problem differs from obtaining an answer in place of a current attempt.Source: §6.3 · p. 11
6.4 Implications for Teaching and AI Design
- Instructors can review attempted proofs and ask students to explain supplied steps, while systems can prompt targeted reasoning and check the mathematical basis of feedback.The authors recommend building on students' own reasoning rather than supplying steps, and suggest course-embedded tutors that require attempts and explanations.Source: §6.4 · pp. 11–12
6.5 Limitations
- The study covers one proof-oriented course, compares access to the two interfaces offered together, and measures ordinary-course exam performance rather than purpose-built transfer or retention.Later usage associations are observational because students chose whether and how much to use each interface; cognitive offloading was not directly measured.Source: §6.5 · p. 12
- The dialogue labels characterize initial requests, and classifier agreement was evaluated on the same 80 sessions used to develop the prompt.The authors propose studying complete conversations alongside written work and later unaided performance, and validating revised labels with independent human coding in new settings.Source: §6.5 · p. 12, §6.6 · p. 12
6.6 Future Work
- Future work could compare answer-providing chat, hint-only chat, and feedback on submitted proofs while holding other features constant where possible.The proposed randomized study would assess both immediate task performance and later performance on problems completed without AI.Source: §6.6 · p. 12
7. Conclusion
Earlier access was associated with higher homework scores but no significant difference on the next exam completed without AI. The authors recommend against open-ended chatbot tutoring and suggest course-embedded tools that build on students' proof attempts.
7. Conclusion
- Students assigned earlier access earned higher homework scores, but no significant difference appeared on the next exam completed without AI.During later self-selected use, lower self-efficacy was associated with more use of both interfaces, and greater chatbot use with lower subsequent exam scores.Source: §7 · p. 13
- The authors recommend against open-ended chatbot tutoring and suggest course-embedded tools that provide feedback on students' proof attempts.They propose evaluating AI-supported homework by whether students can later construct and explain proofs without assistance.Source: §7 · p. 13
Ethics Statement
The study received institutional review board approval and participation was voluntary.
Ethics Statement
- Participation was voluntary, did not affect course grades, and research use of records was covered by the approved protocol.Students consented to use of course, survey, and AI support system records.Source: Ethics Statement · p. 13
Data Availability Statement
Aggregate results, code, and prompts are available, while student-level records are not publicly included.
Data Availability Statement
- Aggregate results and supporting code and prompts are available; student-level course, interaction, and interview records are excluded, with access requiring institutional and protocol approval.Source: Data Availability Statement · p. 13
AI-Use Disclosure
GPT-5.2 was used as a post hoc study-session classifier, and Hermes Agent assisted with manuscript editing and review; the authors take responsibility for the final manuscript.
AI-Use Disclosure
- GPT-5.2, at temperature 0, classified study sessions post hoc.The Methods reported the model snapshot.Source: AI-Use Disclosure · p. 13
- Hermes Agent, powered by OpenAI GPT-5.6 Sol, assisted with language editing, analysis-code review, and consistency checking.The authors initially drafted the manuscript.Source: AI-Use Disclosure · p. 13
- The authors take full responsibility for the final manuscript.Source: AI-Use Disclosure · p. 13