Source-linked AI summary
The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses
Lydia Manikonda, Mei Si, Sirajam Munira, Oshani Seneviratne, Kristin Bennett
TL;DR
The paper examines how students’ reliance, access, confidence, and course context relate to perceived learning with GenAI in AI-related courses. A survey of students across 12 courses found uneven experiences, with reported benefits and impacts associated with different reliance patterns and access conditions.
Problem
Evidence about GenAI learning experiences often comes from broad populations or separate studies, leaving reliance, evaluation, confidence, access, and course context underexamined together in AI-related courses.
Source: §1 · p. 1Method
A survey of 118 students across 12 AI-related courses examined use patterns, group differences, and associations with perceived learning outcomes.
Source: §1 · p. 1, §3.1 · p. 2Results
Premium-access and multiple-tool users reported greater benefits and engagement, while early reliance and attitudinal change were associated with greater negative impact.
Source: §5 · p. 7Takeaways & Limitations
Perceived GenAI learning experiences are not reliably characterized by usage frequency alone, and broader tool use was associated with support but not greater confidence working independently.
Source: §5 · p. 7The survey was conducted at one R1 university, relied on voluntary responses, and measured 13 factors that may not capture all factors associated with GenAI learning and reliance.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Generative artificial intelligence (GenAI) is changing how students learn, yet the roles of course context, cognitive reliance, evaluation literacy, and early reliance remain underexplored. Using survey responses from 118 students across 12 AI-related courses at our institution, we examined differences in GenAI use and perceived learning experiences. We identified four user clusters: high-use students reporting many benefits, light users reporting less reliance and fewer benefits, and two moderate-use groups reporting different levels of benefit. We also found significant differences between free- and premium-version users, single- and multiple-tool users, and students experiencing different instructor policies. In multivariable regression models, academic benefit was associated with early reliance and academic task support; positive impact was associated with cognitive reliance, academic task support, confidence in GenAI reliability, and instructor policy; and negative impact was associated with early reliance and attitudinal change. The association between early reliance and negative impact became stronger as evaluation literacy increased. Finally, perceptions of GenAI-enhanced learning appear to reflect cognitive, performance, and self-efficacy benefits, while concerns about stress and diminished critical thinking are associated with lower perceived learning benefits. These findings suggest that institutions need better policies to address such inequities so that institutions can enable students to benefit from increasingly capable AI systems.
1 Introduction
The introduction frames GenAI in AI-related courses as both a learning resource and a source of concerns, with access differences potentially shaping students’ experiences. It identifies gaps in how prior research examines reliance, evaluation, confidence, access, and perceived learning together.
1 Introduction
- GenAI can support explanations, programming, reading, brainstorming, and academic work, while also raising concerns about inaccuracy, integrity, overreliance, and reduced engagement.Students’ access also varies between free single-system use and premium or multiple-tool use, which may shape supported activities and perceived benefits.Source: §1 · p. 1
- Prior evidence often comes from broad university populations or tool- and task-specific interventions, and rarely considers reliance, confidence, access, and perceived learning outcomes together.The introduction distinguishes cognitively substantive use from consulting GenAI before independent thought or conventional help-seeking.Source: §1 · p. 1
- The study surveys students in 12 AI- and data-intensive courses and asks how use patterns, reliance, access, tools, and instructor policies relate to perceived learning.It measures cognitive and early reliance separately and also asks which factors are associated with perceived overall learning improvement.Source: §1 · pp. 1–2
2 Related Work
Prior work describes both perceived benefits and risks of student GenAI use, while noting limited evidence jointly examining reliance, evaluation, confidence, and course policy in AI-related courses. The reviewed literature motivates treating reliance patterns and instructional context as distinct dimensions of students’ experiences.
2 Related Work
- The related-work framing identifies a lack of survey analysis jointly examining GenAI use, learning support, cognitive reliance, evaluation practices, and course policy among students in AI-related courses.Source: §2 · p. 2
Student Use and Perceptions of GenAI
- Prior surveys of broad university populations provide limited evidence about how GenAI use, perceived support, and learning outcomes relate within AI-related courses.The reviewed studies report perceived efficiency, assistance, confidence, and work-quality benefits alongside concerns about accuracy, integrity, dependence, and reduced engagement.Source: Student Use and Perceptions of GenAI · p. 2, §2 · p. 2
Confidence, Cognitive Reliance, and Critical Evaluation
- Cognitive reliance and early consultation are distinct patterns, and existing surveys rarely examine them alongside confidence and evaluation practices.Prior research links confidence in GenAI with less self-reported critical-thinking effort, while confidence in one’s own abilities is associated with more critical engagement.Source: Confidence, Cognitive Reliance, and… · p. 2
Instructor Policy and Student Practices
- Course policy is both a formal instructional condition and something students perceive, amid reported uncertainty about acceptable uses and possible instructor misestimates of student practices.Students and faculty may differ in views of GenAI’s use, acceptability, and learning effects.Source: Instructor Policy and Student Practices · p. 2
3 Methodology
The study surveyed students in AI-related courses and used complementary analyses to examine patterns of GenAI use and experience, group differences, adjusted associations, and nonlinear relationships. The methods combine construct-based survey measures with clustering, group comparisons, regressions, and exploratory ensemble analysis.
3.1 Study Design and Data Collection
- The final sample included 118 undergraduate and graduate students from 12 AI-related courses, who completed a 60-item survey organized into eight multi-item constructs and five single-item measures.Responses used a 1–4 scale, with Not Applicable coded as 2.5 and multi-item scores calculated by averaging included items.Source: §3.1 · p. 2
3.2 Analytic Approach
- Four complementary analyses examined use patterns, group differences, adjusted associations with four learning outcomes, and potential nonlinear relationships with perceived learning.Hierarchical agglomerative clustering with Ward’s method used responses to the 60 items; the four-cluster solution was selected by inspecting the dendrogram and heatmap and considering interpretability.Source: §3.2 · p. 2
Item-Level Student Group Comparisons
- Group comparisons covered student, school, course activity, access, number of tools, and perceived instructor-policy categories.Two-group comparisons used Mann–Whitney U tests; comparisons of more than two groups used Kruskal–Wallis tests and, when appropriate, Holm-adjusted Dunn comparisons, with Benjamini–Hochberg correction within each comparison family.Source: Item-Level Student Group Comparisons · p. 3
Multiple Regression
- Separate multiple linear regressions modeled academic benefit, positive impact, negative impact, and course-supported GenAI learning using all predictors simultaneously and all 118 respondents.Variables were standardized; “allowed with limits” was the instructor-policy reference category, HC3 standard errors supported primary two-sided p-values, and course-clustered standard errors were a robustness check.Source: Multiple Regression · p. 3
Ensemble Learning and Feature Rankings
- A gradient-boosting classifier and SHAP values explored nonlinear relationships and ranked feature contributions to predictions of whether GenAI improved learning.This exploratory feature-importance analysis was not used to replace the adjusted associations estimated by regression.Source: Ensemble Learning and Feature Rankings · p. 3
4 Results
Across the student clusters, GenAI use and perceived benefits varied alongside course context, access, and tool use. Regression and exploratory SHAP analyses linked perceived learning to reliance, task support, and whether students experienced GenAI as supporting or interfering with learning.
4.1 Clustering
- Four clusters ranged from high usage/high benefits to less usage/less benefits, with two moderate-use groups reporting different benefit levels; usage and benefits also varied with course context.Confidence completing tasks without GenAI was inversely correlated with usage and benefits; cluster composition also differed in graduate-student share, from 46.4% to 16.6%.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
4.2 Item-Level Differences Across Student Groups
- Significant item-level differences emerged across premium versus free-only access, single- versus multi-tool use, and perceived instructor policy; multi-tool users reported higher responses on nearly all significant items.Premium users reported greater perceived benefits on several items, while policy-related effects varied across groups and did not establish a specific ordering.Source: §4.2 · p. 4, Table 2 · p. 5
Main Effects
- Adjusted models associated academic benefit with early reliance and task support, positive impact with cognitive reliance, task support, GenAI reliability confidence, and policy, and negative impact with early reliance and attitudinal change.The models explained more variance in academic benefit and positive impact than in negative impact or course-supported learning; course-clustered inference did not retain the task-support association with positive impact.Source: Main Effects · pp. 4–6, Table 3 · p. 6
Interaction Effects
- The early-reliance association with negative impact strengthened with evaluation literacy, from b = .025 (p = .861) at lower literacy to b = .688 (p < .001) at higher literacy.At average literacy, the association was b = .357 (p = .004); the interaction remained significant with course-clustered standard errors.Source: Figure 2 · p. 6, Interaction Effects · p. 6
4.4 Impact of Factors on Learning with GenAI
- Perceived learning benefits were higher when students experienced GenAI as augmenting their capabilities and lower when they reported stress or diminished critical thinking.The exploratory SHAP analysis showed that work-quality beliefs were positively correlated with model output; other positively associated experiences included creativity, organizing ideas, confidence, and discovering insights.Source: §4.4 · p. 6, Figure 3 · p. 7
5 Discussion
Students’ reported GenAI impacts were associated with how they relied on the tools, their confidence, and their course context—not simply usage frequency. The authors also caution that the findings come from one university and a limited set of survey measures.
5 Discussion
- Premium-access and multiple-tool users reported greater benefits and engagement, but neither group reported significantly greater negative impact.Single-tool users reported greater confidence completing tasks without GenAI than multi-tool users.Source: §5 · p. 7
- Students with similar usage levels could report different academic benefits, and task frequency alone may not reliably characterize GenAI’s learning impacts.The authors distinguish productive cognitive offloading, such as organizing or summarizing ideas, from prolonged reliance; they say domain knowledge and evaluation literacy are needed to determine what has been offloaded.Source: §5 · p. 7
- Early reliance and attitudinal change were associated with greater negative impact, while cognitive reliance, academic task support, confidence in GenAI reliability, and less restrictive instructor policies were associated with positive impact.The association between early reliance and negative impact strengthened as evaluation literacy increased; academic benefit was associated with early reliance and academic task support.Source: §5 · p. 7
- The survey covered one R1 university, voluntary respondents, and thirteen measures, which may not represent broader student populations or capture all relevant factors.The authors describe the findings as preliminary and propose expanding the survey across courses and institutions.Source: §5 · p. 7
6 Conclusion
The conclusion argues that GenAI’s learning effects depend on reliance type, evaluation literacy, and context, not merely whether students use it. It calls for course policies that clarify appropriate uses, support equitable access, and preserve independent reasoning.
6 Conclusion
- Students with similar use and reliance levels did not necessarily report similar academic benefits or consequences.The conclusion frames student learning experiences as uneven and shaped by reliance, evaluation literacy, impacts, and the contexts of use.Source: §6 · p. 7
- Cognitive reliance was generally associated with more positive academic experiences, while early reliance was associated with both academic benefits and negative effects.Cognitive reliance includes organizing ideas, evaluating work, or solving problems; early reliance means turning to GenAI before independent thought, searching, or asking an instructor.Source: §6 · p. 7
- The authors argue that educators should clarify appropriate GenAI uses, support equitable access, and help students extend rather than replace independent reasoning.They identify when and how reliance can support learning without weakening independent thinking and critical reasoning as the central question.Source: §6 · p. 7
A Student Survey Instrument
The student survey instrument organizes questions into sections and constructs, with Likert response formats for many items. Its questions cover evaluation practices, reliance, perceived impacts, academic benefits, and course experiences.
A Student Survey Instrument
- The appendix identifies the instrument as a student survey and organizes it under section, construct, item, question, and response-format headings.The supplied table caption identifies Table A.1 as the Student Survey Instrument.Source: Table A.1 · p. 9, A Student Survey Instrument · p. 10
- Evaluation-literacy items ask about cross-checking outputs, assessing organization and conceptual accuracy, judging logical coherence, combining ideas, and meeting expectations.These questions appear as Q22_1–Q22_6 and use Likert response formats where specified.Source: A Student Survey Instrument · p. 10
- Impact and course-experience items ask about confidence, stress, fairness concerns, changing attitudes, and whether the course helped students use GenAI effectively.The listed items include positive and negative impacts, attitudinal change, and learning GenAI; responses are Likert-type.Source: A Student Survey Instrument · p. 10
- Academic-benefit and reliance items cover confidence, workload, new ideas, work quality, grades, motivation, and when students consult GenAI.The listed questions include consulting GenAI before traditional search or instructor support and using it before thinking independently.Source: A Student Survey Instrument · p. 10
- Other survey questions address long-term learning, clear course guidelines, critical-thinking concerns, changed interactions with instructors or peers, and overall learning impact.The instrument also includes an open-comment prompt for respondents’ feedback.Source: A Student Survey Instrument · p. 10