PaperlayerView exact revision on arXiv

Source-linked AI summary

Impacts of generative AI skill development on academic adaptation and learning outcomes among university students

Lei Fan, Fangxue Liu

arXiv:2610.30599v1cs.HCcs.AI

TL;DR

The paper examines how university students perceive GenAI skill development over time and how it relates to learning practices and academic outcomes. It combines a retrospective survey with interviews and finds heterogeneous trajectories, differentiated academic gains, and a positive indirect association through assignment-process adaptation.

  • Problem

    How students’ GenAI skills change beyond initial familiarity, and how perceived development relates to learning practices and academic outcomes, remains an educational question.

    Source: §1 · p. 1

  • Method

    The study combines a retrospective questionnaire with 313 analyzed responses and interviews with 18 students across disciplines to examine perceived development and associated academic experiences.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    34.2% reported rapid early improvement followed by a plateau; assignment-grade gains (74.7%) exceeded examination-grade gains (61.4%), with a positive indirect association through assignment-process adaptation.

    Source: §6 · p. 19

  • Takeaways & Limitations

    Support should address evaluating outputs, retaining responsibility for reasoning, and demonstrating understanding alongside GenAI-assisted performance.

    Source: §6 · p. 19

    The cross-sectional design cannot establish temporal ordering or causality, and the indirect association should not be interpreted as mediation.

    Source: §5.4 · p. 19

Abstract

from arXiv · show

Understanding how university students perceive the development of their generative artificial intelligence (GenAI) skills over time can clarify how GenAI use relates to learning and academic outcomes. This study examined perceived skill-development trajectories, changes in learning practices, and associated academic gains using a retrospective questionnaire with 313 analyzed responses and interviews with 18 students across subject areas. Descriptive and comparative analyses and exploratory structural equation modeling (SEM) were integrated with thematic analysis. Perceived developmental trajectories were heterogeneous: 34.2% of students reported rapid early improvement followed by a plateau, 28.4% steady improvement, and 21.1% delayed acceleration. Development was also uneven across skills, with improvement in evaluating GenAI outputs perceived as more challenging; assignments were the most frequently selected driver of skill growth. Interviews further indicated increasingly specific prompting, tailored verification, and selective delegation, although some students reported stable routines, dependence, skipped reasoning, or improvements attributable partly to better GenAI tools. Academic gains were reported more often for assignments than for examinations. The SEM indicated a positive indirect association between perceived skill growth and academic gains through adaptation of assignment processes, whereas academic gains did not clearly differ across developmental trajectory groups after multiplicity correction. The findings suggest that perceived GenAI skill development varies in both trajectory and skill domain and is associated with changes in learning practices and differentiated academic benefits.

1. Introduction

The study asks how students’ GenAI skills develop beyond initial familiarity, how that development relates to learning practices, and whether it is associated with academic outcomes. It combines retrospective survey and interview evidence to connect perceived trajectories, skill changes, academic practices, and reported gains.

Research motivation

  • GenAI skill development may involve more precise requests, recognizing unreliable answers, and deciding which academic work requires students’ own reasoning.These dimensions need not develop together or at a constant pace.Source: §1 · p. 1
  • Producing better GenAI-assisted work is distinct from learning that students can demonstrate independently.Prior findings include greater essay-score improvement with ChatGPT support but no statistically significant advantage in knowledge gain or transfer, and opposing indirect associations through self-efficacy and thinking offloading.Source: §1 · p. 1

Research gap

  • Existing studies leave perceived skill development insufficiently connected to changes in learning practices and academic outcomes.Repeated-observation studies can offer temporal detail but often have small or narrowly bounded samples; broader surveys have emphasized adoption-related variables.Source: §1 · p. 2

Study contribution

  • The study uses a retrospective questionnaire with 313 analyzed responses and interviews with 18 students across academic disciplines.It examines perceived trajectories, specific skill improvements, assignment-process changes, reported academic gains, and support demands.Source: §1 · p. 2
  • The study integrates perceived GenAI skill development with learning practices and reported academic outcomes, identifying patterns for further investigation through longitudinal research.Its research questions address reported skills and trajectories, associations among growth, process adaptation and gains, and students’ support needs.Source: §1 · p. 2

2. Literature review

Prior work distinguishes GenAI capability from adoption and confidence, examines changes in academic engagement, and shows that educational outcomes depend on the kind of assistance and assessment. It motivates studying perceived development alongside learning practices and academic outcomes.

2.1. Temporal development of GenAI skills

  • Temporal studies measure different outcomes, and changes in use or confidence do not by themselves establish individual capability development.Prior work includes a two-wave study of 222 students’ use and a comparison of successive cohorts; studies following academic activity report changing practices but do not establish common stages or progression rates.Source: §2.1 · pp. 2–3

2.2. Changing learning practices and critical engagement

  • GenAI engagement includes planning, monitoring, reflection, and directing assistance; increased interaction alone does not demonstrate these processes.Authentic dialogues showed agentic engagement in 12% of analysed dialogues, while other accounts describe increasingly specific but unevenly critical interactions.Source: §2.2 · p. 3

2.3. Academic consequences of developing GenAI skills

  • Academic outcomes vary with the task and assistance conditions, so improved GenAI-assisted performance does not necessarily indicate independent learning gains.Prior studies report different outcomes for essay scores, knowledge transfer, purpose-built tutoring, and unaided examinations; a meta-analysis found positive average performance effects with substantial heterogeneity.Source: §2.3 · pp. 3–4

3. Methodology

The study combines a retrospective survey and interviews, then uses descriptive, comparative, measurement, and exploratory structural analyses to examine perceived GenAI skill development and associated academic experiences.

3.1. Study design and data collection

  • The cross-sectional retrospective questionnaire yielded 313 analyzed responses, and 18 students from diverse disciplines also took part in interviews.Convenience sampling was supplemented with targeted recruitment across gender, disciplines, and regions; both study components relied on retrospective self-report rather than repeated competency measurements or verified performance changes.Source: §3.1 · p. 4

3.2. Questionnaire structure

  • The 39-field questionnaire used five-point agreement items, with perceived-growth questions referring retrospectively to change since initial academic GenAI use.Other item blocks assessed process and academic gains, perceived development sources, assessment views, and responsible-use practices.Source: §3.2 · p. 4

3.3. Interview scope and analysis

  • Interviews examined perceived development, output evaluation, support, delegation, dependence, and academic gains; thematic summaries contextualized and qualified survey findings.Interviewees represented a range of academic disciplines.Source: §3.3 · p. 5

3.4. Sample characteristics

  • The sample included undergraduate and postgraduate students, with approximately balanced gender representation; STEM and HSS served as contextual descriptors, not detailed comparison groups.The largest group had used GenAI for academic purposes for 1–2 years.Source: §3.4 · p. 5

3.5 Descriptive and comparative analyses

  • Descriptive analyses summarized responses and composites, while rank correlations, paired Wilcoxon tests, and trajectory-group comparisons examined associations and contrasts.Multiplicity corrections were applied across paired tests, trajectory tests, and contextual correlations; confidence intervals were not multiplicity-adjusted.Source: §3.5 · p. 6

3.6 Measurement assessment

  • Measurement assessment combined reliability, factorability, validity, and model diagnostics, treating indices jointly as preliminary evidence rather than definitive construct validation.Ordinal DWLS candidate models were assessed for fit and admissibility; comparisons within the same sample were descriptive, not confirmatory validation.Source: §3.6 · pp. 6–7

3.7 Structural associations

  • A focused SEM estimated associations among perceived skill growth, assignment-process adaptation, and academic gains, excluding practice-related development because it overlapped substantially with growth.The saturated structural component was interpreted associationally, not as causal ordering or mediation, and contextual indicators were not covariates.Source: §3.7 · p. 7

4. Results

The results support preliminary measurement quality while showing overlap between growth and practice constructs. Students reported frequent GenAI use and stronger perceived efficiency than evaluation, alongside changes in assignment practices and selective use described in interviews.

4.1 Reliability, validity and model selection

  • Composite reliability and factorability indices were above 0.70 and AVE above 0.50, although practice and process AVE only narrowly exceeded 0.50.Across six composites, alpha estimates were 0.712–0.804, blockwise KMO values 0.701–0.796, CR values 0.794–0.863, and AVE values 0.501–0.668.Source: §4.1 · p. 7, Table 3 · p. 8
  • The exploratory five-factor model merged growth and practice, which were weakly differentiated in the admissible six-factor model; a one-factor model fit substantially worse.Growth and practice had latent correlation 0.954 and HTMT 0.964; the seven-factor model was inadmissible, and the one-factor model had CFI = 0.885, RMSEA = 0.125, and SRMR = 0.102.Source: §4.1 · pp. 7–8, Table 4 · p. 8

4.2 Patterns of GenAI use

  • GenAI use was common: 85.6% reported using it often or very frequently, and 57.5% reported six or more interactions per assignment.Although 40.3% used GenAI for more than half of academic tasks, interaction counts alone could not distinguish efficient prompting from superficial engagement.Source: §4.2 · p. 8

4.3 Perceived skill development and changes in assignment processes

  • Students more often reported improvement in efficiency than in evaluating output accuracy, with a 0.335-point mean difference (dz = 0.413; Holm-adjusted p < 0.001).Improved efficiency was endorsed by 89.1%, versus 74.4% for evaluating output accuracy; measures capture perceived growth rather than absolute competence.Source: §4.3 · pp. 8–9
  • Assignment-related learning was endorsed more often than learning new GenAI uses beyond coursework (91.7% vs. 65.4%), while strategic work and critical evaluation exceeded increased draft revision.Students endorsed strategic working at 84.9% and critical evaluation at 84.4%, versus 69.0% for increased draft revision.Source: §4.3 · p. 10
  • Interview accounts described more targeted prompting and selective assistance, with students adapting GenAI use to changing academic needs while retaining control over substantive work.These accounts did not demonstrate corresponding improvements in work quality.Source: §4.3 · p. 10

4.4 Perceived academic gains and task-specific improvement

Students reported stronger academic gains for assignments than examinations, alongside varied perceived development trajectories. Trajectory groups differed in perceived development and regulation of GenAI use, but not clearly in reported academic gains.

4.4 Perceived academic gains and task-specific improvement

  • 78.9% endorsed overall academic improvement, and assignment-grade gains were reported more often than examination-grade gains (74.7% vs. 61.4%).The assignment–examination mean difference was 0.204 points (dz = 0.294; Holm-adjusted p < 0.001).Source: §4.4 · p. 10, Figure 3 · p. 11
  • Conceptual understanding (65.8%), academic writing (63.6%), and brainstorming (61.7%) were the most frequently selected areas of improvement.Programming (27.8%) and citation work (28.1%) were less frequently selected; differences may reflect task exposure as well as perceived benefit.Source: §4.4 · p. 10, Figure 3 · p. 11
  • Interview accounts distinguished assisted work from underlying understanding or assessed performance, including cases where better language did not raise marks.Some participants linked limited examination benefits to restrictions on GenAI use, while another described learning gains during examination preparation.Source: §4.4 · p. 10

4.5 Retrospectively perceived developmental trajectories

  • 34.2% reported rapid early improvement followed by a plateau, 28.4% steady improvement, and 21.1% delayed acceleration.Fluctuating development was reported by 12.8%; little or no improvement and other patterns were uncommon.Source: §4.5 · p. 11, Figure 4 · p. 12
  • After Benjamini–Hochberg correction, trajectory groups differed in perceived skill growth, practice-related development, assignment-process adaptation, and responsible use, but not clearly in academic gains.The largest group difference was for responsible academic use (ε² = 0.073); the three largest groups had similar academic-gains means (3.82–3.91).Source: §4.5 · p. 11, Table 6 · p. 12, Figure 4 · p. 12
  • Interviewees attributed perceived development to changing strategies, academic demands, and GenAI capabilities, while some described stable routines.P08 reported greater fluency and more precise prompting despite broadly stable uses; P13 and P18 emphasized technological advances.Source: §4.5 · p. 12

4.6 Sources of support and assessment alignment

Course requirements and informal resources were commonly reported as supports for GenAI development, while students were less confident that assessments were ready for widespread GenAI use. Responsible-use reports included frequent verification, but interviews distinguish checking from the ability to judge accuracy.

4.6 Sources of support and assessment alignment

  • 79.8% endorsed course requirements as a development driver, while online resources and peer exchange were rated higher than university teaching or workshops.Online resources and peer support exceeded university teaching by 0.316 and 0.236 points, respectively (both Holm-adjusted p < 0.001).Source: §4.6 · p. 13, Figure 5 · p. 13
  • 73.2% perceived assessment as adapted to GenAI, but only 46.3% considered current assessment suitable for widespread GenAI use.The mean difference was 0.463 points (dz = 0.554; Holm-adjusted p < 0.001); interviewees described uneven exposure and fit with formal teaching.Source: §4.6 · pp. 13–14, Figure 5 · p. 13

4.7 Responsible use

  • Verification was endorsed by 91.6%, yet students’ reported growth in evaluating output accuracy was lower, so checking does not necessarily indicate judgment ability.Interviewees described checking sources, calculations, code, and outputs across tools, while some remained uncertain about evaluation quality or lacked relevant disciplinary knowledge.Source: §4.7 · p. 14

4.8 Recommendations for institutional support

Students’ institutional-support suggestions centered on practical guidance and preserving independent engagement. Interview accounts also show that productive assistance and dependence can coexist, with delegation varying by task and circumstances.

4.8 Recommendations for institutional support

  • Training or guidance was the most common recommendation among 83 valid comments (49.4%), followed by independent thinking (39.8%).Comments could receive multiple codes, so percentages describe the thematic composition of responses rather than prevalence across the full sample.Source: §4.8 · p. 14, Table 7 · p. 15
  • Students connected responsible use with assessment design and academic ownership, but comments showed no clear consensus on suitable institutional policies.Suggestions included oral defenses and assignments emphasizing critical thinking, practical work, and originality; views on detection and monitoring differed.Source: §4.8 · pp. 14–15

4.9 Delegation and dependence

  • Some students described thinking through problems before consulting GenAI, while others linked deadline pressure or skipped steps with reduced independent reasoning.The interviews distinguish using GenAI to support reasoning from using it to replace parts of that process.Source: §4.9 · p. 15
  • Productive assistance and dependence could coexist within one account, and dependence varied with task, circumstances, and use.P12 described learning through stepwise questioning alongside increased laziness; P13 noted that following an AI-generated plan could narrow consideration of the whole problem.Source: §4.9 · p. 15
  • GenAI was also described as lowering interpersonal barriers to seeking help, while raising questions about how much reasoning students delegate.P18 said it reduced embarrassment about asking questions and allowed help-seeking without the pressure of approaching teachers.Source: §4.9 · p. 15

4.10 Exploratory structural and contextual associations

The exploratory model associated perceived skill growth with assignment-process adaptation and, indirectly, with perceived academic gains. Contextual correlations varied by exposure and student-focused measures, and the cross-sectional analyses do not establish causal or temporal relationships.

4.10 Exploratory structural and contextual associations

  • The direct association between perceived skill growth and academic gains was statistically uncertain (β = 0.209, robust-test p = 0.156).Because the structural component was saturated, fit indices primarily reflect measurement restrictions; the indirect association does not establish complete mediation or causation.Source: §4.10 · pp. 15–16, Figure 6 · p. 16
  • The indirect association between perceived skill growth and academic gains through assignment-process adaptation was positive (β = 0.452), resulting in a total association of β = 0.661.Source: §4.10 · p. 16
  • The cross-sectional analysis cannot determine temporal ordering, establish causal mechanisms, or exclude alternative explanations and unmeasured confounding.Model-implied R² values represent variance under the specified model, not causal explanatory power or predictive performance.Source: §4.10 · p. 16
  • Perceived academic gains correlated with GenAI use frequency (ρ = 0.216) and interactions per assignment (ρ = 0.159), but not clearly with task share or time since first use.These were bivariate correlations with BH-adjusted p < 0.001, p = 0.008, p = 0.180, and p = 0.072, respectively.Source: §4.10 · p. 16, Figure 7 · p. 17
  • Self-rated digital literacy was positively associated with skill growth (ρ = 0.419), process adaptation (ρ = 0.289), academic gains (ρ = 0.284), and responsible use (ρ = 0.383).All had BH-adjusted p < 0.001; these bivariate associations do not establish independent effects or causation.Source: §4.10 · p. 17, Figure 7 · p. 17

5. Discussion

Students described heterogeneous, uneven GenAI skill development and differentiated academic benefits, alongside changes in learning practices and assessment. Because the evidence is retrospective and cross-sectional, it cannot establish actual development or causal effects.

5.1. Perceived heterogeneous and uneven development over time

  • Perceived efficiency gains were more common than gains in evaluating output accuracy (89.1% vs. 74.4%), and academic gains did not clearly differ across trajectory groups after correction.Repeated observations on comparable tasks under stable tool conditions would help distinguish learner development from changing task difficulty, standards, feedback, or tool capabilities.Source: §5.1 · pp. 17–18

5.2. Learning processes and perceived performance gains

  • Assignment-grade gains were reported more often than examination-grade gains (74.7% vs. 61.4%), while skill growth showed a positive indirect association with academic gains through assignment-process adaptation.The concurrent, attributional process measures do not establish a developmental sequence; interviews also described delegation that supported learning or bypassed reasoning.Source: §5.2 · p. 18

5.3. Assessment adaptation

  • Although 73.2% perceived assessment practice had adapted to GenAI, only 46.3% considered current assessment appropriate for widespread use.The discussion recommends making students’ reasoning and contribution visible; proposed practices require testing for effectiveness, feasibility, and workload in specific contexts.Source: §5.3 · p. 18

5.4. Limitations and future research

  • Retrospective self-reports capture perceived rather than observed development, and the cross-sectional design cannot establish temporal ordering or causality.The authors recommend prospective longitudinal, multimethod research with repeated competency assessments, interaction logs, assignment artefacts, verified outcomes, and linked interviews.Source: §5.4 · pp. 18–19

6. Conclusion

Students reported varied perceived GenAI skill-development trajectories and changes in learning practices. Academic gains differed by assessment context, and the study concludes that supporting evaluation, reasoning, and visible understanding matters alongside effective use.

6. Conclusion

  • Among 313 respondents, 34.2% reported rapid early improvement followed by a plateau, 28.4% steady improvement, and 21.1% delayed acceleration.Students perceived evaluating GenAI outputs as more challenging to improve than other academic skills.Source: §6 · p. 19
  • Perceived skill growth accompanied assignment-process adaptation, while interviews described more specific prompting, varied verification, and selective delegation alongside stable routines and skipped reasoning.Some students attributed improvement partly to advances in GenAI tools; assignments were the most frequently selected driver of skill development.Source: §6 · p. 19
  • Assignment-grade gains were reported more often than examination-grade gains (74.7% vs. 61.4%), and gains did not clearly differ across trajectory groups.The structural analysis found a positive indirect association between perceived skill growth and gains through assignment-practice adaptation; retrospective, one-time self-reports cannot establish longitudinal development or causal effects.Source: §6 · p. 19
  • The authors highlight support for evaluating outputs, retaining responsibility for reasoning, and demonstrating understanding alongside GenAI-assisted performance.Interviewees called for training tailored to skill levels, disciplines, and learning needs, together with appropriately designed assessment practices.Source: §6 · p. 19

Ethics approval

The survey and interview protocols received ethics approval from the University Research Ethics Review Panel of Xi’an Jiaotong-Liverpool University on 13 Jun 2026.

Ethics approval

  • The University Research Ethics Review Panel of Xi’an Jiaotong-Liverpool University approved the survey and interview protocols.The approval date was 13 Jun 2026.Source: Ethics approval · p. 20
  • The study procedures were conducted in accordance with applicable ethical guidelines and regulations.Source: Ethics approval · p. 20
  • The approval covered both the survey and interview protocols.Source: Ethics approval · p. 20

Informed consent

Participants gave informed consent; their identities were kept anonymous and participation was voluntary.

Informed consent

Loading 2610.30599v1…