Source-linked AI summary
Understanding Generative AI Use in Programming MOOCs: The Role of Course Context and Learner Characteristics
Marina Lepp
TL;DR
Research on GenAI in education reports mixed learning outcomes, and how learners use these tools may matter. This study compares self-reported use across two programming MOOCs and learner backgrounds. Use frequency and purposes differed by course, while measured learning-related outcomes showed limited relationships with GenAI use.
Problem
GenAI is widely available for programming guidance, but its educational impact remains uncertain amid reported benefits and risks.
Source: §1 · p. 1Method
The study analyzed post-course self-reports and recorded outcomes from two programming MOOCs that differed in duration, workload, complexity, and assignment volume.
Source: §1 · p. 1, §3.1 · p. 2Results
Participants in the longer MOOC reported higher overall frequency and more frequent use for some purposes, while no significant differences appeared across learner characteristics and outcome associations were limited.
Source: §5 · pp. 5–6, §6 · p. 6Takeaways & Limitations
The findings suggest that course context may shape GenAI use, although the study cannot separate the contributions of course characteristics that varied together.
Source: §5 · p. 6Use was self-reported, only two MOOCs were studied, and actual GenAI interactions were not examined.
Source: §6 · p. 6
Abstract
from arXiv · showhide
The increasing availability of generative artificial intelligence (GenAI) tools, such as ChatGPT and code-completion assistants, raises questions about how learners integrate these tools into learning activities, particularly in MOOCs that attract diverse participant populations. This study examines the use of GenAI in two programming MOOCs taught in Estonian that differ in duration, workload, topic complexity, and assignment volume: About Programming (4 weeks, 26 expected hours, n = 187) and Introduction to Programming (8 weeks, 78 expected hours, n = 182). Post-course questionnaire data were analyzed using non-parametric statistical methods to examine self-reported adoption, usage frequency, and purposes of GenAI use across courses and learner backgrounds. The results show that GenAI adoption was widespread in both MOOCs, with no statistically significant differences by gender, age, education level, or prior programming experience. However, participants in the longer, more extensive MOOC reported significantly higher usage frequency and were more likely to use GenAI for debugging and idea generation. Reported usage frequency for code explanation and debugging was also higher in the longer course. Exploratory analyses found limited relationships between GenAI use and learning-related outcomes. The findings suggest that course context may play a greater role than learner characteristics in shaping how GenAI tools are used. These results provide implications for instructional design and guidance in programming education.
1 Introduction
GenAI can provide programming guidance and feedback, but its educational effects remain uncertain, with research reporting both benefits and risks. This study compares use across two programming MOOCs and examines differences by course, learner characteristics, and learning-related outcomes.
1 Introduction
- GenAI tools can generate code, explain concepts, and assist with debugging, while research reports both learning benefits and risks such as overreliance.The mixed findings make how learners use the tools relevant alongside whether they use them.Source: §1 · p. 1
- The study compares two programming MOOCs that differ in duration, workload, topic complexity, and assignment volume, examining adoption, purposes, frequency, learner characteristics, and learning-related outcomes.Its four research questions cover between-course use, purpose-specific patterns, learner-background differences, and relationships with recorded performance and activity.Source: §1 · p. 1
2 Related Work
Prior research describes varied GenAI uses and mixed learning-outcome findings, while noting that use may depend on both learner characteristics and context. These mixed demographic patterns motivate separating adoption, purpose prevalence, and use frequency when comparing learners across programming MOOCs.
2 Related Work
- GenAI tools offer on-demand programming guidance, but incorrect or misleading outputs and overreliance may pose risks to learning.Research examines the conditions under which these tools support learning.Source: §2 · p. 2
- Studies report improved performance and learning-related measures, no significant differences, or negative effects, with findings often linked to how learners use GenAI.Explanation and conceptual understanding have been associated with greater benefits than using AI primarily to generate complete solutions.Source: §2 · p. 2
- GenAI use ranges from debugging and explanation-seeking to solution generation, and from iterative refinement to copying outputs without modification.Heavy reliance may contribute to superficial learning and an “illusion of competence.”Source: §2 · p. 2
- Learner habits and motivation, field of study, task complexity, and course level have all been linked to GenAI interaction patterns.Prior research also describes different uses by more experienced learners and novices, and proposes that effectiveness depends on learner characteristics interacting with tool affordances.Source: §2 · p. 2
- Evidence about demographic differences is mixed, including findings on gender, academic level, education, and age.This motivates distinguishing adoption, purpose prevalence, and reported frequency when comparing diverse learners across programming MOOCs.Source: §2 · p. 2
3 Methods
The study compares self-reported GenAI use in two Estonian Python programming MOOCs with different course characteristics. Post-course questionnaire responses and course records were analyzed using non-parametric tests and corrected significance testing.
3.1 Context
- About Programming ran for 4 weeks with 26 expected hours, while Introduction to Programming ran for 8 weeks with 78 expected hours and more assignments.The courses also differed in content coverage; these characteristics varied together, so the comparison does not isolate any one characteristic.Source: §3.1 · p. 2
3.2 Sample and Data Collection
- The final sample comprised 187 About Programming and 182 Introduction to Programming post-questionnaire respondents, recruited from 226 and 230 enrolled participants, respectively.The questionnaire measured self-reported adoption, overall frequency, tools, and use purposes; LMS records supplied performance and activity indicators, and assignments were unproctored.Source: §3.2 · pp. 2–3
3.3 Data Analysis
- Non-parametric tests compared course and learner-group differences, while permutation-based p-values and Holm correction were used across analysis families.Analyses also examined associations between GenAI use and learning-related outcomes; missing values were excluded from relevant analyses.Source: §3.3 · p. 3
4 Results
GenAI adoption was widespread, but reported frequency was higher in the longer MOOC. Purpose-specific patterns also differed, while the analyses found no significant differences by learner characteristics and limited relationships with measured learning outcomes.
4.1 GenAI Use Across Different MOOCs
- MOOC2 reported higher overall frequency (U = 13,243, r = 0.222, 95% CI [0.125, 0.319], p < 0.001), despite equal medians of 2.Adoption was 75.9% in MOOC1 and 82.4% in MOOC2, with no significant between-course difference; weekly or almost-weekly use was 10.2% versus 18.7%.Source: §4.1 · p. 3, Figure 1 · p. 3
4.2 Purposes of GenAI Use
- MOOC2 users more often selected debugging and idea generation, and purpose-specific frequency was higher for debugging and code explanation.The prevalence of selecting a purpose and its frequency among respondents reporting that purpose are distinct patterns.Source: §4.2 · p. 4, Table 1 · p. 4, Table 2 · p. 4
4.3 Differences by Learner Characteristics
- No significant adoption or user-frequency differences were detected across gender, age, education level, or prior programming experience.All sixteen Holm-adjusted p-values were 1.000; the results do not establish equivalence between learner groups.Source: §4.3 · p. 4, Table 3 · p. 4, Table 4 · p. 4
4.4 GenAI Use and Learning Outcomes
- In MOOC1, non-users completed more optional exercises (M = 2.09 versus 0.96; r = −0.251, adjusted p = 0.027), while overall use frequency was not significantly correlated with performance.MOOC2 showed no significant differences across performance indicators, and frequency-performance correlations were nonsignificant in both courses.Source: §4.4 · pp. 4–5, Table 7 · p. 5
5 Discussion
Across two programming MOOCs, GenAI use was widespread, but reported frequency and purposes differed by course context. The findings showed no significant differences across measured learner characteristics and limited associations with learning-related outcomes.
5 Discussion
- GenAI adoption was widespread in both courses, with no significant differences in adoption or reported frequency across gender, age, education level, or prior programming experience.The estimated associations were generally small, but the results do not establish equivalence between learner groups.Source: §5 · p. 5
- Introduction to Programming respondents reported significantly higher overall GenAI-use frequency than About Programming, including within user groups, though the estimated gap was modest.The discussion suggests that greater problem-solving demands, duration, and assignment volume may offer possible explanations, but these were not established.Source: §5 · p. 5
- Debugging and idea generation were more prevalent in the longer MOOC, while debugging and code explanation were used more frequently by respondents reporting those purposes.The required-assignment code-solution frequency difference favored the longer MOOC but was imprecise and did not meet the significance threshold after Holm correction.Source: §5 · p. 5
- Reported purposes and frequency do not reveal interaction quality: problem-solving effort and whether GenAI scaffolded effectively were not measured.Higher frequency therefore does not establish deeper understanding, iterative refinement, or more effective learning.Source: §5 · p. 5
- Exploratory analyses found limited associations between GenAI use and recorded outcomes; more frequent use was not associated with better performance in either course.Users and non-users showed similar measured performance and engagement except for optional exercises in the shorter course.Source: §5 · p. 6
- Course context may shape GenAI frequency and purposes, but duration, complexity, workload, and assignment volume varied together, preventing attribution to any one factor.Source: §5 · p. 6
6 Conclusion
Across two programming MOOCs, GenAI adoption was widespread, but reported use varied by course; exploratory links to learning-related outcomes were limited. The findings support course-aware guidance while leaving instructional effectiveness for further study.
6 Conclusion
- Adoption was widespread without a course difference, while the longer course reported higher overall use and more debugging and idea-generation use; demographic differences were not significant.Respondents in the longer course also reported more frequent use for code explanation and debugging.Source: §6 · p. 6
- Exploratory analyses found limited links to assessment performance and course activity, while non-users completed more optional exercises in the shorter course.Source: §6 · p. 6
- Educators should clarify GenAI expectations, address common activities, encourage critical evaluation, and consider course characteristics when planning guidance.The absence of significant demographic differences does not establish that the same support would work equally well for everyone.Source: §6 · p. 6
- Self-reported use, two MOOCs in a specific context, unproctored assessments, and unobserved GenAI interactions constrain interpretation and generalizability.Course duration, workload, topic complexity, and assignment volume varied simultaneously, so their individual contributions could not be separated.Source: §6 · p. 6
- Future work should combine self-reports with interaction logs, directly assess learning, track usage longitudinally, and test instructional interventions.Source: §6 · p. 6