Source-linked AI summary
Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education
Alessio Ferrari, Minh An Nguyen, Kushal Ramkumar, Liliana Pasquale
TL;DR
Secure software engineering education has limited evidence about how students use LLMs and how interaction strategies relate to outcomes. This exploratory study analysed ChatGPT logs from 26 graduate students completing vulnerability fixing, finding that usage diversity was positively associated with performance even after controlling for cybersecurity expertise. The findings motivate controlled studies of diversified, task-oriented, and reflective LLM use.
Problem
LLM use in secure software engineering education, especially how interaction patterns relate to student outcomes, remains underexplored.
Method
The study analysed double-coded ChatGPT interaction logs from 26 graduate students completing a vulnerability-fixing assignment, examining usage patterns, diversity, expertise, and performance.
Results
Usage diversity was positively associated with higher grades even after controlling for cybersecurity expertise, whereas no individual pattern remained statistically significant after multiple-comparison correction.
Takeaways & Limitations
How students engage with ChatGPT may be more informative than whether they use it, motivating controlled studies of diversified, task-oriented, and reflective use.
Takeaways & Limitations
The observational exploratory design and small sample limit causal interpretation and statistical power, and the participants came from one educational context.
Abstract
from arXiv · showhide
The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored. We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment. To characterise ChatGPT use, we analysed students' ChatGPT interaction logs using a structured double-coding procedure and examined whether usage patterns and prior cybersecurity expertise were associated with assignment performance. The results show that students with varying levels of cybersecurity expertise used broadly similar ChatGPT strategies. Individual usage patterns showed descriptive differences by grade, but none remained statistically significant after correcting for multiple comparisons. In contrast, diversity of ChatGPT usage, i.e., the number of distinct usage patterns adopted, was positively associated with performance, even after controlling for cybersecurity expertise. These exploratory findings suggest that the way students engage with ChatGPT may be more informative than whether they use it, and motivate future controlled studies to guide students toward effective LLM use in secure software engineering education.
1 Introduction
This study examines how students use ChatGPT during secure vulnerability fixing and whether usage strategies and cybersecurity expertise relate to performance. It finds that usage diversity was more consistently associated with grades than individual patterns, while the exploratory evidence is not causal.
- Prior work has paid less attention to LLM interaction patterns in secure software engineering education and their relationship to student outcomes.
- 26 students used ChatGPT during a cybersecurity vulnerability-fixing exercise, enabling analysis of expertise, usage patterns, and usage diversity.The study analysed interaction logs and assignment performance.
- Students with different cybersecurity expertise levels used broadly similar ChatGPT strategies.
- Individual usage patterns showed descriptive grade differences, but none remained statistically significant after correcting for multiple comparisons.
- A larger number of distinct ChatGPT usage patterns was positively associated with higher grades, including after controlling for cybersecurity expertise.
- The findings motivate controlled studies of diversified, task-oriented, and reflective ChatGPT use rather than causal conclusions from this exploratory study.
2 Related Work
Prior research has examined LLMs in programming education, developer workflows, and security-related software engineering tasks. However, the supplied literature indicates that secure software engineering education remains comparatively underexplored, especially regarding interaction patterns.
- Research has studied ChatGPT capabilities, programming behaviours, learning effects, and AI tutoring in programming and software engineering education.
- Secure software engineering education has received less attention, particularly concerning how LLM interaction patterns relate to student outcomes.
- Developer–ChatGPT conversation research describes uses including code generation, explanation, debugging, documentation, and problem solving.
- Security-focused studies find vulnerability tasks challenging and suggest that reasoning and patch-validation feedback improve LLM-based repair.
3 Research Design
The study uses an exploratory mixed analysis of ChatGPT-supported vulnerability fixing by postgraduate students with differing cybersecurity expertise. It combines coded interaction logs, descriptive comparisons, and regression to examine usage patterns, usage diversity, and performance.
- The study characterises ChatGPT usage patterns and diversity during vulnerability fixing while accounting for prior cybersecurity expertise.
- The research questions address observed patterns, individual-pattern performance differences, usage-diversity associations, and expertise-controlled relationships.
- 3.1 Data Collection Procedure: The analysed sample comprised 26 ChatGPT-using participants recruited from a part-time postgraduate Secure Software Engineering module.
- 3.1 Data Collection Procedure: Participants completed a cybersecurity pre-test and a vulnerability-fixing assignment based on OWASP Top 10 2021 and associated CWE entries.
- 3.2 Data Analysis Procedure: Two researchers independently coded ChatGPT conversations using a semi-open procedure, adding emergent codes and resolving disagreements through consensus.
- 3.2 Data Analysis Procedure: Individual patterns were compared descriptively and with exploratory Wilcoxon tests, while usage diversity was measured as the number of distinct patterns.
- 3.2 Data Analysis Procedure: An exploratory linear regression model predicted grade from usage diversity while controlling for cybersecurity expertise, with diagnostic plots used to inspect assumptions.
4 Results
Students with different cybersecurity expertise levels used broadly similar ChatGPT strategies, while usage diversity showed the clearest positive association with vulnerability-fixing performance. Individual usage patterns displayed descriptive grade differences, but none remained statistically significant after multiple-comparison correction.
- 4.1 RQ1: Distribution of ChatGPT Usage Patterns: The expertise groups showed some descriptive differences, but small and unbalanced samples limit interpretation as evidence of systematic group differences.The groups contained 10 medium-low-expertise and 16 high-expertise participants.
- 4.1 RQ1: Distribution of ChatGPT Usage Patterns: Both expertise groups primarily used ChatGPT for direct task support, including generating mitigation implementations, refining code, explaining security controls, and debugging.Generating an implementation for a specific mitigation was observed for 90% of medium-low-expertise participants and 87.5% of high-expertise participants.
- 4.2 RQ2: Usage Patterns and Observed Grade Differences: No individual ChatGPT usage pattern remained statistically significant after Benjamini–Hochberg correction, despite several descriptive grade differences.The largest descriptive difference favored generating code based on a mitigation name (+26.5 points), but that pattern was highly imbalanced across participants.
- 4.3 RQ3: Usage Diversity and Grade: Students using a wider range of ChatGPT patterns tended to obtain higher grades, with a moderate positive association between diversity and grade (ρ = .375, p = .030).The scatterplot showed substantial variability, indicating that usage diversity explained only part of the observed grade differences.
- 4.4 RQ4: Usage Diversity, Grade, and Cybersecurity Expertise: Usage diversity was positively associated with grade after accounting for cybersecurity expertise, with each additional pattern associated with an estimated 3.11-point increase.The exploratory regression model explained approximately 21% of grade variance, although its overall test did not reach the conventional .05 threshold.
5 Discussion
The discussion argues that ChatGPT usage diversity was more consistently associated with vulnerability-fixing performance than any individual pattern, motivating critical-use guidance and controlled follow-up studies.
- 5 Discussion: Usage diversity showed a more consistent positive association with performance than any single ChatGPT usage pattern.Individual patterns had descriptive grade differences, but none remained statistically significant after multiple-comparison correction.
- 5 Discussion: Students may have benefited from using ChatGPT in complementary ways, including generating fixes, debugging, refining code, and exploring alternatives.
- 5 Discussion: Critical-use instruction may be more productive than simply allowing or prohibiting ChatGPT in secure software engineering training.Suggested guidance includes targeted security-control questions, explanations, validation, alternative mitigations, and documenting evaluation of generated suggestions.
- 5 Discussion: Future research should capture interaction-level data and test whether diversified, task-oriented, and reflective use can improve secure software engineering learning.The paper frames these implications as hypotheses and proposes controlled studies rather than conclusions.
6 Threats to Validity
The study’s validity is constrained by coded usage measures, observational design, limited statistical power, and a single educational setting.
- 6 Threats to Validity: Manually coded patterns may miss differences in interaction depth, quality, and critical reflection, while usage diversity does not measure interaction effectiveness.Performance grades may also omit code quality, reasoning, and security robustness.
- 6 Threats to Validity: The observational design prevents isolating causal effects because students self-selected patterns and may differ in motivation, ability, persistence, or problem-solving skill.Other unmeasured confounders may remain, and students often combined multiple patterns.
- 6 Threats to Validity: The small sample limits statistical power, particularly for patterns used by few participants and imbalanced user comparisons.Accordingly, the analysis emphasizes effect direction, confidence intervals, and visual trends alongside significance.
- 6 Threats to Validity: Because participants completed one vulnerability-fixing activity in one educational context, findings may not generalize to professional developers, other courses, or other security tasks.
7 Conclusion
The conclusion reports that broader ChatGPT engagement was more consistently associated with performance than individual patterns or prior cybersecurity expertise alone, while emphasizing the exploratory design.
- 7 Conclusion: Wider ChatGPT usage was associated with higher vulnerability-fixing grades even after accounting for cybersecurity expertise.
- 7 Conclusion: The findings are hypothesis-generating rather than causal evidence because the study was exploratory and had a limited sample size.
- 7 Conclusion: Future controlled studies should test whether diversified, task-oriented, and reflective ChatGPT use improves vulnerability-fixing performance.
Data Availability
An anonymised replication package containing study materials, data, and analysis scripts is available on Zenodo.
- Data Availability: The anonymised replication package is available on Zenodo at DOI 10.5281/zenodo.20453130.It includes the pre-test questionnaire, assignment description, instructor rubric, and code explanations with examples.