Source-linked AI summary
Predicting Struggling Students in CS1 Programming Using Keystroke-Level Editing Features
Yasuyo Kofune, Kazumasa Shimari, Kenichi Matsumoto
TL;DR
The paper asks whether struggling CS1 students can be identified during an exercise rather than only after outcomes become available. Using CodeBench execution and keystroke-level logs with strict Breakthrough and Fully Stuck labels, it finds that early editing behavior adds predictive value and that the first segment is most informative.
Problem
Existing instructor information is often limited to submission outcomes and final grades, motivating earlier identification of students who will remain stuck during CS1 exercises.
Method
The study compares ExecOnly, CMOnly, and Combined features across submission-based exercise segments using CodeBench data and Breakthrough/Fully Stuck outcome labels.
Results
+0.098 AUROC was achieved by Combined over ExecOnly at k=1, while the first segment had the strongest predictive performance across configurations.
Takeaways & Limitations
Keystroke-level editing logs provide preliminary evidence of additional value for early prioritization of students who may remain stuck during programming exercises.
Takeaways & Limitations
The analysis uses one semester at one institution, does not control exercise difficulty, and does not test generalization to unseen exercises.
Abstract
from arXiv · showhide
This paper investigates the feasibility of early detection of struggling students during CS1 programming exercises using keystroke-level logs. Some students fail to reach a correct solution before the exercise ends, and by the time this becomes apparent from grades or final outcomes, the opportunity for timely instructor support aimed at helping them recover may have passed. We use data from the CodeBench platform, which records real-time code editing events at the keystroke level, alongside execution and submission logs. We define two outcome groups: Breakthrough (BT) students, whose prior submissions all receive 0% and whose final submission achieves full credit, and Fully Stuck (FS) students, whose submissions all receive 0% without reaching a correct solution. To examine this feasibility, we focus on two questions: (RQ1) whether adding keystroke-level editing features improves the prediction of FS students over execution-log features alone, and (RQ2) at which stage BT and FS students can be predicted most accurately. Experiments on the 2019-1 semester of the CodeBench dataset, comprising 507 students, compare three feature configurations: execution-based features (ExecOnly), CodeMirror-based features (CMOnly), and their combination (Combined). We evaluate prediction across successive submission-based stages during each exercise. In the earliest stage, CMOnly outperforms ExecOnly (AUROC 0.654 vs. 0.575), and Combined further improves over ExecOnly by +0.098 (AUROC 0.674). Across all configurations, the earliest stage yielded the strongest predictive signal. These findings indicate that behavioral signals present at the very start of an exercise contain useful clues about whether a student will ultimately solve the problem, and that keystroke-level editing logs provide additional value for early prioritization beyond execution logs alone.
I. INTRODUCTION
The paper targets real-time identification of CS1 students who will remain stuck, addressing the limits of submission outcomes and final grades. It tests whether keystroke-level editing logs improve prediction and whether useful signals appear at the exercise’s earliest stage.
- CS1 instructors often lack actionable information about struggling students until exercise submissions or final grades are available.
- The study contrasts Breakthrough students, who move from prior 0% submissions to a final 100%, with Fully Stuck students, whose submissions remain at 0%.
- RQ1 asks whether keystroke-level editing features improve Fully Stuck prediction over execution-log features alone.
- RQ2 evaluates which submission-based exercise stage provides the most accurate prediction of Breakthrough and Fully Stuck students.
- +0.098 AUROC in the initial segment is the reported improvement of adding keystroke-level features over execution logs alone.
- The initial segment provides the strongest segment-wise predictive signal and requires no prior submission history.
II. RELATED WORK
Prior CS1 prediction work largely uses accumulated outcome-level data, while a smaller body of research examines coding-process behavior. This study focuses on keystroke-derived features from individual submission intervals.
- Outcome-level prediction: Outcome-level approaches predict course or assignment performance using grades, submission counts, time-on-task, and other accumulated measures.
- Outcome-level prediction: Post-submission features can achieve strong predictive accuracy but are less suitable for intervention during active problem solving.
- Outcome-level prediction: Prior CodeBench work reached AUROC 0.89, but grade-based and submission-based features dominated while keystroke-derived aggregates contributed minimally.
- Process-level and in-exercise data: Process-level studies use compilation logs, edit-compile cycles, weekly behavior aggregates, or submission sequences to model student outcomes.
- Process-level and in-exercise data: The present study extracts keystroke-derived features from individual submission intervals rather than relying on session-level aggregates.
III. METHODS
The study uses CodeBench logs from Python CS1 exercises, combining execution/submission records with CodeMirror editing events. The analysis uses the 2019-1 semester, comprising 507 students.
- A. Dataset: CodeBench records execution/submission logs and keystroke-level editing logs for each student–exercise pair.
- A. Dataset: CodeMirror is the integrated code editor that generates the study’s keystroke-level editing logs.
- A. Dataset: The dataset covers Python programming exercises in CS1 courses at the Federal University of Amazonas, Brazil.
- A. Dataset: 35,887 student–exercise pairs are represented in the semester’s submission-outcome distribution.
- A. Dataset: 507 students from the 2019-1 semester were analyzed as a representative pre-COVID-19 feasibility-study sample.
B. Label Definition
The study defines a strict Breakthrough-versus-Fully-Stuck classification task from submission histories and analyzes behavior in submission-based segments. Analysis is limited to segments with sufficient instances and recorded editing events.
- Label Definition: Breakthrough instances contain at least one prior 0% submission followed by a final 100% submission.
- Label Definition: Fully Stuck instances receive 0% on every submission and never reach a correct solution.
- Label Definition: Pairs containing partial-credit submissions were excluded from the Breakthrough and Fully Stuck labels.
- Segment Definition: A segment spans the interval between consecutive submissions, with segment 1 running from exercise start to the first submission.
- Segment Definition: The final 100% Breakthrough segment was excluded because its editing behavior reflects solution-oriented activity rather than struggle.
- Segment Definition: Segments beyond k=10 were omitted because available instances declined rapidly, making reliable estimation difficult.
- Segment Definition: At k=1, missing CodeMirror events excluded 10.9% of Breakthrough and 9.3% of Fully Stuck instances.
D. Features
The study extracts execution/submission outcomes and keystroke-level coding behavior for each submission-based segment, then compares separate and combined feature sets.
- ExecOnly uses four execution/submission features describing syntax, runtime, logic errors, and elapsed time since the previous submission.The elapsed-time feature is set to 0 for the first segment.
- CMOnly uses eight CodeMirror features capturing editing activity, character changes, deletion behavior, pauses, and segment duration.These features represent how students write and revise code within each segment.
- The Combined configuration contains 12 features, combining the four execution features with the eight CodeMirror features.
E. Classification Models and Evaluation
Random Forest and Logistic Regression models are trained separately for each segment and feature configuration, with student-level cross-validation and pooled out-of-fold evaluation.
- Classification Models: Random Forest and Logistic Regression classifiers were trained separately for every segment and feature configuration.Random Forest used 200 trees, while Logistic Regression used L2 regularization with C=1.0.
- Evaluation: Student-level GroupKFold cross-validation used five folds, assigning all exercises from each student to the same fold to prevent data leakage.Students contributed multiple exercise records, averaging approximately 27 exercises each.
- Evaluation: AUROC was the primary metric, computed from pooled out-of-fold predictions across the five GroupKFold splits.Paired bootstrap confidence intervals with N=2,000 assessed AUROC differences between feature configurations at each segment.
- Supplementary Analysis: A supplementary FS-vs-rest analysis at k=1 included 31,790 student–exercise pairs and an FS prevalence of 0.060.All other outcome patterns were treated as the negative class after applying the main exclusion criteria.
IV. RESULTS
Prediction was strongest in the first submission-based segment, where CodeMirror and combined features outperformed execution-only features; performance generally declined in later segments.
- Comparison of Feature Configurations (RQ1): 0.654 vs. 0.575 AUROC: CMOnly outperformed ExecOnly at k=1.The bootstrap 95% confidence interval for the AUROC difference was [+0.056, +0.104].
- Comparison of Feature Configurations (RQ1): +0.098 AUROC: Combined outperformed ExecOnly at k=1, reaching AUROC 0.674.Combined improvements over ExecOnly were statistically reliable in the early segments k=1–4.
- Comparison of Feature Configurations (RQ1): 0.281 PR-AUC: Combined ranked above CMOnly at 0.256 and ExecOnly at 0.186 in the first segment.All three exceeded the FS prevalence baseline of 0.151.
- Prediction Performance Across Segments (RQ2): At k=4, ExecOnly reached AUROC 0.484 while Combined maintained AUROC 0.568.The Combined 95% confidence interval for the difference was [+0.052, +0.116].
- Prediction Performance Across Segments (RQ2): 0.674 AUROC: Combined achieved the highest AUROC at k=1, while Combined and CMOnly declined to 0.549 and 0.529, respectively, by k=10.ExecOnly followed a non-monotonic pattern, reaching 0.484 at k=4 and recovering to 0.543 at k=10.
C. Feature Importance
Temporal CodeMirror features were consistently important, while editing volume was especially informative before the first submission; duration alone was insufficient.
- duration_sec and max_pause_sec consistently ranked among the top three features across the examined segments.mean_pause_sec also ranked within the top three from k=3 onward.
- code_change ranked second at k=1 but declined thereafter, making editing volume particularly informative before the first submission.
- 0.473 AUROC: a duration_sec-only Logistic Regression model performed near chance despite duration_sec being the highest-ranked Random Forest feature.This contrasts with the feature-importance ranking and shows that the single feature was not sufficient for strong prediction.
- 0.653 AUROC: Combined Logistic Regression approached Combined Random Forest at 0.674, indicating that early signal arose from multivariate behavioral features.Nonlinear modeling provided a modest additional gain.
- 0.839 AUROC and 0.224 PR-AUC: the supplementary FS-vs-rest Combined model exceeded the main BT/FS AUROC of 0.674.The higher AUROC was partly attributed to including behaviorally distant outcome patterns such as Immediate Success.
V. DISCUSSION
Prediction was strongest in the initial segment across configurations and generally declined thereafter. Early keystroke-level behavior added information beyond submission outcomes, potentially reflecting how students interpret and plan solutions before trial-and-error edits converge.
- The initial segment produced the strongest prediction across all feature configurations, with performance generally declining as later segments were analyzed.Later segments include only students who have already failed multiple times, and fewer instances increase uncertainty in later AUROC estimates.
- Keystroke-level editing patterns captured aspects of problem-solving behavior that execution outcomes alone did not reflect.The distinction concerns how students engage with code before submitting, rather than only the errors produced in submissions.
- The first segment may reveal differences in how Breakthrough and Fully Stuck students interpret the problem and plan a solution.After failed submissions, both groups may make similar incremental trial-and-error edits, reducing their behavioral contrast.
- Comparisons with prior F1 results are not straightforward because predicting outcomes from early in-exercise behavior is a harder task than summarizing completed submissions.The study reports AUROC values up to 0.674 for its in-exercise prediction task.
C. Limitations
The study’s evidence is constrained by its data, labeling, deployment setting, and modeling scope. These limitations restrict generalizability, complicate interpretation of the Fully Stuck label, and limit high-confidence individual use.
- The analysis uses one semester at one institution without controlling exercise difficulty, and it does not test generalization to unseen exercises.Generalizability to other semesters, institutions, and programming languages remains to be verified.
- The 2019-1 data predate widespread generative AI coding assistants, so the predictive value of keystroke features in recent cohorts remains untested.Generated code may produce few subsequent edits and change observed editing behavior.
- The Fully Stuck label may combine persistent struggle with early disengagement, especially because editing duration was the most important feature.Among FS instances, 25.4% had only one submission; excluding those cases reduced Combined AUROC from 0.674 to 0.650.
- At FS recall 0.70, precision was 0.214, indicating substantial false positives and limiting the model to prioritization rather than standalone intervention decisions.Operational thresholds must account for classroom constraints and instructor capacity.
- Each segment was modeled independently, and the maximum AUROC of 0.674 suggests that fine-grained behavioral features alone are insufficient for high-confidence individual prediction.Sequential dependencies across segments were not used.
- The study provides initial evidence for using fine-grained editing behavior as an early prioritization signal, while CodeBench access remains subject to provider terms.Preprocessing and analysis scripts, library information, and reproduction instructions are available through the cited repository.
VI. CONCLUSION
The study evaluates whether CodeMirror keystroke logs can identify students who may remain stuck during CS1 exercises. It finds that adding editing features improves early prediction and that the first submission-based segment contains the strongest signal, while further work is needed to translate prediction into support and improve precision.
- The study defines Breakthrough and Fully Stuck groups and extracts execution-log and CodeMirror editing features for each submission interval.The analysis uses CodeBench data from the 2019-1 semester.
- +0.098 AUROC at k=1 was obtained by adding CodeMirror-based features over execution logs alone.The comparison uses the Combined configuration against ExecOnly.
- AUROC 0.674 for Combined was achieved in the first segment, which consistently performed best across configurations.This indicates above-chance information about eventual problem-solving outcomes at the exercise’s beginning.
- The findings provide preliminary evidence that fine-grained editing logs can support early prioritization of students who may remain stuck.The evidence supports feasibility of in-exercise struggle detection rather than a complete intervention system.
- Further improvements in predictive precision are needed before these findings can support broader early-prioritization and support systems.
- Future work should test instructional support during the earliest stage and characterize behavioral differences between students who recover and those who remain stuck.Suggested support includes targeted hints, instructor check-ins, and prioritization for assistance.