Source-linked AI summary
Which CS1 Students Will Fail? Identifying Digital Markers from Learning Analytics in Computer Systems and Architecture Using Weighted Academic Momentum and Interaction Logs
Lighton Phiri, Mutune Chaibela, Ivy Chisha, David Pungwa, Danny Siabbaba, Bydon Simukoko
TL;DR
Can traditional and digital markers identify failing CS1 students early enough to support practical intervention? Using stakeholder-informed predictive modelling of multi-source data, the study found that an interpretable logistic regression model achieved 87% recall for failing students by week 5.
Problem
Evidence is limited on whether combined traditional and digital markers can predict CS1 failure with sufficient recall for early intervention.
Method
The study used an exploratory sequential mixed-methods design, stakeholder elicitation, and logistic-regression ablation modelling across four CS1 cohorts.
Results
87% recall identified failing students on the held-out test set, while the winning feature subset combined weighted academic momentum, demographics, and LMS engagement.
Takeaways & Limitations
A simple logistic-regression model using early assessments, basic demographics, and a binary LMS flag can predict CS1 failure with high recall by week 5.
Takeaways & Limitations
Generalisability is unknown because the dataset covers one institution and one CS1 course, with COVID-19 cohorts introducing unmeasured confounders.
Abstract
from arXiv · showhide
Digital learning platforms generate rich behavioural traces (digital markers) that offer the potential to identify struggling students early. This paper investigates whether a combination of traditional and digital markers can predict failure in a first-year CS1 course (Computer Systems and Architecture) with sufficient recall to enable timely intervention. Using data from four cohorts (2017-2021, N=284) at a large public university in sub-Saharan Africa, we conducted a mixed-methods stakeholder elicitation to identify ten candidate factors. These were operationalised into a comprehensive feature set spanning demographics, self-reported surveys, Moodle interaction logs, and continuous assessment scores. A systematic ablation study using logistic regression with 5-fold cross-validation and SMOTE+ENN resampling revealed that the most predictive feature subset was Base + Demo + LMS: weighted academic momentum (M = 0.1Q1 + 0.15Q2 + 0.2Q3 + 0.55T1), basic demographics (gender, sponsorship, COVID-19 cohort), and a binary indicator of any LMS activity. On a held-out test set, logistic regression achieved 74.7% accuracy, 0.742 macro F1, and an AUC of 0.800. At the default threshold of 0.5, the model identified 87% of failing students (recall = 0.87) with a 41% false positive rate. SHAP analysis confirmed that weighted academic momentum is the strongest predictor, followed by its interaction with LMS engagement. These results demonstrate that simple digital markers can power a practical early-warning system by the fifth week of the semester. Our main contributions are: (1) a multi-source dataset and a stakeholder-guided methodology; (2) an ablation study quantifying feature group contributions; and (3) an interpretable, high-recall model ready for deployment.
1 Introduction
The paper argues that engineered digital markers combined with traditional student data can support early, high-recall detection of at-risk CS1 students. It contributes a multi-source dataset, a weighted academic momentum score, comparative modelling, and an interpretable early-warning approach.
- Motivation: 39.7% average failure rate was recorded in the institution’s CS1 Computer Systems and Architecture course across four cohorts.Instructors often lack timely information about which students are at risk.
- Research aim: Digital markers from demographics, surveys, LMS interaction logs, and continuous assessment scores are proposed for early detection of at-risk CS1 students with sufficient recall for intervention.LMS platforms record clicks, forum posts, and time spent on course materials, complementing traditional data sources.
- Contributions: The paper contributes a rich multi-source dataset capturing four CS1 cohorts and a weighted academic momentum score that reflects instructional importance.The dataset is anonymised and made available to researchers; the score weights early quizzes and the first test.
- Contributions: 72% accuracy and 87% recall for the failing class were achieved by an interpretable logistic regression model accompanied by SHAP explanations.The paper also compares seven classifiers, handles class imbalance with SMOTE+ENN, tunes thresholds, and discusses classroom operationalisation.
2 Related Work
AIED and educational data mining have shifted toward predictive models for identifying students at risk of failure or dropout. Related work emphasizes digital markers, careful feature selection and imbalance handling, and interpretable models suitable for instructor action.
- Predictive educational data mining: Predictive AIED research increasingly uses large-scale datasets and complex algorithms to identify students at risk of failure or dropout.The field has moved beyond descriptive analysis toward advanced predictive modelling aimed at improving learning outcomes.
- Digital markers: Digital markers from LMS behavioural traces can strengthen prediction beyond static demographic data, especially in high-failure CS1 courses.Prior studies describe demographic-only indicators as insufficient for revealing students’ underlying learning obstacles.
- Models and data challenges: 96% accuracy was reported for an SVM with hyperparameter optimization, while irrelevant features can reduce performance and increase processing time.The related work also identifies class imbalance as a critical challenge when passing students substantially outnumber failing students.
- Interpretability: Interpretable models are increasingly considered necessary in education because instructors need explanations to act on predictions.SHAP supports global and local transparency by quantifying the influence of specific digital markers on individual predictions.
3 Methodology · 3.1 Study Context and Participants · 3.2 Stakeholder Elicitation of Factors
The study used a longitudinal exploratory sequential mixed-methods design across four CS1 cohorts at a large public university in sub-Saharan Africa. Stakeholder elicitation identified and validated factors that were mapped to interpretable predictive features.
- 3 Methodology: The exploratory sequential mixed-methods design began with qualitative factor identification, followed by quantitative measurement and predictive modelling across four academic cohorts.The longitudinal cohorts were 2017/18, 2018/19, 2019/20, and 2020/21.
- 3.1 Study Context and Participants: CS1 S+A was a mandatory first-year course in the Bachelor of Information and Communication Technologies with Education at a large public university in sub-Saharan Africa.The course spans two semesters and covers computer systems organisation, architecture, and introductory programming concepts.
- 3.1 Study Context and Participants: Four cohorts were studied, with a binary COVID-19 flag controlling for fully online teaching disruptions in 2019/20 and 2020/21.The disruptions affected the 2019/20 and 2020/21 cohorts.
- 3.2 Stakeholder Elicitation of Factors: Stakeholder elicitation formed the CRISP-DM “Business/Domain Understanding” step, grounding predictive modelling in course pedagogy rather than available data alone.The process was designed to connect model development with pedagogical realities.
- 3.2.1 Qualitative exploration (interviews and focus groups): Interviews with three teaching staff and two student focus groups involving 12 participants identified ten factors believed to influence student performance.Factors included Time Management, Teaching Mode, Programme Workload, Prior Knowledge, Motivation, Lack of Orientation, Lack of Computer, Lack of Equipment, Interest, and Assessment Structure.
- 3.2.2 Quantitative survey: A structured survey of current and past CS1 students (N=29) used a 5-point Likert scale to assess factor prevalence and impact.Time Management, Motivation, and Lack of Computer each exceeded 75% combined Agree/Strongly Agree ratings.
- 3.2.2 Quantitative survey: Assessment Structure was the only factor rated as not influential by a majority, exceeding 50% Disagree/Strongly Disagree responses.The survey findings corroborated the qualitative results and provided a severity ranking.
- 3.2.3 Mapping to feature set: Validated factors were mapped directly to data-mining variables, linking survey, Student Information System, and prior-experience measures to an interpretable early-warning feature set.Examples included SurveyOwnComputer, SurveyMajorMotivation, SurveyMinorMotivation, CourseWorkload, computing experience, and prior training.
3.3 Dataset Preparation
The analytical dataset combined SIS demographics, pre-course survey responses, Moodle activity logs, and assessment scores through standardized preprocessing and source merging. After excluding students without final exam scores, the clean dataset contained 284 complete student records.
- Data sources: Four sources contributed to the dataset: SIS demographics, pre-course surveys, Moodle logs aggregated to daily unique hits, and assessment scores.The survey captured prior computing experience, computer ownership, motivation, and perceived barriers; Moodle contained 17,870 records before aggregation.
- Preprocessing: Moodle event-level duplicates were removed by aggregating multiple daily events per student into daily unique hits.The final Moodle measures included total unique days and component counts.
- Source integration: The final dataset was assembled with left joins on StudentID and AcademicYear, except surveys were joined using StudentID only because they were not cohort-specific.The merge order was Base assessments, SIS demographics, pre-course surveys, then Moodle engagement metrics.
- Missing data: Fewer than 5% of records had missing values after merging, and numeric and categorical features were imputed with the median and mode, respectively.Textual survey responses were reserved for qualitative analysis rather than predictive modelling.
- Final dataset: 284 complete student records remained after preprocessing and exclusion of records with missing final exam scores.Features were organized by factor group and classified by source and data mining attribute type.
3.4 Feature Engineering
The feature engineering combined a pedagogically weighted early-assessment momentum score with aggregated Moodle activity and engineered nonlinear and interaction terms. Categorical and numeric variables were then transformed into a 47-column feature matrix for modelling.
- Assessment features: Weighted momentum M = 0.1Q1 + 0.15Q2 + 0.2Q3 + 0.55T1 prioritised later assessments because they cover advanced and comprehensive material closer to the first test.Qi are quiz scores and T1 is the first-test score, each ranging from 0–100; the weights were selected through iterative instructor consultation.
- Digital markers: Moodle logs were aggregated into any-LMS-activity, distinct active days, and component-specific access counts, then joined by StudentID and AcademicYear.The binary indicator recorded whether a student had at least one semester log entry, while MoodleHits counted distinct dates with activity.
- Derived features: Degree-2 polynomial terms were added for MoodleHits, CourseWorkload, and M, alongside M_times_LMS = M×has_lms_record to model nonlinear and conditional effects.The interaction represented the effect of academic momentum depending on whether any LMS activity was present.
- Feature transformation: Nominal categorical variables were one-hot encoded with drop_first=True, whereas ordinal variables remained numeric codes.This encoding avoided multicollinearity for nominal features such as Gender and MinorDescription.
- Feature transformation: 47 columns remained after imputation, encoding, polynomial expansion, and standardisation, following removal of identifiers, qualitative-only text fields, and FinalExamination.The resulting matrix differed from the original 69 raw columns after excluding non-predictive or target-related fields.
3.5 Class Imbalance and Resampling
The dataset was imbalanced, with 34.5% failing students versus 65.5% passing. SMOTE+ENN was applied within stratified cross-validation to address imbalance and reduce overfitting compared with SMOTE alone.
- 3.5 Class Imbalance and Resampling: 65.5% of students passed, while 34.5% failed, creating a majority-class imbalance.The dataset contained 186 passing and 98 failing students.
- 3.5 Class Imbalance and Resampling: SMOTE+ENN was applied inside a 5-fold stratified cross-validation loop to prevent majority-class prediction.SMOTE generated minority-class examples, while ENN removed noisy samples from both classes.
- 3.5 Class Imbalance and Resampling: SMOTE+ENN reduced overfitting compared with SMOTE alone by combining minority oversampling with noisy-sample removal.SMOTE creates synthetic minority examples, and ENN removes noisy samples from both classes.
3.6 Model Training and Hyperparameter Tuning · 3.7 Threshold Tuning · 3.8 Model Interpretability with SHAP
The study compared seven tuned classifiers using resampled training folds and original test folds, then adjusted decision thresholds to prioritize failing-student recall. SHAP surrogate analysis decomposed predictions into feature contributions and examined interactions for actionable interpretation.
- 3.6 Model Training and Hyperparameter Tuning: Seven classifiers were evaluated: LR, RF, XGB, LGB, RBF-SVC, a stacking ensemble, and a shallow ANN.The ANN used two hidden layers with 64 and 32 neurons, ReLU activation, dropout 0.4, and Adam optimisation.
- 3.6 Model Training and Hyperparameter Tuning: Hyperparameters were tuned with RandomizedSearchCV using 3-fold cross-validation, with search spaces reported in Table 4.
- 3.6 Model Training and Hyperparameter Tuning: Models were trained on SMOTE+ENN-resampled training folds, evaluated on original test folds, and scored using metrics averaged over 5 folds.
- 3.7 Threshold Tuning: Thresholds from 0.1 to 0.9 were tested in 0.05 steps, selecting the value maximizing macro F1 on validation data; logistic regression’s optimum was 0.65.
- 3.8 Model Interpretability with SHAP: SHAP explanations used XGBoost as a surrogate to decompose each prediction into additive feature contributions.The decomposition satisfies local accuracy, missingness, and consistency.
- 3.8 Model Interpretability with SHAP: Summary and dependence plots were generated to identify feature interactions and provide actionable insights for instructors.
3.9 Experimental Evaluation and Ablation Study
The study evaluated failing-student prediction with standard binary-classification metrics, prioritising recall for the failing class. A logistic-regression ablation study tested whether stakeholder-aligned feature groups improved performance beyond a weighted academic-momentum baseline.
- Evaluation Metrics: Recall for the failing class was the primary optimisation target, while precision and false positive rate captured intervention trade-offs.Recall measures actual failing students correctly identified; precision measures flagged students who fail, and false positive rate measures passing students incorrectly flagged.
- Evaluation Protocol: All results used a 30% hold-out test set after 5-fold stratified cross-validation on the training set.The ablation retained the same 70/30 train/test split and applied SMOTE+ENN only to training folds.
- Ablation Design: The ablation trained logistic-regression models on nested feature subsets to quantify contributions beyond the formative-assessment baseline.Logistic regression was selected as the best-performing classifier in preliminary experiments, and each feature set was evaluated using accuracy, macro F1, and AUC.
- Feature Sets: The Base feature set contained only weighted academic momentum M, while +Demo+LMS added gender, sponsorship, COVID-19 cohort, and LMS activity markers.The expanded set tested the effects of engagement and basic demographics relative to the purely formative-assessment baseline.
- Feature Sets: Additional ablation groups included course-registration features testing whether programme workload affects performance.These features covered course workload, minor classification, and minor programme.
4 Results
Results show substantial course failure and declining LMS engagement, especially during COVID-19 cohorts. The best-performing early-warning model used weighted academic momentum, demographics, and LMS markers, with logistic regression achieving high recall and strong held-out performance.
- Course outcomes: 39.7% was the average overall failure rate across cohorts, with the 2019/20 COVID-19 cohort showing increased NE and D grades.Failure comprised D+, D, and NE grades.
- Stakeholder factors: Time Management, Motivation, and Lack of Computer received the highest combined Agree/Strongly agree ratings among ten performance factors.Assessment Structure was the only factor rated not influential by a majority.
- LMS engagement: The lowest Moodle-engagement cluster predominantly failed, while a small high-hit cluster also tended to fail, indicating a U-shaped engagement–performance relationship.K-means used k=5 and achieved a silhouette score of 0.32.
- Feature ablation: The best cross-validated feature set, Base + Demo + LMS, achieved AUC = 0.7399 and accuracy = 69.25%.It combined weighted academic momentum with gender, sponsorship, COVID-19 cohort flag, has_lms_record, and MoodleHits; additional registration, survey, and interaction groups did not improve metrics.
- Classifier comparison: 74.7% accuracy, 0.742 macro F1, and AUC = 0.800 were achieved by logistic regression on the held-out test set using Base + Demo + LMS.SVC matched accuracy and macro F1 but had a lower AUC of 0.731, while tree-based models underperformed.
- Threshold analysis: 0.87 recall and approximately 0.59 precision were obtained at threshold 0.5; lowering the threshold to 0.4 would raise recall to about 0.92 while reducing precision.The default threshold was recommended because it maximised overall accuracy and AUC, despite the precision–recall trade-off.
5 Discussion
The findings support high-recall, interpretable early-warning models based on weighted academic momentum, demographics, and LMS activity. However, added survey or interaction features did not improve performance, and generalisability remains limited by the dataset and course context.
- Interpretation: 87% recall by week 5 supports using early assessment scores and behavioural traces as digital markers for identifying CS1 failure.The discussion contrasts these markers with static demographics and links the result to first-level engagement data.
- Interpretation: Students with strong initial performance but zero Moodle activity remain at risk, indicating a potentially unproductive form of engagement.This interaction is interpreted as possible overconfidence or unproductive engagement.
- Interpretability: SHAP analysis improves transparency by showing how predictors contribute, addressing stakeholder demands for interpretable educational models.The study extends prior combinations of admission scores and early grades with a theoretically motivated momentum score and its engagement relationship.
- Feature ablation: The ‘Base + Demo + LMS’ feature set achieved AUC = 0.7399 and accuracy = 69.25% in cross-validation.It combined weighted academic momentum, gender, sponsorship, COVID-19 cohort, and LMS engagement; registration and survey features did not improve metrics.
- Feature ablation: AUC fell to 0.6813 when interaction terms were added, indicating that main effects were sufficient for this sample.The discussion suggests larger datasets may be needed to reassess interaction effects.
- Limitations: Generalisability is unknown because the data come from one institution and one CS1 course, with COVID-19 confounders, N = 284, and an institution-specific failure definition.The discussion also notes that the modest sample size explains poor ANN performance.
6 Conclusion and Future Work
The study concludes that a simple, interpretable model combining weighted academic momentum, demographics, and LMS activity can identify failing CS1 students with high recall by week 5. Future work will test temporal and cross-context stability, deploy live warnings, enrich behavioural signals, and develop prescriptive interventions.
- Conclusion: The ‘Base + Demo + LMS’ feature subset achieved the best cross-validated training performance, with AUC = 0.7399 and accuracy = 69.25%.It combined weighted academic momentum, gender, sponsorship, COVID-19 cohort, and LMS engagement; adding registration, survey, or interaction features did not improve metrics.
- Conclusion: 74.7% accuracy, 0.742 macro F1, and an AUC of 0.800 were achieved on the held-out test set.At threshold 0.5, recall = 0.87 and the false positive rate was 41%; SHAP ranked weighted academic momentum highest, followed by its interaction with LMS activity.
- Conclusion: The resulting logistic regression model offers a practical early-warning approach using early assessment scores, basic demographics, and a binary LMS engagement flag.The paper reports high recall (87%) by week 5 and actionable SHAP explanations for instructors.
- Future work: Validation on the unseen 2022/23 cohort and replication across other universities and courses will assess temporal stability and generalisability.Planned contexts include Software Engineering, Computer Graphics and Visual Computing, Humanities, and Social Sciences courses.
- Future work: A Moodle REST API dashboard is planned to compute M after the first test, flag at-risk students, and display SHAP explanations to instructors.A pilot deployment is planned for the next academic year.
- Future work: Future extensions will extract weekly activity, activity decay, dwell time, richer survey semantics, counterfactual recommendations, personalised study plans, and scalable imbalance-handling methods.Candidate methods include LSTMs, BERT, cost-sensitive learning, and focal loss as datasets expand.
Funding
The research received no specific grant funding from public, commercial, or not-for-profit funding agencies.
- No specific grant was received from any public, commercial, or not-for-profit funding agency.