PaperlayerView exact revision on arXiv

Source-linked AI summary

Likely to stop? Predicting Stopout in Massive Open Online Courses

Colin Taylor, Kalyan Veeramachaneni, Una-May O'Reilly

arXiv:1408.3382v1cs.CYcs.LG

TL;DR

This paper asks whether MOOC persistence can be predicted from limited early-course data and develops an end-to-end stopout-prediction methodology for Fall 2012 6.002x. Stopout prediction was tractable, reaching AUCs as high as 0.95 one week ahead and about 0.7 for end-of-course prediction using only one week of data.

  • Problem

    The paper asks whether machine-learning algorithms can accurately predict MOOC persistence using only a few weeks of course data, including whether first-week data can predict completion.

  • Method

    The study analyzes 6.002x interaction data, engineers complex predictive features, represents temporal and non-temporal data, and evaluates many machine-learning models for stopout prediction.

  • Results

    AUC reached 0.95 for one-week-ahead prediction and approximately 0.7 for predicting end-of-course stopout from only one week of data.

  • Takeaways & Limitations

    Stopout prediction is tractable, and early predictors of stopout exist.

  • Takeaways & Limitations

    Only 132.3 million navigational events were sufficiently well formed to be reliably included in the compendium.

Abstract

from arXiv · show

Understanding why students stopout will help in understanding how students learn in MOOCs. In this report, part of a 3 unit compendium, we describe how we build accurate predictive models of MOOC student stopout. We document a scalable, stopout prediction methodology, end to end, from raw source data to model analysis. We attempted to predict stopout for the Fall 2012 offering of 6.002x. This involved the meticulous and crowd-sourced engineering of over 25 predictive features extracted for thousands of students, the creation of temporal and non-temporal data representations for use in predictive modeling, the derivation of over 10 thousand models with a variety of state-of-the-art machine learning techniques and the analysis of feature importance by examining over 70000 models. We found that stop out prediction is a tractable problem. Our models achieved an AUC (receiver operating characteristic area-under-the-curve) as high as 0.95 (and generally 0.88) when predicting one week in advance. Even with more difficult prediction problems, such as predicting stop out at the end of the course with only one weeks' data, the models attained AUCs of 0.7.

1. Introduction

The paper frames MOOC stopout as a large-scale learning problem and develops an end-to-end predictive methodology for 6.002x. It combines interpretive features, temporal representations, and machine-learning models to predict persistence from limited course history.

  • 1. Introduction: 94% of registrants did not make it past the midterm, motivating analysis of why students stopout in MOOCs.The paper reports that only 6% passed the midterm and fewer than 5% earned certificates.
  • 1. Introduction: The paper asks whether a few weeks of course data can predict persistence, including completion by the final week and the history needed for accurate forecasts.
  • 1.2. Our contributions through this paper: The compendium develops an end-to-end stopout prediction methodology spanning raw data preparation, feature engineering, model evaluation, and outcome analysis.
  • 1.2. Our contributions through this paper: The study extracts 27 interpretive features, creates temporal and non-temporal feature sets, and evaluates more than 10,000 predictive models.

2. Data organization into MOOCdb

The study consolidates heterogeneous 6.002x learning records into MOOCdb, a normalized schema designed for scalable analysis and reusable feature extraction. The source data include clickstreams, forum and Wiki activity, learner state, and course-calendar information.

  • 2. Data organization into MOOCdb: The 6.002x dataset contains clickstream logs, forum and Wiki records, learner-state data, and an XML course calendar.
  • 2. Data organization into MOOCdb: 154,763 registered learners and 17.8 million submission events were included among the reported raw data.
  • 2. Data organization into MOOCdb: Only 132.3 million navigational events were considered reliably usable because the received data also contained additional malformed events.
  • 2. Data organization into MOOCdb: MOOCdb organizes the raw records into a standardized schema intended to capture pertinent information across MOOC platforms and support reusable analysis scripts.
  • 2. Data organization into MOOCdb: Schema normalization reduced approximately 70GB of raw data to approximately 7GB, enabling the entire database to be loaded into RAM for prompt queries and feature extraction.

3. Prediction problem definition and assumptions

The paper defines stopout and prediction windows around 6.002x’s weekly course structure, then forms many learner-level prediction problems from historical features. It also analyzes separate collaboration-based learner cohorts using 27 interpretive features.

  • 3.3. Lead and Lag: Weekly modules provide the time slices, while lead denotes prediction distance and lag denotes the number of historical weeks used as features.
  • 3.2. Stopout definition: Stopout is defined as the first week after a learner’s last assignment or exercise submission, excluding passive course-page access.
  • 3.2. Stopout definition: 52,683 of 105,622 learners who accessed the course stopped out in week one and were excluded because they never submitted an assignment.
  • 3.3. Lead and Lag: Under the definition, 91 independent prediction problems are formed across future weeks and historical-data lengths.
  • 3.4. Cohorts: Learners are divided into passive, Wiki-contributor, forum-contributor, and fully collaborative cohorts based on forum and Wiki participation.
  • 3.5. Features: The models use 27 interpretive per-learner features to represent behavior for prediction.

4. Logistic Regression

The paper represents learner behavior with feature covariates and applies logistic regression to estimate binary stopout probabilities. It evaluates predictions with ROC AUC across varying prediction horizons and historical-data windows.

  • 4. Logistic Regression: Logistic regression combines feature covariates through a weighted sum and maps the result to a probability between 0 and 1.The model uses coefficients for feature values and a sigmoid/logit function for binary classification.
  • 4. Logistic Regression: Training estimates coefficients from covariates and binary labels, iteratively optimizing them with maximum likelihood estimation.The fitted model should assign higher probabilities to positive examples and lower probabilities to negative examples.
  • 4. Logistic Regression: The feature matrix records each feature value for each learner and week, while flattened temporal representations assemble learner-week values as separate model variables.The paper uses these representations to support prediction problems with different lead and lag settings.
  • 4. Logistic Regression: Model evaluation applies the learned logistic function and a decision rule to test data, then uses ROC curves and AUC to summarize performance.The decision rule produces class labels from estimated probabilities, enabling confusion-matrix and ROC analysis.
  • 4. Logistic Regression: Heat maps organize predictive performance by target week and lag, with color representing ROC AUC; later target weeks and longer lags generally make prediction harder.The heat-map layout summarizes multiple prediction problems simultaneously.

5. Predicting stopout with logistic regression

The study applies logistic regression to 27 interpretive features across cohorts and lead-lag prediction problems. Stopout prediction is generally strong, including substantially ahead-of-time predictions, but performance varies with cohort size and the availability of midterm data.

  • 5. Predicting stopout with logistic regression: AUC reached 0.95 for fully collaborative students predicting week 8 with a lag of 7, while every lead-one passive-collaborator experiment exceeded 0.88.These results show high predictive accuracy across multiple lead and lag settings, not just one experiment.
  • 5. Predicting stopout with logistic regression: Passive collaborators achieved the highest overall accuracies, whereas wiki contributors often performed poorly because small cohorts lacked enough examples for training or AUC computation.Cohort size is explicitly linked to stability and evaluability of the predictions.
  • 5. Predicting stopout with logistic regression: Using only one week of data to predict course completion produced AUCs from 0.64 across cohorts to 0.78 for wiki contributors.The result indicates that early behavioral data contains predictive information about later persistence.
  • 5. Predicting stopout with logistic regression: Including midterm data raised passive-collaborator AUC from 0.75 to 0.91 and fully collaborative AUC from 0.68 to 0.81 for next-week prediction.The paper identifies the midterm as a major prediction milestone.
  • 5. Predicting stopout with logistic regression: Wiki participation reflects high engagement and is associated with strong persistence signals, suggesting that similarly engaging technologies may also inform prediction.The paper contrasts this engagement pattern with the smaller wiki-contributor cohort’s data limitations.

6. Multiple classifiers

The study compares multiple classifiers through the Delphi machine-learning service on selected difficult prediction problems. Model choice mattered little for larger cohorts but substantially improved results for smaller cohorts.

  • 6. Multiple classifiers: Delphi automatically searched across algorithms and parameters using a hybrid Bayesian and multi-armed-bandit optimization system.The service returned optimized models and evaluation outputs for selected lead-lag datasets.
  • 6. Multiple classifiers: For larger cohorts, Delphi models were within 0.02 AUC of logistic regression, indicating that predictive strength primarily came from the engineered features.The tested algorithms included logistic regression, stochastic gradient descent, support vector machines, and random forests.
  • 6. Multiple classifiers: For the smaller wiki and fully collaborative cohorts, Delphi produced substantially better accuracy, with every wiki-contributor setting exceeding 0.85 AUC.The strongest classifiers included k-nearest neighbors and stochastic gradient descent.
  • 6. Multiple classifiers: The paper concludes that when a cohort is small relative to its number of features, multiple models should be investigated to identify the most accurate classifier.This recommendation follows the stronger model dependence observed in the smaller cohorts.

7. Related work and literature

Prior MOOC persistence studies commonly used limited historical intervals, performance-related variables, surveys, or aggregated behavioral measures. This work emphasizes granular trace data, longitudinal feature engineering, and systematically varied prediction problems evaluated with AUC.

  • Predictive modeling: 91 predictive modeling problems span different historical lags and forecasting horizons within one MOOC, enabling week-by-week prediction through the course.Variables are restricted to information available before each prediction point.
  • Evaluation: The authors use AUC rather than R2 because stopout prediction involves both stopout and persistence errors and supports intervention threshold selection.AUC also allows an intervention designer to choose a trade-off point on the receiver operating characteristic curve.
  • Data and variables: The study prioritizes behavioral trace data over demographic, perceptual, and prior academic attributes that may be unavailable in MOOCs.Its models use learner interactions with resources, assignments, forums, and other platform activity.
  • Data and variables: Unlike much prior work using single-interval summaries, the approach operationalizes variables at multiple time points to capture longitudinal behavior.Examples include time-varying downloads, video minutes, and weekly participation measures.
  • Contribution: Feature engineering from granular clickstream data produced AUCs of 0.88-0.90 one week ahead for the passive collaborator cohort.The authors attribute this success more to engineered variables than to the modeling techniques themselves.

8. Summary of Research Findings

The study finds that MOOC stopout is predictably tractable, especially with recent and engagement-oriented features. Complex temporal and peer-relative features generally matter more than model choice, while collaboration signals vary by activity type.

  • Predictive performance: AUC reaches 0.95 one week ahead and approximately 0.7 for end-of-course prediction using only one week of data.The results indicate that early stopout predictors exist even for the more difficult long-horizon task.
  • Predictive performance: The models generally find only the most recent four weeks of data predictive across prediction weeks.
  • Feature importance: Feature engineering contributes more consistently to accuracy than switching among modeling techniques.More informative features yielded superior accuracy across the techniques examined.
  • Feature importance: Problem-submission engagement is the most predictive overall feature family, consistent with the study’s submission-based stopout definition.
  • Feature importance: Complex features using peer-relative comparisons and temporal trends are more predictive than simple submission counts.Examples include student percentiles relative to peers and lab-grade trajectories over time.
  • Collaboration features: Forum post length predicts stopout, whereas post counts, response counts, and Wiki edits provide little or no predictive power.The findings suggest that collaboration mechanism and interaction content matter, not merely participation frequency.

9. General reflections for the entire compendium

The compendium argues that effective MOOC analytics requires systematic feature engineering, large-scale exploration of modeling choices, and scalable infrastructure. It also aims for reusable methods that can extend beyond one course.

  • Feature engineering: Careful data preparation and creative feature engineering are foundational because vague assumptions and weak conditioning undermine later modeling.The process may require manual alignment of course events, such as problem-set deadlines.
  • Machine learning and scale: Large-scale MOOC modeling requires exploring temporal and non-temporal representations, feature definitions, cohorts, and multiple classifier families.The authors argue that considering only a narrow set of choices produces more limited results.
  • Machine learning and scale: Building 10,000 models depended on cloud-scale infrastructure capable of running hundreds of nodes simultaneously.The computational framework supported repeated iterations over data, features, and cohort definitions.
  • Transfer learning prospects: The methodology is designed for reuse because investigating stopout in only one course would limit the impact of the results.The authors envisioned open-source software standardized through the shared MOOCdb schema.
Loading 1408.3382v1…