PaperlayerView exact revision on arXiv

Source-linked AI summary

Next-Term Student Performance Prediction: A Recommender Systems Approach

Mack Sweeney, Huzefa Rangwala, Jaime Lester, Aditya Johri

arXiv:1604.01840v1cs.CYcs.IR

TL;DR

The paper addresses the limited understanding of semester-to-semester interventions that support student course selection and retention. It predicts next-term grades from historical transcripts and student, course, and instructor information, finding that a hybrid FM-RF method performs best across cold-start settings.

  • Problem

    Semester-to-semester interventions supporting course selection and student retention remain insufficiently understood.

  • Method

    The paper applies recommender-system methods to predict next-term course grades from historical transcript data and student, course, and instructor features.

  • Results

    The hybrid FM-RF method outperforms the individual methods and predicts grades for new and returning students taking new and existing courses.

  • Takeaways & Limitations

    The techniques hold promise for degree planning, instructor interventions, and personalized advising aimed at improving retention and academic performance.

  • Takeaways & Limitations

    The FM-RF hybrid may be outperformed or made computationally cheaper by more sophisticated methods for cold-start matrix-factorization prediction.

Abstract

from arXiv · show

An enduring issue in higher education is student retention to successful graduation. National statistics indicate that most higher education institutions have four-year degree completion rates around 50 percent, or just half of their student populations. While there are prediction models which illuminate what factors assist with college student success, interventions that support course selections on a semester-to-semester basis have yet to be deeply understood. To further this goal, we develop a system to predict students' grades in the courses they will enroll in during the next enrollment term by learning patterns from historical transcript data coupled with additional information about students, courses and the instructors teaching them. We explore a variety of classic and state-of-the-art techniques which have proven effective for recommendation tasks in the e-commerce domain. In our experiments, Factorization Machines (FM), Random Forests (RF), and the Personalized Multi-Linear Regression model achieve the lowest prediction error. Application of a novel feature selection technique is key to the predictive success and interpretability of the FM. By comparing feature importance across populations and across models, we uncover strong connections between instructor characteristics and student performance. We also discover key differences between transfer and non-transfer students. Ultimately we find that a hybrid FM-RF method can be used to accurately predict grades for both new and returning students taking both new and existing courses. Application of these techniques holds promise for student degree planning, instructor interventions, and personalized advising, all of which could improve retention and academic performance.

1. INTRODUCTION

The paper frames next-term grade prediction as a recommender-systems problem using historical transcripts and content features to support retention. It evaluates hybrid approaches for cold-start prediction and identifies a hybrid FM-RF system as the strongest overall method.

  • Next-term grade predictions can support course planning, advising, and early identification of students needing assistance.
  • The system predicts next-term course grades from historical transcript data plus student, course, and instructor information.
  • The study blends collaborative-filtering and content-based recommender ideas while incorporating transfer-student data and feature-importance analysis.
  • FM, RF, and PMLR achieve the lowest prediction error, while selected content features improve performance in cold-start scenarios.
  • The hybrid FM-RF method outperforms the three individual methods by combining their strengths to address FM cold-start limitations.

2. LITERATURE REVIEW

The literature review situates this study within recommender systems, student modeling, and university grade-prediction research. It emphasizes the paper’s focus on traditional university settings and content-enhanced methods.

  • Prior recommendation research includes nearest neighbors, matrix factorization, restricted Boltzmann machines, and topic modeling, largely in e-commerce.
  • Much recent educational prediction work targets MOOCs and intelligent tutoring systems, whereas this study predicts grades in a traditional university environment.
  • Related intelligent-tutoring research uses matrix factorization and tensor-factorization methods, including models for higher-order interactions that this study leaves for future work.
  • Other related approaches include multi-relational matrix factorization and text-derived features for predicting university performance.
  • The present work extends prior next-term grade prediction by adding content features and blending collaborative-filtering and content-based modeling.

3. PROBLEM FORMULATION

The task predicts grades for relevant next-term student-course pairs using a sparse grade matrix and associated content features. Models train on prior terms and predict selected courses in the current term.

  • The goal is to predict each student’s next-term grade for relevant student-course dyads using student, course, and instructor features.
  • The grade data form an n×m sparse matrix with values from 0 to 4, providing the primary input for matrix-factorization methods.
  • Each dyad is represented as a feature vector containing one-hot student and course identifiers plus content features.
  • Models train on feature vectors from preceding terms and predict grades for courses students select in the current term rather than completing the entire matrix.

4. DATASET DESCRIPTION

The dataset covers 15 academic terms at a public university and combines anonymized student, course, instructor, and grade records. The evaluation distinguishes regression targets and several cold-start conditions.

  • The data span 15 terms from Summer 2009 through Spring 2014 at a public university with 30,754 students, 144 majors, and 13 colleges.
  • After filtering, the dataset contains 894,736 student-course dyads, including 584,179 transfer and 310,557 non-transfer records.
  • Because A-F grades are ordered values with nominal 0–4 equivalents, the study formulates prediction as regression rather than classification.
  • Student records include demographics, academic measures, majors, and prior-term information, while course records include discipline, credits, level, enrollment, and GPA features.
  • Cold-start records contain a new student, a new course, or both relative to the training terms; the dataset is evenly split between cold-start and non-cold-start dyads.

5. METHODS

The paper evaluates baselines, matrix-factorization methods, and regression models for next-term grade prediction, including approaches that address sparsity and cold-start cases.

  • 5. METHODS: Factorization Machines, Random Forests, and Personalized Multi-Linear Regression form the principal model families evaluated for next-term grade prediction.The regression group includes SGD, kNN, and PMLR; the matrix-factorization group includes SVD, SVD-kNN, and FM.
  • 5. METHODS: Mean-based baselines progress from random guessing to global, student-level, and course-level central tendencies.Uniform Random samples grades from [0, 4], Global Mean uses all observed grades, and Mean of Means averages global, per-student, and per-course means.
  • 5.2. MATRIX FACTORIZATION METHODS: Matrix-factorization models learn latent course and student vectors whose interactions capture grade-related structure in a reduced feature space.The evaluated methods are SVD, SVD-kNN, and FM; SVD-kNN post-processes predictions using similarity in latent space.
  • 5.2. MATRIX FACTORIZATION METHODS: Factorization Machines combine central tendencies, bias terms, and factorized interactions while incorporating arbitrary content features for cold-start prediction.Unlike pure collaborative filtering, FM can use student, course, and other content features alongside the sparse grade matrix.
  • 5.3. REGRESSION MODELS: Random Forest regression averages trees built from bootstrap samples and random feature subsets, with maximum depth used to limit overfitting.Each tree predicts from the mean grade at the reached leaf, while ensemble averaging reduces variance.
  • 5.3.1. REGRESSION MODELS: SGD regression fits least squares with L1 regularization, encouraging sparse parameters and functioning as online feature selection.Incremental parameter updates reduce overfitting and improve training time relative to optimization over the entire training set.
  • 5.3.4. Personalized Multi-Linear Regression: PMLR predicts grades with student and course biases plus a student-specific weighted combination of regression models over dyad features.Pi contains per-student model weights, W contains regression coefficients, and RMSE is used as the loss function.

6. EXPERIMENTAL RESULTS AND DISCUSSION

Across cold-start, non-cold-start, native, and transfer settings, FM, RF, and PMLR provide the strongest predictions, with a hybrid FM-RF system achieving the best overall RMSE. Feature analyses highlight the importance of bias terms, instructor characteristics, and population-specific patterns, while covariate shift limits FM performance for cold-start records.

  • Feature importance: MADImp addresses FM’s poor feature-selection capability and identifies bias terms, course discipline, major, race, and instructor rank as especially informative.Less important but still informative features include cohort, instructor class and tenure, student term, transfer indicator, and sex.
  • Grade prediction results: For non-cold-start records, FM outperforms all other methods by a wide margin, whereas cold-start performance improves for every method except MoM.FM avoids the overconfident patterns affecting MoM through Bayesian complexity control and regularization.
  • Grade prediction results: FM produces the lowest-error predictions with content features overall, while RF is best for cold-start records and PMLR follows FM in the broader comparison.For non-cold-start records, content-enabled FM, PMLR, and RF outperform MoM; content-enabled FM improves over FM without content features only in this setting.
  • Grade prediction results: Transfer students are predicted more accurately than native students, with FM best for native students and RF best for transfer students.The gap narrows when content features are included; 42.62% of transfer dyads are cold-start compared with 7.37% of native dyads.
  • Grade prediction results: 0.7443 overall RMSE is achieved by swapping RF for FM when prior student information is absent, outperforming FM at 0.7709 and RF at 0.7775.This hybrid combines FM for records with prior student information and RF when that information is missing.
  • Feature importance: The top three FM features account for 97.3% of importance: student bias, course bias, and instructor bias, with roughly two thirds arising from two-way interactions.Course discipline and major are the only non-bias features with notable importance, accounting for 1.1% and 0.8%.
  • Feature importance: PMLR feature importance shifts across terms, with instructor attributes, student enrollment history, and course discipline prominent later, while last-term GPA gains importance.The first term shows the clearest shift, followed by less drastic changes in subsequent terms.
  • Feature importance: PMLR assigns nearly 60% of overall bias to course bias for native students but none for transfer students, producing different importance patterns across populations.The transfer population instead receives larger weights for other features, including enrollment duration, instructor classification, and student and instructor bias terms.

7. DISCUSSION

The system’s predictions support degree planning, early-warning interventions, and personalized advising, while cold-start prediction remains an important limitation. The authors identify hybrid modeling and distribution shift as central to addressing these challenges.

  • 7.1. PREDICTIVE PERFORMANCE: The FM-RF hybrid is the most effective method overall and overcomes FM cold-start limitations.The authors also note that attribute-to-latent-feature mappings may offer better or cheaper alternatives.
  • 7.1. PREDICTIVE PERFORMANCE: Covariate shift can cause FMs to learn overconfident 2-way interactions that reduce cold-start generalization.RFs avoid this problem but cannot capture the same 2-way interactions.
  • 7.2. BENEFICIAL APPLICATIONS: Predicted grades can support degree planning by comparing alternative semester course selections.The system is intended to help students choose suitable majors and balance course difficulty.
  • 7.2. BENEFICIAL APPLICATIONS: Expected low grades can help educators identify at-risk students and proactively provide support.The system can serve as a component of an early-warning system.
  • 7.2. BENEFICIAL APPLICATIONS: Personalized predictions can help advisors tailor recommendations when students may struggle with course material or teaching style.This extends beyond generic assumptions about course difficulty.

8. CONCLUSIONS AND FUTURE WORK

The paper concludes that a MADImp-selected FM-RF hybrid predicts next-term grades across new and returning students and new and existing courses. Future work targets cold-start modeling, live deployment, and evaluation of effects on decision-making and retention.

  • 8. CONCLUSIONS AND FUTURE WORK: The MADImp-selected FM-RF hybrid predicts grades for new and returning students taking new and existing courses.It significantly outperforms random guessing and noticeably outperforms the other tested regression models.
  • 8. CONCLUSIONS AND FUTURE WORK: Tensor factorization, multi-relational matrix factorization, and customized tree-based methods are identified as promising future directions.These methods could leverage instructor features or provide interpretable decision rules.
  • 8. CONCLUSIONS AND FUTURE WORK: Cold-start is a limiting factor in matrix-factorization applications, requiring complementary methods such as combining FMs with RFs.The authors also propose attribute-to-latent-feature mappings and methods handling covariate shift while learning 2-way interactions.
  • 8. CONCLUSIONS AND FUTURE WORK: The authors plan live deployment in degree-planning and early-warning systems after improving prediction of failing and passing grades.They also plan A/B testing of performance, user decision-making, retention, and learning outcomes.

APPENDIX I: FEATURE DESCRIPTIONS

The appendix defines student, course, and instructor variables used to represent historical performance, demographics, course context, and teaching characteristics. These features include identifiers, academic measures, enrollment statistics, and instructor attributes.

  • APPENDIX I: FEATURE DESCRIPTIONS: Student features include identifiers, prior GPA measures, credit-hour totals, academic level, term sequence, demographics, major, SAT score, and high-school code.Academic level is derived by binning total credit hours.
  • APPENDIX I: FEATURE DESCRIPTIONS: Student demographic and background variables include major, race, sex, age, ZIP code, SAT score, and high-school CEEB code.Race, sex, and SAT may be unspecified or unavailable.
  • APPENDIX I: FEATURE DESCRIPTIONS: Term sequence records a student’s chronological enrollment position, with the first term numbered 0.This feature is defined relative to each student rather than globally.
  • APPENDIX I: FEATURE DESCRIPTIONS: Course features include identifiers, discipline, credit hours, level, offering term, enrollment counts, and aggregate prior course GPAs.The aggregate GPA measures cover the previous term and all prior offerings.
  • APPENDIX I: FEATURE DESCRIPTIONS: Instructor features include unique identifiers, classification, academic rank, and tenure status.Instructor identifiers are used to learn instructor bias terms in training data.

APPENDIX II: MODEL PARAMETER SETTINGS

The appendix reports parameter settings for the Factorization Machine, Personalized Multi-Linear Regression, Random Forest, Boosted Decision Trees, SGD Regression, and OLS models.

  • APPENDIX II: MODEL PARAMETER SETTINGS: The Factorization Machine uses rank 8, 200 iterations, and initial standard deviation 0.2.
  • APPENDIX II: MODEL PARAMETER SETTINGS: Personalized Multi-Linear Regression uses 4 models, regularization 0.01 on P and W, λB = 0.5, and learning rate 0.001.
  • APPENDIX II: MODEL PARAMETER SETTINGS: Random Forest uses 100 trees with maximum depth 10, while Boosted Decision Trees use 100 trees with maximum depth 11.
  • APPENDIX II: MODEL PARAMETER SETTINGS: SGD Regression uses learning rate 0.001, regularization 0.001, and 15 iterations; OLS has no parameters.

APPENDIX III: MADIMP EXAMPLE

The appendix illustrates Mean Absolute Deviation Importance (MADImp) using a Factorization Machine example with one-hot-encoded user, item, and season features. It then uses hypothetical parameters to show how a prediction and feature importance calculation are constructed.

  • MADImp is illustrated with a Factorization Machine, although the method can also be used with generalized linear models.
  • The example predicts a user’s response to an item in a particular season using one-hot-encoded user, item, and season features.
  • The hypothetical design contains 34 encoded features: 10 users, 20 items, 4 seasons, and a global-intercept feature.
  • The assigned parameters represent global, user, item, seasonal, and user-item-season response tendencies.
  • The example calculates a Summer prediction for a user-item combination and then computes the importance of the three active dyad features using Equation (10).
Loading 1604.01840v1…