Source-linked AI summary
Tag-Aware Ordinal Sparse Factor Analysis for Learning and Content Analytics
Andrew S. Lan, Christoph Studer, Andrew E. Waters, Richard G. Baraniuk
TL;DR
The paper addresses the limits of binary-response SPARFA for educational data that include partial credit and question tags. It develops Ordinal SPARFA-Tag to model ordinal responses and use tags while discovering additional associations, and reports better missing-response prediction than SPARFA and collaborative filtering.
Problem
SPARFA models binary correct/incorrect responses, limiting its use for ordinal partial-credit data and incomplete question–concept information.
Method
Ordinal SPARFA-Tag jointly models ordinal learner responses and uses question tags as support for estimating question–concept associations.
Results
Ordinal SPARFA-Tag outperforms SPARFA-related methods and state-of-the-art collaborative filtering in predicting missing ordinal learner responses.
Takeaways & Limitations
The framework yields interpretable learner concept knowledge and can support personalized recommendations and discovery of new question–concept associations.
Takeaways & Limitations
User-specified tags may be imprecise or incomplete, so the algorithm must discover unsupported associations and remove predefined ones unexplained by the data.
Abstract
from arXiv · showhide
Machine learning offers novel ways and means to design personalized learning systems wherein each student's educational experience is customized in real time depending on their background, learning goals, and performance to date. SPARse Factor Analysis (SPARFA) is a novel framework for machine learning-based learning analytics, which estimates a learner's knowledge of the concepts underlying a domain, and content analytics, which estimates the relationships among a collection of questions and those concepts. SPARFA jointly learns the associations among the questions and the concepts, learner concept knowledge profiles, and the underlying question difficulties, solely based on the correct/incorrect graded responses of a population of learners to a collection of questions. In this paper, we extend the SPARFA framework significantly to enable: (i) the analysis of graded responses on an ordinal scale (partial credit) rather than a binary scale (correct/incorrect); (ii) the exploitation of tags/labels for questions that partially describe the question{concept associations. The resulting Ordinal SPARFA-Tag framework greatly enhances the interpretability of the estimated concepts. We demonstrate using real educational data that Ordinal SPARFA-Tag outperforms both SPARFA and existing collaborative filtering techniques in predicting missing learner responses.
1. INTRODUCTION
The paper motivates personalized learning analytics as an alternative to one-size-fits-all education and extends SPARFA to use ordinal responses and question tags. Ordinal SPARFA-Tag improves interpretability and outperforms prior SPARFA methods and collaborative filtering for predicting missing ordinal responses.
- Motivation: Personalized learning systems aim to tailor educational experiences to individual learners’ backgrounds, interests, goals, and performance.Existing systems include labor-intensive expert rule systems and more scalable machine-learning systems that mine learner data.
- Motivation: Learning analytics estimates learner understanding, while content analytics organizes educational materials to support targeted feedback.SPARFA provides a statistical and convex-optimization framework for these analytics using latent knowledge components.
- Contributions: Ordinal SPARFA-Tag extends SPARFA by modeling partial-credit responses and exploiting instructor-provided question tags.The framework also discovers concept–question relationships absent from the predefined tags but supported by graded response data.
- Evaluation: Ordinal SPARFA-Tag produces more interpretable question–concept associations and outperforms SPARFA-related methods and collaborative filtering in predicting missing ordinal learner responses.The comparison uses synthetic ground-truth simulations and experiments on real-world educational datasets.
2. STATISTICAL MODEL
Ordinal SPARFA-Tag models graded learner responses through latent concept knowledge, question–concept associations, question difficulty, and quantization, while using tags to structure and interpret those associations. The model addresses an ill-posed ordinal factor-estimation problem under low-dimensionality, sparsity, non-negativity, and oracle-support assumptions.
- Ordinal learner response model: Ordinal responses are modeled by quantizing a latent score formed from question–concept associations, learner knowledge, question difficulty, and Gaussian observation noise.The precision parameter controls the noise reliability, and the quantizer maps real-valued latent scores into ordered response labels.
- Ordinal learner response model: The model represents each question’s concept associations in w_i, each learner’s concept knowledge in c_j, and intrinsic question difficulty in µ_i.The observed-response index set allows the response matrix to be incompletely observed.
- Ordinal learner response model: For ordinal responses, the precision parameter τ is estimated from data because it significantly affects model behavior, whereas the original binary SPARFA model fixes P = 2 and τ = 1.Binary SPARFA is recovered as a special case with bin boundaries {−∞, 0, ∞}.
- Fundamental assumptions: Estimating W, µ, and C from ordinal observations is generally ill-posed because there can be more unknowns than observations.The framework therefore relies on structural assumptions about the educational data.
- Fundamental assumptions: The assumptions impose low-dimensionality, sparse question–concept associations, and non-negative associations so positive learner factors represent stronger concept knowledge.Under non-negativity, greater concept knowledge cannot reduce the chance of receiving a good score.
- Fundamental assumptions: Instructor-provided tags supply prior information that can partially or fully determine where nonzero entries occur in W.Tags are used as oracle support within the model, but the estimation algorithm must still discover unsupported associations and remove predefined ones unexplained by the data.
3. ALGORITHM
The paper estimates W, C, and τ from ordinal response data by alternating convex subproblem updates, extending SPARFA with ordinal modeling and tag-informed regularization. Ordinal SPARFA-Tag uses predefined question–concept supports while permitting discovery of additional relationships, and its tri-convex optimization has local-convergence guarantees.
- Model and optimization: Ordinal SPARFA-M and Ordinal SPARFA-Tag estimate W, C, and τ from ordinal response data by maximizing the constrained log-likelihood.The optimization is tri-convex when two of the three variable blocks are held fixed.
- Model and optimization: Block coordinate descent alternates updates of learner knowledge C, question-association rows W, and the precision parameter τ.Each outer iteration optimizes these three blocks in sequence, beginning with C and ending with τ.
- First-order methods: FISTA solves the ordinal regression subproblems for W and C, while the secant method optimizes τ.The W and C updates use gradient and shrinkage or projection steps.
- Evaluation: As problem size increases, estimation errors EW, EC, and Eµ decrease for all considered algorithms.The study varies learner count N and question count Q in the Figure 1 comparisons against K-SVD+ variants.
- Ordinal SPARFA-Tag: Tag information partitions W into predefined-support and non-support entries, applying ℓ2 regularization to the former and ℓ1 regularization to the latter.Reducing λ enables discovery of new question–concept relationships outside the predefined support Γ.
- Convergence: Ordinal SPARFA-Tag is tri-convex and converges to a local optimum from any starting point, while sufficiently close initialization to a global optimum yields convergence to that global optimum.The method does not necessarily converge globally from arbitrary initialization.
4. EXPERIMENTS
The experiments evaluate Ordinal SPARFA-Tag and Ordinal SPARFA-M on synthetic and educational data, showing improved factor estimation, interpretable tag-based concepts, and strong ordinal-response prediction.
- Synthetic data: Estimation errors decrease as the numbers of learners or questions increase, and Ordinal SPARFA-M outperforms K-SVD+ across tested error metrics.The comparison covers errors for question–concept associations, learner concept knowledge, and intrinsic difficulties.
- Synthetic data: Without prior knowledge of precision τ, Ordinal SPARFA-M performs as well as when τ is known, indicating that precision can be learned from data.
- Synthetic data: All error measures decrease by about half with 6 bins compared with 2 bins, while performance approaches K-SVD+ on unquantized data as bins increase.Ordinal SPARFA-M consistently outperforms K-SVD+ in the quantization-bin experiment.
- Real-world data: Instructor-provided tags produce unique concept labels, making learner concept profiles directly interpretable as tag knowledge profiles for targeted feedback.The estimated association matrix also indicates missing and nonexistent tag–question associations for instructors and domain experts.
- Real-world data: In the Earth Science dataset, estimated associations mostly match expert-defined associations, with some nonexistent links identified and no new associations discovered.The dataset contains 145 learners, 80 questions, and only 13.5% observed responses.
- Predicting unobserved learner responses: The nuclear-norm Ordinal SPARFA-M variant outperforms OrdRec, while other variants are comparable to OrdRec and all outperform SVD++.Variants optimizing precision or bin boundaries deliver almost identical performance.
- Predicting unobserved learner responses: Ordinal SPARFA-M combines prediction performance with interpretable factors, a property emphasized as important for educational applications.
5. RELATED WORK
The related work contrasts this framework with Bayesian, item-response, and ordinal factor-analysis approaches that rely on predefined dependencies or features or lack interpretable factors.
- Bayesian belief networks model learner responses and estimate concept knowledge and question difficulty but rely on predefined question–concept dependencies.
- Existing item-response-related models may rely on predefined question features, lack disciplined data-only parameter estimation, or fail to produce interpretable interactions.
- Ordinal factor-analysis methods can predict missing entries, whereas this model adds interpretability through nonnegative sparse learner–concept factors and unique tags per concept.
6. CONCLUSIONS
Ordinal SPARFA-Tag extends SPARFA with ordinal responses and instructor tags while estimating both predefined and newly discovered question–concept associations. It supports interpretable learner feedback and improves prediction of unobserved responses.
- Ordinal SPARFA-Tag exploits ordinal learner responses and instructor-generated tags as support for question–concept associations.
- The framework estimates strengths of predefined question–concept associations while discovering new associations.
- A nuclear norm constraint on the learner concept knowledge matrix improves prediction of unobserved responses over state-of-the-art collaborative filtering techniques.
- The estimated concepts provide readily interpretable feedback and can support personalized remedial or enrichment recommendations.
- Instructors can use the framework to discover new question–concept associations in learning materials.