Source-linked AI summary
Mind the Student: Behavioral and Contextual Cues for Automated Engagement Prediction in Online Learning
Alperen Kantarci, Visvanathan Ramesh, Gemma Roig
TL;DR
Student engagement prediction from online tutoring videos is difficult because engagement is multidimensional, multimodal, variable across people, and subjectively annotated. The paper proposes multimodal fusion with Bayesian participant context and uncertainty-aware regression and classification heads. On CASED, methods perform near chance, while the framework remains competitive and provides calibrated uncertainty analysis.
Problem
CASED engagement prediction is difficult because training data are limited, students vary substantially, and engagement annotations are subjective.
Method
The framework fuses pretrained video, screen, and audio encoders with explicit behavioral features through Perceiver IO and models participant context with Bayesian embeddings and uncertainty-aware heads.
Results
All participating CASED methods perform near random chance, while the proposed system achieves competitive leaderboard performance.
Takeaways & Limitations
The results support evaluating engagement models not only by prediction, but also by uncertainty handling, multimodal structure, and generalization to unseen students.
Takeaways & Limitations
The work is limited by ambiguous labels, identity-discriminative representations, weakly constrained population priors, and privacy and demographic-representation concerns.
Abstract
from arXiv · showhide
The prediction of student engagement from the online tutoring videos is difficult because engagement is a multidimensional construct comprising distinct behavioral, emotional, and cognitive states. A reliable prediction requires bringing together different types of behavioral signals as well as expressive cues. Through our analysis of the CASED dataset, it is clear that engagement prediction gets even harder due to the high inter-person variability as well as the subjectivity of the engagement annotation. To tackle these challenges, we develop a multimodal framework that integrates the implicit spatiotemporal features extracted from pretrained video, audio, and image encoders along with structured behavioral modalities like head pose, gaze, facial action units, emotion, and wavelet-based audio features. We integrate these modalities via a Perceiver IO latent bottleneck. Moreover, student and instructor personalities are modeled as variational posteriors over learnable embeddings to enable partial pooling across participants. We employ evidential regression and spectral-normalized Gaussian process classification heads for uncertainty-aware prediction to further improve robustness and calibration. Benchmark on the CASED challenge test set shows that all participating methods converge near random-chance performance, revealing the difficulty of the dataset. In this highly ambiguous regime, our framework achieves competitive performance while uniquely offering well-calibrated uncertainty metrics, demonstrating that reliable risk-quantification is an essential prerequisite for deploying engagement models in real-world educational tools.
1 Introduction and Background
Student engagement is multidimensional and difficult to predict from online tutoring clips because informative signals span modalities, vary across people, and are subjectively annotated. The proposed framework combines multimodal encoders and behavioral cues with Bayesian participant context and uncertainty-aware prediction.
- Engagement reflects behavioral, emotional, cognitive, and social dimensions associated with learning outcomes, retention, motivation, and dropout.
- Engagement prediction is difficult because cues are distributed across modalities and can become unreliable under occlusion or low resolution.
- CASED prediction is challenged by limited data, inter-person variability, subjective labels, and potentially insufficient signal, motivating uncertainty-aware evaluation.
- Proposed method: The framework combines pretrained encoders with head pose, gaze, facial action units, emotion, and other behavioral features, then fuses them using Perceiver IO.
- Proposed method: Student and instructor effects are modeled with Bayesian variational embeddings and partial pooling toward a shared prior, including learned-prior handling for unseen students.
- Proposed method: Evidential NIG regression and SNGP classification heads jointly produce engagement predictions with aleatoric and epistemic uncertainty estimates.
2 Experiments and Dataset Analysis
Experiments use strict student-independent evaluation and reveal strong class imbalance, compressed engagement scores, and substantial differences between students. Instructor engagement rates are comparatively stable, but student and instructor identities are perfectly collinear.
- Evaluation protocol: Evaluation uses student-independent splits, five student-level cross-validation folds, and CASED test-set leaderboard reporting across classification and regression metrics.Pretraining uses DaiSEE and Aff-Wild2, while CASED is used for fine-tuning.
- Dataset characteristics: Approximately 69% of clips are engaged and 31% are not engaged, while scores have mean 3.7, median 4.0, and a long lower tail.A majority-class classifier achieves approximately 40–45% macro F1 without learning discriminative features.
- Student analysis: Individual students differ substantially in mean engagement and within-session variability, with some engaged in over 90% of clips and others below 40%.These differences indicate strong between-student variation in the labels.
- Instructor analysis: Instructor engagement rates do not vary systematically: all three instructors have mean engagement of 3.37, similar standard deviation, and approximately 68% engaged clips.Each student is paired with exactly one instructor, making student and instructor identity perfectly collinear.
3 Results
On the CASED challenge leaderboard, participating methods perform near the limits imposed by the dataset, while the proposed framework remains competitive. Ablations favor combining student, instructor, and content modalities, and the model’s uncertainty and personalization analyses show strong epistemic dominance and individualized student posteriors.
- Leaderboard results: The best classification method reaches F1 Macro 0.52, while regression submissions show near-zero or negative R2, Pearson correlation, and CCC.The small leaderboard margins and similar metrics indicate broadly comparable performance across participating methods.
- Ablation results: The S+I+C modality combination achieves CCC 0.018 and F1-macro 0.498, the highest reported ablation performance.S, I, and C denote student, instructor, and content inputs.
- Ablation results: The Bayesian Context layer, NIG head, and SNGP head yield the top F1-macro and strong CCC, although the base transformer has lower RMSE and MAE.The component comparison uses the three-modality input and metrics averaged over five student-grouped folds.
- Uncertainty analysis: Total NIG uncertainty is dominated by epistemic uncertainty across engagement bins, with negligible Pearson relationships between engagement scores and either uncertainty component.The bins divide the 1–5 engagement scale into five equal-width groups.
- Bayesian context analysis: Student posterior personalization exceeds instructor personalization, with mean KL 14.32 versus 13.12, and student clip count correlating negatively with KL divergence at r=-0.748, p=0.001.Students with abundant training clips converge closer to the population norm.
4 Limitations and Discussion
The framework remains constrained by weak test performance and several data and architectural factors that obscure engagement dynamics.
- Validation CCC below 0.02 and F1-macro at most 0.52 indicate most configurations do not consistently outperform a constant mean predictor.
- Annotation-boundary ambiguity near the Likert midpoint makes small annotator perturbations flip the binary engagement label.
- Identity-discriminative representations can cause the model to fit student appearance rather than engagement dynamics.
- A weakly constrained population prior does not fully compensate for identity-driven representations.
- Mean-pooling 64 frames under a single clip label suppresses within-clip fluctuations, motivating finer-grained temporal supervision.
5 Conclusion
The paper concludes with an uncertainty-aware multimodal framework combining fusion, Bayesian context modeling, and specialized regression and classification heads. On CASED, it achieves competitive regression performance while providing calibrated epistemic uncertainty estimates.
- The framework combines Perceiver IO fusion, a hierarchical Bayesian context layer, evidential NIG regression, and an SNGP classifier.
- Combining student, instructor, and screen inputs produces the lowest error across all reported metrics in the ablation study.
- The Bayesian context layer learns distinguishable per-student posteriors, while prior regularization varies with data availability.Clip count is strongly negatively correlated with KL divergence from the population prior.
- On the CASED challenge leaderboard, the system achieves competitive regression performance with calibrated epistemic uncertainty estimates at inference time.
- Explicit personality-trait conditioning and cross-session identity tracking are identified as promising future directions.
Safe and Responsible Innovation Statement
The work addresses privacy and demographic-representation concerns associated with analyzing student tutoring videos. It is intended as a research tool rather than a deployed surveillance or performance-evaluation system.
- The study processes student tutoring videos, creating inherent privacy concerns around face analysis and behavioral signal extraction.
- Performance disparities may arise across demographic groups that are under-represented in the relatively small training cohort.
- The system is scoped to understanding engagement dynamics, not surveillance or performance evaluation in deployed educational settings.