Source-linked AI summary
Prediction and Localization of Student Engagement in the Wild
Amanjot Kaur, Aamir Mustafa, Love Mehta, Abhinav Dhall
TL;DR
The paper addresses limited public evidence for automatic student-engagement assessment and localization in online-learning settings. It introduces an in-the-wild dataset and applies deep multi-instance learning to weakly supervised engagement prediction and localization. Sequence-based networks using fused head-pose and eye-gaze features perform best among the reported approaches, while fine-grained separation of engagement levels remains difficult.
Problem
Public datasets for student engagement in online learning are limited, although engagement is dynamic and relevant to understanding when learners concentrate or lose interest.
Method
The paper introduces a diverse engagement dataset and formulates video-level prediction and localization as weakly supervised multiple-instance learning using spatio-temporal and motion features.
Results
The proposed sequence network with fused head-pose and eye-gaze features performs better than other reported networks and distinguishes level 0 from levels 1–3, but not levels 2 and 3 reliably.
Takeaways & Limitations
The dataset and localization framework provide benchmarks for studying student engagement in applications such as MOOCs and intelligent tutoring systems.
Takeaways & Limitations
The study uses students from one institute, and the proposed network does not correctly distinguish the fine-grained difference between engagement levels 2 and 3.
Abstract
from arXiv · showhide
In this paper, we introduce a new dataset for student engagement detection and localization. Digital revolution has transformed the traditional teaching procedure and a result analysis of the student engagement in an e-learning environment would facilitate effective task accomplishment and learning. Well known social cues of engagement/disengagement can be inferred from facial expressions, body movements and gaze pattern. In this paper, student's response to various stimuli videos are recorded and important cues are extracted to estimate variations in engagement level. In this paper, we study the association of a subject's behavioral cues with his/her engagement level, as annotated by labelers. We then localize engaging/non-engaging parts in the stimuli videos using a deep multiple instance learning based framework, which can give useful insight into designing Massive Open Online Courses (MOOCs) video material. Recognizing the lack of any publicly available dataset in the domain of user engagement, a new `in the wild' dataset is created to study the subject engagement problem. The dataset contains 195 videos captured from 78 subjects which is about 16.5 hours of recording. We present detailed baseline results using different classifiers ranging from traditional machine learning to deep learning based approaches. The subject independent analysis is performed so that it can be generalized to new users. The problem of engagement prediction is modeled as a weakly supervised learning problem. The dataset is manually annotated by different labelers for four levels of engagement independently and the correlation studies between annotated and predicted labels of videos by different classifiers is reported. This dataset creation is an effort to facilitate research in various e-learning environments such as intelligent tutoring systems, MOOCs, and others.
1 INTRODUCTION
Student engagement is multidimensional and dynamic, yet existing assessment methods and datasets provide limited coverage for automatic localization in diverse online-learning settings. The paper addresses this gap by framing engagement prediction and localization as a weakly supervised deep multi-instance learning problem.
- 1 INTRODUCTION: Existing engagement measures, including self-reports and expert observation, are insufficient across contexts, motivating automatic assessment from facial, postural, and other behavioral cues.Prior work also uses physiological signals, student responses, and facial movements to estimate engagement.
- 1 INTRODUCTION: Engagement has emotional, cognitive, and behavioral dimensions and varies over time, making localization within online lectures practically informative.Temporal analysis can identify when learners concentrate or lose interest.
- 1 INTRODUCTION: Public datasets for student engagement in online learning remain limited, while diverse settings are needed for deep-learning-based assessment.The paper introduces an engagement database for benchmarking under diverse, in-the-wild conditions.
- 1 INTRODUCTION: The study positions its dataset and DMIL baselines as resources for engagement research in e-learning environments such as MOOCs and intelligent tutoring systems.The dataset is intended to support common benchmarking and development of engagement assessment.
- 1 INTRODUCTION: The paper formulates engagement prediction and localization as a multiple-instance learning problem because frequent video-level labeling is expensive and noisy.Deep neural networks learn complex features within the weakly supervised MIL framework.
2 DATA COLLECTION
The dataset records subjects watching educational stimuli in unconstrained environments, combines video responses with verbal feedback, and assigns engagement labels through multiple annotators. The resulting recordings and annotations support engagement prediction and localization.
- 2 DATA COLLECTION: Subjects watched educational videos and then gave 15 seconds of verbal feedback about interest, replay preferences, engaging portions, and improvement suggestions.The stimuli included language learning, faster-learning tips, and research-writing videos.
- 2 DATA COLLECTION: 195 videos from 78 subjects aged 19–27 were collected in unconstrained locations, including computer labs, hostel rooms, open ground, and video-conferencing setups.The recordings use approximately five-minute stimuli videos.
- 2 DATA COLLECTION: The dataset and accompanying baseline code were planned for public release to facilitate open research.The paper presents the collection as one of the first publicly available engagement datasets recorded in diverse conditions.
- 2 DATA COLLECTION: Five annotators rated perceived engagement intensity from facial expressions on a four-level scale from 0 to 3, with audio muted during labeling.The categories were inspired by prior engagement-labeling work.
- 2 DATA COLLECTION: Annotator reliability was assessed with quadratically weighted Cohen’s kappa, and labels below 0.4 were discarded before averaging and rounding the remaining ratings.The processed annotations form the video-level ground truth engagement rating.
3 PROPOSED FRAMEWORK
The framework predicts video-level engagement and localizes engaging or disengaging segments from facial-video cues using weakly supervised deep multiple-instance learning. It extracts temporal features from video segments, ranks segment intensities, and pools them into a video-level score.
- Framework overview: The pipeline detects faces, divides videos into segments, extracts features, and computes engagement prediction and localization with deep MIL pooling.OpenFace-based processing supplies facial landmarks and segment features before mean or top-k pooling produces predictions.
- Feature extraction: LBP-TOP captures spatio-temporal facial changes from either complete videos or shorter sliding-window segments.The shorter-window approach uses k=20 frames, approximately three seconds in the sampled video, with a stride of 10 frames.
- Feature extraction: Eye-gaze and head-pose movements are fused as social-signal features because both provide cues related to students’ attention or interest.The proposed feature representation combines head-pose and eye-gaze information extracted with OpenFace.
- Multiple-instance formulation: Multiple-instance learning represents each video as a bag of instances with one video-level label, enabling prediction of engagement intensity and localization of salient segments.The method targets whole-video prediction while identifying the most engaging or disengaging segments.
- Pooling: Top-k pooling averages the highest-ranked segment intensities, while mean pooling averages instance outputs to obtain the video-level engagement score.The top-k scheme uses the first k ranked instances rather than the single-instance assumption common in binary MIL.
- Sequence model: The LSTM-based architecture processes the segment sequence through recurrent and dense layers before average pooling, reflecting engagement variation across segments.Its design is motivated by segment-dependent annotations and temporal variation in engagement.
4 EXPERIMENTS
The experiments compare conventional and deep approaches for engagement regression under subject-independent evaluation. They use whole-video or segment-level representations, alternative segment relabeling strategies, and deep MIL pooling variants.
- Experimental setup: Subject-independent training and testing are used so the engagement prediction approaches are evaluated on users separate from those used for training.The dataset split and cross-validation are both subject independent.
- Feature representations: The experiments compare full-video LBP-TOP vectors with temporal bags of segment features for engagement prediction.The second representation extracts features from sliding-window segments and organizes them as bags.
- Segment labeling: Segment labels are assigned either uniformly from the corresponding video label or by K-means cluster relabeling using cluster modes or means.The cluster-count hyperparameter is selected from k ∈{5, 10, 15, 20, 25, 30, 35, 40}.
- Traditional regressors: The study trains traditional regressors such as SVR and compares their mean-squared-error performance across segment-relabeling strategies.For SVR, Gaussian-kernel parameters are selected by grid search, with different settings reported for mode and mean relabeling.
- Deep MIL: The Deep MIL approach divides each video into 100 segments, maps each segment through dense layers, and pools segment intensities using Top-k or mean pooling.Top-k pooling averages the largest contributing instance intensities, whereas mean pooling averages intensities across instances.
- LSTM-based Deep MIL: The proposed LSTM-based Deep MIL model represents each segment with a fused nine-dimensional head-pose and eye-gaze feature vector.The features are derived from standard deviations of pose coordinates and gaze points within each segment.
5 RESULTS AND DISCUSSIONS
DMIL improves engagement prediction over baseline approaches and supports segment-level localization, while fused head-pose and eye-gaze features improve correlation with annotations but remain limited in fine-grained class separation.
- Baseline comparison: DMIL outperformed baseline machine-learning approaches for LBPTOP features, although linear regressors were substantially worse than SVR.The authors conclude that a linear function is unsuitable for this problem.
- Classwise evaluation: Dense-only DNNs failed to capture classwise accuracy well, particularly for engagement levels 0 and 1, despite reduced overall mean squared error.The paper therefore evaluates classwise MSE alongside overall MSE and PCC.
- Engagement prediction: PCC increased to 0.37 with fused Head Pose and Eye Gaze features, while mean pooling with ReLU after the LSTM minimized classwise and overall MSE.This configuration outperformed the other tested fused-feature networks.
- Engagement localization: The proposed sequence network differentiated level 0 from levels 1–3, but could not reliably distinguish the finer-grained difference between levels 2 and 3.The paper attributes this limitation to feature-space noise and overlapping distributions.
6 FUTURE WORK
Future work extends engagement modeling with body-movement features and broader participant recruitment to improve generalization beyond the institute-based sample.
- Feature expansion: Future experiments will add optical-flow and body-part movement features to the engagement model.The authors specifically identify body movement as a planned feature vector.
- Participant diversity: Recruiting older participants from outside the institute is planned to provide greater freedom for generalizing the results.The current study used students from different institute departments.
7 CONCLUSION
The paper presents deep multiple-instance learning for predicting and localizing student engagement from behavioral cues. Segment-level predictions are strongest for lower engagement levels, while level 3 remains difficult to predict accurately.
- Deep multiple-instance learning predicts and localizes engagement from behavioral cues, supporting automatic analysis of engaging and non-engaging video segments.The approach addresses engagement assessment and localization in the recorded stimuli videos.
- Level 0 predictions generally fall between 0 and 1, while level 1 averages higher and has a high mean squared error.
- Level 2 predictions are near ground truth, whereas level 3 is not predicted with high accuracy and remains below ground truth on average.The same pattern is shown for examples from videos across engagement levels.