Source-linked AI summary
Multimodal Engagement Analysis from Facial Videos in the Classroom
Ömer Sümer, Patricia Goldberg, Sidney D'Mello, Peter Gerjets, Ulrich Trautwein, Enkelejda Kasneci
TL;DR
Automated engagement analysis has focused mainly on computer-based settings, while classroom studies face limited data, inconsistent assessment, and noisy or intrusive measurement conditions. The study uses classroom audiovisual recordings, continuous observer ratings, deep attention and affect embeddings, engagement classifiers, fusion, and person-specific personalization. Average AUC ranged from .560 to .719, and 60 seconds of person-specific data improved performance on average by +.084, with a largest gain of +.124.
Problem
Automated engagement analysis has focused mainly on computer-based settings, while classroom studies face limited data, inconsistent assessment, and noisy or intrusive measurement conditions.
Method
The study uses classroom audiovisual recordings, continuous observer ratings, deep attention and affect embeddings, engagement classifiers, fusion, and person-specific personalization.
Results
Average AUC ranged from .560 to .719, and 60 seconds of person-specific data improved performance on average by +.084, with a largest gain of +.124.
Takeaways & Limitations
Visual classroom analysis can support manual classroom-video analysis for research on teaching quality and teacher training, with personalization improving classifier performance.
Takeaways & Limitations
Limited sample size prevented learning representations directly from engagement data, and models failed to detect low engagement amid highly imbalanced labels.
Abstract
from arXiv · showhide
Student engagement is a key construct for learning and teaching. While most of the literature explored the student engagement analysis on computer-based settings, this paper extends that focus to classroom instruction. To best examine student visual engagement in the classroom, we conducted a study utilizing the audiovisual recordings of classes at a secondary school over one and a half month's time, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement levels from faces in the classroom. We trained deep embeddings for attentional and emotional features, training Attention-Net for head pose estimation and Affect-Net for facial expression recognition. We additionally trained different engagement classifiers, consisting of Support Vector Machines, Random Forest, Multilayer Perceptron, and Long Short-Term Memory, for both features. The best performing engagement classifiers achieved AUCs of .620 and .720 in Grades 8 and 12, respectively. We further investigated fusion strategies and found score-level fusion either improves the engagement classifiers or is on par with the best performing modality. We also investigated the effect of personalization and found that using only 60-seconds of person-specific data selected by margin uncertainty of the base classifier yielded an average AUC improvement of .084. 4.Our main aim with this work is to provide the technical means to facilitate the manual data analysis of classroom videos in research on teaching quality and in the context of teacher training.
1 INTRODUCTION
The study addresses the challenge of estimating student engagement in unconstrained classroom video, where existing self-report and observer-rating approaches have practical limitations. It proposes visual attention and affect representations with engagement classification and personalization for classroom analysis.
- Motivation: Classroom engagement analysis is motivated by limitations of self-reports, observer ratings, and noisy or occluded field-camera recordings.Self-reports depend on student compliance and understanding, while observer ratings require trained raters and reliable labeling.
- Contributions of the Study: Deep representations of head pose and facial expression approximate behavioral and emotional engagement without requiring precise facial alignment.Attention-Net learns attentional features from head pose estimation, while Affect-Net learns affective features from facial expression recognition.
- Contributions of the Study: The study evaluates engagement classifiers on learned attention and affect embeddings, including feature and score-level fusion and person-specific personalization.The evaluation goes beyond person-independent classification because students show intrapersonal variation in engagement.
- Contributions of the Study: The work extends automated engagement analysis from predominantly computer-based settings to large-scale classroom video classification.The authors describe the study as one of the first large-scale video-based classroom engagement classification efforts.
2 RELATED WORK
Prior automated engagement research spans computer-based learning, classroom analytics, and human interactions, but classroom studies remain limited in scale and assessment consistency. The reviewed work motivates visual, sensor-free approaches while highlighting the difficulty of interpreting engagement from classroom behavior alone.
- Related Work Overview: The review categorizes automated engagement analysis by learning situation, nonverbal features, computational methodology, and final objectives, while focusing this study on visual modalities without intrusive sensors.The scope includes classroom, computer-based, and human-human or human-robot interaction domains.
- Learning Analytics in the Classroom: Classroom analytics capture richer student-teacher and peer interactions than computer-based settings, but field-camera video is noisier and less fine-grained.Corner cameras avoid intrusive per-student sensors while introducing noise and occlusion.
- Learning Analytics in the Classroom: Existing classroom studies use varied cues and labels, including head pose, gaze, motion, body pose, facial expressions, self-reports, and observer ratings.Reported approaches include SVMs, bagged decision trees, and systems combining multiple nonverbal features.
- Learning Analytics in the Classroom: Some behavioral proxies are insufficient indicators of engagement because students may attend while looking away, taking notes, or listening to a voice source.This limitation complicates interpreting gaze direction or screen orientation as attention.
- Learning Analytics in the Classroom: Classroom computer-vision studies remain limited by small samples, and no consensus exists for assessing student attention or engagement.Many reviewed systems estimate selected behaviors rather than validated engagement levels.
- Related Work Overview: The literature is concentrated in computer-based learning and interaction settings, whereas classroom research is important but comparatively underdeveloped.Classroom studies are often small, conducted with university courses, or designed mainly to test computer-vision systems.
3 DATA COLLECTION FOR AUTOMATED ENGAGEMENT ESTIMATION IN THE CLASSROOM
The study collected repeated classroom recordings and continuous engagement annotations to support automated estimation from visible student behavior. Data preparation combined multi-camera video, manual rating, face-based preprocessing, and discretized engagement labels.
- 3.1 Participants: 47 classes from Grades 5–12 yielded over 360 participant-class samples, with 15 repeatedly visible students selected for automated-method development and benchmarking.Participants attended 3.84 classes on average, and the selected subsample was chosen based on occurrence and visibility across recordings.
- 3.4 Continuous Manual Annotation: Students’ observable behavior was continuously annotated in one-second steps using CARMA, combining on-task/off-task behavior with ICAP-inspired behavioral cues.The annotation used a symmetric engagement scale, although the supplied passage truncates its full range description.
- 3.2 Video Recording: Three synchronous cameras recorded each classroom, while the pipeline selected the left or right stream where each student was more visible.One camera covered the classroom and teacher from the rear; two others faced the class from either side of the teacher area.
- 3.5 Preprocessing: Face preprocessing detected faces with RetinaFace, matched query images using ArcFace embeddings and cosine similarity, and formed student-specific face tracklets despite occlusion and camera-angle variation.The method used query images rather than conventional face tracking and generally localized and recognized faces because the models were trained on large unconstrained datasets.
- 3.5 Preprocessing: Because disengagement labels were rare and imbalanced, continuous ratings were discretized into low [-2, 0.35], medium (0.35, 0.65], and high (0.65, 2.0] engagement.Figure 3 presents the continuous and discrete label distributions for Grades 8 and 12.
4 METHODOLOGY
The study represents classroom engagement through facial attention and affect features, then evaluates classifiers, fusion, temporal modeling, and personalization across grades. Person-specific labeling and limited data improved classification while revealing grade- and student-level variation.
- 4.3 Engagement Classification: Engagement is modeled from one-second, 24-frame sequences using either a middle frame or temporal learning over all frames.Labels have three levels, and the same one-second interval is retained to support per-second prediction without deployment delay.
- 4.2 Feature Representation: Attention-Net and Affect-Net extract complementary engagement cues from faces through head-pose estimation and categorical facial-expression recognition.The two feature branches address attentional and emotional aspects of engagement while avoiding reliance on fragile facial action-unit processing in unconstrained classroom footage.
- 4.5 Results: Attention features produced the strongest overall Grade 12 classifiers, with AUCs of .708 for RF and .719 for LSTM.Across grades, average AUC ranged from .560 to .719, while linear SVM lagged other classifiers by .03 to .06 AUC and RF, RBF-SVM, and MLP showed no consistent winner.
- 4.5 Results: Engagement classification varied within students, with affect features helping some students and attention features outperforming them for others.Temporal information improved MLP only for Affect-Net in Grade 8 (+.013 AUC) and Attention-Net in Grade 12 (+.018 AUC).
- 4.5 Results: Fusion added +.012-.013 AUC over the best Grade 8 modality, while Grade 12 score-level fusion remained on par with attention features.Feature-level fusion was slightly above Affect-Net performance in Grade 12.
- 4.4 Personalization of Engagement Classifiers: Personalization with 60 seconds of person-specific data improved performance by an average of +.084 AUC and by up to +.12 AUC.The largest reported gain was +.124 AUC for Affect-Net with RF in Grade 12; improvements varied from .03 to .29 AUC and typically used only 2-3% of each student’s data.
5 DISCUSSION
The study demonstrates classroom engagement classification from facial representations and argues that personalization can improve automated processing, while emphasizing limits involving data scale, visible engagement, privacy, and ethical deployment.
- 5.1 Main Findings: Three-class classroom engagement prediction was demonstrated using learned facial representations rather than primarily handcrafted or precomputed features.
- 5.1 Main Findings: Observer-rated engagement data from Grades 8 and 12 included 15 students, providing a basis for validating automated classroom engagement analysis.
- 5.1 Main Findings: Person-specific data substantially improved engagement classification, consistent with visible behavioral differences across students and engagement levels.
- 5.2 Limitations and Future Work: The study focused only on visible engagement, leaving mind wandering and students’ affective and cognitive behaviors for possible future integration.
- 5.2 Limitations and Future Work: The study’s limited sample size constrained representation learning on engagement data, and models failed to detect low engagement amid highly imbalanced continuous labels.
- 5.3 Applications and Ethical Considerations: Automated engagement analysis may support classroom-instruction research, but real-world monitoring is opposed because ethical risks and possible negative effects lack sufficient empirical evidence.