Source-linked AI summary
Student Engagement Detection Using Emotion Analysis, Eye Tracking and Head Movement with Machine Learning
Prabin Sharma, Shubham Joshi, Subash Gautam, Sneha Maharjan, Salik Ram Khanal, Manuel Cabral Reis, João Barroso, Vítor Manuel de Jesus Filipe
TL;DR
Assessing engagement in distance learning is challenging. This paper proposes a real-time webcam system that combines eye and head movement with facial emotion to estimate concentration. In tests with 15 students, it identified engagement periods, and higher quiz scores were associated with higher concentration indexes.
Problem
Assessing how well students receive lecture content is a challenge for instructors, and virtual classrooms also face student dropout.
Source: §1 · pp. 1–2Method
The system uses a laptop’s built-in webcam to combine eye and head movement with facial emotion in a real-time concentration index with three engagement categories.
Source: §1 · p. 2, §5 · p. 16Results
In tests with 15 students, the system identified engagement periods; 92% of students scoring 3 and above were in the highly engaged category.
Source: §4 · p. 15, §5 · p. 16Takeaways & Limitations
The authors report that students with higher scores also had higher concentration indexes.
Source: §5 · p. 16The system was unable to handle partial face occlusion, such as a face covered by a hand or another object.
Source: §4 · p. 15
Abstract
from arXiv · showhide
With the increase of distance learning, in general, and e-learning, in particular, having a system capable of determining the engagement of students is of primordial importance, and one of the biggest challenges, both for teachers, researchers and policy makers. Here, we present a system to detect the engagement level of the students. It uses only information provided by the typical built-in web-camera present in a laptop computer, and was designed to work in real time. We combine information about the movements of the eyes and head, and facial emotions to produce a concentration index with three classes of engagement: "very engaged", "nominally engaged" and "not engaged at all". The system was tested in a typical e-learning scenario, and the results show that it correctly identifies each period of time where students were "very engaged", "nominally engaged" and "not engaged at all". Additionally, the results also show that the students with best scores also have higher concentration indexes.
1 Introduction
The introduction frames student engagement as important to learning and difficult for instructors to assess in virtual classrooms. It proposes a real-time webcam system combining eye movements and facial emotions to estimate concentration.
1 Introduction
- Instructors face the challenge of determining how well students receive lecture content, while engagement is treated as important to meaningful, high-quality education.The introduction describes engagement as meaningful involvement in the learning environment.Source: §1 · pp. 1–2
- Virtual classrooms face student dropout, and disengagement is presented as a more serious concern than low achievement,.The introduction also links engagement with achievement and describes attention as an indicator of disengagement.Source: §1 · p. 2
- The proposed system uses a laptop’s built-in webcam to analyze eye movements and facial emotions in real time, helping instructors assess student engagement.The paper also identifies head movement as a signal used to estimate students’ attention.Source: §1 · pp. 2–3
2 Related Works
Related work explores engagement measurement through facial emotion, eye and head movement, gaze, and behavioral cues. Prior systems vary in setting and method, with some reported limitations in classification or accuracy.
2 Related Works
- Engagement indexes are particularly relevant to autonomous e-learning, where no instructor is present to observe students’ reactions and emotions.Prior work also identifies eye tracking and facial emotion as signals for assessing attention or participation.Source: §2 · pp. 3–4
- Eye and head movement studies addressed computer-vision syndrome alerts and concentration monitoring in e-learning.The latter used video frames, face detection, and Viola-Jones features to detect negative emotions.Source: §2 · p. 3
- Turabzadeh et al.'s facial-emotion system using LBP features and K-NN regression with dimensional labels reached 51.28% accuracy in MATLAB Simulink and 47.44% in Xilinx simulation.An MKL-SVM model reached 43.98% average accuracy and 50.77% maximum accuracy.Source: §2 · pp. 3–4
- A gaze-based engagement classifier trained with expert observations and an HMM classified students only as “engaged” or “not engaged”.Although eight behavior categories were proposed, the reported classifier could distinguish only those two states.Source: §2 · p. 3
- A real-time emotion-based system linked expressed emotions with concentration and assigned high, medium, and low concentration levels.It also proposed adapting content according to the student’s concentration level.Source: §2 · p. 4
- A deep-learning engagement model initialized from a facial-expression model was trained on 4,627 engaged and disengaged images and reportedly outperformed earlier architectures.Source: §2 · p. 4
- Other frameworks measured participation in classroom and online settings using facial, gaze, head-attitude, and action-unit features.One framework used domain-adapted facial-expression recognition for MOOC engagement and reported superiority to state-of-the-art methods.Source: §2 · p. 4
3 System Architecture
The system processes webcam images through attention and emotion classifiers, then combines their outputs into a concentration index and three engagement categories. Emotion weights are derived from quiz performance, and the system is designed for real-time monitoring.
3 System Architecture
- The concentration index is the dominant emotion probability multiplied by that emotion’s weight, producing a score from 0% to 100%.Emotion weights were derived from quiz means for groups of 30 students grouped by their majority facial expression.Source: c) Engagement Classification · p. 6, c) Concentration Index · pp. 7–8
- The pipeline detects a face and eyes, classifies attention as focused or distracted, analyzes emotion when focused, then calculates concentration and engagement.The system is intended for real-time use in remote learning or other laptop-based learning environments.Source: §3 · p. 5, c) Engagement Classification · p. 6
- The Viola–Jones Haar cascade detects the student’s frontal face and locates the eye region within it.The detector applies this process to each video frame before the eye images are classified.Source: a) Distraction Detector · p. 5, a) Haar Cascade Algorithm · p. 6
- A CNN trained on eye images performs binary attention classification, identifying whether the student faces the webcam as focused or distracted.Its architecture includes a 64×64 input, 3×3 convolution filters, 2×2 pooling, and a fully connected layer.Source: b) Convolution Neural Network (CNN) · pp. 6–7
- A grayscale CNN classifies seven facial emotions, and the highest-probability emotion is selected as dominant.The mini-Xception architecture was evaluated at 66% accuracy on FER2013.Source: b) Convolution Neural Network (CNN) · p. 7
- The emotion CNN used 35,887 grayscale face images from Kaggle; accuracy increased with training epochs but stopped improving substantially after epoch 81.Figure 2 plots accuracy against epoch.Source: b) Convolution Neural Network (CNN) · p. 7, Figure 2 · p. 8
- Engagement is very engaged at 50–100% concentration when focused, nominally engaged below 50% when focused, and not engaged when distracted.A below-threshold concentration index can trigger an alert to the instructor or learner.Source: §3 · p. 5, d) Categories of engagement · p. 9
- The system presents facial emotion and eye-head movement information for real-time monitoring while learning content is delivered.Its modules are presented as a system for student engagement evaluation.Source: §3 · pp. 4–5, d) Categories of engagement · p. 9
4 Experimental Results and Discussion
The experiment used a short lecture and quiz with 15 students, then examined concentration traces from eye/head movement, facial emotion, and combined signals alongside quiz performance. Quiz outcomes broadly aligned with concentration, but face occlusion and glasses produced mismatches.
4 Experimental Results and Discussion
- The test involved 15 students aged 20–30 watching a 1-minute-56-second lecture, followed by a 3-minute quiz.The video included a prompt asking students to identify a missing component in a pyramid diagram.Source: §4 · pp. 9–10
- Across full-video plots, concentration fluctuations are attributed to eye/head movements and negative emotions.The figure pairs examples of students watching the video with their concentration-index plots.Source: §4 · p. 11, Figure 5 · p. 12
- Using only eye and head movements, the binary concentration index is 0% (“Distracted”) or 100% (“Focused”); the example is focused from frames 10th to 70th and distracted from frame 70th to 105th.The rule assigns 100 when c > 0.5 and 0 otherwise; c is the value given by the Distraction Detector.Source: §4 · p. 11
- The emotion-only trace shows high concentration at frames 90–120 and low concentration at frames 150–190.The index is based on the dominant emotion and weights assigned to seven basic emotions.Source: Figure 7 · pp. 12–13, §4 · pp. 12–13
- In the combined trace, frames 12–47 averaged above 45 concentration, while frames 109–140 registered 0 as totally distracted.The system first analyzes eye/head movement and predicts facial emotions only if the student is focused.Source: §4 · p. 13, Figure 8 · p. 14
- All 15 students answered question 3 correctly (mean concentration 60%, “Highly engaged”); 92% scoring 3+ were also “Highly engaged” (50–100%).Table 2 reports the quiz results and partial concentration indexes.Source: §4 · pp. 13, 15, Table 2 · p. 14
- Scores and concentration indexes disagreed for students 2, 6, and 8: occlusion impaired emotion extraction for 2 and 6, while glasses made 8’s index fluctuate.The paper notes that students 2 and 6 were focused despite low concentration estimates when their faces were partly covered.Source: §4 · p. 15, Figure 9 · p. 15
5 Conclusion and Future Work
The paper presents a real-time engagement detector using a typical laptop’s built-in webcam to combine eye and head movements with facial emotions. In a typical e-learning test, it correctly identified three engagement levels; the authors also describe possible instructional uses and future extensions.
5 Conclusion and Future Work
- The system uses only built-in laptop webcam images, combining eye and head movements with facial emotions to produce three engagement classes in real time.The classes are “very engaged,” “nominally engaged,” and “not engaged at all.”Source: §5 · p. 16
- The authors propose that facial-emotion feedback could help teachers or learning systems adapt learning content to students’ needs.They state that this could dynamically enrich the learning environment and improve student performance.Source: §5 · p. 16
- In a typical e-learning test with 15 students, the system correctly identified periods of very engaged, nominally engaged, and not engaged at all.Students with the best scores also had higher resultant concentration indexes.Source: §5 · p. 16
- Future work includes combining the system’s information with other sensors and moving to 3D facial-expression detection, which would require cameras beyond a typical built-in laptop webcam.Source: §5 · p. 16