Source-linked AI summary
DAiSEE: Towards User Engagement Recognition in the Wild
Abhay Gupta, Arjun D'Cunha, Kamal Awasthi, Vineeth Balasubramanian
TL;DR
User engagement recognition lacks publicly available datasets that capture affective states in natural settings. DAiSEE introduces a multi-state, publicly available dataset with crowd and expert-supported annotations and benchmarks video classification methods, establishing a baseline for further research.
Problem
Existing user-engagement datasets are small and not publicly available, while commercial systems operate in constrained settings with limited real-world applicability.
Method
DAiSEE constructs a publicly available dataset covering engagement, boredom, confusion, and frustration, using crowd annotations strengthened by expert-psychologist gold standards and benchmarking video-classification models.
Results
DAiSEE establishes a benchmark baseline for four affective states, with aggregated labels matching the expert gold standard for 70% of videos.
Takeaways & Limitations
DAiSEE provides a real-world dataset and baseline for research on feature extraction, context-based inference, and machine-learning methods for affective-state recognition.
Takeaways & Limitations
Recognition can be misclassified when faces are occluded or users turn away from the camera because of real-world distractions.
Abstract
from arXiv · showhide
We introduce DAiSEE, the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration in the wild. The dataset has four levels of labels namely - very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. We have also established benchmark results on this dataset using state-of-the-art video classification methods that are available today. We believe that DAiSEE will provide the research community with challenges in feature extraction, context-based inference, and development of suitable machine learning methods for related tasks, thus providing a springboard for further research. The dataset is available for download at https://people.iith.ac.in/vineethnb/resources/daisee/index.html.
1 INTRODUCTION
User engagement recognition matters across digital applications, but existing datasets and systems are limited by small scale, restricted availability, and constrained real-world settings. DAiSEE addresses this gap with a publicly available dataset covering multiple affective states and benchmark results.
- Motivation: Large-scale datasets have enabled progress in computer vision, motivating a comparable resource for user engagement recognition.The paper frames DAiSEE as a dataset and benchmark contribution for a contemporary vision problem.
- Motivation: User engagement recognition is relevant to advertising, healthcare, e-learning, and autonomous driving applications.These applications include assessing attention, confusion, frustration, fatigue, and engagement during interactions with digital content.
- Research gap: Existing user-engagement datasets are small and unavailable publicly, while commercial affective-recognition systems work mainly in constrained settings.The paper motivates research on recognition in natural, real-world environments.
- E-learning motivation: E-learning is a central use case because current systems lack real-time interactive feedback and face high dropout rates linked to poor design, lecture fatigue, and frustration.The paper reports dropout rates of 91–93% and first-assignment completion near 45%.
- Contribution: DAiSEE introduces a publicly available dataset focused on engagement, frustration, confusion, and boredom in e-environments.The dataset is intended to support further research and development in user engagement recognition.
- Applications: The dataset supports potential applications including personalized e-learning and advertising informed by users’ interests and engagement.The paper presents these as possible application domains rather than exclusive uses.
2 BACKGROUND AND RELATED WORK
Affect-recognition datasets have expanded beyond posed facial expressions, but existing resources remain limited in their coverage of natural, application-specific affect and real-world engagement. DAiSEE addresses this gap with a publicly available dataset designed around four affective states and real-world conditions.
- Affect Recognition Datasets: Early facial-expression datasets were largely collected in lab-controlled settings with posed expressions, limiting their correspondence to spontaneous affect.Later datasets introduced stimuli-driven or unconstrained recordings, but constraints in subjects, head pose, or environmental diversity remained.
- Affect Recognition Datasets: Existing datasets variously model basic expressions, continuous valence-arousal, facial action units, or broad affect categories rather than the specific states needed for user-engagement applications.AffectNet includes bored, tired, confused, and focused expressions in a combined “None” class without intensity annotation.
- Affect Recognition Datasets: No reviewed dataset jointly addresses boredom, confusion, engagement, and frustration in-the-wild with intensity labels.Belfast includes frustration but is stimuli-driven and laboratory-based, while AffectNet combines several relevant states into one non-intensity-annotated class.
- User Engagement: User-engagement recognition is relevant to advertising, healthcare, e-learning, and autonomous driving, yet existing engagement datasets were small and not publicly available.These applications include detecting attention, confusion, frustration, or engagement during interactions with content or systems.
- DAiSEE and E-Learning: E-learning offers a defined context for studying engagement because existing systems provide limited real-time interactive feedback during content delivery.The authors motivate this setting through the growth of MOOCs and state that the dataset can extend to other application domains.
- DAiSEE and E-Learning: DAiSEE provides 2,723,882 frames from 9068 snippets totaling 25 hours of video to support training state-of-the-art deep-learning models.The dataset is intended to capture real-world conditions and to promote more reliable engagement-recognition methods.
3 THE DAISEE DATASET
DAiSEE is a publicly available, in-the-wild video dataset for four engagement-related affective states, built from student recordings and multi-stage annotation. It provides four-level labels, high-fidelity aggregated annotations, raw crowd votes, and benchmark support for applications and further research.
- Data collection: DAiSEE contains 9,068 ten-second video snippets from 112 student subjects recorded in varied real-world settings.The recordings use educational and recreational videos to capture focused and relaxed behavior, with subjects able to scroll naturally.
- Affective states: The dataset labels engagement, frustration, confusion, and boredom at four levels: very low, low, high, and very high.These states are selected as affective states related to user engagement rather than relying only on basic facial expressions.
- Data annotation: Each snippet receives votes from ten annotators, with faulty annotations removed before Dawid-Skene aggregation estimates the ground-truth labels.Annotators complete instructions and standardized test questions, while the aggregation procedure estimates observer error rates iteratively.
- Annotation quality: 70% of videos match the expert-psychologist gold standard, while 84 of 100 videos receive majority user-study agreement with the aggregated labels.Among disagreements, 10 of the remaining 16 videos concern intensity differences, while 6 concern opposite-polarity affective states.
- Research platform: DAiSEE publicly provides raw crowd annotations and high-fidelity aggregated annotations for research on temporal affect recognition and related applications.The dataset is positioned for e-learning, healthcare, advertising, and autonomous-vehicle applications, and its video format preserves temporal information.
4 BENCHMARK RESULTS
DAiSEE is benchmarked with static and temporal CNN-based models under subject-exclusive train, validation, and test splits. LRCN generally performs best, while temporal classifiers outperform static approaches for these affective states.
- Evaluation setup: The dataset is split using a 60:20:20 train:validation:test ratio, with mutually exclusive subjects and matched male:female ratios across splits.These principles are used to prepare DAiSEE for benchmarking.
- Evaluation setup: The benchmark evaluates single-frame classification, pooled frame prediction, fully trained C3D, transfer-learned C3D, and LRCN sequence learning.Each affective state is benchmarked individually.
- Model comparison: Temporal classifiers generally outperform static classifiers, indicating that the affective states persist over time rather than appearing only instantaneously.InceptionNet frame-level prediction was only slightly better than random guessing for boredom.
- Model comparison: LRCN generally achieves better performance than the other evaluated models on DAiSEE.Baseline results are reported using Top-1 accuracy averaged over three runs.
- Annotation analysis: Dawid-Skene aggregation is compared with and without removing bad annotators to assess label-quality improvement against expert annotations.The comparison is summarized in Table 4.
- Binary engagement analysis: LRCN reaches 94.6% Top-1 accuracy when engagement is reduced to binary engaged versus not-engaged classification.The binary labels combine low and very-low categories as not-engaged and high and very-high categories as engaged.
5 BENCHMARKING CHALLENGES OF DAISEE
DAiSEE’s uncontrolled settings create several sources of benchmark difficulty and possible video misclassification. These include poor visibility, occlusion or non-frontal poses, changing affective states, and complex label relationships.
- Visual and temporal challenges: Low illumination can make faces difficult to detect, preventing models from learning features needed for correct classification.The issue is illustrated in Figure 10.
- Visual and temporal challenges: Face occlusion and non-frontal poses can hide facial features and cause misclassification.Occlusion may result from hair or hands, while distractions can make users turn away from the camera.
- Visual and temporal challenges: Changes in affective state within a video snippet make it difficult for models to learn features representing the video correctly.Such temporal variation is illustrated in Figure 12.
- Label relationships: Boredom and engagement are often complementary, but both can be low when confusion or frustration is high.Figure 13 shows complementary cases, whereas Figure 14 shows that the relationship is not universal.
6 DAISEE RELEASE
DAiSEE is released for download through the project website.
- DAiSEE is available for download at the project website.
7 CONCLUSION AND FUTURE WORK
The paper presents DAiSEE as a publicly available dataset covering multiple affective states in realistic e-environment settings. It provides benchmarks and raw annotations while motivating future models based on richer behavioral cues.
- Dataset contribution: DAiSEE covers engagement, boredom, confusion, and frustration with four ordinal levels from very low to very high.The labels exclude a neutral category and are based on crowd annotation.
- Dataset contribution: The dataset captures real-world variation in user poses, positions, and background noises within e-learning-like environments.It is presented as a publicly available dataset for studying these four affective states.
- Dataset contribution: DAiSEE provides benchmark results and raw annotation data for research on vote aggregation and improved models.The raw annotations can also be used during training.
- Future work: Future approaches may use facial geometry, action units, body and head pose, gaze, and gesture as model inputs.These mid-level cues may support cognitive models involving engagement, attentional focus, and intention.
- Applications: The dataset is intended to support more responsive human-computer interaction in e-learning, advertising, healthcare, and autonomous vehicles.