PaperlayerView exact revision on arXiv

Source-linked AI summary

Dimension-Specific Imbalance and an Adaptive Hybrid Label Strategy for Multi-Task Affective State Recognition in Classroom Video

Xiangqian Li

arXiv:2609.31750v1cs.CV

TL;DR

DAiSEE class imbalance varies across affective dimensions, complicating approaches that treat it uniformly. The paper proposes per-dimension label granularity with a null-class placeholder, and reports the highest mean Macro-F1 alongside a possible annotation-density limitation.

  • Problem

    DAiSEE studies rarely report minority-class recall, and results are difficult to reconcile when reweighting or label mapping varies; imbalance also differs across dimensions.

    Source: §I · p. 1, §III · p. 3

  • Method

    AHLS assigns four-level labels to dimensions with manageable imbalance and binary labels to severely long-tailed dimensions, with a null-class placeholder to stabilize multi-task optimization.

    Source: §I · p. 2

  • Results

    47.70% mean Macro-F1, the highest reported, while Confusion minority recall (42.9%) matched models up to 167× larger.

    Source: A. Cross-Model Collapse Analysis: The… · p. 10

  • Takeaways & Limitations

    The authors suggest that future gains on DAiSEE-style minority affective states may depend increasingly on annotation density and inter-rater consensus rather than architectural scaling alone.

    Source: E. The Fundamental Limitation:… · p. 12

    High and Very High affective levels are sparse in DAiSEE, including only 100 and 87 Very High clips for Confusion and Frustration, respectively, limiting the sample density available to characterize these states.

    Source: E. The Fundamental Limitation:… · p. 12

Abstract

from arXiv · show

Recognition of student affective states from classroom video is constrained by a class imbalance problem whose true nature is, we argue, under-analyzed. We show that on DAiSEE -- the de facto benchmark for this task -- class imbalance is fundamentally a dimension-specific phenomenon: raw imbalance ratios reach as high as approximately 79:1 (Frustration), and the structure of imbalance -- not only its magnitude -- varies across dimensions, so uniform algorithmic treatments (aggressive class reweighting, focal loss, uniform binary simplification) fail in dimension-dependent ways. We propose the Adaptive Hybrid Label Strategy (AHLS), which assigns four-level classification to dimensions with manageable imbalance and binary classification to severely long-tailed dimensions, coupled with a null-class placeholder mechanism that stabilizes multi-task optimization by compressing the maximum effective inverse-frequency weight ratio from as high as ~79$\times$ to at most ~8.5$\times$ across the four dimensions. Validated on a lightweight FERShuffleNetV2 + LTCN architecture (0.39 M parameters, 0.07 G FLOPs), the strategy attains the highest mean Macro-F1 of 47.70% across all four affective dimensions among 11 compared methods, including five state-of-the-art deep models (up to 167$\times$ larger) and five traditional machine-learning baselines. A cross-model behavioral analysis on DAiSEE indicates that, in this setting, classification strategy may influence minority-state recognition more strongly than further backbone scaling alone. We also report evidence of an annotation-density limitation of DAiSEE on minority affective states, which suggests that benchmark design may bound future progress as much as algorithmic refinement. Index Terms Affective computing, classroom video analysis, class imbalance, multi-task learning, lightweight deep learning, DAiSEE, engagement recognition.

I. Introduction

DAiSEE imbalance varies across affective dimensions, making uniform treatments poorly suited to the task. The paper proposes per-dimension label granularity and evaluates it through cross-model analysis.

I. Introduction

  • Minority-class recall on Confusion and Frustration is rarely reported, while differing reweighting and label-mapping choices make results difficult to reconcile.The paper frames these issues as a gap in evaluation and comparison across DAiSEE methods.Source: §I · p. 1
  • Uniform aggressive reweighting destabilizes the most long-tailed dimensions, whereas binary simplification discards information on moderately imbalanced dimensions.The authors argue that imbalance magnitude and structure differ across dimensions, so a single treatment can trade recall in one dimension for collapse in another.Source: §I · p. 2
  • AHLS assigns four-level classification to dimensions with manageable imbalance and binary classification to severely long-tailed dimensions, using a null-class placeholder to stabilize multi-task optimization.The mechanism compresses the maximum effective inverse-frequency weight ratio from as high as ~79× to at most ~8.5× across the four dimensions.Source: §I · p. 2
  • The proposed strategy achieves higher mean Macro-F1 than a 64.97 M parameter model trained without it, although the authors describe the cross-model pattern as suggestive rather than proof that architecture is irrelevant.The study compares eleven methods; the validation architecture has 0.39 M parameters.Source: §I · p. 2
  • The authors provide initial evidence that sparse annotation of minority affective states may constrain DAiSEE progress, motivating benchmark refinement.They frame this as a possible limitation rather than a definitive ceiling on algorithmic progress.Source: §I · p. 2

A. Engagement and Affective State Recognition on DAiSEE

Prior DAiSEE work scaled model capacity while giving limited direct attention to minority-class detection. This paper re-examines the challenge as dimension-specific imbalance.

A. Engagement and Affective State Recognition on DAiSEE

  • DAiSEE provides four-level annotations for four affective dimensions across 9,068 short clips and has become a community standard for classroom-style video recognition.Unlike multimodal physiological corpora such as AMIGOS, it relies on webcam video alone.Source: A. Engagement and Affective State… · p. 3
  • DAiSEE methods progressed from CNN and temporal models to 3D video transformers, while most prior work reported accuracy only.The passage describes steady improvements in mean accuracy alongside limited direct reporting of minority-class detection.Source: A. Engagement and Affective State… · p. 3
  • In the authors’ re-implementation, Frustration minority recall stays below 40% for four of five deep baselines, motivating an analysis of dimension-specific imbalance.Source: A. Engagement and Affective State… · p. 3

B. Class Imbalance in Deep Learning

Deep-learning imbalance responses span data-level methods, reweighting, and loss engineering. The paper questions whether one mechanism can address imbalance uniformly across heterogeneous task outputs.

B. Class Imbalance in Deep Learning

  • Common imbalance responses include oversampling, undersampling, and synthetic data generation; inverse-frequency, effective-number, and label-distribution-aware reweighting; and focal or asymmetric losses.Source: B. Class Imbalance in Deep Learning · p. 3
  • These methods share an assumption that imbalance is a single task property addressable by one mechanism applied uniformly across outputs.Source: B. Class Imbalance in Deep Learning · p. 3
  • The paper argues that uniform treatment suited to single-label classification does not generalize to multi-task settings with fundamentally different imbalance structures across heads.Source: B. Class Imbalance in Deep Learning · p. 3

C. Multi-Task Learning under Heterogeneous Label Structures

The paper situates per-dimension label granularity within multi-task learning, where heterogeneous output structures remain relatively underexplored in affective state recognition.

C. Multi-Task Learning under Heterogeneous Label Structures

  • Traditional multi-task learning assumes comparable label spaces across tasks and balances losses with scalar weights or uncertainty-based combinations.Source: C. Multi-Task Learning under… · p. 3
  • A small but growing literature considers structurally heterogeneous outputs, including joint regression and classification tasks.Source: C. Multi-Task Learning under… · p. 3
  • The authors know of no prior affective-state recognition work that varies classification granularity per dimension according to imbalance structure.Their strategy treats label granularity as a per-task design variable informed by each dimension’s distribution.Source: C. Multi-Task Learning under… · p. 3

III. Problem Analysis: Dimension-Specific Imbalance

DAiSEE methods often treat class imbalance as a uniform dataset property, addressed with a single mechanism. The paper argues this assumption is incorrect and relates it to poor minority-class performance across baselines.

III. Problem Analysis: Dimension-Specific Imbalance

  • DAiSEE methods commonly assume that class imbalance is uniform and can be addressed with one algorithmic mechanism.Examples include focal loss, class weighting, and label smoothing.Source: §III · p. 3
  • The paper argues that DAiSEE's imbalance is not a uniform property of the dataset.Source: §III · p. 3
  • The authors argue that this incorrect assumption helps explain poor minority-class performance across the baselines in TABLE V.Source: §III · p. 3

A. Quantifying Per-Dimension Imbalance on DAiSEE

DAiSEE's four affective dimensions have different imbalance magnitudes and structures. The paper distinguishes the original label distribution from the optimizer-facing effective ratios after AHLS mapping.

A. Quantifying Per-Dimension Imbalance on DAiSEE

B. Failure Modes of Uniform Treatments

Uniform treatment creates distinct failure modes across DAiSEE dimensions: standard training and aggressive reweighting struggle with long tails, while universal binarization discards recoverable intensity information. The paper motivates dimension-specific granularity, a null-class placeholder, and bounded task weighting.

B.1 Full four-level training with standard cross-entropy

  • Even under binary labels, EfficientNetB7+LSTM and VideoSwinEmotion show low Confusion-positive and Frustration-positive recall despite VideoSwinEmotion leading mean accuracy.EfficientNetB7+LSTM has 42.5% and 27.0% recall, respectively; VideoSwinEmotion has 10.9% and 14.8%.Source: §B.1 · p. 5

B.2 Aggressive class reweighting at the four-level granularity

  • 79× is the maximum-to-minimum class-weight ratio for four-level Frustration reweighting; logistic regression's 32.93% mean Macro-F1 is the lowest in the comparison.Its minority recalls are 62.2% / 62.4% on Confusion and Frustration, while majority recalls fall to 38.1% / 41.4%.Source: §B.2 · p. 5

B.3 Uniform binary simplification

  • Uniform binarization discards intensity information in Engagement and Boredom, whose post-merger ratios are 8.5:1 and 1.7:1 and support graded classification.Source: §B.3 · p. 5

C. Design Principles for Heterogeneous Treatment

A. Adaptive Hybrid Label Strategy (AHLS)

  • AHLS binarizes dimensions whose Very Low class holds a clear majority and otherwise retains four-level labels with rare-class merging; the placeholder contributes no gradient.The effective inverse-frequency ratio falls from approximately 79× under naive Frustration reweighting to at most 8.5× across dimensions.Source: A. Adaptive Hybrid Label Strategy (AHLS) · pp. 6–7

B. Validation Architecture: FERShuffleNetV2 + LTCN

  • The validation model uses a FER-pretrained ShuffleNetV2 backbone and an LTCN to process ten-frame clips, with four AHLS-configured task heads trained end-to-end.The architecture has 0.39 M parameters and 0.07 G FLOPs; it validates the strategy rather than serving as the paper's contribution.Source: B. Validation Architecture:… · p. 7, D. Conceptual Summary · pp. 7–8

C. Multi-Task Loss with Hybrid Granularity

  • Four-level heads use inverse-frequency weighted cross-entropy with label smoothing, binary heads use class-weighted focal loss, and task losses are combined by a weighted sum.Higher task weights for binarized dimensions address under-training observed with uniform weights in preliminary validation runs.Source: C. Multi-Task Loss with Hybrid… · p. 7

A. Dataset and Splits

  • Experiments use 9,068 DAiSEE clips and a subject-disjoint split, with all reported results evaluated on the 1,638-clip test set.The split is 5,358 training, 1,429 validation, and 1,638 test clips.Source: A. Dataset and Splits · p. 8

B. Comparison Methods

The comparison includes deep-learning and traditional machine-learning baselines alongside the proposed AHLS model. The study uses a fixed validation-based checkpoint-selection protocol and reports Macro-F1 as its primary metric, with minority recall on Confusion and Frustration.

B. Comparison Methods

  • Eleven methods comprise five deep-learning baselines, five machine-learning baselines, and the proposed FERShuffleNetV2 + LTCN model trained with AHLS.The machine-learning baselines use FERShuffleNetV2 features and class or sample reweighting; all methods share the same data partition and random seed.Source: B. Comparison Methods · pp. 8–9

C. Implementation and Training Protocol

D. Evaluation Metrics

  • Macro-F1 is the primary metric because majority-class collapse can yield high accuracy but low Macro-F1; minority recall is also reported for Confusion and Frustration.The paper identifies minority recall as operationally meaningful because false negatives translate into missed instructor-intervention opportunities.Source: D. Evaluation Metrics · p. 9

A. Cross-Model Collapse Analysis: The Natural Ablation

Across the compared methods, the proposed model achieved the highest mean Macro-F1 while matching much larger models on Confusion minority recall. Accuracy and per-dimension results show that strong performance on one measure or dimension does not guarantee balanced performance across all four.

A. Cross-Model Collapse Analysis: The Natural Ablation

  • 47.70% mean Macro-F1 was highest; the proposed model’s 42.9% Confusion minority recall matched models up to 167× larger, while Frustration minority recall remained 15.3%.The authors interpret this as an association between per-dimension granularity and balanced performance, not proof that architecture is irrelevant.Source: Table V · p. 9, A. Cross-Model Collapse Analysis: The… · p. 10
  • VideoSwinEmotion achieved the comparison’s highest mean Accuracy (56.68%), alongside 94.5%/94.4% recall on corresponding majority classes; balanced LR had the table’s highest minority recall and lowest mean Macro-F1 (32.93%).Source: A. Cross-Model Collapse Analysis: The… · p. 10

B. Macro-F1 vs. Accuracy: Why the Metric Matters

  • VideoSwinEmotion’s mean Accuracy of 56.68% exceeded the proposed model’s mean Macro-F1 by a different metric, but its mean Macro-F1 was 43.92%, 3.78 points lower.On Confusion, VideoSwinEmotion’s 68.86% Accuracy accompanied 10.9% minority recall; the authors argue Macro-F1 is more operationally meaningful on imbalanced dimensions.Source: B. Macro-F1 vs. Accuracy: Why the… · p. 10

C. Per-Dimension Macro-F1 Comparison

  • The proposed model led no individual dimension in Macro-F1, but its 19.3-point spread was close to SE-ResNet50+LSTM’s 19.1-point spread.SE-ResNet50+LSTM led Engagement (35.27) and Boredom (38.40), ResNet18+TCN led Frustration (56.73), and EfficientNetB7+LSTM led Confusion (54.73); the proposed model’s spread was comparable despite a 65× smaller parameter budget.Source: Table VI · p. 10, C. Per-Dimension Macro-F1 Comparison · pp. 10–11

D. Efficiency Analysis

The proposed model combines low computational cost with strong mean Macro-F1, while the paper argues that sparse minority-state annotations constrain further gains on DAiSEE. The deployment study demonstrates real-time operation in a small controlled session, but uses a heavier backbone and leaves field validation of the lightweight model for future work.

D. Efficiency Analysis

  • 0.39 M parameters, 0.07 G FLOPs, and 5.4 ms single-sample A100 latency accompany higher mean Macro-F1 than VideoSwinEmotion and EfficientNetB7+LSTM.The proposed model has approximately 1/71 the parameters of VideoSwinEmotion and 1/167 those of EfficientNetB7+LSTM; the figure plots mean Macro-F1 against parameter count.Source: D. Efficiency Analysis · p. 11, Figure 3 · p. 11

E. The Fundamental Limitation: Annotation-Density Argument

  • 841 Confusion and 425 Frustration High/Very High clips, including only 100 and 87 Very High clips, support the authors’ annotation-density explanation for modest minority recall.The authors note moderate inter-rater agreement on Frustration and report that scaling to EfficientNetB7+LSTM did not yield proportional gains; they propose larger, more balanced datasets with multi-annotator consensus.Source: E. The Fundamental Limitation:… · p. 12

A. Real-Time Classroom Pipeline

B. Backbone Choice for the Deployed System

  • The deployed system used VideoSwinEmotion rather than the proposed lightweight model, selected earlier for mean Accuracy; the authors identify the lightweight model as preferable when minority-state detection is prioritized.Field validation of the lightweight model in classroom conditions remains future work.Source: B. Backbone Choice for the Deployed… · pp. 12–13

C. System-Level Validation

  • Below 100 ms end-to-end per-student latency supported real-time per-clip operation in a ten-student session, but quantitative human-rater agreement remains pending.The authors characterize this as a single-session, small-N validation.Source: C. System-Level Validation · p. 13

D. Privacy Considerations

  • Face detections were processed locally without transmitting or retaining identifiable face crops or raw video; student trajectories were shown under anonymous tracker indices.The authors state that GDPR-style data-minimisation compatibility depends on appropriate workstation access controls.Source: D. Privacy Considerations · p. 13

E. Edge Deployment Future Work

  • Future deployment work targets embedded GPUs and field validation under varying illumination, occlusion, and student populations.The authors identify the lightweight model’s 0.07 G FLOPs and 5.4 ms latency profile as advantageous for embedded deployment.Source: E. Edge Deployment Future Work · p. 13

VIII. Discussion and Conclusion

  • Across eleven methods, classification strategy was associated with balanced Macro-F1 more strongly than backbone scale alone; the authors argue that DAiSEE progress may increasingly depend on annotation density.The authors also acknowledge single-run results, no same-backbone comparison of four-level, binary, and hybrid heads, pending lightweight-model field validation, and the need to replicate beyond DAiSEE.Source: §VIII · p. 14

Related papers

Loading 2609.31750v1…