PaperlayerView exact revision on arXiv

Source-linked AI summary

Fairness-Aware Multimodal Transformer Modeling for Real-Time Student Attention Estimation

Christoforos Fragkiadakis, Seyed Sahand Mohammadi Ziabari, Ali Mohammed Mansoor Alsahag

arXiv:2609.02232v1cs.CY

TL;DR

Aggregate metrics can hide demographic disparities in automated student-attention estimation. This study evaluates fairness-aware multimodal temporal models on naturalistic DIPSER data and finds modest multimodal performance gains, while validation fairness improvements do not consistently generalize to held-out subjects.

  • Problem

    Aggregate predictive metrics can conceal demographic disparities, while fairness-aware modeling remains comparatively underexplored in student attention estimation.

  • Method

    The study compares Visual GRU, Sensor GRU, and Residual Fusion Transformer models on DIPSER, then applies demographic MAE-gap regularization and subgroup-aware evaluation.

  • Results

    The Residual Fusion Transformer achieves the best mean test performance, with MAE 0.283±0.009 and RMSE 0.363 ± 0.009, and the lowest worst-group errors among baseline models.

  • Takeaways & Limitations

    Multimodal fusion can modestly improve attention estimation and worst-group performance, but fairness gains require subgroup-aware evaluation and repeated subject-level validation.

  • Takeaways & Limitations

    Validation disparity reductions from fairness regularization did not consistently generalize to held-out subjects or repeated subject-level test splits, amid small subgroup samples and inferred demographic attributes.

Abstract

from arXiv · show

Automated student-attention estimation can support learning analytics, but aggregate predictive metrics can conceal demographic disparities. This study evaluates fairness-aware multimodal temporal models on DIPSER, a naturalistic classroom dataset combining facial images, wearable-sensor measurements, attention annotations, and automatically inferred demographic metadata. Three baselines are compared across 10 training seeds: a Visual GRU, a Sensor GRU, and a Residual Fusion Transformer. The multimodal model achieves the best mean test performance (MAE 0.283, RMSE 0.363) and the lowest worst-group error among the evaluated baselines, although its gain over the Visual GRU is modest. Gender- and age-targeted MAE-gap regularization reduces disparities on validation data, but these gains do not consistently transfer to held-out subjects or repeated subject-level splits. On an NVIDIA A100-SXM4-40GB GPU, the warm end-to-end pipeline averages 50.65 ms per prediction window at a one-second stride, while the temporal model itself requires 1.02 ms. The findings show that multimodal fusion can modestly improve prediction and worst-group performance, but validation-level fairness gains should not be assumed to generalize. Robust fairness assessment therefore requires subgroup-aware evaluation, repeated subject-level validation, and larger, better balanced demographic samples.

1 Introduction

Automated attention estimation is feasible from behavioral signals, but aggregate metrics can conceal demographic disparities. This study therefore evaluates multimodal temporal models and fairness-aware mitigation for continuous student attention estimation.

  • Student attention reflects focus on learning activities and is strongly associated with academic performance.
  • Facial behavior, eye tracking, physiological responses, and interaction data can provide signals for inferring attentional states.
  • Aggregate predictive metrics can conceal systematic demographic differences in model error across age, gender, and race.
  • Fairness-aware modeling remains comparatively underexplored for multimodal student attention estimation.
  • The study compares Visual GRU, Sensor GRU, and Residual Fusion Transformer baselines, then evaluates subgroup disparities and demographic MAE-gap regularization.
  • The evaluation jointly considers overall error, worst-group error, and best-to-worst subgroup gaps because fairness constraints can create accuracy–fairness trade-offs.

2 Related Work

Prior student-engagement research emphasizes predictive performance and global metrics, while fairness-aware subgroup analysis remains comparatively underexplored. Related mitigation strategies intervene before, during, or after training to address demographic disparities.

  • Student-engagement models have used vision, physiological, audio, textual, wearable, and interaction signals to capture behavioral and latent cognitive states.
  • Hierarchical and temporally aware multimodal architectures have combined several sensing modalities, with cited ablations reporting stronger performance for broader modality combinations.
  • Engagement recognition is primarily evaluated with aggregate metrics, leaving fairness-aware modeling and demographic subgroup analysis comparatively underexplored.
  • High predictive accuracy does not necessarily correspond to fairness, and multimodal fusion can sometimes amplify group-level disparities.
  • Pre-processing methods address data imbalance through demographic oversampling or reweighting strategies.
  • In-processing methods incorporate fairness constraints or regularization into optimization, while post-processing modifies outputs through calibration or threshold adjustment.

3 Methodology

The methodology uses naturalistic multimodal classroom data and compares temporal visual, sensor, and fusion architectures. Preprocessing cleans synchronized image and wearable streams while retaining demographic information for subgroup-aware analysis.

  • The methodology comprises dataset preprocessing, unimodal and multimodal model design, training and evaluation, and fairness-aware optimization.
  • DIPSER synchronizes camera recordings with smartwatch data collected during scheduled educational activities without experimental staging or behavioral constraints.
  • Subject records include image sequences, attention annotations, wearable sensor data, and metadata with automatically derived demographic attributes.
  • Preprocessing removes invalid physiological records, excludes subjects with extensive or absent sensor data, and filters corrupted image frames.
  • Three architectures separately model facial embeddings, wearable physiological and motion signals, or their multimodal fusion over rolling temporal windows.
  • The Residual Fusion Transformer uses visual features for the primary estimate and sensor streams for a bounded residual correction.

3.3 Model Training and Evaluation

Training uses subject-level splits, attention-range reweighting, weighted regression, repeated seeds, and subgroup-aware metrics. The complete warm pipeline is benchmarked for compatibility with the dataset’s one-second prediction stride.

  • The 57-subject dataset uses subject-level partitioning with 39 training subjects and 9 subjects each for validation and testing.
  • Attention-bin inverse-frequency weighting addresses the concentration of labels in moderate attention ranges.
  • The models optimize weighted mean squared error using inverse-frequency sample weights.
  • Performance is measured with MAE and RMSE overall and across demographic subgroups, with each architecture trained across 10 independent random seeds.
  • The dataset’s subject-level demographic distributions are reported separately across training, validation, and test splits.
  • Warm end-to-end benchmarking includes preprocessing, temporal alignment, CLIP feature extraction, window construction, and final prediction.

3.5 Fairness-Aware Optimization

The fairness-aware optimization modifies the best validation baseline by adding a subject-level demographic fairness regularizer while keeping the rest of the training and evaluation setup unchanged. The regularizer aggregates subject-level MAEs into demographic-group errors, penalizes the gap between the highest and lowest groups, and uses balanced sampling with a two-epoch prediction-only warm-up.

  • Regularized baseline: The fairness-aware variant adds a subject-level demographic regularizer to the lowest-mean-validation-error baseline, leaving the remaining setup unchanged.The architecture, preprocessing, split, optimizer, and evaluation setting match the unregularized baseline.
  • Fairness measurement: Subject-level MAE is computed first, then averaged across subjects within each demographic group to obtain group-level error.This design avoids computing fairness directly at the sample level.
  • Fairness measurement: The fairness penalty is the difference between the highest and lowest group-level subject MAEs.It therefore targets unequal predictive performance across demographic subgroups.
  • Optimization objective: The final objective combines the task loss with the fairness penalty, with λ≥0 controlling the accuracy–fairness trade-off.Fairness regularization is added after the first 2 epochs, during which optimization uses only prediction MSE.
  • Balanced sampling: Subject- and group-balanced batches support more stable fairness estimates during training.Gender batches use 2 subjects per gender and 8 temporal windows per subject; age-group batches use 1 subject from each of 4 groups.

4 Results

Across repeated baseline runs, the Residual Fusion Transformer delivered the strongest average test performance and lowest worst-group errors, but multimodal gains over visual modeling were modest. Fairness regularization improved validation disparities inconsistently and failed to generalize reliably to unseen subjects, while inference latency was dominated by visual feature extraction.

  • Baseline predictive performance: Physiological and motion signals alone produced limited predictive information, while visual representations supplied the main information for attention prediction.The Sensor GRU slightly underperformed the train-mean baseline, whereas the Visual GRU achieved MAE 0.287 and RMSE 0.366.
  • Baseline predictive performance: MAE 0.283 and RMSE 0.363 were the Residual Fusion Transformer’s best average test results across the evaluated architectures.The improvement over the Visual GRU was small and within training-run variability.
  • Demographic fairness: Worst-group MAE was lowest for the Residual Fusion Transformer for age at 0.328 and gender at 0.289.It also had the lowest gender gap, 0.015, among the evaluated baselines.
  • Demographic fairness: The Sensor GRU achieved the smallest age gap, 0.096, but its weaker overall predictive performance complicates interpreting this as improved fairness.The Visual GRU had the largest age gap, 0.147, despite strong overall predictive performance.
  • Fairness-aware regularization: Age MAE-gap regularization reduced validation age-gap error from 0.105 to 0.093 and worst-group performance from 0.352 to 0.345, with only a minor overall MAE increase.The regularization strength was λ=0.8.
  • Runtime: The warm end-to-end pipeline averaged 50.65 ms per prediction window at a one-second stride, while the temporal fairness model required 1.02 ms.CLIP-based visual feature extraction dominated computational cost on the NVIDIA A100-SXM4-40GB GPU.

5 Discussion

Visual information was the primary signal for attention estimation, while multimodal fusion added only modest predictive benefit. Fairness patterns varied by demographic attribute, and validation improvements did not reliably generalize to unseen subjects.

  • Visual representations provided the primary information source for student attention estimation in DIPSER.
  • Multimodal fusion modestly improved prediction over the Visual GRU, while sensor signals added limited information beyond visual representations.Residual fusion therefore used the visual stream for the primary estimate and sensors for a bounded residual correction; early and late fusion did not outperform the Visual GRU.
  • Each 10-second sequence contained 10 visual frames and 10 aggregated sensor vectors after one-second sampling and sensor aggregation.
  • Gender MAE gaps were relatively small, whereas age gaps were substantially larger across the Visual GRU and Residual Fusion Transformer.The age disparity reflected heterogeneous performance across groups rather than consistent deterioration with age; four age groups were compared with two gender groups.
  • Validation fairness improvements did not consistently transfer to unseen subjects, with targeted gaps and overall errors increasing on average for regularized models.The limited number of independent subjects, including 3 male participants and 1 participant in the oldest age group, made subgroup metrics sensitive to individual behavior.

6 Conclusion

The study finds that multimodal temporal modeling modestly improves attention estimation and worst-group performance, while fairness regularization gains remain unreliable for held-out students. Real-time feasibility was demonstrated on the evaluated GPU, but broader efficiency and fairness require stronger evidence.

  • MAE 0.283±0.009 and RMSE 0.363 ± 0.009 were the Residual Fusion Transformer’s best mean test performance, with only modest improvement over the Visual GRU.
  • The Residual Fusion Transformer produced the lowest worst-group errors and smallest gender gap, while the Sensor GRU had the smallest age gap but substantially weaker overall performance.Disparity metrics should therefore be interpreted jointly with predictive accuracy and worst-group performance.
  • Fairness regularization reduced targeted MAE gaps on validation data but did not consistently improve fairness for held-out subjects or repeated subject-level splits.The evidence does not support reliable generalization of demographic fairness gains to unseen students.
  • 50.65 ms was the mean warm end-to-end latency per prediction window at a one-second stride on an NVIDIA A100-SXM4-40GB GPU.This supports real-time feasibility on the evaluated hardware but does not establish efficiency on resource-constrained devices.

A Training and Validation Error Curves

Figure 2 shows training and validation error curves for a representative Residual Fusion Transformer run, with epoch selection based on validation RMSE. The widening curve separation after epoch 9 supports validation-based early stopping.

  • Increasing separation between training and validation curves after epoch 9 supports validation-based early stopping.
  • Figure 2 plots training and validation MAE and RMSE curves for the Residual Fusion Transformer using seed 100.
  • The dashed line marks the epoch selected according to the lowest validation RMSE.

B Repeated Subject-Split Robustness Analysis

Table 7 reports paired changes relative to the unregularized Residual Fusion Transformer across 10 repeated subject-level splits, where positive values indicate improvement.

  • Table 7 reports paired changes relative to the unregularized Residual Fusion Transformer across 10 repeated subject-level splits.
  • Positive values in the table indicate improvement over the baseline.

C Frame Sampling Rate Sensitivity Analysis

This analysis compares 1 frame per second with 10 frames per second using the Residual Transformer across five matched seeds, reporting means across those seeds.

  • C Frame Sampling Rate Sensitivity Analysis: The Residual Transformer is evaluated at 1 frame per second and 10 frames per second across 5 matched seeds.Values are reported as means across the matched seeds.

D Architecture and Training Hyperparameters

The models use documented architecture and training hyperparameters, with fairness-aware variants retaining the Residual Fusion Transformer architecture and modifying only the training objective.

  • D Architecture and Training Hyperparameters: Fairness-aware models use the same Residual Fusion Transformer architecture as the unregularized multimodal model.The architectural configuration is unchanged between these model variants.
  • D Architecture and Training Hyperparameters: The fairness-aware models modify only the training objective through a demographic MAE-gap regularizer.Their architecture remains fixed while the demographic fairness regularizer changes optimization.
  • D Architecture and Training Hyperparameters: Table 10 reports the training hyperparameters used across the temporal models.The passage identifies the table as the source for these training settings.
Loading 2609.02232v1…