PaperlayerView exact revision on arXiv

Source-linked AI summary

An Intelligent Decision Support System for Emotion Monitoring using Microscopic Fixational Dynamics

Xiangyu Shen, Feiyang Deng, Zijian Dai, Aibin Chen, Jizheng Yi, Jie Li, Hongbo Jiang

arXiv:2609.00846v1cs.CV

TL;DR

Emotion monitoring needs less intrusive and more privacy-conscious alternatives to facial or physiological sensing. EmoGaze extracts microscopic fixation dynamics with an interpretable edge-computing pipeline and evaluates them under subject-independent validation. The system reports an 83.6% F1-score after few-shot personalized calibration, while its operating envelope excludes vigorous exercise and extreme sudden noise.

  • Problem

    Conventional emotion-recognition methods face privacy and practicality challenges, motivating emotion inference from less intrusive physiological signals.

  • Method

    EmoGaze uses smart glasses and a companion smartphone to decompose fixation into microsaccades, ocular drifts, and ocular microtremors, then processes them with an interpretable MHA-XGBoost-SVM pipeline.

  • Results

    83.6% F1-score is reported after minimal few-shot personalized calibration in evaluations of 60 volunteers using leave-one-subject-out validation across controlled and naturalistic mobile scenarios.

  • Takeaways & Limitations

    The framework provides a physiologically interpretable and locally processed approach to unobtrusive emotion monitoring that accommodates individual emotional variability through personalization.

  • Takeaways & Limitations

    EmoGaze is positioned as opportunistic rather than unconditional continuous monitoring because signal quality degrades during vigorous exercise and extreme high-decibel sudden noises.

Abstract

from arXiv · show

The rising prevalence of psychological disorders necessitates effective emotion monitoring, yet current methods relying on facial or physiological signals often suffer from intrusiveness and privacy issues. This paper proposes an intelligent decision support system and pervasive edge-computing framework that leverages smart glasses and a companion smartphone to infer emotional states from microscopic visual fixation patterns. Moving beyond traditional macroscopic gaze metrics, the proposed system extracts and decomposes three distinct neurophysiological micro-movements: microsaccades, ocular drifts, and ocular microtremors. We introduce an interpretable hybrid artificial intelligence pipeline combining a multi-head attention mechanism, extreme gradient boosting, and a support vector machine to extract deep temporal features, quantify their physiological importance, and perform efficient on-device classification. Through an extensive evaluation involving 60 volunteers, we rigorously validate the framework under a strict leave-one-subject-out cross-validation protocol across both controlled and naturalistic mobile scenarios. Ablation studies unequivocally demonstrate that these fixational micro-movements are substantially more discriminative for emotion inference than traditional macroscopic features. Furthermore, aligned with contemporary affective science, the system incorporates a few-shot personalization mechanism to bridge universal physiological baselines with individual emotional heterogeneity, achieving a highly robust personalized F1-score of 83.6%. This work establishes a physiologically interpretable, unobtrusive, and deployable paradigm for continuous real-time emotion monitoring.

1. Introduction

EmoGaze addresses privacy and practicality limitations in conventional emotion recognition by analyzing involuntary fixational micro-movements with an interpretable edge-based system. Evaluations with 60 volunteers and leave-one-subject-out validation report improved personalized performance.

  • Traditional facial and voice-based emotion recognition can require direct observation of sensitive cues, creating privacy and real-world practicality challenges.
  • Microsaccades, ocular drifts, and ocular microtremors are proposed as emotion-related physiological markers during fixation.
  • EmoGaze uses smart glasses as a lightweight sensor node and a companion computing hub to support unobtrusive mobile emotion monitoring.
  • The MHA-XGBoost-SVM pipeline combines temporal representation, feature-importance ranking, and classification while exposing physiological importance to domain experts.
  • 83.6% F1-score is reported after minimal few-shot personalized calibration in evaluations involving 60 volunteers across controlled and naturalistic mobile scenarios.
  • The framework decomposes fixation into neurophysiological components and reports stronger discrimination than macroscopic gaze features through modality ablation studies.

2. Related Work

Prior emotion-recognition approaches use facial, speech, physiological, and macroscopic gaze signals, but face, task, environmental, and computational constraints limit their applicability. EmoGaze instead targets microscopic fixation dynamics for interpretable mobile edge deployment.

  • Facial-expression methods can be disrupted by occlusion, lighting, and intentional suppression, while speech methods face environmental background interference.
  • Macroscopic gaze metrics such as fixation duration, saccade amplitude, and pupil dilation can be confounded by visual tasks, ambient lighting, and cognitive load.
  • EmoGaze analyzes involuntary microscopic fixation dynamics as task-independent physiological markers for lightweight continuous emotion monitoring.
  • Existing gaze research has largely focused on macroscopic behavioral trajectories and overall visual attention or saccadic patterns.
  • The paper frames emotions as individualized and context-dependent while using microscopic movements as lower-level physiological indicators related to arousal and valence.
  • The system is engineered for smart-glasses and companion-smartphone deployment and extracts deep temporal features from previously overlooked microscopic dynamics.

3. Motivation

The motivation section links fixation micro-movements to autonomic arousal and valence and uses a separate pilot cohort to examine their emotional variation. The observed patterns support microscopic fixation dynamics as discriminative emotion indicators.

  • Microsaccades, ocular drifts, and ocular microtremors are treated as ocular motion metrics associated with autonomic arousal and valence.
  • A separate cohort of 14 volunteers was recruited to measure fixation patterns during discrete emotional states without overlapping the 60-subject system evaluation.
  • Anger produced relatively high frequency and amplitude across all three eye-movement signals, reflecting heightened physiological activity.
  • Sadness showed lower signal frequency and amplitude, whereas joy and pleasure showed moderate, stable ocular patterns.
  • Signal variability was higher in the reported high-arousal states and more stable in sadness.
  • The pilot analysis reports a correlation between the three micro-eye-movement signals and emotional states, supporting their use as discriminative emotion indicators.

4. System Design

EmoGaze is an edge-based five-module system that collects binocular infrared eye data, preprocesses fixation signals, decomposes them into physiological micromovements, extracts and ranks features, and classifies emotions. Its preprocessing and detection stages target reliable recovery of microsaccades, ocular drifts, and ocular microtremors from wearable recordings.

  • System architecture: The five modules are data collection, preprocessing, fixation decomposition, feature extraction and ranking, and emotion classification.The glasses capture eye videos while the companion edge device performs subsequent processing and classification.
  • System architecture: Smart glasses capture cropped binocular ROI infrared videos, which are transmitted to a companion smartphone for DSP-based coordinate and angle extraction.The glasses operate as a sensor node, while the smartphone serves as the edge hub.
  • Fixation decomposition: The system decomposes fixation sequences into microsaccades, ocular drifts, and ocular microtremors, whose physiological fidelity is assessed against hardware noise.Ocular drifts occupy approximately 0–40 Hz, while ocular microtremors occupy approximately 40–100 Hz; the stationary-eye baseline has PSD < 10^-4 deg^2/Hz.
  • Microsaccade extraction: Microsaccades are detected through velocity calculation, sliding-window filtering, peak screening by duration, and acceleration-based verification.The duration constraint retains events typically lasting 40–70 ms, while acceleration profiles distinguish microsaccades from macroscopic saccades.
  • Microsaccade extraction: Extracted microsaccade amplitudes and durations align with established physiological norms, including amplitudes below 1°.The paper uses this alignment as external validation of the decomposition pipeline.

4.4. Feature Extraction & Ranking

EmoGaze transforms the three decomposed fixation micromovement sequences into attention-based global features and uses XGBoost to rank their physiological importance. The resulting weighted representation preserves deep temporal information while exposing contributions from microsaccades, ocular drifts, and ocular microtremors.

  • Feature extraction: The feature-extraction stage addresses the difficulty of inferring emotions directly from the three micromovement sequences.Multi-head attention is selected because the signals are time series and the mechanism can model their relationships.
  • Feature extraction: The framework decomposes fixation input into OD, MS, and OMT matrices, concatenates their features, projects them into a shared embedding space, and adds sine positional encoding.This design retains temporal information while combining complementary component information.
  • Feature extraction: The MHA module produces a 128-dimensional global representation using 2 Transformer encoder layers, 4 attention heads, and dropout rate 0.3.The chosen dimensionality balances representation capacity against overfitting and mobile memory overhead.
  • Feature extraction: Multi-head attention uses independently learned queries, keys, and values for the micromovement components to capture global dependencies across their sequences.The head outputs are combined into attention features, then refined with residual connections, layer normalization, and a feed-forward network.
  • Feature importance ranking: XGBoost ranks the attention-derived features by information gain and aggregates dimension-wise scores back to MS, OD, and OMT modalities.This provides a quantitative mapping from model features to physiological drivers of emotional arousal and valence.
  • Feature importance ranking: Min-Max-normalized XGBoost scores become feature-scaling weights before the weighted attention representation is passed to the SVM.A baseline shift of 0.1 suppresses low-contributing features without setting them exactly to zero.

4.5. Emotion Classification

Emotion Classification uses an RBF-kernel SVM on attention features scaled by fold-specific XGBoost importance weights. Nested validation within each leave-one-subject-out split tunes the classifier without using the held-out subject, while SVM support-vector boundaries enable few-shot personalization.

  • Classifier: The final classifier is an RBF-kernel SVM operating on 128-dimensional attention features weighted by XGBoost-derived importance scores.The RBF kernel is used to model nonlinear decision boundaries in physiological data.
  • Validation: All standardization, XGBoost weighting, and SVM training steps are confined to the training folds to prevent data leakage.The feature standardizer is fitted only on training data, followed by fold-specific weighting and classification.
  • Validation: Nested cross-validation tunes SVM C and γ within the remaining 59 training subjects during each LOSO iteration.The internal search evaluates C values from 0.1 to 100 and γ values from 0.001 to 1 plus scale.
  • Validation: The internally selected parameters are then used to train the fold-specific SVM and evaluate the completely unseen test subject.The parameters typically converge around C = 10 and γ = 0.01.
  • Personalization: SVM support-vector decision boundaries can be fine-tuned using a minimal subset of target-user physiological data for personalized deployment.This provides the algorithmic basis for the paper’s few-shot personalization mechanism.

5. Evaluation

The evaluation examines EmoGaze’s accuracy, interpretability, and real-world viability through subject-independent validation, ablation studies, feature-importance analysis, and mobile profiling.

  • Evaluation design: The evaluation uses a two-stage architecture separating workstation-based training and validation from lightweight mobile feature extraction and inference.This design targets pervasive mobile deployment while retaining rigorous offline optimization.
  • System accuracy and personalization: System accuracy is assessed with rigorous Leave-One-Subject-Out Cross-Validation to measure generalization to unseen individuals, complemented by few-shot personalization.
  • Ablation and interpretability: Ablation studies isolate ocular drifts, microsaccades, and ocular microtremors, while XGBoost ranks their feature importance.
  • Mobile viability: Mobile viability is evaluated under naturalistic visual tasks and environmental variation, with end-to-end latency, CPU usage, and memory overhead reported on mobile hardware.

5.1. Experiment Setup and Metrics

The experiment evaluates EmoGaze with 60 volunteers in controlled and naturalistic settings, using PANAS labels and strict LOSO-CV to measure generalization, personalization, feature contributions, and mobile overhead.

  • Participants and apparatus: 60 volunteers, evenly split by sex and spanning ages 18–73, provide data representing varied emotional expressiveness, including acting-trained and untrained participants.
  • Participants and apparatus: The smart glasses capture both eyes at 360p and 500 fps, a sampling rate selected to capture ocular microtremors typically occurring at 40–100Hz.
  • Emotion elicitation: The two-phase protocol combines controlled fixation with naturalistic mobile activities, eliciting Anger, Joy, Sadness, and Pleasure through standardized or embedded stimuli.
  • Ground truth and deployment: PANAS responses immediately after each trial establish self-reported affective ground truth, while processing is profiled on a Snapdragon 8 Gen 2 Android smartphone.
  • Validation protocol: LOSO-CV trains on 59 subjects and tests exclusively on the remaining subject, preventing identity leakage when evaluating unseen-user generalization.
  • Validation protocol: Few-shot personalization fine-tunes the baseline using a minimal fraction of unseen-user data, reflecting inter-individual physiological variability.
  • Metrics: The evaluation reports macro-averaged precision, recall, and F1-score, personalization gain, modality-ablation performance, feature importance, latency, and CPU/memory utilization.

5.2. System Accuracy and Personalization

Under strict LOSO-CV, EmoGaze provides a measurable zero-shot baseline for unseen users, improves with small personalized calibration, and remains effective across demographic groups and mobile-model comparisons.

  • Subject-independent accuracy: 75.5% macro-averaged F1-score is achieved in zero-shot LOSO-CV, alongside 77.4% precision and 73.9% recall on unseen individuals.
  • Subject-independent accuracy: The LOSO-CV confusion matrix shows strong diagonal performance, with minor predictable confusion between neighboring positive states such as Joy and Pleasure.
  • Demographic robustness: EmoGaze remains effective across acting experience, sex, and age groups, with higher accuracy among actors and only marginal variation across sex and age.
  • Personalization: The system addresses cold start by using generalized zero-shot boundaries before transitioning to personalized inference from sporadic user labels.
  • Personalization: Support Vector Transfer retains global support vectors and adds weighted user-specific samples to shift RBF decision hyperplanes without retraining the entire pipeline.
  • Personalization: 83.6% F1-score is reached with 10% calibration data, while performance reaches 84.2% at 20% and shows diminishing returns beyond 10%.
  • Personalization: A 10% calibration proportion corresponds to approximately 4.5 seconds of active physiological data given an average of 45.3 one-second windows per user.
  • Mobile classification: SVM matches heavy deep-learning models at lower computational complexity, supporting real-time mobile classification of extracted features.

5.3. Ablation Study and Feature Importance

The ablation and feature-importance analyses show that emotion recognition depends on combining microsaccades, ocular drifts, and ocular microtremors, whose relative contributions vary by emotion.

  • Modality ablation: The study compares full MS+OD+OMT fusion with macroscopic features and single- or dual-micro-movement subsets.
  • Modality ablation: 47.3% F1-score for Only MS and 54.6% for MS+OD fall below the 62.1% Macroscopic baseline, demonstrating the partial representation paradox.
  • Modality ablation: 75.5% F1-score is achieved when MS, OD, and OMT are fused through multi-head attention, exceeding the macroscopic baseline through cross-modality integration.
  • Feature importance: XGBoost information gain quantifies the relative MS, OD, and OMT contributions for each discrete emotion, making attention-derived features physiologically interpretable.
  • Feature importance: Emotion-specific importance patterns support the interpretation that distinct fixation micro-movements provide robust markers for real-time emotion monitoring.
  • Feature importance: For anger, microsaccades contribute 55.2% and ocular microtremors 30.5% of feature importance, whereas sadness relies on ocular drifts for 63.1%.
  • Feature importance: Pleasure shows a more balanced distribution involving ocular drifts and stable microsaccade activity.

5.4. Mobile Viability and Naturalistic Robustness

EmoGaze remains robust across routine lighting, sounds, and seated or ambulatory tasks, while vigorous movement and strong acoustic disturbances define practical reliability boundaries. A smartphone-centered edge architecture enables real-time processing with modest mobile resource use.

  • Environmental robustness: Precision mildly improves under strong light (L_is, 280–300 lux), while the system remains robust under weak light (L_iw, 110–150 lux).Stronger light provides clearer camera captures.
  • Environmental robustness: Sudden sounds cause a slight accuracy drop, whereas natural environmental and everyday activity sounds preserve relatively high accuracy.Abrupt noises can capture attention and temporarily disrupt focus on the target.
  • Environmental robustness: Sound intensity significantly decreases system accuracy as levels rise from 30–50 dB to 80–90 dB.Levels above 60 dB can interfere with neural activity and confound baseline emotional states.
  • Environmental robustness: Front- and back-originating sounds minimally affect performance, while side sounds have a minor disruptive effect through reflexive visual-attention shifts.Lateral sounds can prompt involuntary eye movements that disrupt stable fixation.
  • Naturalistic robustness: Accuracy remains highly comparable across seated, controlled ambulation, and free indoor/outdoor tasks, declining noticeably only during intensive running.The decline is attributed to severe physical vibrations and motion artifacts.
  • Mobile viability: The architecture assigns sensing to smart glasses and computation to a companion smartphone, avoiding cloud latency and glasses-side battery depletion.The smartphone performs image processing, feature extraction, weighting, and classification locally.
  • Mobile viability: 38.5 ms end-to-end latency, 4.2% background CPU utilization, and 125 MB memory support real-time processing on the smartphone edge hub.The measurements use a continuous 1-second sliding window at 500 fps.

6. Discussions

The discussion identifies scope boundaries involving emotion representation, ecological validity, hardware conditions, privacy, and vigorous or extreme environments. These constraints motivate context-aware, longitudinal, privacy-preserving, and opportunistic deployment strategies.

  • Context-dependent emotion construction: EmoGaze currently uses four discrete valence-arousal categories, whereas future work should model continuous, context-aware affective trajectories.The paper notes that emotions combine physiological signals, cognitive states, and environmental surroundings.
  • Longitudinal in-the-wild deployments: True ecological validity remains untested because evaluation has not yet included prolonged, unsupervised deployments spanning weeks or months.Such studies are proposed to examine fatigue, stress, circadian rhythms, and temporal changes in fixational patterns.
  • Hardware robustness in dynamic contexts: Direct outdoor sunlight of 10,000–100,000 lux can severely saturate the infrared sensors, beyond the evaluated indoor range of 110–300 lux.This establishes a strict operational limitation for the current optical hardware.
  • Secondary privacy implications: Continuous ocular monitoring may expose sensitive physiological or cognitive traits, including neurological conditions and reading habits.The paper calls for on-device anonymization and strict consent protocols.
  • Operating envelope and opportunistic sensing: EmoGaze should operate opportunistically rather than as an unconditional 24/7 monitor because vigorous exercise and extreme high-decibel sudden noises degrade its signals.Future deployments could suspend inference at detected boundary conditions and resume it within the established operating envelope.

7. Conclusion

EmoGaze presents microscopic fixational dynamics as a real-time, unobtrusive emotion-recognition framework implemented through smart glasses and a smartphone edge hub. Subject-independent evaluation and ablation analyses support the informativeness of micro-level eye movements across controlled and naturalistic scenarios, with few-shot personalization addressing individual variability.

  • Conclusion: EmoGaze decomposes visual fixations into microsaccades, ocular drifts, and ocular microtremors for emotion recognition.The framework uses smart glasses with a companion smartphone as an edge-computing hub for millisecond-level analysis and local processing.
  • Conclusion: A rigorous LOSO-CV evaluation involving 60 diverse volunteers demonstrates robust capabilities across controlled and naturalistic mobile scenarios.The conclusion presents these results alongside ablation and feature-importance analyses.
  • Conclusion: Ablation studies and feature-importance analyses validate that micro-level eye movements are more informative and discriminative for emotion inference than macroscopic features.The conclusion frames this as support for the paper’s core hypothesis.
Loading 2609.00846v1…