Source-linked AI summary
Self-Supervised Graph Representation Learning for In-The-Wild Wearable and Smartphone based Emotion Recognition
Ioannis N. Ziogas, Leontios J. Hadjileontiadis, Ahsan H. Khandoker, Aamna Al Shehhi
TL;DR
In-the-wild wearable and smartphone emotion recognition is constrained by scarce, biased labels and inter- and intra-subject emotional variability. The paper uses inductive graph node classification with sampled subgraphs, labeled and unlabeled data, and supervised, semi-supervised, and SSL graph-masking tasks. On K-EmoPhone, the approach improves accuracy over the full-resource setting while using only a fraction of labels for binary arousal and valence recognition.
Problem
In-the-wild wearable and smartphone emotion recognition faces difficult, biased label collection and high inter- and intra-subject emotional variability.
Method
The approach trains a multi-task inductive graph neural network on sampled subgraphs using labeled and unlabeled data with supervised, semi-supervised, and graph-masking SSL objectives.
Results
4.3% and 7.8% average accuracy gains over the full-resource setting were achieved for binary arousal and valence, using 25% and 20% of labels, respectively.
Takeaways & Limitations
Subgraph-based SSL supports limited-resource in-the-wild emotion recognition and can outperform a single large graph with more labels in the evaluated setting.
Abstract
from arXiv · showhide
Wearable and smartphone-based emotion recognition (WER) remains a challenging setting in affective computing, due to the notorious difficulty and bias associated with in-the-wild label collection. The high inter-and intra-subject emotional variability motivates us to explore WER modeling through graph node classification in a limited resources learning scheme powered by Self-Supervised Learning (SSL) graph masking augmentation tasks. We employ a subgraph sampling approach during training, utilizing labeled and unlabeled data, along with supervised, semi-supervised, and SSL mechanisms in a multi-task inductive graph neural network architecture. Our evaluations on K-EmoPhone through leave-one-group-out cross-validation in the binary arousal and valence tasks yield average accuracy gains of 4.3% and 7.8%, compared to the full resource setting, utilizing only 20% and 25% of the labels, respectively. Our model analysis sheds light on the relation of SSL graph augmentations to emotional arousal and valence and justifies the approach of SSL-driven subgraph training for in-the-wild WER.
I. INTRODUCTION
In-the-wild wearable and smartphone emotion recognition is difficult because affect labels are scarce and emotional variability is high. The paper addresses this setting with SSL-based graph representation learning that uses sampled subgraphs and labeled and unlabeled data.
- In-the-wild emotion recognition remains challenging because daily-life emotional manifestations are difficult to analyze accurately.
- Scarce affect labels make efficient use of limited resources central to wearable and smartphone-based emotion recognition.
- SSL can learn without labeled instances, but wearable emotion-recognition applications and GNN-based SSL remain limited.
- Graph representations connect emotionally loaded samples across biosignal and smartphone modalities while embedding labeled and unlabeled instances in a structural similarity space.
- The proposed approach formulates emotion recognition as inductive graph node classification, trains on sampled subgraphs, and integrates graph-masking SSL tasks in a limited-label setting.The authors report that smaller graphs reduce the influence of unlabeled instances and that K-EmoPhone results support limited-resource SSL over full-resource graph baselines.
A. Graph Construction
The method represents time-series segments as graph nodes with embedding attributes, constructs sampled graphs from labeled and unlabeled subsets, and trains a shared graph encoder with multiple objectives.
- A feature encoder maps time-series segments into embeddings, which become graph nodes with embeddings as node attributes.
- Training samples labeled and unlabeled subsets whose union forms the available segment set, enabling inductive subgraph construction.
- Labeled nodes connect to same-label nearest neighbors with weight 1 and farthest labeled neighbors with weight −1.
- A shared graph encoder supports semi-supervised node classification with pseudolabeling and a second self-supervised submodule.
- The semi- and self-supervised losses are weighted by regularization parameters λ1 and λ2, respectively.
1) Supervised Node Classification Task:
The supervised task trains emotion classification on labeled nodes by comparing predicted class distributions with their observed emotion labels.
- For labeled subgraph nodes, the supervised objective computes cross-entropy between softmax emotion logits and their labels.
2) Semi-Supervised Pseudolabeling Task:
The semi-supervised task regularizes predictions on unlabeled nodes, while graph perturbations provide self-supervised signals by aligning representations of original and altered subgraphs.
- Predictions on unlabeled nodes form a regularization term that penalizes uncertainty through low-entropy output distributions.
- The procedure generates pseudolabels at each training iteration and reuses them as additional supervision during inference.
- Self-supervision perturbs each sampled input subgraph and minimizes the difference between original and perturbed node embeddings.
- The framework uses node masking, node attribute masking, Gaussian noise addition, and edge removal as graph augmentations.
C. Inference
At inference, the inductive model forms a subgraph from unseen labeled and unlabeled data, then assigns unlabeled nodes their most frequent pseudo-label across training iterations.
- The model samples labeled and unlabeled nodes from unseen data to construct an inference subgraph.
- Unlabeled nodes receive the most commonly appearing pseudo-label for each instance across training iterations.These pseudo-labels are calculated from the semi-supervised loss.
III. EXPERIMENTS
Experiments use K-EmoPhone, combining experience-sampling affect labels with diverse smartphone and physiological signals. The data are segmented across multiple temporal windows and represented by selected low-level descriptors.
- K-EmoPhone contains data from 77 participants over seven days, including 5589 experience-sampling responses serving as affective labels.Modalities include smartphone logs, battery status, activity, location, accelerometer, electrodermal activity, skin temperature, and heart rate.
- Each experience-sampling response marks the end of a segment for window-wise analysis.
- Segments span 30 seconds to six hours, with resampling, segment-wise normalization, and low-level descriptor extraction.
- The resulting feature vector has 3356 dimensions before feature selection reduces it to approximately 10% using labeled training nodes within each subgraph.
C. Evaluation and Training Settings
Training and evaluation use LOGO cross-validation with limited labeled subjects and sampled subgraphs, while the model uses a shared three-layer GCN and task-specific linear projectors.
- LOGO cross-validation uses a stratified 43/3/1 subject train/dev/test split.
- Approximately 20% of training participants are labeled, with L = 11 and U = 32 subjects designated labeled and unlabeled.
- Subgraph sampling uses 11 labeled and six unlabeled subjects for arousal, versus nine labeled and five unlabeled subjects for valence.
- The shared graph encoder is a three-layer GCN with tanh activations and hidden size 96, while the projectors are one-layer linear heads.
- Table I reports average and standard deviation across four evaluation metrics for the LOGO validation variants.
- Figure 2 plots accuracy against labeled-data percentage and SSL-task selection probability for arousal and valence.
IV. RESULTS
Limited-resource subgraph SSL performs on par with or better than the full-resource setting. Accuracy gains occur for both arousal and valence despite using substantially fewer labels.
- 4.3% higher accuracy is observed for arousal with node masking and pseudolabeling SSL tasks versus the fully supervised setting.
- 7.8% higher accuracy and a 3% macro-F1 increase are observed for valence with node masking and node attribute masking tasks.
- The arousal and valence results use only 25% and 20% of labels, respectively, compared with prior efforts using 100%.
- Semi-supervised performance varies considerably with labeled-data percentage, so adding labels does not always increase accuracy.
- Larger graphs with more labels may not always benefit heterogeneous in-the-wild emotion recognition tasks.
2) Effect of SSL Graph Augmentation Probability:
The study examines how SSL augmentation strength affects emotion recognition in sampled subgraphs. Valence and arousal respond differently, suggesting that their useful information is distributed across graph connectivity and node attributes.
- Valence performance generally increases with stronger node, edge, and attribute perturbations, except for edge removal.The authors attribute edge removal’s behavior to its stronger structural effect on graph connectivity.
- Valence appears to encode emotion-related inter- and intra-subject differences mainly in graph connectivity rather than segment features.
- Arousal performance is more affected by node masking and noise addition than by edge removal.
- The results support SSL-driven subgraph training as an alternative to a single large graph for low-resource wearable emotion recognition.The broader evaluations and ablations were conducted on the 7 days-long K-EmoPhone dataset.