PaperlayerView exact revision on arXiv

Source-linked AI summary

Take it Personally: The Limits of General SSL Representations for Real-Life PPG Emotion Detection

Dominika Kunc, Przemysław Kazienko, Stanisław Saganowski

arXiv:2608.14675v1cs.LGcs.AI

TL;DR

The paper tests whether general PPG self-supervised representations transfer to real-life subjective emotion detection. They transfer strongly to objective physical activity recognition, but emotion performance remains unsatisfactory under LOSO, with personal data driving performance in Across-Time validation.

  • Problem

    Evidence remains limited on whether general PPG self-supervised representations suit subjective, highly individualized emotion detection in unconstrained real-life settings.

  • Method

    The authors pretrain an RL-PPG encoder on unconstrained real-life data and evaluate it across objective activity recognition and subjective emotion detection using LOSO and Across-Time validation.

  • Results

    Almost 5-fold higher F1-score than uniform baselines appears for physical activity recognition, whereas generalized SSL fails to surpass naive emotion-detection baselines under LOSO and personal data leads performance in Across-Time validation.

  • Takeaways & Limitations

    Within the evaluated setup, general SSL representations transfer to objective physical tasks but may be insufficient for subjective emotion inference, making personalization a likely focus for practical affect recognition.

  • Takeaways & Limitations

    The emotion-detection findings were tested on a single dataset, limiting their demonstrated scope.

Abstract

from arXiv · show

While Self-Supervised Learning (SSL) effectively extracts general representations from noisy, unconstrained physiological signals such as photoplethysmography (PPG), its suitability for highly subjective tasks remains unproven. In this work, we evaluate the efficacy of PPG-based SSL for real-life intense emotion detection. First, we pretrain a Real-Life PPG encoder (RL-PPG) on unconstrained, real-life data. As a rigorous sanity check, we demonstrate that these representations transfer exceptionally well to an objective physical activity recognition task, yielding almost 5-fold increase in performance over baselines in a leave-one-subject-out evaluation (LOSO). However, when applied to a~subjective real-life emotion detection task, these same general representations fail to surpass naive baselines under the LOSO protocol. Using an Across-Time validation strategy, we establish that incorporating an individual's personal data during fine-tuning is the main driver of predictive performance, outweighing the benefits of population-level pretraining. Ultimately, our findings indicate that in the evaluated scenario, general SSL representations may be insufficient for subjective affective inference, suggesting that personalization is likely a key component for real-world emotion recognition. To support future research, we share the code and pretrained RL-PPG~encoder~weights.

I. INTRODUCTION · II. RELATED WORK · A. SSL for Physiological Time-Series

The paper motivates real-life physiological affect detection by highlighting gaps in continuous mental-health monitoring and the subjectivity of emotion-related signals. It reviews SSL’s promise for noisy physiological time series while establishing the need to test whether general representations transfer to real-life emotion detection.

  • I. INTRODUCTION: Wearable devices provide detailed physical-health feedback, but comparable tools for continuous mental-health support remain limited.The paper frames emotional awareness as relevant to stress regulation, psychological resilience, and well-being.
  • I. INTRODUCTION: Physiological responses to emotion vary across individuals and contexts, making stimulus- or activity-based laboratory labels potentially misleading.Such labels can cause models to recognize the external stimulus rather than the subject’s affective state.
  • I. INTRODUCTION: LarField uses unconstrained everyday data and reduces the task to detecting the presence or absence of an intense emotional experience.This binary formulation avoids dependence on specific subjective emotion categories while retaining relevance for mental-health applications.
  • I. INTRODUCTION: SSL is motivated by the expense and difficulty of labeling unconstrained real-world data, but affect-related physiological patterns remain deeply subjective.The paper therefore questions whether generalizable SSL patterns from other domains transfer to affective physiological signals.
  • I. INTRODUCTION: The work evaluates real-life PPG SSL representations, finding robust transferability to objective physical tasks but notable limitations for subjective emotion detection.It also investigates personalized modeling through Across-Time validation and releases pretrained RL-PPG encoder weights.
  • II. RELATED WORK: ECG SSL methods such as CLOCS and WildECG demonstrate strong generalization through temporal-spatial augmentation and state-space modeling on real-world data.These approaches establish SSL as an effective methodology for physiological time-series representation learning.
  • II. RELATED WORK: Wearable PPG SSL is less explored because PPG has lower signal quality and greater noise susceptibility than ECG, with existing methods often using multimodal setups.COCOA is cited as an example that learns cross-modal representations through contrastive learning with high label efficiency.
  • A. SSL for Physiological Time-Series: TS-TCC offers a modality-agnostic SSL backbone for noisy real-life PPG, combining temporal and contextual contrasting with competitive low-label and transfer-learning performance.The reviewed results suggest SSL can handle signal noise, subject variability, and limited annotations, motivating its application to real-life emotion detection.

B. Emotion Recognition using Physiological Signals

Emotion recognition research is moving from controlled laboratory datasets toward unconstrained real-life physiological sensing. This shift exposes substantial challenges from ecological variability, noisy signals, sparse subjective labels, and inter-subject differences.

  • Laboratory-based emotion recognition: Laboratory emotion-recognition studies use datasets such as DEAP, AMIGOS, Emognition, and WESAD to support feature-based and deep-learning approaches.These datasets enable diverse modeling approaches but rely on controlled experimental conditions.
  • From laboratory to real life: Controlled stimuli constrain the natural complexity of emotional experiences, whereas real-life settings introduce variability from individual circumstances, appraisals, and environments.The contrast reflects the trade-off between experimental control and ecological complexity.
  • Wearable sensing challenges: Consumer-grade wearables enable everyday emotion capture but face motion noise, sparse subjective ground-truth labels, and substantial inter-individual variation.These devices create opportunities in healthcare and human-computer interaction while complicating real-world affect recognition.
  • Methodological motivation: Laboratory-trained models rarely generalize to real-life scenarios, motivating real-world data for both SSL pretraining and downstream binary detection of subjectively intense emotion.The methodology avoids relying on laboratory datasets or fine-grained categories that struggle to generalize in the wild.

III. SSL REPRESENTATION · A. Pretraining Dataset · B. Architecture

RL-PPG was pretrained on real-life PPG data from the LarField dataset and built on TS-TCC with modifications for 10-second, 25 Hz inputs. The resulting encoder uses compact convolutional architecture and low-magnitude augmentations designed to preserve PPG morphology.

  • A. Pretraining Dataset: The pretraining subset came from LarField’s four-week daily-life monitoring of 48 participants using Samsung Galaxy 3 smartwatches.The watches collected continuous PPG at 25 Hz and ACC at 50 Hz through a custom mobile application.
  • A. Pretraining Dataset: Half of the available subjects were allocated to the test set because subjective emotion labels could otherwise bias evaluation toward individual quirks.The passage states that a smaller test set risks this bias.
  • A. Pretraining Dataset: Preprocessing split the signals into non-overlapping 10-second windows, excluded irregular-length samples, cleaned signals with NeuroKit2, and resampled them to 25 Hz.The 10-second window length followed common practice in prior physiological time-series SSL work.
  • B. Architecture: RL-PPG adopts TS-TCC’s temporal contrasting and contextual contrasting objectives for self-supervised time-series representation learning.Temporal contrasting predicts future representations from past context, while contextual contrasting aligns augmented views of the same signal and separates other signal pairs.
  • B. Architecture: For 10-second PPG windows sampled at 25 Hz, the model adds two convolutional layers, adjusts kernel parameters, uses 10 timesteps for temporal contrasting, and replaces ReLU with GELU.The architectural changes were made to fit the large-scale real-life dataset and improve pretraining stability.
  • B. Architecture: The resulting encoder contains five convolutional layers and approximately 345K trainable parameters.The paper characterizes it as a compact representation model for noisy, real-world physiology.
  • B. Architecture: Contextual contrasting uses jittering and permutation with low-magnitude parameters to avoid destroying PPG’s delicate morphological features.These features are described as important for downstream physiological and affective inference.

IV. EXPERIMENTAL SETUP · A. Datasets and downstream tasks · 1) ProSi:

The experiments test transfer from unconstrained, noisy wearable data to real-life intense emotion detection under strict domain consistency. ProSi provides matched smartwatch PPG data for binary and multiclass physical-activity recognition.

  • IV. EXPERIMENTAL SETUP: The experiments assess whether representations learned from unconstrained, noisy wearable data transfer to the deeply subjective task of real-life intense emotion detection.The evaluation maintains strict domain consistency between pretraining and downstream data to avoid severe domain shifts and support ecological validity.
  • 1) ProSi:: ProSi is a multimodal physiological dataset for physical activity recognition recorded in laboratory and semi-naturalistic settings with 11 participants.Experiments use only Samsung Galaxy Watch 3 recordings, whose PPG is sampled at 25 Hz and matches the pretraining hardware setup.
  • 1) ProSi:: ProSi comprises seven five-minute activities grouped into stationary and running categories.The running session includes treadmill activities at 1, 5, and 9 km/h, designed to produce increasing physical intensity and varying artifacts.
  • 1) ProSi:: Stationary activities include unconstrained sitting, paced breathing, periodic arm raising, and repetitive finger tapping simulating smartwatch interactions.These activities introduce different movement-related and physiological artifacts under semi-naturalistic conditions.
  • 1) ProSi:: The ProSi evaluation includes binary classification distinguishing stationary from running activities.This task measures whether representations transfer to a coarse physical-activity distinction.
  • 1) ProSi:: The ProSi evaluation also includes multiclass classification across the seven individual activities.Together, the two downstream tasks test both coarse and activity-specific recognition.

2) Real-life Intense Emotion: … 1) Leave-One-Subject-Out (LOSO):

The paper evaluates real-life emotion and physical-activity recognition using carefully constructed physiological datasets and subject-disjoint validation. The emotion dataset is balanced overall but highly variable across participants, while LOSO simulates cold-start generalization without prior data from the test subject.

  • 2) Real-life Intense Emotion:: The Real-life Intense Emotion dataset is a hold-out test set from the pretraining LarField dataset, with questionnaires triggered by predicted intense emotional events.The mobile application continuously monitored participants’ physiology and triggered a questionnaire when a feature-based model was confident an intense event was present.
  • 2) Real-life Intense Emotion:: The dataset contains 24 subjects and 3066 labeled samples: 1475 intense and 1591 non-intense emotions.Although the overall label distribution is well-balanced, intense versus non-intense labels vary substantially across participants.
  • 2) Real-life Intense Emotion:: This participant-level label variability makes real-life emotion detection difficult to generalize across people.The passage links the variability in label distributions to differences in people’s real-life experiences.
  • 3) Real-life Physical Activity Recognition:: Physical-activity ground-truth labels were generated with an unsupervised heuristic pipeline based on accelerometer data from each temporal window.The pipeline used triaxial magnitude to reduce device-placement effects before deriving movement-related statistics.
  • 3) Real-life Physical Activity Recognition:: Movement intensity was estimated from the variance of triaxial acceleration magnitude after a natural logarithmic transformation.The transformation addressed right skew in the variance measurements.
  • 3) Real-life Physical Activity Recognition:: A three-component Gaussian Mixture Model was fit separately per subject to distinguish stationary, ambiguous intermediate, and other movement states.The per-subject GMM accounted for inter-subject variability and ambiguous real-world motion.
  • B. Cross-Validation Scenarios: Window-level random splits can leak information when neighboring samples from one individual appear in both training and test sets.Such leakage can artificially inflate performance and misrepresent model generalization.
  • 1) Leave-One-Subject-Out (LOSO):: LOSO holds out one participant per fold with disjoint subjects across splits, simulating cold-start deployment without prior data for a new user.The protocol generally produces lower performance than random splits but more robustly assesses cross-subject generalization.

2) Across-Time (AT): · C. Evaluation

The Across-Time evaluation chronologically splits each participant’s data and compares cold-start, general, and personal training variants under a shared test set. Downstream models use identical encoder–MLP architectures with either scratch or RL-PPG initialization, weighted training, naive baselines, and macro-F1 evaluation.

  • 2) Across-Time (AT):: The initial 30% of days form training, the subsequent 20% validation, and the remaining 50% test, compensating for declining self-report compliance.The chronological split is applied per participant in the Across-Time setup.
  • 2) Across-Time (AT):: Cold Start excludes the test participant’s data from training to measure temporally constrained leave-one-subject-out generalization.This variant evaluates pure generalization without participant-specific training data.
  • 2) Across-Time (AT):: General combines the test participant’s initial days with data from all other subjects, reflecting deployment with prior user data and a global dataset.This setup leverages both participant-specific history and population data.
  • 2) Across-Time (AT):: Personal trains exclusively on each test participant’s initial data, isolating participant-specific characteristics while keeping the test set identical across variants.The procedure is conducted for all 24 participants in the real-life intense emotion dataset.
  • C. Evaluation: Downstream classification uses the same representation-encoder–MLP architecture with either scratch initialization or pretrained RL-PPG weights.The compared models differ only in whether SSL-pretrained weights initialize the encoder.
  • C. Evaluation: Class imbalance is addressed with weighted cross-entropy, assigning class-specific weights inversely proportional to class frequencies.The weighting uses the total sample count, per-class sample count, and total number of classes.
  • C. Evaluation: Performance is reported as mean ± standard deviation of macro-averaged F1-score across participant folds, contextualized against majority-class and uniform-random baselines.Learning rates are selected per task from {0.1, 0.05, 0.005, 0.0005}.

V. RESULTS · A. Physical Activity Recognition Sanity Check

The physical-activity sanity check shows that SSL-based RL-PPG representations transfer strongly across laboratory and real-life PPG tasks. End-to-end fine-tuning consistently yields the best results, including macro F1-scores of 0.90, 0.64, and 0.88 across evaluated settings.

  • A. Physical Activity Recognition Sanity Check: Frozen FS and RL-PPG encoders substantially outperform naive baselines in laboratory binary activity recognition, with frozen RL-PPG improving F1 by roughly 0.08 over frozen FS.The task distinguishes stationary from running activity under LOSO validation.
  • A. Physical Activity Recognition Sanity Check: +0.11 and +0.07 are added by end-to-end fine-tuning to the FS and RL-PPG encoders, respectively, in the laboratory binary task.Finetuned RL-PPG reaches a macro F1-score of 0.90 despite PPG indirectly capturing movement through cardiovascular shifts.
  • A. Physical Activity Recognition Sanity Check: SSL pretraining strictly outperforms training from scratch in laboratory multiclass recognition of seven activities, while end-to-end fine-tuning achieves the highest performance.Overall performance decreases relative to the binary setting, but the performance ordering remains consistent.
  • A. Physical Activity Recognition Sanity Check: SSL approaches consistently outperform FS variants in both frozen and finetuned configurations on real-life binary activity recognition.This setting distinguishes stationary from non-stationary activity under LOSO validation.
  • A. Physical Activity Recognition Sanity Check: Table I reports F1 macro scores for laboratory and real-life physical-activity recognition under LOSO validation, with FS denoting training from scratch.The results establish transferability of the learned representations across the evaluated activity-recognition settings.
  • A. Physical Activity Recognition Sanity Check: 0.88 is the highest overall F1-score achieved by Frozen RL-PPG on the real-life binary task.The simpler labeling scheme isolates extreme stationary and non-stationary states, unlike the more nuanced laboratory behaviors.

B. Intense Emotion Detection (LOSO)

Under LOSO validation, PPG models and naive baselines show little separation on subjective intense emotion detection, unlike the physical-activity sanity check. The results suggest that excluding each participant’s data removes personalized context needed for individualized emotional responses, motivating Across-Time validation.

  • LOSO results: Macro F1-scores remain narrowly clustered from 0.43 to 0.48 across all evaluated LOSO methods, including naive baselines.The frozen FS encoder exceeds the uniform baseline by only 0.02.
  • LOSO results: 0.46 is the identical performance of both end-to-end-finetuned FS and RL-PPG encoders, matching the uniform baseline.In the frozen setting, RL-PPG also fails to outperform the FS encoder.
  • Cross-subject limitation: LOSO excludes the test participant’s data during training, leaving models without personalized context for decoding highly individualized emotional responses.The findings identify externally learned general representations as potentially insufficient for subjective internal experiences.
  • Personalization: Across-Time validation experiments are introduced to test whether participant-specific data can bridge the subjectivity gap.These experiments follow the observed drop in cross-subject generalization.

C. Intense Emotion Detection (Across-Time)

Across-Time validation shows that generalized representations do not beat naive baselines in cold-start emotion detection, whereas incorporating a participant’s personal data yields the strongest and only consistently superior performance. External-population pretraining adds no supplementary benefit beyond personalized affective data.

  • Cold-Start: A uniform baseline reaches the maximum cold-start F1-score of 0.49, while none of the evaluated models surpasses naive baselines.Cold-start excludes validation and test participants from training, mirroring LOSO with smaller, temporally constrained datasets.
  • General: Access to subject-specific data produces minor improvements ranging from +0.01 for the finetuned FS encoder to +0.05 for the frozen FS encoder.The General variant adds participants’ initial days to training alongside the general dataset, but overall gains remain modest.
  • Personal: Finetuned encoders exceed the uniform baseline by 0.05 in the Personal variant, the only Across-Time scenario where models consistently outperform naive guessing.Training and validation use only the test participant’s historical data, eliminating inter-subject noise.
  • Personal: The FS and RL-PPG encoders achieve identical performance in both frozen and finetuned settings, showing no supplementary benefit from external-population pretraining.The performance gain is driven entirely by isolating and using highly personalized affective data.
  • Participant-level analysis: Personal data yields the highest F1-scores for 19 of 24 subjects, and finetuned RL-PPG significantly outperforms cold-start (p=0.004), general (p=0.007), and majority (p=0.009).Wilcoxon signed-rank tests were conducted across 24 folds.

VI. DISCUSSION AND LIMITATIONS · VII. CONCLUSIONS

The study finds that general SSL representations transfer strongly to objective physical activity recognition but fail to overcome cross-subject limitations in subjective real-life emotion detection. The authors emphasize personalization, acknowledge substantial data and sensing constraints, and outline directions for more ecologically valid affective AI.

  • VI. DISCUSSION AND LIMITATIONS: Almost 5-fold increase in F1-score over uniform baselines was achieved by finetuned SSL in multiclass physical activity recognition under LOSO.This sanity check supports transfer of meaningful, universally transferable PPG structures to objective real-world tasks.
  • VI. DISCUSSION AND LIMITATIONS: Generalized SSL pretraining did not improve subjective intense emotion detection, which showed severe cross-subject generalization limitations under LOSO.The results for real-life emotion detection with general SSL models remained unsatisfactory.
  • VI. DISCUSSION AND LIMITATIONS: Personalization is supported as a crucial factor for effective affect recognition from physiological signals.The discussion links this interpretation to recent work on personalized physiological affect recognition.
  • VI. DISCUSSION AND LIMITATIONS: The evaluation used a single real-life emotion dataset, while subject-level personalization left relatively little labeled data available.The dataset choice reflects a lack of fully unconstrained, open-access longitudinal affective datasets with physiological signals.
  • VI. DISCUSSION AND LIMITATIONS: Real-world PPG responses to reported intense emotions may be absent or obscured, creating a fundamental limit for ambulatory sensing.Low physiological arousal or unrecorded contextual factors may override the affective response.
  • VI. DISCUSSION AND LIMITATIONS: Future work should explore variable window lengths, alternative architectures, and pretraining on mixed real-life and laboratory datasets.Fully personalized SSL pretrained on an individual’s unlabeled data is theoretically optimal for subjective tasks, but required longitudinal data remains a severe practical bottleneck.
  • VII. CONCLUSIONS: Generalized SSL captures transferable physiological patterns for objective physical tasks but remains limited by the subjective nuances of ambulatory emotion detection.The severe cross-subject generalization gap indicates that personalization is likely necessary for the evaluated affective AI setup.

ETHICAL IMPACT STATEMENT

The study addresses ethical risks through approved data-collection protocols, transparency, and plans for open data sharing. It also highlights substantial limitations in participant diversity, clinical representation, sensor hardware, personal context, and potential misuse.

  • Ethical safeguards: Approved protocols governed participant consent, data handling, and study procedures for all datasets used in the work.The protocols were approved by the Wrocław Tech Ethics Committee under approval no. O-23-08.
  • Transparency and data sharing: The Real-life Emotion Dataset is not yet publicly available, but procedures are being finalized to enable safe and open sharing.The authors frame this commitment as part of transparency and open science.
  • Representation limitations: The dataset includes healthy participants from a single nationality and language, limiting investigation of how cultural backgrounds influence personalized modeling.The authors call for multi-site, cross-cultural datasets to address this limitation.
  • Clinical limitations: Participants with cardiovascular disease or mental health disorders were excluded because these conditions or related medications could alter physiological baselines, affective reactivity, or subjective reporting.The study consequently emphasizes that its findings should not be generalized to individuals with such conditions.
  • Hardware bias: BMI of 30 or higher led to participant exclusion because of PPG signal degradation, while darker skin tones may experience sensor performance disparities.The authors recommend multimodal architectures pairing PPG with physiological sensors less affected by these optical limitations.
  • Inference limitations and dual use: Personalized affective inference cannot capture full personal context and is vulnerable to temporal drift in subjective self-assessments, while emotion-recognition systems may be misused for surveillance or harm.The authors advocate transparency, participant protection, responsible innovation, and stronger technical and ethical safeguards.
Loading 2608.14675v1…