Source-linked AI summary
Modeling Shared and Individual Structure for Cross-Subject Continuous Affect Regression from EEG-fNIRS
Xuan Wang, Bing Wang, Shuai Chang, Hao Yuan, Xinbo Qi, Xinyue Zhang
TL;DR
The paper addresses continuous valence-arousal prediction for new subjects whose labels are unavailable, using synchronized EEG-fNIRS recordings. It combines a shared stimulus-based trajectory with an EEG-derived per-subject gain; the authors report that it outperforms EEGNet and ASAC-Net baselines on the subject-independent task.
Problem
Zero-shot cross-subject affect regression asks whether valence and arousal can be predicted for new users without labeled target-subject data, a constraint relevant to deployment without lengthy per-user calibration.
Source: §1 · p. 1Method
The model combines a population-average trajectory for shared stimuli with a per-subject gain estimated from label-free alpha-band synchrony in the test subject's EEG.
Source: §1 · p. 1, §4.2 · p. 2Results
The authors report that the model substantially outperforms EEGNet and ASAC-Net on the same subject-independent evaluation protocol, and no searched architecture, feature, or target improved on the single synchrony scalar.
Source: §7 · p. 4, §5.4 · p. 4Takeaways & Limitations
The reported calibration is close to the achievable limit for arousal, while the authors identify more headroom for valence and suggest a different valence-specific marker may be needed.
Source: §6 · p. 4Calibration validation relies on 24 training subjects, limiting the precision of the synchrony-to-gain regression and leave-one-out ceiling estimates.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Continuous, second-by-second valence-arousal estimation from physiological signals is typically studied in a subject-dependent setting, where the model sees labeled data from the same person it is later evaluated on. We study the harder zero-shot cross-subject variant on a synchronized EEG-fNIRS dataset: predict raw-scale ([1, 255]) valence and arousal trajectories for subjects whose labels the model never observes, given only their unlabeled EEG/fNIRS recordings while watching the same video stimuli as a disjoint set of training subjects. We decompose the affect trajectory into a structure shared across subjects who watch the same stimuli and an individual structure estimated for each test subject from a label-free EEG marker (alpha-band cross-channel synchrony), which rescales the shared trajectory around the scale midpoint. We validate the per-subject calibration mechanism on four independent axes: leave-one-subject-out correlation between the marker and each subject's true optimal gain, a functional-form comparison against non-linear alternatives, a repeated leave-4-out component ablation isolating each part of the pipeline's contribution, and a ceiling analysis bounding the remaining headroom for per-subject scaling. On held-out subjects, the model reaches an overall MAE of 25.96 / 22.80 across two evaluation batches (valence 21.94 / 19.6, arousal 29.98 / 26.0), well below EEGNet and ASAC-Net baselines reported for the same subject-independent split (raw scale score 60.6 and 55.0 respectively). We further report a systematic negative-result search across model architectures, feature representations, and prediction targets that found no signal able to improve on the single alpha-synchrony marker.
1 Introduction
The paper targets zero-shot cross-subject valence-arousal regression, where target subjects provide no labels and personalization must use their unlabeled recordings. It combines a shared stimulus-linked trajectory with an EEG-derived subject gain and validates the calibration mechanism alongside a systematic search for alternatives.
1 Introduction
- Zero-shot evaluation must predict for new subjects without labeled target data, despite EEG non-stationarity and substantial inter-subject variability.The setting matters for deployment without lengthy per-user calibration.Source: §1 · p. 1
- The model combines a population-average trajectory for shared stimuli with a per-subject gain derived from EEG alpha-band synchrony.The report describes an auditable pipeline intended to outperform published EEGNet and ASAC-Net baselines on the same subject-independent protocol.Source: §1 · p. 1
- The calibration mechanism is evaluated through leave-one-out correlation, functional-form comparison, component ablation, and a ceiling analysis.A three-axis negative-result search also covers architectures, feature representations, and prediction targets.Source: §1 · p. 1
2 Related Work
Prior cross-subject EEG emotion work spans several approaches, while continuous affect datasets with direct brain activity and zero-label personalization remain less represented in the cited landscape. This work combines continuous EEG-fNIRS affect data with label-free test-time calibration for regression.
2 Related Work
- Cross-subject EEG emotion recognition addresses the failure of same-subject or same-distribution assumptions, with solutions spanning alignment, graph modeling, representation learning, generative modeling, and multimodal fusion.Personalization approaches with reported cross-subject gains typically use labeled target-subject samples.Source: §2 · pp. 1–2
- Most benchmark datasets cited use EEG with largely static per-clip labels, whereas CASE and CEAP-360VR provide continuous annotations from peripheral physiology.The dataset used here combines continuous annotation with synchronized EEG and fNIRS.Source: §2 · p. 2
- SATTC and T3A-style adjustment use unlabeled target-subject structure, and this paper applies that test-time calibration family to continuous regression with one physiological scalar.The cited EEG-fNIRS fusion literature includes graph convolutions and cross-modal attention; the paper reports its comparable learnable fusion network does not transfer under its zero-shot protocol.Source: §2 · p. 2
3 Dataset and Task Formulation
The task uses synchronized EEG-fNIRS recordings and continuous joystick reports to evaluate raw-scale valence-arousal prediction under a strict subject-independent split.
3 Dataset and Task Formulation
- Thirty-two participants watched 15 emotion-eliciting clips while 64-channel EEG and 51-channel fNIRS were recorded simultaneously.The clips span five targeted valence-arousal quadrants.Source: §3 · p. 2
- Participants continuously reported affect with a joystick, producing 1 Hz valence-arousal trajectories on the integer scale [1, 255] with neutral point (128, 128).Source: §3 · p. 2
- The strict subject-independent protocol fits every pipeline component on 24 participants and holds out eight others in two disjoint batches of four.Primary performance is MAE on the raw [1, 255] scale, averaged across valence and arousal.Source: §3 · p. 2
4 Method
The method estimates a shared trajectory for each video and adjusts it for each test subject using an EEG-derived scalar gain. The gain rescales deviations around the affect-scale midpoint.
4.1 Population-Mean Trajectory
- For each video and timestep, a Huber M-estimator across the 24 training subjects followed by a length-5 median filter produces a robust central label estimate.The procedure is applied independently to valence and arousal to form one shared trajectory per video.Source: §4.1 · p. 2
4.2 Per-Subject Gain from EEG Alpha-Band Synchrony
- The prediction is 128 + gain_subject * (popmean[video, t] − 128), scaling the shared trajectory's deviation from the scale midpoint.The gain comes from a label-free marker: mean absolute off-diagonal cross-channel correlation of alpha-band (8–13 Hz) power across 64 EEG channels.Source: §4.2 · pp. 2–3
- A clipped linear mapping from synchrony to gain uses an OLS fit of training subjects' synchrony against their optimal gains, with shrinkage and sensitivity parameters.If the recording is too short or the synchrony estimate is non-finite, the method uses a fallback gain of 0.90.Source: §4.2 · p. 2
- Figure 1 depicts the combination of the training-derived population trajectory and a scalar gain computed from the test subject's EEG.Source: Figure 1 · p. 3
5 Experiments
The experiments compare the model under a subject-independent protocol and examine component contributions, calibration validity, and alternative per-subject signals.
5.1 Main Results
- Table 1 reports overall test MAE on the raw [1, 255] scale, comparing the model with EEGNet and ASAC-Net on the same 24-train/8-test split.Both baselines use their published EEG+fNIRS configuration.Source: Table 1 · p. 3, §5.1 · p. 3
5.2 Component Ablation
- The population trajectory dominates; fixed shrinkage is indistinguishable from noise, while per-subject calibration consistently improves MAE by 0.45.The same qualitative ordering appears on the held-out ladder.Source: §5.2 · p. 3
- A learnable multimodal residual network showed no statistically distinguishable cross-subject generalization gain and was excluded from the deployed model (λ=0).The ablation used a graph-convolution, cross-modal-attention, and GRU network.Source: §5.2 · p. 3
5.3 Validating the Calibration Mechanism
- Sync correlated with deployed shared oracle gain (r = −0.465, 95% CI [−0.731, −0.076], p < .05), driven mainly by arousal; valence alone was not significant.Nonlinear alternatives did not improve LOO-MAE (0.194 vs. 0.195-0.210). The deployed calibration was within 1.44 ± 0.52 MAE for arousal and 3.01 ± 1.00 MAE for valence of a perfect scale-only oracle.Source: §5.3 · p. 4
5.4 Negative-Result Search
- Searches across architectures, feature representations, and prediction targets found no approach that improved on the single sync scalar.The search included trajectory-matching and residual-learning models, fNIRS and EEG features, and alternative prediction targets.Source: §5.4 · p. 4, Table 2 · p. 4
6 Discussion and Limitations
The calibration uses a single multiplicative scalar, with different remaining headroom for arousal and valence; validation precision is limited by the subject pool.
6 Discussion and Limitations
- The calibration reduces each subject’s individual differences to a single multiplicative scalar, which is close to the arousal ceiling but leaves more valence headroom.The sync marker’s correlation with valence gain was not significant in this sample; the authors suggest a different valence-specific marker may be needed.Source: §6 · p. 4
- Validation on 24 training subjects limits the precision of both the sync-to-gain regression and leave-one-out ceiling estimates.A larger subject pool would sharpen both estimates.Source: §6 · p. 4
- The sync marker’s valence correlation did not reach significance in this sample, leaving more headroom for valence than for arousal.The authors note that closing the valence gap likely requires a different valence-specific marker.Source: §6 · p. 4
7 Conclusion
The paper concludes that a population-average trajectory rescaled by a physiologically derived per-subject gain outperforms trained baselines on the subject-independent task.
7 Conclusion
- A population-average trajectory rescaled by a single physiologically derived per-subject gain substantially outperforms trained EEGNet and ASAC-Net baselines on the subject-independent EEG-fNIRS affect-regression task.Source: §7 · p. 4
- The calibration was assessed through leave-one-out correlation, functional-form robustness, component ablation, and ceiling analysis; a systematic search found no improvement over the single alpha-synchrony marker.Source: §7 · p. 4