PaperlayerView exact revision on arXiv

Source-linked AI summary

IDEAL: A Multimodal Domain Adaptation Framework for EEG-Eye Emotion Recognition

Yang Wu, Jinpeng Li

arXiv:2609.39421v1cs.HC

TL;DR

EEG–eye emotion recognition must contend with individual variability and multimodal heterogeneity, while common adaptation methods emphasize feature alignment and may overlook data-level discrepancy. IDEAL combines target-instance expansion with hierarchical adversarial feature alignment. It is reported to outperform state-of-the-art methods across four benchmark datasets.

  • Problem

    Domain adaptation methods commonly focus on feature-level alignment and do not adequately address multimodal heterogeneity or data-level discrepancy in EEG–eye emotion recognition.

    Source: §I · p. 1, C. Multimodal Emotion Recognition · p. 3

  • Method

    IDEAL screens high-confidence target samples for instance-level expansion, then uses multimodal Transformer features and hierarchical adversarial alignment with angular-contrastive constraints.

    Source: §I · p. 2, 2) Hierarchical Feature Interaction · p. 5, 5) Hierarchical Adversarial Alignment… · p. 6

  • Results

    IDEAL consistently outperforms state-of-the-art methods across four benchmark datasets under two LOSO protocols.

    Source: F. Discussion · p. 9

  • Takeaways & Limitations

    The results support IDEAL as a framework for cross-subject emotion recognition with EEG and eye-movement signals.

    Source: §V · p. 9

    The two-stage architecture incurs higher offline computational cost due to Stage 1 precomputation, although deployable training time is reported as comparable to end-to-end methods.

    Source: F. Discussion · p. 9

Abstract

from arXiv · show

Electroencephalography (EEG) emotion recognition serves as a pivotal interface for human-computer interaction, yet the physiological variability across individuals complicates the already challenging task of fusing heterogeneous physiological signals (e.g., EEG and eye movements). However, most prevalent domain adaptation paradigms are tailored for unimodal scenarios, failing to address the heterogeneity of multimodal signals. Furthermore, they predominantly rely on feature-level alignment, overlooking the fundamental data-level discrepancy, which risks compromising fine-grained discriminative information during aggressive adaptation. To bridge these coupled gaps, we propose Instance-based Domain Expansion and Adversarial Learning (IDEAL), a unified framework that synergizes instance-level curriculum expansion with feature-level hierarchical adversarial alignment. IDEAL first introduces a multi-model collaborative screening mechanism, which propagates high-confidence target samples to explicitly bridge the distributional gap at the data level via a quantity-quality equilibrium strategy. We provide a theoretical analysis that this instance expansion strategy strictly tightens the upper bound of the target risk. Subsequently, a hierarchical adversarial network, augmented with the angular-contrastive constraints, progressively aligns representations from low-level statistics to high-level semantics while preserving class separability. Extensive experiments on four benchmark datasets demonstrate that IDEAL significantly outperforms state-of-the-art methods. To facilitate reproducibility and future research, our source code is publicly available at https://github.com/WY-BCI-Club/IDEAL.

I. INTRODUCTION

EEG’s low signal-to-noise ratio and non-stationarity, together with cross-modal heterogeneity and individual differences, complicate robust EEG–eye emotion recognition. IDEAL combines instance-level expansion with hierarchical feature adaptation to address these gaps, and is reported to outperform state-of-the-art methods across four public datasets.

I. INTRODUCTION

  • Existing EEG–eye fusion methods can exhibit modality bias and superficial cross-modal integration, leaving hierarchical alignment between neural and behavioral signals insufficiently addressed.Unimodal EEG is also constrained by low signal-to-noise ratio and non-stationarity.Source: §I · p. 1
  • Feature-level adaptation risks negative transfer by compromising emotional-feature discriminability, while instance-level methods struggle to balance sample quantity and quality.The cited approaches also remain sensitive to hyperparameter settings.Source: §I · p. 1
  • IDEAL unifies instance-level curriculum expansion and feature-level hierarchical adaptation to address individual-induced domain shift and multimodal heterogeneity.Its design screens high-confidence target samples, then progressively aligns features from shallow statistics to deep semantics.Source: §I · p. 2
  • The curriculum expansion mechanism selects high-value target samples without target labels using prediction consistency and a quantity–quality balance objective.The stated aim is to narrow the distribution gap at the data level.Source: §I · p. 2
  • A multimodal Transformer mines complementary EEG and eye-movement information, while hierarchical adversarial learning progressively aligns features at different levels.The feature discriminability objectives are described separately in the paper’s contribution summary.Source: §I · p. 2
  • IDEAL achieves state-of-the-art performance on cross-subject tasks across four public datasets.The framework combines angular and contrastive objectives to enhance feature discriminability and mitigate negative transfer.Source: §I · p. 2

B. EEG Emotion Recognition

Cross-subject EEG emotion recognition faces individual-driven distribution shifts, while multimodal EEG–eye methods must address both signal integration and domain adaptation. The framework formulates this setting as domain adaptation and motivates a shared-structure subset for transfer before hierarchical feature alignment.

B. EEG Emotion Recognition

  • Unimodal EEG methods remain constrained by signal non-stationarity and low signal-to-noise ratio, motivating integration with complementary modalities such as eye movements.Prior EEG work described in the passage spans deep graph, adaptive sampling, attention, and recurrent approaches.Source: B. EEG Emotion Recognition · p. 2

C. Multimodal Emotion Recognition

  • Existing multimodal methods predominantly emphasize feature-level alignment and neglect instance-level transfer, leaving fusion and data-level domain shift coupled in cross-subject settings.The section characterizes this as the joint question of how to fuse and what to transfer.Source: C. Multimodal Emotion Recognition · p. 3

A. Problem Formulation, Modeling and Analysis

  • The cross-subject task is formulated as domain adaptation under covariate shift, with source and target marginal distributions differing due to individual physiological variability.The source contains labeled samples from multiple subjects, whereas the target comprises unlabeled samples from a new subject.Source: A. Problem Formulation, Modeling and… · p. 3
  • The subset-selection assumption is a shared latent subspace in which selected target samples are both close to the source distribution and representative of the target.These properties are intended to support transfer while preserving target relevance.Source: A. Problem Formulation, Modeling and… · p. 3
  • The subset objective maximizes the number of selected target samples while bounding their divergences from both source and target distributions.The formulation frames this as balancing transfer quantity against negative-transfer risk.Source: A. Problem Formulation, Modeling and… · p. 3
  • Under effective screening, augmenting the source domain with a non-empty target subset strictly reduces domain divergence and tightens the target generalization upper bound.The proposition’s stated condition is the criterion for effective screening; the empty subset is the degenerate case.Source: A. Problem Formulation, Modeling and… · p. 4
  • IDEAL proceeds in two stages: collaborative screening builds an intermediate domain, then a multimodal Transformer applies hierarchical adversarial alignment from low-level statistics to high-level semantics.The framework figure describes the same two-stage structure.Source: Figure 1 · p. 3, A. Problem Formulation, Modeling and… · p. 4

B. Instance-Level Domain Expansion

IDEAL screens unlabeled target samples through agreement among multiple source-trained observers, then selects a subset using a quantity–quality objective that adapts to domain shift.

B. Instance-Level Domain Expansion

  • Three heterogeneous observers use EEG-only, eye-only, and concatenated views to produce aggregated target-sample predictions, with consensus serving as a proxy for proximity to shared structure.The observers are Random Forest, Extremely Randomized Trees, and CatBoost, with view aggregation weights 0.3, 0.3, and 0.4.Source: B. Instance-Level Domain Expansion · p. 4
  • The selection ratio balances conservative high-quality selection against aggressive selection that risks including low-consensus samples and negative transfer.The objective uses the minimum consensus among selected samples as a conservative quality measure and penalizes excessive quantity.Source: B. Instance-Level Domain Expansion · p. 4
  • The trade-off parameter adapts to estimated target-sample quality: severe domain shift prioritizes quality, while greater source–target overlap favors quantity.The passage says this adaptive design avoids manual threshold tuning.Source: B. Instance-Level Domain Expansion · p. 4
  • A traversal search over the threshold selects the final expanded source domain, which the proposition states yields a strictly tightened target generalization bound.The expanded domain is the source domain combined with the selected target subset.Source: B. Instance-Level Domain Expansion · p. 4

C. Network Framework

The network progressively transforms low-level signals into semantic embeddings through three blocks.

C. Network Framework

  • The pipeline uses multi-view embedding for feature disentanglement, a hierarchical Transformer for intra- and inter-modal interaction, and a decision head for emotion prediction.Its stated aim is robust cross-subject emotion recognition.Source: C. Network Framework · p. 4

1) Multi-view Embedding and Attention-Fusion:

The multi-view embedding projects each modality into parallel semantic views, then gates and attention-fuses them into compact EEG and eye-movement features.

1) Multi-view Embedding and Attention-Fusion:

  • Each modality is projected into V parallel views to decompose physiological signals into diverse semantic subspaces.The view projections use independently initialized learnable parameters intended to encourage semantic diversity.Source: 1) Multi-view Embedding and… · p. 5
  • A global gating vector weights view informativeness, and the weighted embeddings are stacked and batch-normalized.The stated purpose is to suppress irrelevant noise while enhancing affect-salient features.Source: 1) Multi-view Embedding and… · p. 5
  • Attention fusion aggregates the EEG and eye-movement feature banks into compact representations.For EEG, a learnable attention vector computes the global importance score; the eye features are obtained similarly.Source: 1) Multi-view Embedding and… · p. 5

2) Hierarchical Feature Interaction:

A two-stage Transformer refines each modality independently before modeling joint EEG–eye information to produce a deeply fused representation.

2) Hierarchical Feature Interaction:

  • Independent Unimodal Transformers use multi-head self-attention to model long-range temporal or spectral dependencies within EEG and eye features.Their outputs are refined modality-specific features.Source: 2) Hierarchical Feature Interaction · p. 5
  • A Mix-Transformer processes concatenated refined EEG and eye features to model their joint distribution and mine complementary information.Unlike simple summation, it yields a fused representation containing modality-invariant and modality-specific affective cues.Source: 2) Hierarchical Feature Interaction · p. 5

3) Emotion Prediction:

The fused representation is classified over emotion categories, while class-separation objectives constrain both feature geometry and decision boundaries.

3) Emotion Prediction:

  • A classifier maps the fused representation to probabilities over emotion categories, and cross-entropy loss trains the predictions.The passage defines the target indicator and predicted probability for each observation and class.Source: 3) Emotion Prediction · p. 5

4) Discriminative Manifold Learning (Lcs):

  • Class Separation loss combines contrastive constraints with an angular-margin objective to promote intra-class compactness and inter-class separability.The constraints act at the feature level through contrastive learning and at the decision-boundary level through angular margin loss.Source: 4) Discriminative Manifold Learning… · pp. 5–6
  • Supervised intra-modal InfoNCE uses same-class samples as positives, while cross-modal consistency aligns EEG and eye pairwise similarity patterns.The total contrastive loss balances modality-specific structure and cross-modal alignment, with γ set to 0.3.Source: 4) Discriminative Manifold Learning… · pp. 5–6
  • ArcFace adds an angular margin to the fused representation’s classification criterion, encouraging angularly separable features.The stated margin makes the classification criterion stricter.Source: 4) Discriminative Manifold Learning… · p. 6

5) Hierarchical Adversarial Alignment (Ldomain):

Hierarchical adversarial alignment applies domain confusion at shallow, intermediate, and deep feature levels; the final objective combines classification with the stated constraints.

5) Hierarchical Adversarial Alignment (Ldomain):

  • Three domain discriminators target low-, mid-, and high-level features, while gradient reversal makes the feature extractor minimize domain-classification accuracy.The three levels correspond to shallow, intermediate, and deep representations.Source: 5) Hierarchical Adversarial Alignment… · p. 6
  • The alignment weights are ηa,b,c = {0.2, 0.3, 0.5}, and the text states that the multi-granularity strategy captures domain-invariant patterns from raw statistics to semantic abstractions.The final optimization objective integrates classification with the previously described constraints; λ1,2 = {0.3, 0.5} balance discrimination and alignment.Source: 5) Hierarchical Adversarial Alignment… · p. 6

IV. EXPERIMENTS AND RESULTS

IDEAL is evaluated on four multimodal emotion datasets under two leave-one-subject-out protocols, with results reported as mean accuracy and standard deviation. It outperforms baselines across the reported results, with especially strong performance under the All-Session protocol.

A. Experimental Setup

  • The evaluation covers SEED, SEED-IV, SEED-V, and SEED-VII, which have 3, 4, 5, and 7 emotion classes, respectively; features are Z-score normalized per subject and session.All four datasets contain EEG and eye-movement signals.Source: A. Experimental Setup · p. 6
  • The two protocols are Intra-Session LOSO, with one subject held out within each session, and All-Session LOSO, with all sessions of one subject held out together.Results are reported as mean accuracy with standard deviation.Source: A. Experimental Setup · p. 6
  • Training first screens M high-confidence target samples, then combines them with N source instances, where N = min(max(1500, NS ·M), 3000).NS is the total number of source samples.Source: A. Experimental Setup · p. 6
  • IDEAL is benchmarked against classic transfer-learning and multimodal methods using identical partitioning protocols.The listed classic baselines are DANN, JDA, and MWACN; the comparison also includes multimodal methods such as CFDA-CSF, MMDA, and RHPRNet.Source: A. Experimental Setup · p. 6

B. Emotion Classification Performance

  • IDEAL reaches 95.69% average accuracy on SEED, ahead of runner-up CSMM (94.96%), and leads all baselines on SEED-IV and SEED-V in the Intra-Session results.Its first-session SEED accuracy is 97.91%, 4.86% above CFDA-CSF; it also has the lowest standard deviation on SEED (5.21%) and SEED-V (8.46%).Source: B. Emotion Classification Performance · pp. 6–7
  • Under All-Session evaluation, IDEAL reaches 86.79% on SEED-IV, 1.25% above MMDA (85.54%), and 88.13% on SEED-V, above MMDA (86.77%).On SEED, IDEAL scores 94.31%, compared with MMDA (94.82%) and CFDA-CSF (90.47%); on SEED-IV its standard deviation is 4.50% versus MMDA's 8.11%.Source: B. Emotion Classification Performance · p. 7, Table II · p. 7

C. Ablation Studies

Ablations identify instance-level domain expansion and hierarchical adversarial alignment as central components, while quantitative analyses examine screening quality, its limits, and alignment during training. Visualizations show distributions moving closer through expansion and feature alignment before forming emotion clusters.

C. Ablation Studies

  • Removing domain expansion or hierarchical adversarial loss causes the largest accuracy drops across datasets (p < 0.001), while the multi-level adversarial strategy outperforms single-layer DANN.Removing ArcFace or contrastive loss individually yields moderate declines; their combination is reported as important for optimal class separability.Source: C. Ablation Studies · p. 7

D. Quantitative Analysis of Domain Expansion Strategy

  • Screening ratio and accuracy show an inverted-U relationship: under-screening transfers too little data, while over-screening admits noisy samples and can cause negative transfer.The adaptive objective selects subject-specific ratios, including 14.73% and 36.46%, and reaches peak performance without manual tuning.Source: D. Quantitative Analysis of Domain… · p. 7, Figure 3 · p. 8
  • The screening analysis uses class-aware MMD as a proxy for H∆H-divergence to test whether the selected subset is closer to the target domain than the source domain.The reported condition is violated for Subject 10, while the other listed subjects satisfy it.Source: D. Quantitative Analysis of Domain… · p. 7, Figure 4 · p. 8
  • Across more than 90% of subjects, the screened subset satisfies the stated distribution condition and achieves positive transfer; Subject 10 is a counterexample associated with observer errors.The reported failure rate is below 10% (p < 0.001 across trials); violation in Subject 10 leads to marginal performance degradation.Source: D. Quantitative Analysis of Domain… · pp. 7–8, Figure 4 · p. 8

E. Convergence Analysis and Visualization

  • IDEAL's deep-feature PAD reaches approximately 0.002, while deep-layer PAD is consistently lower than shallow-layer PAD, indicating progressively refined domain invariance.DANN converges to dA ≈ 0.17 with considerable fluctuations; without domain expansion, convergence is slower and more variable.Source: E. Convergence Analysis and… · p. 8, Figure 5 · p. 8
  • UMAP visualizations show initially disjoint source and target distributions drawing closer after domain expansion and becoming well-aligned after feature-level adaptation.The final features form distinct, compact clusters for Positive, Neutral, and Negative emotions.Source: E. Convergence Analysis and… · p. 9, Figure 6 · p. 9

F. Discussion

IDEAL consistently outperforms state-of-the-art methods across four benchmark datasets under two LOSO protocols. The paper frames its approach as curriculum expansion followed by hierarchical adversarial alignment.

F. Discussion

  • IDEAL consistently outperforms state-of-the-art methods across four benchmark datasets under two LOSO protocols.The discussion attributes gains to data-level expansion that preconditions feature alignment, progressive domain invariance across abstraction levels, and adaptive thresholds that balance quantity and quality by subject.Source: F. Discussion · p. 9
  • Screening depends on consensus among RF, ET, and CatBoost; weak classifier diversity or accuracy can violate its condition and marginally degrade performance.The discussion identifies Subject 10 as a rare failure case and notes that the two-stage architecture raises offline precomputation cost, though deployable training time remains comparable to end-to-end methods.Source: F. Discussion · p. 9
  • Future work targets missing-modality handling through cross-modal imputation and improved generalization by strategically weighting multiple source domains.Source: F. Discussion · p. 9

V. CONCLUSION

  • IDEAL’s curriculum-first design screens high-confidence target samples to bridge the data-level gap, then learns domain-invariant, semantically discriminative representations with three-level adversarial constraints.The conclusion calls this sequence the “Curriculum-then-Alignment” philosophy and reports experiments on the SEED series outperforming state-of-the-art baselines.Source: §V · p. 9
Loading 2609.39421v1…