PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning the Incomplete: Joint Distribution Calibration for Multimodal EEG-Eye Emotion Recognition

Yang Wu, Jinpeng Li

arXiv:2609.39413v1cs.HC

TL;DR

Cross-subject EEG shifts and missing eye-movement signals jointly distort the multimodal distribution, while existing methods address adaptation and imputation separately. GUARD combines gradient-weighted source anchoring, perceptually supervised generation, and cycle-consistent unsupervised target calibration. It reports state-of-the-art results on SEED-series benchmarks and resilient decoding under complete auxiliary-modality failure.

  • Problem

    Domain adaptation and missing-modality imputation have been treated as separate problems, although their concurrent occurrence distorts the joint distribution used for cross-subject multimodal emotion recognition.

    Source: Introduction · pp. 1–2

  • Method

    GUARD combines gradient-weighted source anchoring, perceptually supervised ocular-feature generation, and cycle-consistent unsupervised calibration on target EEG.

    Source: Introduction · p. 2

  • Results

    GUARD reports state-of-the-art results on SEED-series benchmarks and resilient performance under complete auxiliary-modality failure.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The reported results support GUARD as a unified approach to cross-subject emotion recognition when eye-movement inputs are unavailable.

    Source: Introduction · p. 2, Related Work · p. 2

Abstract

from arXiv · show

The success of cross-subject multimodal emotion recognition hinges on maintaining the consistency of the joint data distribution across individuals. However, real-world deployment frequently triggers the \emph{asymmetric joint distribution collapse}: EEG signals suffer from severe cross-subject distribution shifts, while eye movements sensors are susceptible to packet loss and tracking failures. Existing methods treat domain adaptation and missing-modality imputation as disjoint tasks. Consequently, they fail to resolve the compounded errors when both degradations co-occur, either propagating domain shifts through imputed signals or destroying the joint decision boundary. To tackle this unified challenge, we propose GUARD (\textbf{G}radient-guided \textbf{U}nsupervised \textbf{A}symmetric \textbf{R}ecovery of \textbf{D}istributions). First, GUARD establishes a reliable anchor manifold in the source domain by employing a theoretically grounded gradient-weighted objective, which forces the robust EEG modality to preemptively entangle task-discriminative ocular features. Next, to structurally recover the collapsed joint distribution, we constrain a generative module with downstream perceptual losses, prioritizing emotion-discriminative semantics over mere signal fidelity. Finally, we formulate target-domain adaptation as an ill-posed inverse problem. By driving a cycle-consistent flow, we achieve unsupervised calibration of the recovered joint distribution directly on the target-domain manifold. Extensive experiments demonstrate that GUARD significantly outperforms state-of-the-art methods, maintaining resilient discriminative performance even under complete auxiliary modality failure. Our code and models are made publicly available to ensure complete reproducibility.

Introduction

The paper frames cross-subject EEG shifts and missing eye-movement signals as coupled degradations of the multimodal joint distribution. It introduces GUARD to recover and calibrate that distribution when the auxiliary modality is absent.

Introduction

  • Missing eye-movement data removes discriminative information and cross-modal dependencies while amplifying the effective domain shift.The resulting collapse can make domain-agnostic completion synthesize signals that increase semantic confusion rather than support emotion discrimination.Source: Introduction · pp. 1–2
  • GUARD combines gradient-weighted channel reconstruction, perceptually supervised ocular-feature generation, and cycle-consistent unsupervised target calibration.Its source-domain objective gives greater reconstruction weight to channels with larger sample-specific classifier gradients.Source: Introduction · p. 2
  • GUARD reports state-of-the-art results on the SEED series and resilient emotion decoding under complete auxiliary-modality failure.The supplied passage gives no benchmark values.Source: Introduction · p. 2

Related Work

Related work treats cross-subject adaptation and missing-modality imputation as separate problems, despite their shared vulnerability to distribution mismatch. GUARD formulates them as an asymmetric joint-distribution alignment task, matching the complete source joint distribution to the incomplete, shifted target EEG marginal.

Related Work

  • Conventional adaptation assumes complete, matching modality structures, while imputation methods often assume training and test data are identically distributed.These assumptions make the approaches fragile to target modality deficiencies and new subjects with distinct physiological traits.Source: Related Work · p. 2
  • Existing domain-adaptation and imputation pipelines address separate challenges, while GUARD unifies recovery and alignment over an asymmetric joint distribution.The paper identifies disjoint pipelines as a source of propagated imputation errors and suboptimal solutions.Source: Related Work · p. 2

A. Problem Formulation

  • GUARD frames the target task as aligning the complete source joint distribution with a structurally collapsed and distributionally shifted target EEG marginal.Eye-sensor loss removes the target conditional eye-given-EEG distribution, while subject variability shifts both EEG marginals and cross-modal mappings.Source: A. Problem Formulation · p. 3
  • GUARD first entangles task-discriminative cross-modal features into source EEG representations to establish a robust anchor manifold.That anchor is intended to keep the target EEG marginal from becoming semantically vacuous.Source: A. Problem Formulation · p. 3

2. Latent joint distribution recovery to model the condi-

The paper models the missing ocular conditional distribution with a generator, then calibrates the recovered target joint space using a cycle-consistent self-supervised flow.

2. Latent joint distribution recovery to model the condi-

  • A generative mapping models the lost conditional eye-given-EEG term to reconstruct the collapsed joint structure.The mapping is constrained by task-perceptual losses to maximize emotion discriminability rather than relying on signal fidelity alone.Source: §2 · p. 3
  • The generated target joint distribution remains misaligned because the target eye-given-EEG conditional differs from the source conditional.Source: §2 · p. 3
  • A cycle-consistent self-supervised flow treats calibration as an overdetermined equation-solving problem to align the recovered target joint space with the source anchor manifold.Source: §2 · p. 3

B. Multimodal entanglement and alignment

The method uses a Transformer-based classifier and cross-modal reconstructor to form a source-domain anchor. Its training objective combines classification, gradient-weighted reconstruction, and adversarial alignment.

B. Multimodal entanglement and alignment

  • The classifier processes modality-specific Transformer features and their concatenation through another Transformer encoder before MLP classification.Source: B. Multimodal entanglement and alignment · p. 4
  • The cross-modal reconstructor is a Transformer decoder that uses one modality’s representation as queries and the other’s as keys and values.It supports complementary-information capture during source training and provides an anchor for self-supervised calibration.Source: B. Multimodal entanglement and alignment · p. 4
  • Reconstruction error’s influence on prediction is modulated by channel-gradient magnitude, motivating sample-adaptive channel weights.The standard unweighted reconstruction loss remains part of the objective to ensure basic signal fidelity.Source: B. Multimodal entanglement and alignment · p. 4
  • The source-domain objective combines classification, G-MSE reconstruction, and adversarial alignment to produce an entangled, subject-invariant joint manifold.The reconstruction term uses an exponential moving-average buffer with momentum m = 0.9.Source: B. Multimodal entanglement and alignment · p. 4

C. Latent Joint Distribution Recovery

GUARD recovers the missing ocular conditional distribution by mapping EEG to eye-movement signals. It trains this generator with both physical-space consistency and downstream perceptual supervision from a frozen main model.

C. Latent Joint Distribution Recovery

  • GUARD learns the missing conditional P(x_eye | x_eeg) with a cross-modal MLP generator that maps EEG to eye-movement signals.The generator synthesizes pseudo eye movements while the pre-trained main model remains frozen.Source: C. Latent Joint Distribution Recovery · p. 4
  • The generator objective combines physical-space consistency with downstream perceptual supervision because raw-signal fidelity alone is insufficient for discriminative tasks.The synthesized signals pass through the frozen main model for this supervision.Source: C. Latent Joint Distribution Recovery · p. 4
  • Classification and G-MSE losses on synthesized ocular signals prioritize discriminative semantics over signal fidelity and anchor recovery to the source manifold.Freezing the other modules prevents spurious drift in the recovered conditional distribution.Source: C. Latent Joint Distribution Recovery · p. 4

D. Unsupervised Asymmetric Alignment

GUARD calibrates a source-biased recovered joint distribution on the target manifold without target eye-movement labels. It treats calibration as an ill-posed inverse problem and uses cyclic consistency and knowledge distillation to constrain the recovery.

D. Unsupervised Asymmetric Alignment

  • Although the generator recovers the conditional distribution's form, it is biased toward the source condition, while target conditional shift lacks ground-truth ocular labels for direct adaptation.GUARD therefore formulates target calibration as an ill-posed inverse problem, using the source joint manifold as a regularizer.Source: D. Unsupervised Asymmetric Alignment · p. 4
  • With only target EEG observed, the reconstruction target is that EEG, yielding an overdetermined, highly nonlinear, severely ill-posed system with no closed-form solution.The EEG-reconstruction component is denoted W_eeg.Source: D. Unsupervised Asymmetric Alignment · p. 5
  • GUARD introduces a latent MLP mapping conditioned on target EEG and relaxes the overdetermined constraint into a cyclic reconstruction objective.A Lagrangian-style consistency regularizer confines optimization to a physiologically meaningful subspace.Source: D. Unsupervised Asymmetric Alignment · p. 5
  • Closed-loop consistency preserves structural dependencies of the joint distribution, and knowledge distillation transfers the refined representation back to the generator.The consistency penalty relaxes the hard constraint into a least-squares penalty, mitigating ill-posedness.Source: D. Unsupervised Asymmetric Alignment · p. 5
  • Unsupervised calibration aligns recovered ocular representations with target EEG and the classifier's decision boundary without ground-truth target eye-movement data.At inference, complete target data bypass the generator; missing-eye data are reconstructed before multimodal classification.Source: D. Unsupervised Asymmetric Alignment · p. 5

Experiments

Experiments use three EEG–eye movement datasets with leave-one-subject-out settings described for the reported evaluation. The paper also specifies training stages and a broad set of adaptation and classification baselines.

Experiments

  • The experiments use SEED, SEED-IV, and SEED-V, with 12, 15, and 16 subjects and 3, 4, and 5 emotional states, respectively.EEG has 62 channels across five frequency bands; eye movements have 33 features for SEED and SEED-V or 31 for SEED-IV.Source: A Experiments · p. 5
  • Training alternates through three stages each epoch, with 300 main-model epochs and self-supervised adaptation every 10 epochs.The main model uses batch size 2000; adaptation uses batch size 256 and 50 internal steps per trigger.Source: A Experiments · p. 5
  • GUARD is benchmarked against domain-adaptation, multimodal, and conventional classifiers, including DANN, MMDA, CSMM, CMGNN, and SVM.The listed baselines also include TCA, DGCNN, RGNN, BiDANN, BiHDM, PGCN, and UDDA.Source: A Experiments · p. 5

B. Emotion Classification Performance

GUARD reports the best results across the three complete multimodal datasets and leads on SEED and SEED-IV when target inputs contain EEG only. The comparisons use mean accuracy across subjects under leave-one-subject-out cross-validation.

B. Emotion Classification Performance

  • GUARD led all three multimodal datasets: SEED-V averaged 95.30% (+4 percentage points over MMDA), SEED-IV reached 92.26% vs CMGNN (90.21%), and SEED reached 94.99% vs CSMM (94.96%).Table 1 reports mean accuracy across subjects under leave-one-subject-out cross-validation for complete multimodal input and the missing-modality setting.Source: B. Emotion Classification Performance · p. 5, Table 1 · p. 6
  • With EEG-only target inputs, GUARD led SEED (91.06%, vs BGAGCN-MT 89.66%) and SEED-IV (77.33%, vs BGAGCN-MT 75.78%), gains of 1.40 and 1.55 percentage points.The reported missing-modality setting reconstructs eye movements from EEG.Source: B. Emotion Classification Performance · pp. 5–6, Table 1 · p. 6

C. Ablation Study

GUARD is evaluated under increasing target eye-modality loss and through component ablations. The reported comparisons favor joint recovery and alignment over decoupled pipelines, while ablations identify contributions from the model’s components.

Remove component Results

  • A 20.05% performance drop on SEED-V follows zeroing eye signals, supporting the complementary role of eye movements.The ablation compares the full model with eye signals zeroed on SEED-V Session 2.Source: C. Ablation Study · p. 6
  • Removing G-MSE reduces performance, indicating uniform reconstruction weights do not emphasize discriminative features.The reported ablations also find lower performance with signal-fidelity-only generator training and without adversarial alignment.Source: C. Ablation Study · p. 6

Discard proportionally

  • GUARD outperforms Gen→DA and DA→Gen across all target eye-missing ratios, with the gap widening as modality loss increases.The comparison uses the challenging SEED-V Session 2, Subject 6 setting; Figure 3 reports accuracy.Source: Figure 3 · p. 6, C. Ablation Study · p. 6

Remove component Results

  • Removing the self-supervised flow significantly hurts performance; cycle-consistent calibration bridges the conditional distribution gap without target ocular labels.The flow ablation is reported in the component results.Source: C. Ablation Study · p. 6

Discard proportionally

  • GUARD achieves the lowest and most stable domain discrepancy over training epochs, while the baselines diverge and fluctuate.Figure 4 plots Proxy A-distance (PAD); GUARD’s periodic ripples coincide with episodic self-supervised triggers.Source: C. Ablation Study · p. 6, Figure 4 · p. 7

D. Visualization

The visualizations examine feature distributions and channel sensitivity. GUARD-generated eye movements reduce the observed source–target discrepancy, while channel sensitivities vary across samples, motivating sample-adaptive weighting.

Distribution visualization

  • With EEG alone, source and target distributions show a clear shift; adding GUARD-generated eye movements substantially reduces the discrepancy.In the classification feature space, clusters become more compact and domain boundaries nearly vanish.Source: D. Visualization · p. 7

Importance analysis

  • Channel sensitivity varies considerably across samples within the same class, motivating sample-wise gradient weighting rather than uniform attention.Figure 6 heatmap values are gradient magnitudes, with warmer colors indicating higher decision sensitivity; similar patterns can also occur across classes.Source: D. Visualization · p. 7, Figure 6 · p. 7

Conclusion

GUARD frames cross-subject adaptation and missing-modality completion as joint-distribution recovery and calibration. Its gradient-weighted EEG anchor, perceptually constrained eye-signal recovery, and cycle-consistent target calibration are evaluated on SEED-series datasets.

Conclusion

  • GUARD reframes cross-subject adaptation and missing-modality completion as joint-distribution recovery and calibration.It uses sample-adaptive gradient weighting to build a task-discriminative EEG anchor, perceptual constraints to recover P(xeye | xeeg), and a cycle-consistent self-supervised flow to calibrate on the target manifold without target ocular labels.Source: Conclusion · p. 7
Loading 2609.39413v1…