PaperlayerView exact revision on arXiv

Source-linked AI summary

RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

Xingyi He, Ziwei Wang, Dongrui Wu

arXiv:2609.11372v1cs.AI

TL;DR

AAD seeks to identify the attended speaker, but EEG alone can provide incomplete evidence in naturalistic audio-visual scenes and existing fusion methods have limited interaction and robustness. RAMamba-Net combines modality-specific EEG-EOG encoding, cross-modal attention, and sample-wise reliability-aware fusion. Across two AAD benchmarks, it improves decoding accuracy and robustness while producing more discriminative representations.

  • Problem

    AAD requires complementary EEG-EOG evidence because EEG alone provides incomplete information in naturalistic audio-visual scenes, while existing approaches have weak cross-modal interaction, inefficient temporal modeling, and limited robustness to sample variations.

  • Method

    RAMamba-Net combines modality-specific EEG and EOG encoders, Mamba-based temporal modeling, cross-modal attention, and reliability-aware sample-wise modality weighting.

  • Results

    RAMamba-Net consistently improves decoding performance, robustness, and representation quality on the AVGC and DTU benchmarks.

  • Takeaways & Limitations

    Explicit cross-modal interaction improves multimodal alignment, while reliability-aware fusion uses complementary information and suppresses unreliable modality evidence.

Abstract

from arXiv · show

Auditory attention decoding (AAD) identifies the attended speaker from physiological signals, supporting neuro-steered hearing devices and natural human-machine interaction. Electroencephalography (EEG) is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes, motivating EEG and electrooculography (EOG) fusion. Existing approaches remain limited by weak cross-modal interaction, inefficient temporal modeling, and low robustness to sample variations. To address the limitations, we propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD. RAMamba-Net employs a Mamba-enhanced band-aware convolutional Transformer to capture band-specific EEG patterns and long-range temporal dynamics. A dual-branch temporal-spatial encoder models EOG temporal and inter-channel dependencies. Cross-modal attention enables explicit modality interaction. Then, a reliability-aware module is introduced to estimate sample-wise modality weights for feature and prediction consistency, thereby enhancing multimodal fusion. Experiments on two AAD benchmarks demonstrate that RAMamba-Net effectively exploits complementary EEG-EOG information, yielding accuracy gains of 5.76% over unimodal baselines, together with more robust decoding and discriminative representations. Further analyses show that explicit cross-modal interaction improves multimodal alignment, while the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.

Introduction

AAD decodes the attended speaker from physiological signals but faces incomplete EEG evidence, weak cross-modal modeling, and variable signal quality in realistic scenes. RAMamba-Net addresses these challenges through modality-specific encoding, cross-modal interaction, and reliability-aware fusion.

  • AAD infers the attended speaker from physiological signals for neuro-steered hearing devices, target-speaker enhancement, and natural human-machine interaction.
  • EEG is the dominant AAD modality, but low signal-to-noise ratio, artifacts, and poor cross-subject and cross-session generalization limit its evidence.
  • EOG provides complementary behavioral and ocular information, yet confounding ocular patterns require explicit modeling rather than naïve concatenation.
  • EEG-EOG fusion remains challenged by insufficient cross-modal dependency modeling and inefficient long-sequence temporal modeling.
  • RAMamba-Net integrates modality-specific encoding, cross-modal interaction, and sample-wise reliability estimation in a unified EEG-EOG AAD framework.

Related Work

Prior AAD work has advanced EEG representation learning and explored complementary eye-movement signals, but unified EEG-EOG modeling remains insufficient. These limitations motivate explicit modality-specific and cross-modal integration.

  • EEG-only methods model spectral, spatial, and temporal neural patterns, but low signal-to-noise ratios and artifacts hinder stable representations across recording conditions.
  • Eye tracking and EOG provide behavioral evidence related to attended speech and can support attention labeling or target-speaker detection alongside neural signals.
  • Existing multimodal work remains constrained by indirect eye-movement supervision and simplified EOG modeling, leaving EEG-EOG complementarity insufficiently captured.

Method

RAMamba-Net combines band-aware EEG encoding, dual-branch EOG modeling, bidirectional cross-modal attention, and sample-wise reliability-aware fusion. Its composite training objective supervises multimodal, unimodal, and band-specific predictions.

  • Architecture: RAMamba-Net uses an EEG encoder, an EOG encoder, cross-modal fusion, and classifiers in an end-to-end multimodal pipeline.The encoders produce modality-specific representations, fusion enables bidirectional interaction, and classifiers predict attended speakers from fused and auxiliary features.
  • EEG Encoder: Mamba blocks selectively propagate temporal information before and within the EEG band-aware convolutional Transformer.The front block reinforces raw band-specific temporal structure, while the middle block enriches representations before Transformer processing.
  • EOG Encoder: The EOG encoder combines temporal and spatial Transformer branches to capture fine-grained temporal dependencies and inter-channel relationships.Their outputs are concatenated and projected into dimension-aligned tokens for subsequent cross-modal interaction.
  • Cross-Modal Interaction: Cross-modal attention lets each modality query the other, retrieving complementary information through symmetric token-wise interaction.Residual learning, normalization, and feed-forward refinement preserve modality-specific information while promoting cross-modal alignment.
  • Reliability-Aware Fusion: RAM estimates sample-wise modality reliability from pooled descriptors and feature- and prediction-consistency scores, then regulates interacted representations before classification.Mean pooling forms fixed-dimensional descriptors, while the reliability mapping produces modality weights that modulate the fused representations.
  • Training Objective: A composite objective jointly supervises multimodal predictions, unimodal branches, and each EEG frequency band.The unimodal loss preserves modality-specific discriminative capability, while the band-wise loss supervises independent frequency-band classifiers.

Experiments and Results

RAMamba-Net is evaluated on two public AAD benchmarks using standardized preprocessing, chronological subject-wise partitioning, and multiple decoding metrics. The experiments use repeated seeded training and report accuracy, balanced accuracy, Macro-F1, and Cohen’s Kappa.

  • Datasets: RAMamba-Net is evaluated on AVGC and DTU, two public AAD datasets containing EEG and EOG recordings from competing-speaker tasks.AVGC evaluates 12 subjects after exclusions, whereas DTU includes all 18 participants.
  • Datasets: AVGC contains 64 EEG and 4 EOG channels at 8,192 Hz, while DTU contains 64 EEG and 6 EOG channels at 512 Hz.The datasets differ in speaker directions, trial durations, and recording conditions.
  • Preprocessing: Signals undergo dataset-specific filtering and resampling, followed by modality-specific spatial-pattern extraction and frequency-band decomposition.AVGC is resampled to 128 Hz and DTU to 64 Hz; common spatial pattern filters are estimated from training data.
  • Data Partitioning: Chronological partitioning yields an approximately 8:1:1 training, validation, and testing split after independent 2-second decision-window segmentation.The split is performed before windowing to keep each continuous recording segment within an independent subset.
  • Implementation: Experiments use five random seeds, Adam optimization, early stopping, and dataset-specific batch sizes on a single NVIDIA GeForce RTX 3090 GPU.Models train for up to 200 epochs with patience 10, learning rate 5 × 10^-4, and weight decay 3 × 10^-4.

Unimodal Decoding

Unimodal evaluation shows that EEG is consistently stronger than EOG, while EOG remains above chance and contributes complementary attention-related information.

  • EEG consistently outperformed EOG across datasets and models, while EOG remained clearly above chance level.
  • FAConformer achieved the best EEG decoding, whereas DBConformer achieved the best EOG decoding.The results suggest frequency-aware temporal-spectral modeling suits EEG, while temporal-spatial dual-branch modeling suits EOG.
  • EEG provides stronger neural evidence, while EOG captures complementary ocular dynamics that motivate modality-specific encoders.

Main Results

Main-result comparisons show that explicit EEG-EOG interaction improves over direct concatenation, while RAMamba-Net achieves the strongest performance across both datasets and all reported metrics.

  • Explicit cross-modal interaction produced more consistent improvements than direct concatenation by facilitating information exchange between EEG and EOG.Direct concatenation yielded only limited gains, indicating that simple feature fusion does not fully exploit complementary information.
  • RAMamba-Net achieved the best results across all metrics and both datasets, with larger gains on AVGC.Table 2 compares unimodal and multimodal approaches, including direct concatenation and RAMamba-Net without reliability-aware modulation.
  • Reliability-aware modulation jointly improved accuracy, balance, and robustness by adaptively regulating modality contributions.

Feature Visualization

t-SNE comparisons show progressively better organization from EEG-only features to direct fusion and then RAMamba-Net, whose representations form the clearest class structure on both datasets.

  • The visualization compared EEG-only, direct concatenation, and RAMamba-Net features on AVGC and DTU before classification.
  • EEG-only features showed limited separability, with attention samples still mixed in the embedding space.
  • Direct concatenation introduced EOG information but left feature distributions insufficiently organized, indicating weak cross-modal alignment.
  • RAMamba-Net yielded the clearest class structure on both datasets, with more compact intraclass clusters and larger inter-class margins.The improvement was more evident on DTU, where the attention classes formed well-structured regions.

Robustness to Noise Injection

Noise-perturbation experiments show that direct concatenation is especially vulnerable to EEG degradation, whereas RAMamba-Net maintains higher accuracy across nearly all modality-noise conditions.

  • The robustness comparison used heatmaps for classification accuracy under modality-specific noise combinations and for the resulting accuracy gain.
  • RAMamba-Net maintained higher accuracy across nearly all EEG and EOG noise conditions, with gains remaining largely positive under joint corruption.The experiment independently varied each modality’s noise level from 0 for clean signals to 1 for severe corruption.
  • Direct concatenation lost accuracy mainly as EEG noise increased, while EOG noise had less pronounced effects on both datasets.This pattern indicates that direct fusion remains strongly dominated by EEG and is less robust when EEG is degraded.
  • Bidirectional interaction preserves complementary information, while RAM suppresses unreliable inputs and rebalances modality contributions under perturbation.

Ablation and Sensitivity Analysis

The complete RAM module achieved the highest accuracy on both datasets, while RAMamba-Net remained stable across broad settings of its key hyperparameters.

  • Ablation Study: The complete RAM module achieved the highest accuracy on both AVGC and DTU by combining complementary reliability terms.Modality preservation, feature consistency, and prediction consistency contributed differently across datasets, while their combination produced the strongest result.
  • Ablation Study: Feature consistency produced the largest term gain on AVGC, whereas modality preservation was most effective on DTU.
  • Sensitivity Analysis: Accuracy remained stable across broad ranges of λuni, β, and Hc, indicating limited sensitivity to hyperparameter variation.The analysis covered auxiliary-supervision weighting, reliability-module strength, and cross-modal attention heads on AVGC and DTU.

Conclusion

RAMamba-Net combines modality-specific EEG and EOG encoders, cross-modal attention, and sample-wise reliability weighting for multimodal AAD. Experiments on AVGC and DTU showed improved decoding performance, robustness, and representation quality, while future work targets distribution shifts and efficient deployment.

  • Conclusion: RAMamba-Net combines modality-specific EEG and EOG encoders, cross-modal attention, and sample-wise reliability weighting for multimodal AAD.
  • Conclusion: Experiments on AVGC and DTU showed improved decoding performance, robustness, and representation quality through explicit interaction and reliability modeling.
  • Conclusion: Future work targets cross-subject and cross-dataset generalization under distribution shifts, along with efficient adaptation and deployment.
Loading 2609.11372v1…