PaperlayerView exact revision on arXiv

Source-linked AI summary

RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

Xingyi He, Ziwei Wang, Dongrui Wu

arXiv:2609.11372v1cs.AI

TL;DR

AAD seeks to identify the attended speaker, but EEG alone can provide incomplete evidence in naturalistic audio-visual scenes and existing fusion methods have limited interaction and robustness. RAMamba-Net combines modality-specific EEG-EOG encoding, cross-modal attention, and sample-wise reliability-aware fusion. Across two AAD benchmarks, it improves decoding accuracy and robustness while producing more discriminative representations.

  • Problem

    AAD requires complementary EEG-EOG evidence because EEG alone provides incomplete information in naturalistic audio-visual scenes, while existing approaches have weak cross-modal interaction, inefficient temporal modeling, and limited robustness to sample variations.

    Source: Introduction · p. 1

  • Method

    RAMamba-Net combines modality-specific EEG and EOG encoders, Mamba-based temporal modeling, cross-modal attention, and reliability-aware sample-wise modality weighting.

    Source: Introduction · p. 2, Reliability-Aware Fusion · p. 4

  • Results

    RAMamba-Net consistently improves decoding performance, robustness, and representation quality on the AVGC and DTU benchmarks.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Explicit cross-modal interaction improves multimodal alignment, while reliability-aware fusion uses complementary information and suppresses unreliable modality evidence.

    Source: Main Results · p. 5, Feature Visualization · p. 6, Ablation Study · p. 7

Abstract

from arXiv · show

Auditory attention decoding (AAD) identifies the attended speaker from physiological signals, supporting neuro-steered hearing devices and natural human-machine interaction. Electroencephalography (EEG) is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes, motivating EEG and electrooculography (EOG) fusion. Existing approaches remain limited by weak cross-modal interaction, inefficient temporal modeling, and low robustness to sample variations. To address the limitations, we propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD. RAMamba-Net employs a Mamba-enhanced band-aware convolutional Transformer to capture band-specific EEG patterns and long-range temporal dynamics. A dual-branch temporal-spatial encoder models EOG temporal and inter-channel dependencies. Cross-modal attention enables explicit modality interaction. Then, a reliability-aware module is introduced to estimate sample-wise modality weights for feature and prediction consistency, thereby enhancing multimodal fusion. Experiments on two AAD benchmarks demonstrate that RAMamba-Net effectively exploits complementary EEG-EOG information, yielding accuracy gains of 5.76% over unimodal baselines, together with more robust decoding and discriminative representations. Further analyses show that explicit cross-modal interaction improves multimodal alignment, while the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.

Introduction

AAD decodes the attended speaker from physiological signals but faces incomplete EEG evidence, weak cross-modal modeling, and variable signal quality in realistic scenes. RAMamba-Net addresses these challenges through modality-specific encoding, cross-modal interaction, and reliability-aware fusion.

  • AAD infers the attended speaker from physiological signals for neuro-steered hearing devices, target-speaker enhancement, and natural human-machine interaction.Source: Introduction · p. 1
  • EEG is the dominant AAD modality, but low signal-to-noise ratio, artifacts, and poor cross-subject and cross-session generalization limit its evidence.Source: Introduction · p. 1
  • EOG provides complementary behavioral and ocular information, yet confounding ocular patterns require explicit modeling rather than naïve concatenation.Source: Introduction · p. 1
  • EEG-EOG fusion remains challenged by insufficient cross-modal dependency modeling and inefficient long-sequence temporal modeling.Source: Introduction · p. 1
  • RAMamba-Net integrates modality-specific encoding, cross-modal interaction, and sample-wise reliability estimation in a unified EEG-EOG AAD framework.Source: Introduction · pp. 1–2

Related Work

Prior AAD work has advanced EEG representation learning and explored complementary eye-movement signals, but unified EEG-EOG modeling remains insufficient. These limitations motivate explicit modality-specific and cross-modal integration.

Method

RAMamba-Net combines band-aware EEG encoding, dual-branch EOG modeling, bidirectional cross-modal attention, and sample-wise reliability-aware fusion. Its composite training objective supervises multimodal, unimodal, and band-specific predictions.

Architecture

  • RAMamba-Net uses an EEG encoder, an EOG encoder, cross-modal fusion, and classifiers in an end-to-end multimodal pipeline.The encoders produce modality-specific representations, fusion enables bidirectional interaction, and classifiers predict attended speakers from fused and auxiliary features.Source: Model Structure · p. 2, Method · p. 2

EEG Encoder

  • Mamba blocks selectively propagate temporal information before and within the EEG band-aware convolutional Transformer.The front block reinforces raw band-specific temporal structure, while the middle block enriches representations before Transformer processing.Source: EEG Encoder · pp. 2–3

EOG Encoder

  • The EOG encoder combines temporal and spatial Transformer branches to capture fine-grained temporal dependencies and inter-channel relationships.Their outputs are concatenated and projected into dimension-aligned tokens for subsequent cross-modal interaction.Source: EOG Encoder · p. 3

Cross-Modal Interaction

  • Cross-modal attention lets each modality query the other, retrieving complementary information through symmetric token-wise interaction.Residual learning, normalization, and feed-forward refinement preserve modality-specific information while promoting cross-modal alignment.Source: Cross-Modal Interaction · p. 3

Reliability-Aware Fusion

  • RAM estimates sample-wise modality reliability from pooled descriptors and feature- and prediction-consistency scores, then regulates interacted representations before classification.Mean pooling forms fixed-dimensional descriptors, while the reliability mapping produces modality weights that modulate the fused representations.Source: RAM Optimization · p. 4, Reliability-Aware Fusion · p. 4, Feature Consistency · p. 4

Training Objective

  • A composite objective jointly supervises multimodal predictions, unimodal branches, and each EEG frequency band.The unimodal loss preserves modality-specific discriminative capability, while the band-wise loss supervises independent frequency-band classifiers.Source: Training Objectives · p. 4

Experiments and Results

RAMamba-Net is evaluated on two public AAD benchmarks using standardized preprocessing, chronological subject-wise partitioning, and multiple decoding metrics. The experiments use repeated seeded training and report accuracy, balanced accuracy, Macro-F1, and Cohen’s Kappa.

Datasets

  • RAMamba-Net is evaluated on AVGC and DTU, two public AAD datasets containing EEG and EOG recordings from competing-speaker tasks.AVGC evaluates 12 subjects after exclusions, whereas DTU includes all 18 participants.Source: Datasets · p. 4, AVGC Dataset · p. 4, DTU Dataset · p. 5
  • AVGC contains 64 EEG and 4 EOG channels at 8,192 Hz, while DTU contains 64 EEG and 6 EOG channels at 512 Hz.The datasets differ in speaker directions, trial durations, and recording conditions.Source: AVGC Dataset · p. 4, DTU Dataset · p. 5

Preprocessing

  • Signals undergo dataset-specific filtering and resampling, followed by modality-specific spatial-pattern extraction and frequency-band decomposition.AVGC is resampled to 128 Hz and DTU to 64 Hz; common spatial pattern filters are estimated from training data.Source: Preprocessing · p. 5, Experimental Settings · p. 5

Data Partitioning

  • Chronological partitioning yields an approximately 8:1:1 training, validation, and testing split after independent 2-second decision-window segmentation.The split is performed before windowing to keep each continuous recording segment within an independent subset.Source: Experimental Settings · p. 5

Implementation

  • Experiments use five random seeds, Adam optimization, early stopping, and dataset-specific batch sizes on a single NVIDIA GeForce RTX 3090 GPU.Models train for up to 200 epochs with patience 10, learning rate 5 × 10^-4, and weight decay 3 × 10^-4.Source: Experimental Settings · p. 5

Unimodal Decoding

Unimodal evaluation shows that EEG is consistently stronger than EOG, while EOG remains above chance and contributes complementary attention-related information.

  • EEG consistently outperformed EOG across datasets and models, while EOG remained clearly above chance level.Source: Unimodal Decoding · p. 5, Table 1 · p. 5
  • FAConformer achieved the best EEG decoding, whereas DBConformer achieved the best EOG decoding.The results suggest frequency-aware temporal-spectral modeling suits EEG, while temporal-spatial dual-branch modeling suits EOG.Source: Unimodal Decoding · p. 5
  • EEG provides stronger neural evidence, while EOG captures complementary ocular dynamics that motivate modality-specific encoders.Source: Unimodal Decoding · p. 5

Main Results

Main-result comparisons show that explicit EEG-EOG interaction improves over direct concatenation, while RAMamba-Net achieves the strongest performance across both datasets and all reported metrics.

  • Explicit cross-modal interaction produced more consistent improvements than direct concatenation by facilitating information exchange between EEG and EOG.Direct concatenation yielded only limited gains, indicating that simple feature fusion does not fully exploit complementary information.Source: Main Results · p. 5
  • RAMamba-Net achieved the best results across all metrics and both datasets, with larger gains on AVGC.Table 2 compares unimodal and multimodal approaches, including direct concatenation and RAMamba-Net without reliability-aware modulation.Source: Main Results · pp. 5–6, Table 2 · p. 6
  • Reliability-aware modulation jointly improved accuracy, balance, and robustness by adaptively regulating modality contributions.Source: Main Results · p. 6

Feature Visualization

t-SNE comparisons show progressively better organization from EEG-only features to direct fusion and then RAMamba-Net, whose representations form the clearest class structure on both datasets.

  • The visualization compared EEG-only, direct concatenation, and RAMamba-Net features on AVGC and DTU before classification.Source: Feature Visualization · p. 6, Figure 2 · p. 6
  • EEG-only features showed limited separability, with attention samples still mixed in the embedding space.Source: Feature Visualization · p. 6
  • Direct concatenation introduced EOG information but left feature distributions insufficiently organized, indicating weak cross-modal alignment.Source: Feature Visualization · p. 6
  • RAMamba-Net yielded the clearest class structure on both datasets, with more compact intraclass clusters and larger inter-class margins.The improvement was more evident on DTU, where the attention classes formed well-structured regions.Source: Feature Visualization · p. 6, Figure 2 · p. 6

Robustness to Noise Injection

Noise-perturbation experiments show that direct concatenation is especially vulnerable to EEG degradation, whereas RAMamba-Net maintains higher accuracy across nearly all modality-noise conditions.

  • The robustness comparison used heatmaps for classification accuracy under modality-specific noise combinations and for the resulting accuracy gain.Source: Robustness to Noise Injection · p. 6, Figure 3 · p. 6
  • RAMamba-Net maintained higher accuracy across nearly all EEG and EOG noise conditions, with gains remaining largely positive under joint corruption.The experiment independently varied each modality’s noise level from 0 for clean signals to 1 for severe corruption.Source: Robustness to Noise Injection · p. 6, Figure 3 · p. 6
  • Direct concatenation lost accuracy mainly as EEG noise increased, while EOG noise had less pronounced effects on both datasets.This pattern indicates that direct fusion remains strongly dominated by EEG and is less robust when EEG is degraded.Source: Robustness to Noise Injection · p. 6
  • Bidirectional interaction preserves complementary information, while RAM suppresses unreliable inputs and rebalances modality contributions under perturbation.Source: Robustness to Noise Injection · p. 7

Ablation and Sensitivity Analysis

The complete RAM module achieved the highest accuracy on both datasets, while RAMamba-Net remained stable across broad settings of its key hyperparameters.

Ablation Study

  • The complete RAM module achieved the highest accuracy on both AVGC and DTU by combining complementary reliability terms.Modality preservation, feature consistency, and prediction consistency contributed differently across datasets, while their combination produced the strongest result.Source: Ablation Study · p. 7
  • Feature consistency produced the largest term gain on AVGC, whereas modality preservation was most effective on DTU.Source: Ablation Study · p. 7

Sensitivity Analysis

  • Accuracy remained stable across broad ranges of λuni, β, and Hc, indicating limited sensitivity to hyperparameter variation.The analysis covered auxiliary-supervision weighting, reliability-module strength, and cross-modal attention heads on AVGC and DTU.Source: Sensitivity Analysis · p. 7, Figure 4 · p. 7

Conclusion

RAMamba-Net combines modality-specific EEG and EOG encoders, cross-modal attention, and sample-wise reliability weighting for multimodal AAD. Experiments on AVGC and DTU showed improved decoding performance, robustness, and representation quality, while future work targets distribution shifts and efficient deployment.

Conclusion

  • RAMamba-Net combines modality-specific EEG and EOG encoders, cross-modal attention, and sample-wise reliability weighting for multimodal AAD.Source: Conclusion · p. 7
  • Experiments on AVGC and DTU showed improved decoding performance, robustness, and representation quality through explicit interaction and reliability modeling.Source: Conclusion · p. 7
  • Future work targets cross-subject and cross-dataset generalization under distribution shifts, along with efficient adaptation and deployment.Source: Conclusion · p. 7

Related papers

Loading 2609.11372v1…