Source-linked AI summary
MOV-AAD: A Large-Scale Multimodal Dataset for Auditory Attention Decoding During Moving Conversations
Xiaomin He, Vishal Choudhari, Tristan J. Spratt, Aarya Raghavan, Richard T. Lee, Nima Mesgarani
TL;DR
Auditory attention decoding is often studied with static, simplified speech scenes, while available datasets commonly lack synchronized physiology and behavioral measures. MOV-AAD combines EEG and multimodal recordings from moving-conversation tasks; validation analyses show reliable EEG attentional modulation, cross-subject temporal alignment, and consistent physiological responses.
Problem
Many AAD datasets use stationary or simplified speech scenes and lack synchronized autonomic and behavioral measures, limiting study of attention and effort in dynamic scenes.
Source: §1 · p. 1Method
Results
Validation analyses show reliable EEG attentional modulation, cross-subject temporal alignment, and consistent multimodal physiological responses.
Source: §5 · p. 8Takeaways & Limitations
The publicly available dataset supports investigation of neural and autonomic dynamics of selective attention and decoding in complex acoustic environments.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Auditory attention decoding (AAD) is often evaluated on static, simplified speech scenes that poorly match everyday listening. We introduce MOV-AAD, a large-scale dataset for studying auditory attention under moving, naturalistic conversations. MOV-AAD combines 64-channel EEG with synchronized physiological recordings, including eye tracking, respiration, galvanic skin response, heart rate, peripheral oxygen saturation, body temperature, body motion, and photoplethysmography, enabling analysis of cross-modal neural and physiological markers of attention and listening effort. This dataset uses a more ecologically valid listening paradigm with dynamically moving conversational speech sources and behavioral measures of attentional engagement. MOV-AAD supports research on robust AAD in realistic spatial dynamics, multimodal attention modeling, listening effort, and intersubject neural responses, providing a resource for benchmarking selective auditory attention in naturalistic listening.
1. Introduction
MOV-AAD addresses the shortage of synchronized multimodal recordings and naturalistic moving-speaker scenes in auditory attention research. It combines EEG, physiological and behavioral recordings with dynamic conversational listening tasks to support attention and listening-effort research.
1. Introduction
- Selective auditory attention is the ability to focus on a target speaker amid noise, and EEG/MEG studies show cortical activity tracking attended speech envelopes.Peripheral physiology and eye tracking have also been linked to cognitive load, arousal, and task engagement.Source: §1 · p. 1
- Existing AAD datasets often lack synchronized autonomic and behavioral measures and use stationary speakers, synthetic mixtures, or constrained paradigms.This limits investigation of how neural and physiological processes jointly support attention and effort in dynamic acoustic scenes.Source: §1 · p. 1
- Compared with DTU and KULeuven’s stationary two-speaker EEG paradigms and PhyAAt’s primarily single-speaker noise task, MOV-AAD combines moving conversations with multimodal recordings.The dataset also has a substantial participant cohort (N=50).Source: §1 · p. 1
- Participants completed single-conversation and multi-conversation tasks with spatially moving speech and repeated-word detection to track attentional engagement.The multi-conversation task required attending to one of two dynamically moving conversational streams spanning ±90° azimuth.Source: §1 · p. 1
- MOV-AAD records 64-channel EEG and synchronized physiological and behavioral modalities from 50 participants across 4,800 attentive-listening trials.Signals include eye tracking, respiration, GSR, heart rate, SpO2, temperature, body motion, and PPG.Source: §1 · pp. 1–2
2. Dataset Description
The dataset pairs recordings from 50 screened participants with naturalistic conversational speech, spatialization, and tasks for EEG reliability and spatial localization. Stimuli used moving sources, repeated words, and multiple azimuth locations.
2.1. Participants
- Fifty participants aged 18–38 years, with self-reported normal hearing and no history of neurological or attention disorders, were recruited.The cohort included 28 male-at-birth participants; mean age was 24 years (SD = 4.5).Source: §2.1 · p. 2
2.2. Speech Materials and Spatialization
- Naturalistic continuous conversational speech by native American English talkers included occasional repeated words, and streams were spatialized with HRTFs for movement across −90° to +90° azimuth.Multi-conversation streams were RMS-matched; diotic pedestrian noise or speech babble was added at 9 or 12 dB below stream power.Source: §2.2 · p. 2
2.3.1. Repeated Sentence Task
- The repeated-sentence task presented six short sentences 20 times to support split-half or odd-even reliability analyses of speech-evoked EEG responses.Sentences were spoken by three male and three female talkers, and participants pressed a buzzer when they detected a repeated word.Source: §2.3.1 · p. 2
2.3.2. Localization Task
- The localization task familiarized participants with spatial cues and assessed perception using 54 sentences at nine azimuths from −90° to +90°, followed by mouse-click direction reports.Locations were evenly spaced.Source: §2.3.2 · p. 2
1 Repeated Sentence Task
The conversation tasks use moving single- and multi-speaker streams with repeated-word monitoring. Figure 1 outlines the recording setup and experimental pipeline, while synchronized recordings and aligned behavioral and audio data are released with the dataset.
2.4. Data Acquisition and Preprocessing
- Figure 1 summarizes the multimodal recording setup and synchronized traces alongside the experimental pipeline and stimuli.EEG and peripheral signals were acquired at 1200 Hz and synchronized with eye-tracking streams.Source: Figure 1 · p. 3, §2.4 · p. 3
2.3.3. Single Moving Speaker Conversation (SC)
- The single-conversation task comprised 40 trials with one stream moving across −90° to +90° azimuth, variable speaker identity, and 1-back repeated-word detection.Source: §2.3.3 · p. 3
2.3.4. Multiple Moving Speakers Conversation (MC)
- The multi-conversation task comprised 56 trials with two moving streams; participants attended the stream that began first and detected its repeated words.The unattended stream began 3 seconds later; talker switches occurred once in the attended stream and twice in the unattended stream, at approximately 50% and 25%/75% of trial duration, respectively.Source: §2.3.4 · p. 3
2.4. Data Acquisition and Preprocessing
- The release includes task recordings and behavioral reports, with aligned stimulus audio and repeated-word performance metrics for SC and MC trials.Abnormal-variance EEG channels were replaced by spherical interpolation; peripheral modalities are minimally processed, and each trial includes a 3-second pre-onset baseline.Source: §2.4 · pp. 3–4
3.1. Behavioral Validation Metrics
Behavioral validation quantifies spatial localization and repeated-word detection during conversation tasks. The section defines localization accuracy and error, detection metrics, and paired-condition statistical comparisons.
3.1.1. Localization Perception
- Localization accuracy is the proportion of correctly identified source locations, with chance set at 1/9; mean absolute error compares reported and true azimuths.Smaller MAE indicates higher localization accuracy, and chance-level MAE is approximately 66.7° for random responses among nine locations.Source: §3.1.1 · p. 4
3.1.2. Repeated-word Detection Metrics
- Repeated-word detections count as hits when a space-bar press occurs within 2 seconds of an attended repeated word; participant-level hit rate, precision, and F1 are computed separately for SC and MC.F1 scores are compared between MC and SC and between attended and unattended MC streams using two-tailed Wilcoxon signed-rank tests, with p-values and rank-based effect sizes reported.Source: §3.1.2 · p. 4
3.2. Physiological Data Quality Assessment
The assessment extracts physiological features from the multi-conversation session and compares their distributions with expected ranges. Features capture cardiovascular, electrodermal, respiratory, movement, oxygenation, and pupil measures.
3.2. Physiological Data Quality Assessment
- Heart rate, skin temperature, respiration frequency, SpO2, accelerometer magnitude, and pupil dilation were extracted as basic physiological features.The analysis used data from the multi-conversation task.Source: §3.2 · p. 4
- GSR was summarized with tonic skin conductance level and phasic spontaneous skin conductance response rate.SCL captures baseline level; NS-SCR rate counts spontaneous responses per minute.Source: §3.2 · p. 4
- PPG pulse peaks and inter-beat intervals yielded SDNN and RMSSD heart-rate variability measures.SDNN reflects overall variability, while RMSSD reflects parasympathetic activity.Source: §3.2 · p. 4
- Features were computed in 40-second windows, averaged across trials and participants, and compared with expected physiological ranges.Accelerometer magnitude includes static gravity, so a baseline near 1 g is expected; deviations can reflect movement or orientation.Source: §3.2 · p. 4
3.3. Baseline Analyses
The baseline analyses use backward models to reconstruct speech envelopes and spatial trajectories from EEG, then decode attention from reconstruction correlations. Intersubject correlation (ISC) assesses shared stimulus-locked responses across modalities.
3.3.1. Envelope Reconstruction
- Speech envelopes were extracted with the Hilbert transform and low-pass filtered at 8 Hz; EEG was resampled to 100 Hz, normalized, compressed, and filtered to 1–8 Hz.Backward-model decoders used 0–400 ms time lags, with ridge parameters selected by leave-one-trial-out cross-validation; performance was Pearson correlation on held-out trials.Source: §3.3.1 · p. 4
3.3.2. Trajectory Reconstruction
- Spatial trajectories represent time-varying azimuths from −90° to +90°, generated at 1° resolution and interpolated to 100 Hz.EEG was analyzed in delta (0.05–2 Hz) and alpha (8–12 Hz) bands; backward-model reconstruction used 0–250 ms lags, and attended and unattended streams were reconstructed separately.Source: §3.3.2 · pp. 4–5
3.3.3. Auditory Attention Decoding (AAD)
- AAD used participant-specific LDA on attended and unattended reconstruction correlations, with swapped correlation pairs forming the alternative class.Leave-one-trial-out evaluation tested held-out true-order pairs; envelope-only, trajectory-only, and combined features were evaluated against a permutation-based null with multiple-comparison correction.Source: §3.3.3 · p. 5
3.3.4. Intersubject Correlation (ISC)
- ISC was computed from modality-specific signals using spatial filters that maximize between-subject covariance relative to within-subject covariance.Slowly varying measures were aggregated in non-overlapping blocks of 28 trials; phase-randomized surrogates supplied chance baselines for subject- and group-level tests.Source: §3.3.4 · pp. 5–6
4. Results and Discussion
Across the dataset’s baseline analyses, envelope features provide the strongest AAD evidence, while trajectory features offer weaker, feature-dependent support. Behavioral performance, localization, physiological checks, and cross-subject response similarity also characterize the dataset’s naturalistic listening tasks.
4.1.1. Localization Performance
- Localization retained the correct hemifield structure despite reduced discriminability at extreme lateral positions; most participants performed above chance.Figure 2 compares perceived and true angles and plots subject-level accuracy and mean absolute error against chance.Source: §4.1.1 · p. 6
4.1.2. Repeated-word Detection Performance
- Repeated-word detection was significantly higher for attended than unattended multi-conversation streams, and single-conversation performance modestly exceeded attended multi-conversation performance.The reported comparisons were p < 10−4, r = 0.87 for MC-A versus MC-U and p < 10−4, r = 0.52 for SC versus MC-A.Source: §4.1.2 · p. 6
4.2. Physiological Signal Sanity Check
- Most extracted physiological measurements fell within or near expected ranges, indicating overall good signal quality, with a small number of outliers.The distributions include heart rate, skin temperature, GSR, respiration, movement, SpO2, HRV, and pupil dilation.Source: §4.2 · p. 6, Figure 4 · p. 6
4.3. Stimulus Tracking Performance and AAD accuracy
- Envelope features separated attended from unattended reconstruction correlations and produced AAD accuracy above chance, while delta-trajectory decoding remained near chance.Alpha trajectories performed better than delta and showed modest gains with longer windows.Source: §4.3 · p. 7, Figure 5 · p. 7
- Envelope reconstruction correlations and AAD accuracy were consistently stronger across participants, whereas trajectory-feature tracking was weaker and evident in only a subset.The subject-level comparison is shown in Figure 6.Source: §4.3 · p. 7, Figure 6 · p. 7
- Full-trial LDA gave envelope features the highest accuracy; alpha trajectories were significantly above chance, delta trajectories were not, and combining trajectories with envelopes did not outperform envelopes alone.The combined features significantly improved over trajectory-only decoding.Source: §4.3 · p. 7, Figure 7 · p. 7
4.4. Inter-subject Correlation
- EEG, gaze location, pupil dilation, and PPG showed consistent cross-subject temporal alignment; temperature showed little trial-level shared structure.Heart rate and SpO2 had group-level effects but limited subject-level prevalence; overall, multiple modalities contained reproducible stimulus-locked structure.Source: §4.4 · p. 8, Figure 8 · p. 8
5. Conclusion
MOV-AAD combines EEG with synchronized physiological and behavioral recordings from 50 participants during spatially moving listening tasks. Validation found reproducible stimulus- and attention-related structure, and the dataset is publicly available for reproducibility and cross-laboratory comparison.
5. Conclusion
- MOV-AAD integrates EEG with synchronized physiological and behavioral recordings from 50 participants during spatially moving listening tasks.Source: §5 · p. 8
- Validation showed reliable EEG attentional modulation, cross-subject temporal alignment, and consistent multimodal physiological responses, indicating reproducible stimulus- and attention-related structure.Analyses included stimulus reconstruction, auditory attention decoding, and intersubject correlation across modalities.Source: §5 · p. 8
- MOV-AAD supports investigations of neural and autonomic selective-attention dynamics and decoding approaches in complex acoustic environments, and is publicly available for reproducibility and cross-laboratory comparison.Source: §5 · p. 8
7. Generative AI Use Disclosure
Generative AI tools were used only for language polishing and grammar checking; the authors produced and verified the scientific content, analyses, interpretations, and conclusions.
7. Generative AI Use Disclosure
- Generative AI tools were used only for language polishing and grammar checking.Source: §7 · p. 9
- The authors produced the scientific content, analyses, interpretations, and conclusions.Source: §7 · p. 9
- The authors verified the scientific content, analyses, interpretations, and conclusions.Source: §7 · p. 9