MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

• Many prior datasets use two-speaker, fixed-SNR, low-noise or non-English stimuli, limiting their match to realistic listening conditions.Source: §I · p. 1
• Most datasets record EEG alone or use controlled visual stimuli, leaving spontaneous gaze, head movements, and egocentric visual experience uncaptured.Source: §I · p. 1
• MAESTRO synchronously records EEG, gaze, pupillometry, egocentric video, and head IMU from 16 participants across 100 trials, with four English speakers and attended-speaker SNRs from 0 to 18 dB.Source: §I · pp. 1–2
• The benchmark identifies the attended speaker among four simultaneous speakers and evaluates 15 modality input configurations.Source: §I · p. 2
• Behavioral signals combined with EEG consistently improve baseline performance in the four-speaker benchmark.Source: §I · p. 2

A. Study Configuration

• Four front loudspeakers presented speech at ±67.5° and ±22.5°, while two rear speakers at ±135° presented background noise.Source: A. Study Configuration · p. 2
• The study used five familiarization trials followed by 100 experimental trials across two 50-trial sessions, with a mandatory 10-minute break.Source: A. Study Configuration · p. 2

B. Participant Recruitment and Screening

• Sixteen adults aged 18–30 participated; eligibility required native English and screening for cognition, hearing, and conductive-gel allergies.Source: B. Participant Recruitment and Screening · p. 3

C. Audio Stimuli and Comprehension Questions

• Each trial used four unique 30-second LibriSpeech speech signals and two unique 30-second CHiME-Home noise signals, with no speaker or audio file repeated.Source: C. Audio Stimuli and Comprehension… · p. 3
• The attended-speaker SNR ranged from 0 to 18 dB, averaging approximately 12 dB with a standard deviation of 3.16 dB.Source: C. Audio Stimuli and Comprehension… · p. 3
• After each trial, participants answered a transcript-based question generated automatically and reviewed for consistency and relevance.Source: C. Audio Stimuli and Comprehension… · p. 3
• Each question offered one correct answer, three distractors, and an option to report inability to pay attention; per-trial correctness labels were released.Source: C. Audio Stimuli and Comprehension… · p. 3

D. Multimodal Sensory Data Streams

• EEG, gaze, egocentric video, and IMU were synchronized to audio onset, resampled to 64 Hz, and z-score normalized per channel and trial.Source: D. Multimodal Sensory Data Streams · p. 3

1) EEG:

• EEG was recorded continuously from 32 electrodes at 500 Hz, filtered and cleaned, then downsampled to 64 Hz.Source: 1) EEG · pp. 3–4

2) Gaze:

• The sample gaze figure pairs video frames with overlaid gaze points and plots gaze coordinates and bilateral pupil diameters over the trial.Source: Figure 2 · p. 4
• Eye tracking recorded gaze direction, scene-projected coordinates, and pupil diameter; missing samples were interpolated onto a 64 Hz grid and low-pass filtered.Source: 2) Gaze · p. 4

3) Egocentric video:

• Egocentric video was recorded at 25 fps and 1920 × 1080, converted to grayscale, and represented by four optical-flow features per frame before resampling to 64 Hz.The flow features are mean and standard deviation of magnitude and mean horizontal and vertical components.Source: 3) Egocentric video · p. 4

4) IMU:

• Head movement was recorded as six-axis accelerometer and gyroscope data, interpolated to a uniform grid, resampled to 64 Hz, and low-pass filtered at 20 Hz.The filter was fourth-order Butterworth.Source: 4) IMU · p. 4

5) Audio:

• Audio streams were timestamp-aligned before broadband envelope extraction, then low-pass filtered at 20 Hz, downsampled to 64 Hz, and standardized per trial.Speech and background-noise recordings were stored as separate stereo 16 kHz FLAC streams.Source: 5) Audio · p. 4

A. Comprehension Accuracy and Attentional Compliance

• 83.2% mean comprehension accuracy across 16 participants, and the “I could not pay attention” option was selected on only 1.4% of trials.Most participants improved in session two, with group mean accuracy rising from 80% to 87%.Source: A. Comprehension Accuracy and… · pp. 4–5

B. Eye and Head Movement Characteristics

• Mean gaze-data validity was 80%, with nine participants above 90%; tracking loss primarily came from blinks, partial occlusions, and gaze outside the camera’s range during natural movement.Subjects 8 and 12 had substantially lower validity at 27.3% and 42.4%.Source: B. Eye and Head Movement Characteristics · p. 5
• Gaze dispersion peaked at trial onset and offset but stayed stable between them; head movement peaked at onset and stabilized from Bin 3 onward without an offset rise.Mean gyroscope RMS was 6.95 deg/s.Source: B. Eye and Head Movement Characteristics · p. 5, Figure 5 · p. 6

IV. EVALUATION PROTOCOL AND BASELINE

• The benchmark identifies one attended speaker among four simultaneous speakers; within-subject evaluation uses five-fold cross-validation, while LOSO tests each participant held out from training and model selection.Within-subject folds divide trial contents, and each LOSO test set contains 20 trials for the held-out participant.Source: §IV · p. 5, A. Training and Evaluation Splits · p. 5

A. Training and Evaluation Splits

• Decision windows range from 5 to 30 seconds, with half-window hops below 30 seconds, varying the temporal context consistently across both evaluation settings.This gives eleven overlapping windows at 5 seconds and one at 30 seconds.Source: A. Training and Evaluation Splits · p. 5
• 56% of windows were classified correctly by an envelope-statistics probe against 25% chance, using the attended envelope’s lower crest factor (11.1 dB versus 20.5 dB).SNR scaling before 16-bit PCM encoding clipped the attended channel’s peaks but not the competitors’.Source: A. Training and Evaluation Splits · p. 5
• Histogram equalization reduced the envelope-only probe to 26% by matching value distributions across envelopes; the transform may remove genuine information, so reported accuracies are conservative.Each envelope sample is replaced with the shared value at its rank across the four envelopes.Source: A. Training and Evaluation Splits · pp. 5–6

B. Baseline Network

• EEG and audio use dilated-convolution encoders, extended with gaze, IMU, and scene-video encoders for multimodal decoding.Each encoder uses five layers and produces a 16-dimensional embedding; the EEG encoder adds a spatial convolution across 32 channels.Source: B. Baseline Network · p. 6
• EEG embeddings are scored against four speaker-envelope embeddings by correlation, while gaze, IMU, and video predict speaker position without audio input.The correlation scores are formed from 16 per-dimension correlations, combined with learned weights and temperature.Source: B. Baseline Network · p. 6
• A second score pools each modality’s mean and temporal standard deviation, then fuses the embeddings into four speaker-position logits.EEG is included in this pathway because attention to one side produces lateralized cortical activity.Source: B. Baseline Network · p. 6
• Modality dropout withholds each available modality independently at probability 0.3 during training, while the modalities are fused in one end-to-end model.The dropout is disabled during evaluation and is not used for single-modality configurations.Source: B. Baseline Network · p. 6
• The objective combines four-slot cross-entropy with label smoothing 0.1 and auxiliary speaker-position, batch-contrastive, hinge, embedding-collapse, and audio-adversarial losses.Speaker-position loss is applied separately to each modality’s logits. The hinge margin is 0.5; embedding collapse requires temporal standard deviation 0.5 and penalizes off-diagonal covariance (λ = 0.04).Source: B. Baseline Network · p. 7

A. Within-subject

• 8.03% (15 s) to 13.01% (5 s): the best multimodal configuration’s gains over EEG alone across window sizes, significant at every size.Reported means and standard deviations are computed across five cross-validation folds, with each fold trained once.Source: A. Within-subject · p. 7
• EEG is the strongest single modality at every window size except 5 s, indicating that the gains come from complementary modalities rather than replacing EEG.Every best-performing combination contains EEG.Source: A. Within-subject · p. 7
• The optimal combination is all four modalities at 5, 15, and 20 s, and EEG, gaze, and video at 10 and 30 s.The full four-modality configuration is optimal for three of the five window sizes.Source: A. Within-subject · p. 7

B. Leave-One-Subject-Out

• Under LOSO, multimodal gains over EEG alone range from 5.58% (30 s) to 10.77% (5 s) across window sizes.The gains are narrower than the within-subject gains of 8.03–13.01%, and across-subject standard deviations are roughly four times larger.Source: B. Leave-One-Subject-Out · p. 7
• The best overall configuration beats EEG alone in 75 of 80 subject-window cases (93.8%), ties once, and trails in four (5.0%).The best EEG-containing configuration is also the overall best in 72 of 80 cases (90.0%).Source: Figure 6 · p. 8, B. Leave-One-Subject-Out · p. 8

C. SNR-Stratified Analysis

• The 10–12 dB bin contains only 3 of 20 test trials per listener, omits some attended-speaker positions, and repeats the same three contents across LOSO folds.The authors cannot rule out that this SNR range is genuinely harder, but comprehension accuracy is lowest in the 3–10 dB bin.Source: C. SNR-Stratified Analysis · p. 8
• Across four SNR bins, multimodal accuracy remains 6.2 to 10.7 points above EEG, and every winning configuration contains EEG.Accuracy changes little between the lowest and highest bins; both curves dip in the 10–12 dB bin.Source: Figure 7 · p. 8, C. SNR-Stratified Analysis · p. 8

D. Attributing Accuracy to the Recorded Signals

• Permutation-based contribution is positive for every tested configuration and window, rising from +11.1 points for EEG alone to +25.4 points with behavioral modalities at 5 s.The comparison permutes listener signals across test windows while keeping each window’s speech envelopes and label.Source: D. Attributing Accuracy to the… · pp. 8–9, Figure 8 · p. 9

A. Interpretation of Multimodal Benefit

• Adding behavioral modalities to EEG improves decoding at every window size: 8.03 to 13.01% within-subject and 5.58 to 10.77% under LOSO.The optimal combination varies by window size, and every winning combination contains EEG.Source: A. Interpretation of Multimodal Benefit · p. 9

B. Relation to Prior Work

• EEG alone reaches 43.63%-59.06% within-subject, below the 80-99% reported for binary two-speaker decoding; the four-speaker task includes varying SNRs and background noise.Scene video reaches 39.57%-44.69% under LOSO, while gaze with video reaches 44.61%-47.10%.Source: B. Relation to Prior Work · p. 9

C. Implications for Hearing Aid Design

• Wearable eye-tracking and head-mounted inertial sensors carry attention-related information, and the authors suggest adaptive, user- and context-dependent modality weighting for hearing aids.MAESTRO’s systematic SNR variation provides a platform for studying responses to increasingly challenging acoustic conditions.Source: C. Implications for Hearing Aid Design · p. 9

D. Limitations and Future Directions

• The fixed four-speaker layout and native-English participant pool limit transfer to other languages and environments with moving or previously unknown sources.Gaze quality also varied across participants, and the 16-person cohort limits cross-subject analyses.Source: D. Limitations and Future Directions · p. 10
• SNR adjustment clipped the attended audio channel but not competitors, flattening its waveform; the artifact affects downstream use of released raw audio.The benchmark removes the resulting envelope cue, but the recording distortion remains.Source: D. Limitations and Future Directions · p. 10
• The baseline uses the same encoder design across modalities despite differing temporal and noise characteristics, with spatial mixing only for EEG.The authors propose modality-specific encoders, signal-quality-weighted fusion, and adaptive modality dropout as future directions.Source: D. Limitations and Future Directions · p. 10
• MAESTRO supports extending EEG-guided attended-speaker extraction from two-speaker settings to four-speaker environments using video, gaze, head motion, and mixed audio.Its annotations also support studying gaze confounds and listening effort, with potential hearing-aid, fatigue-monitoring, and personalized-assistance applications.Source: D. Limitations and Future Directions · p. 10

VII. CONCLUSION

• The paper frames AAD as multimodal sensing, with EEG, gaze, head motion, and scene video each providing complementary attention information.The authors call for a shift toward understanding how modalities encode attention and combine into robust, interpretable decoders.Source: §VII · p. 10
• Adding behavioral signals to EEG improves decoding at every window size and under both splits, while the best combination varies with window size.This motivates adaptive rather than fixed modality weighting.Source: §VII · p. 10