PaperlayerView exact revision on arXiv

Source-linked AI summary

Merit: Making every modality earn its contribution in multimodal attention decoding

Ali Alavi, Donald S. Williamson

arXiv:2610.17507v1cs.LG

TL;DR

Auditory attention decoding uses EEG to identify the talker a listener attends to, but multimodal systems also receive behavioral streams that can indicate facing rather than listening. MERIT measures each stream’s accuracy contribution and designs its decoder to block shortcuts. It improves on the MAESTRO benchmark while showing that behavioral inputs can raise accuracy as EEG’s share of credit falls.

  • Problem

    Accuracy alone does not reveal whether a multimodal attention decoder uses EEG or behavioral streams, even though gaze and head motion indicate facing rather than listening.

    Source: §1 · p. 2

  • Method

    MERIT shuffles each input stream across test windows to measure accuracy loss, allocates contribution with Shapley values, and blocks shortcuts through candidate-audio checks and scoring design.

    Source: §1 · p. 2

  • Results

    0.82 four-way accuracy within listeners and 0.74 on unseen listeners exceed the benchmark’s 0.698 and 0.674, while adding behavioral streams almost halves EEG’s share of credit.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    On two other public datasets, the same tests find label imbalance in a widely used benchmark and results that accuracy alone cannot confirm as brain activity.

    Source: §1 · p. 3, §8 · p. 10

    The per-listener correction behind the largest within-listener gain cannot help a listener the model has never seen, so gains on the brain signal are shown within listeners.

    Source: §7.3 · p. 9

Abstract

from arXiv · show

More than 1.5 billion people have hearing loss. When several people talk at once, hearing aids amplify every voice alike. Auditory attention decoding (AAD) aims to solve this by reading a listener's brain signals to find the attended talker, letting the device amplify only them. A real device will not see the brain alone; smart glasses and earbuds also track eyes, head, and scene. These behavioral signals help, as people often look at who they listen to. However, they show where a listener looks, not whom they hear. A decoder relying heavily on them fails when a listener cannot move their head or listens without looking - exactly when a brain-based device is needed. A multimodal decoder must use these signals without letting them replace the electroencephalogram (EEG). Accuracy alone cannot show whether it does. We present MERIT, a decoder and evaluation method asking: how much accuracy does each input earn? MERIT shuffles each input stream across test examples, measures the accuracy drop, and splits the result across streams using Shapley values. It also blocks two shortcuts: candidate audio is checked before training so it cannot reveal the answer, and the scoring function cannot beat chance if it ignores the brain. On a public dataset with EEG, eye tracking, head motion, and scene video, MERIT improves on the published benchmark, reaching 0.82 four-way accuracy within listeners and 0.74 on unseen listeners (chance is 0.25). The measurement reveals what accuracy hides: adding behavioral streams raises overall accuracy but almost halves the brain's share of it. On two other datasets, the same tests find a label imbalance in a widely used benchmark and flag results that accuracy alone would report as brain decoding.

1 INTRODUCTION

MERIT evaluates how much each input stream contributes to auditory attention decoding, rather than relying on accuracy alone. It improves on the MAESTRO benchmark while finding that behavioral streams can raise accuracy as the brain’s share of credit falls.

1 INTRODUCTION

  • AAD uses EEG responses that follow attended speech to identify the talker and steer a hearing device toward them.The motivation is the cocktail-party setting, where hearing aids otherwise amplify all voices, including ignored ones.Source: §1 · p. 1
  • Behavioral streams can help because listeners often turn toward the talker they follow, but gaze and head motion indicate facing rather than listening.A decoder that lets those streams replace EEG risks failing when facing and listening diverge or listeners cannot move freely.Source: §1 · pp. 1–2
  • MERIT shuffles each recorded stream across test windows and measures the resulting accuracy drop to estimate that stream’s contribution.It splits total contribution across streams using Shapley values and uses contribution, rather than accuracy, to choose checkpoints.Source: §1 · p. 2
  • 0.82 four-way accuracy within listeners and 0.74 on unseen listeners exceed the benchmark’s 0.698 and 0.674, respectively.These are the reported Section 7.3 comparisons.Source: §1 · p. 2
  • Candidate audio is checked before training, and the scoring function is designed so ignoring the brain cannot beat chance.The orientation heads receive no audio, preventing them from using audio cues.Source: §1 · p. 2
  • Adding behavioral streams almost halves the brain’s share of credit, and at two-second windows EEG earns almost none.On two other public datasets, the tests also find a label imbalance in a widely used benchmark and results that cannot be confirmed as brain activity.Source: §1 · p. 3

2 RELATED WORK

Prior work spans EEG-based matching and direction decoding, alongside research on leakage, datasets, and modality balance. MERIT brings stream-level contribution measurement into model selection and reporting for multimodal attention decoding.

2 RELATED WORK

  • Linear envelope-reconstruction decoders need tens-of-seconds windows, while deep match-mismatch models learn EEG–speech compatibility and direction decoders classify attended side without audio.Direction-decoding results can reflect eye movements in EEG; removing within-trial train-test leakage reduced recent decoders by 17 to 45 points, with several falling to chance.Source: §2 · p. 3
  • Existing AAD datasets limit multimodal credit analysis: some record little beyond EEG, AV-GC-AAD makes non-gaze streams uninformative, and MAESTRO records multiple streams during free movement.KU Leuven also has an imbalance: one talker is attended in 256 of 320 trials; MAESTRO has an audio confound in its usual candidate construction.Source: §2 · p. 3
  • Prior modality-balancing methods equalize learning speed, whereas MERIT measures each stream’s test-time effect and uses stream-level Shapley values to choose models and report results.The cited foundations include permutation importance and Shapley values.Source: §2 · p. 3

3 CANDIDATE SETS AND SPLITS

Candidate construction determines whether speech inputs themselves can reveal the attended talker. MERIT uses matched candidates for four-way evaluation and stricter same-talker matched candidates to check EEG–speech tracking.

3 CANDIDATE SETS AND SPLITS

  • UNMATCHED candidates are the trial’s four talkers standardized to zero mean and unit variance, but scale-invariant properties can still distinguish them.Such properties include skew, sparsity, and frequency-band balance.Source: §3 · pp. 3–4
  • MATCHED candidates share the same set of values after quantile matching while retaining their individual timing, removing value-statistic cues without removing timing.MERIT uses MATCHED for the four-way task; SAME-TALKER MATCHED applies matching to candidates drawn from the attended talker’s non-overlapping trial segments.Source: §3 · p. 4
  • Within-listener evaluation uses five audio-content-disjoint folds, while leave-one-listener-out evaluation holds out a listener and 20% of audio content.Each test set is scored once, with validation drawn from the training portion.Source: §3 · p. 4

4 THREE WAYS TO BE RIGHT FOR THE WRONG REASON

The section identifies audio cues, model architecture, and behavioral streams as routes to apparently successful decoding without reliable EEG use. These shortcuts motivate candidate checks and a shuffle-based test of each stream’s contribution.

4 THREE WAYS TO BE RIGHT FOR THE WRONG REASON

  • On MAESTRO, attended and ignored talkers differ in scale-invariant audio statistics; the peak-to-average envelope ratio separates them with d = −1.22.The dataset’s authors trace this difference to loudness adjustment that clipped attended-talker peaks.Source: §4 · p. 4
  • The UNMATCHED acceptance probe scores +0.3081 above chance; MATCHED scores 0.2603 against 0.25, and same-talker candidates score 0.5134 against 0.50.The probe is logistic regression on eight scale-independent statistics, evaluated on held-out content.Source: §4 · p. 4
  • With a constant brain embedding, the standard score becomes a linear classifier on time-averaged audio, making an audio-only shortcut reachable.The passage reports that training the standard model on this data exhibits the constant-encoder collapse.Source: §4 · p. 4
  • Gaze, head motion, and scene video indicate facing, which can predict the attended talker in a loudspeaker room, but only EEG follows what the talker is saying.A decoder relying on gaze can report higher accuracy while supporting a weaker claim; this matters for listeners unable or unwilling to face the talker they hear.Source: §4 · pp. 4–5
  • MERIT’s coupling branch compares EEG with candidate speech over time, while its orientation branch receives no audio and therefore cannot use audio cues.The model encodes once and scores separately, allowing re-scoring under stream shuffles without re-encoding.Source: Figure 1 · p. 5
  • Shuffling a recorded stream across test windows while retaining each window’s candidates and label costs accuracy only when the decoder uses that stream.Shuffling labels instead would break the audio-answer link and hide the shortcut being tested; the shuffled accuracy measures what the decoder actually uses.Source: §4 · p. 5

5 MERIT

MERIT combines EEG-based speech coupling with audio-free orientation predictions, while constraining its score so a constant brain signal cannot outperform chance. It measures each stream’s contribution with Shapley values and selects checkpoints to reward accuracy alongside credit for the weakest stream.

5 MERIT

  • MERIT’s EEG-only coupling branch tests speech tracking, while audio-free orientation heads predict loudspeaker direction from gaze, head motion, scene motion, foveal video, and EEG.The orientation scores are fused with coupling scores; an audio-only adversary uses gradient reversal to train readable candidate cues out of the audio encoder.Source: §5 · pp. 5–6
  • A constant EEG embedding produces equal candidate scores and exactly chance accuracy, making that shortcut impossible under the coupling score.The score removes each embedding’s time average; correlation also ignores scale, and centering removes additions shared by all candidates.Source: §5 · pp. 5–6
  • The task loss trains the final candidate scores, while each orientation head receives its own loudspeaker prediction objective so the strongest stream cannot take all the gradient.The full objective combines task, auxiliary, contrastive, null, collapse, and adversarial losses.Source: §5 · p. 6
  • MERIT’s contrastive, null, and collapse losses constrain EEG–speech matching, require real EEG to beat shuffled or zeroed EEG by a 0.5 margin, and discourage collapsed or redundant embeddings.The contrastive batch comes from one listener, preventing matching by listener identity; the collapse loss maintains embedding spread above 0.5.Source: §5 · p. 6
  • Checkpoint selection favors validation contribution plus the weakest stream’s credit, rather than accuracy alone, so a slightly less accurate epoch can win if it uses all streams.MERIT also tested a contribution hinge on final decisions, but it did not raise the brain’s share of credit and was not used in training.Source: §5 · p. 7
  • The Shapley value splits total contribution across streams despite overlap and interactions, and MERIT computes it exactly over all 32 subsets for five streams.The same random shuffles are used for each subset to reduce noise in their differences.Source: §5 · p. 7

6 EXPERIMENTAL SETUP

Experiments use MAESTRO’s synchronized EEG, behavioral, and scene recordings to evaluate candidate selection across listeners and streams. The protocol pairs accuracy with shuffled baselines and contribution measures, and controls the comparison by re-running the benchmark architecture within the same training loop and splits.

6 EXPERIMENTAL SETUP

  • MAESTRO provides 16 listeners with 100 trials each, four simultaneous talkers, and synchronized EEG, gaze, head motion, scene flow, and foveal video embeddings.Each trial lasts about 30 seconds; the attended talker is about 15 dB louder, and talker recordings do not overlap between training and test.Source: §6 · p. 7
  • Every accuracy is paired with its own shuffled baseline and reported alongside contribution, permutation p-value, flip rate, zeroed-stream accuracy, and embedding collapse.With 20 test shuffles, the smallest possible permutation p-value is 0.048.Source: §6 · p. 7
  • Table 1 compares EEG-only and fused models within listeners, with chance at 0.25; its caption says more streams bring higher accuracy but less brain credit.The fused models share streams, data, folds, architecture, and listener adaptation, differing in contribution hinge and modality dropout.Source: Table 1 · p. 8
  • The benchmark architecture is re-run inside MERIT’s training loop and splits, isolating model differences from protocol differences; these runs are not intended to reproduce published benchmark numbers.Settings were fixed in advance without hyperparameter search.Source: §6 · p. 8
  • MAESTRO is described as the only known public AAD dataset combining EEG, eye tracking, head motion, and scene video while listeners move freely among loudspeakers.MERIT’s audio probe and coupling score, which do not need these behavioral streams, are also tested on two other datasets.Source: §6 · p. 8

7 RESULTS

MERIT tests what each stream contributes, and its multimodal decoder improves on the benchmark in three of four comparisons. The results also show that higher overall accuracy can coincide with less EEG credit, while checks across datasets distinguish brain decoding from accuracy alone.

7.1 WHAT THE MATCHING TASK ALLOWS

  • With unmatched candidates, the benchmark model reaches 0.4991 four-way accuracy but +0.0000 EEG contribution; with matched candidates, it scores 0.2833, which is chance.A better encoder raises unmatched-candidate accuracy to 0.5800 while EEG contribution remains +0.0000.Source: §7.1 · p. 8

7.2 MORE STREAMS, LESS CREDIT FOR THE BRAIN

  • Adding behavioral streams raises accuracy from 0.6915 to 0.8173, while EEG credit falls from +0.3971 alone to +0.2217 in the fused model.The foveal embedding earns +0.2866, more than the EEG in the fused model.Source: §7.2 · pp. 8–9
  • The contribution hinge did not raise EEG credit above the matching model without it; with modality dropout, EEG credit fell from +0.2229 to +0.1600.The interaction can make the hinge push whichever stream appears unused, rarely the EEG; choosing checkpoints by contribution raised EEG credit from +0.1039 to +0.1182 in an earlier experiment.Source: §7.2 · p. 9
  • MERIT omits the contribution hinge and modality dropout, keeping the most EEG credit on unseen listeners (+0.1762) and tying for the most within listeners, at an accuracy cost below 0.02.The authors select it because credit, rather than maximum accuracy, is the criterion.Source: §7.2 · p. 9

7.3 THE FULL SYSTEM, AND THE BENCHMARK

  • One decision per trial raises contribution from +0.1974 to +0.2866; after-sound delays add a little, and per-listener correction and an onset channel also improve results.Before-sound delays reach only +0.2127; correction raises contribution from +0.3271 to +0.3971, while the onset channel raises accuracy from 0.6709 to 0.6915.Source: §7.3 · p. 9
  • 0.8173 with all streams within listeners and 0.7438 with all streams on unseen listeners; three of four benchmark comparisons improve on both accuracy and contribution.EEG alone on unseen listeners is the exception: 0.6000 against the benchmark's 0.619.Source: §7.3 · p. 9

7.4 IS THE EEG SIGNAL REAL?

  • On two other datasets, high coupling-branch accuracy with same-talker candidates does not pass the direction-in-time check, so the results are not counted as brain decoding.The audio probe is +0.1947 above chance on KU Leuven because one talker is attended in 256 of 320 trials; at 5 s, the coupling branch is more accurate than NeuroToken on unseen listeners of both datasets.Source: §7.4 · p. 10

8 CONCLUSION

The conclusion argues that accuracy can rise while a decoder stops relying on the brain signal, and reports that MERIT measures each stream’s credit while blocking two shortcuts. Its appendices explain the candidate checks, scoring and shuffling choices, related contributions, and limitations.

8 CONCLUSION

  • Adding behavioural streams almost halves the brain’s share of credit; on other public datasets, high accuracy can hide label imbalance or a signal that cannot be confirmed as neural.Training against the measurement did not raise the brain’s share.Source: §8 · p. 10

A.1 WHY MAKING THE CANDIDATES EQUALLY LOUD IS NOT ENOUGH

  • Equalising candidate loudness removes scale differences, but other envelope statistics survive and can distinguish attended from ignored talkers.The peak-to-average ratio separates them with effect size d = −1.22; the proposed fixes make candidate distributions identical or use same-talker candidates.Source: §A.1 · p. 14

A.2 HOW THE STANDARD SCORE REDUCES TO AN AUDIO CLASSIFIER

  • A constant brain embedding lets the standard score act as an audio classifier, while the time-centred score gives every candidate the same score in that case.With a constant embedding, the centred brain term is zero and all correlations are zero.Source: §A.2 · p. 14

A.3 WHY A JOINT SHUFFLE CANNOT SEE CROWDING OUT

  • A joint shuffle measures total listener-signal contribution but can hide a falling EEG contribution; per-stream Shapley values expose both.The passage illustrates this with total contribution rising from 0.40 to 0.55 while EEG contribution falls from 0.40 to 0.20.Source: §A.3 · p. 14

A.4 WHY WE SHUFFLE THE RECORDING, NOT THE LABELS

  • Shuffling the recording across test windows while retaining each window’s candidates and label isolates the recording’s contribution; the remaining accuracy, v(∅), measures the model’s audio-only pathway.A decoder relying only on audio keeps its accuracy under this test.Source: §A.4 · p. 14

B RELATED WORK IN MORE DETAIL

  • MERIT measures each stream’s effect on test-time decisions, rather than balancing learning rates; it uses this credit to choose checkpoints and report the brain’s share.Modality dropout changes EEG credit little, while EEG credit remains far below what it earns alone.Source: B RELATED WORK IN MORE DETAIL · p. 15

C CHECKING THE CANDIDATE SETS

  • The audio-only acceptance probe uses eight scale-invariant envelope statistics on held-out content; a candidate set without an audio shortcut should score 1/K.The accompanying table describes the probe as having no access to any recording.Source: C CHECKING THE CANDIDATE SETS · p. 15, Table 3 · p. 16

E FURTHER LIMITATIONS

  • Four-way contribution can partly reflect trial identity, and results are limited to one room, one language, and sixteen listeners; the audio mark’s cause is untested.The same-talker results avoid the trial issue because their candidates come from the same trial and voice; the probe measures mark readability regardless of its cause.Source: E FURTHER LIMITATIONS · p. 16

H THE AUDIO PROBE ON TWO OTHER DATASETS

The audio-only probe was run on KU Leuven and DTU to test whether candidate speech itself predicts attention labels. It stayed near chance on DTU but exposed label imbalance on KU Leuven, which distribution matching only partly reduced.

H THE AUDIO PROBE ON TWO OTHER DATASETS

  • The probe was applied unchanged to KU Leuven and DTU, using two-talker candidates, content-disjoint testing, and chance accuracy of 0.5.Neither dataset records gaze, head motion, or video, so only MERIT’s audio-dependent probe applies.Source: H THE AUDIO PROBE ON TWO OTHER DATASETS · p. 17
  • Across four window lengths, DTU’s audio-only probe stayed within 0.005 of chance, while KU Leuven showed excess above chance at every window.The KU Leuven 30 s result is the least powered cell, based on 4,608 windows from eight stimulus pairs.Source: H THE AUDIO PROBE ON TWO OTHER DATASETS · p. 17, Table 6 · p. 18
  • 0.6947 accuracy on KU Leuven’s 10 s windows, against chance 0.5, came from audio alone and reflected an attended-talker label imbalance.Track 1 was attended in 256 of 320 trials because the experiment repeated its first part, where listeners always attended track 1.Source: H THE AUDIO PROBE ON TWO OTHER DATASETS · pp. 17–18
  • Quantile matching reduced KU Leuven’s excess from +0.1947 to +0.0974 but retained temporal frequency-band balance that could identify the talker.The authors recommend balancing labels when two voices are imbalanced and reporting the probe for each dataset rather than treating a candidate set as universally safe.Source: H THE AUDIO PROBE ON TWO OTHER DATASETS · p. 18

I THE COUPLING BRANCH ON TWO OTHER DATASETS

The coupling branch transferred to KU Leuven and DTU, but its direction-in-time check did not confirm that the apparent decoding reflected brain responses. Comparisons with published results also show that split design strongly affects reported accuracy.

I THE COUPLING BRANCH ON TWO OTHER DATASETS

  • Within-listener accuracy was 0.7252 on KU Leuven and 0.7590 on DTU, rising to 0.8149 and 0.8170 in same-talker tests, which remove talker identity; direction checks did not confirm brain decoding.The authors consider electrical leakage the most likely explanation for the failed direction check, but state that they have not shown it.Source: I THE COUPLING BRANCH ON TWO OTHER… · p. 19
  • On both public datasets, the coupling branch remained positive across every fold and held-out listener, but this did not establish brain-based decoding.The same-talker test removes a route based on talker identity or position, while the direction check remains necessary to assess whether timing is consistent with a brain response.Source: I THE COUPLING BRANCH ON TWO OTHER… · p. 19
  • On KU Leuven, direction decoders fell by 17 to 45 points after leakage was removed, while MERIT exceeded NeuroToken on unseen listeners at 0.6664 and 0.6635.Within listeners, NeuroToken exceeded MERIT’s 0.6763 and 0.6909 on both datasets; the five re-run direction decoders had no unseen-listener results.Source: I THE COUPLING BRANCH ON TWO OTHER… · p. 19, Table 8 · p. 19
  • The published-split direction decoders reached 0.64 to 0.93, but after leakage removal their accuracy ranged from 0.35 to 0.70, with several results near or below chance.NeuroToken holds out trials, whereas MERIT holds out stimuli; KU Leuven repeats stories across trials, so a trial split can place the same story in training and test.Source: I THE COUPLING BRANCH ON TWO OTHER… · p. 19
  • The evaluation sequence on other data is audio probing, same-talker testing, then direction checking; omitting the final check could leave accuracy reported as brain decoding.The authors state that the KU Leuven and DTU direction checks do not confirm brain decoding, and do not claim the same check was run on the comparison methods.Source: I THE COUPLING BRANCH ON TWO OTHER… · p. 20

J THE HINGE EXPERIMENT ON UNSEEN LISTENERS

The hinge experiment favors MERIT’s no-hinge, no-dropout model for EEG credit on unseen listeners while showing that short-window fusion can be dominated by behavioral information. The section also reports the training safeguards and the evaluation settings used to assess those results.

J THE HINGE EXPERIMENT ON UNSEEN LISTENERS

  • The model without the hinge and dropout has the most EEG credit on unseen listeners and ties for the most within listeners, while the hinge/no-dropout model is most accurate within listeners and least accurate on unseen ones.No arm is best in every Table 9 column.Source: J THE HINGE EXPERIMENT ON UNSEEN… · p. 20, Table 9 · p. 20

K EVERY STREAM SET, BOTH SPLITS

L CHECKS ON THE EEG SIGNAL

  • The cited summary identifies two analyses: Figure 3 shows how contribution changes with the decision window, and Table 12 gives same-talker results plus the direction-in-time check.Source: L CHECKS ON THE EEG SIGNAL · p. 20

L.1 SHORT DECISION WINDOWS, BY STREAM SET

  • At W = 2 s, EEG alone contributes +0.0263, while the frozen foveal embedding alone reaches 0.7130 accuracy with +0.4570 contribution, and the five-stream model reaches 0.7291.The authors caution that this fused short-window accuracy should not be read as a brain result.Source: §L.1 · p. 20

M THE HINGE EXPERIMENT AT TEN SECONDS

  • At W = 10 s, selecting checkpoints by contribution instead of accuracy is the one reported training choice that raises EEG credit.Table 14 varies checkpoint selection, hinge, and modality dropout while keeping streams, data, and folds identical.Source: M THE HINGE EXPERIMENT AT TEN SECONDS · p. 21

N THE TRAINING OBJECTIVE, EXPLAINED

  • Training uses single-listener batches because raw EEG identifies listeners at 0.90 accuracy in a sixteen-way test, and the contrastive and anti-collapse terms discourage a time-constant EEG representation.With a constant brain embedding, the time-centred score is zero for every pair, the contrastive loss equals its chance value log B, and the anti-collapse term measures lost temporal spread.Source: N THE TRAINING OBJECTIVE, EXPLAINED · p. 21

O REPRODUCIBILITY

The paper packages its decoder, data code, and tests as reproducible runs with scripted result reporting. It also explains how Shapley credits are computed and how the reported per-stream measures differ.

O REPRODUCIBILITY

  • The decoder, data code, and tests share one configuration file, and experiments run from one command with released job scripts and automatically inserted results.Models within a run share data and folds, and each result is used only after every fold finishes.Source: O REPRODUCIBILITY · pp. 21–22
  • The reproducibility materials include the stream-set and short-window runs, Table 12’s same-talker and directionality checks, and Figure 3’s window-length comparison.Figure 3 plots contribution and shuffled accuracy by window length; Table 13 covers all stream sets at W = 2 s and W = 5 s.Source: O REPRODUCIBILITY · p. 22, Table 12 · p. 22, Figure 3 · p. 22

Q WHY THE SHAPLEY CREDITS ADD UP

  • All 32 subsets are computed with shared shuffles, and each stream’s Shapley credits sum exactly to the total contribution.The exact sum is checked against the total for every run; most computational cost comes from rescoring because embeddings are computed once.Source: Q WHY THE SHAPLEY CREDITS ADD UP · p. 23

R THREE PER-STREAM MEASURES

S DATASET

MAESTRO pairs EEG with behavioral and scene streams in four-talker listening trials. Its windowed examples randomize candidate order and use non-overlapping negatives to avoid audio shortcuts.

S DATASET

  • MAESTRO includes 16 listeners completing 100 trials each, with four talkers at fixed positions and the attended talker about 15 dB louder.Each trial lasts about 30 seconds; recordings are not reused across training and test.Source: S DATASET · p. 24
  • The dataset records EEG, gaze, head motion, scene-video optical flow, and per-talker speech envelopes, plus frozen gaze-centered V-JEPA 2 embeddings.Signals are resampled to 64 Hz and standardized per channel over each trial; the added embeddings are never trained.Source: S DATASET · p. 24
  • Examples use 10-second windows with a 5-second hop by default, yielding 640 samples and 7,990 windows.Each example contains exactly one correct candidate envelope among K candidates, its index, attended loudspeaker, candidate permutation, and listener identity.Source: S DATASET · p. 24
  • Candidates are randomly ordered in each window, so candidate position carries no answer information and test order is shared across compared models.At test time, order is fixed by window index.Source: S DATASET · p. 25
  • Furthest-window same-talker negatives let an audio probe exploit trial position, reaching 0.60 on a task with 0.50 chance.The method instead samples random non-overlapping negatives from the same trial and lowers K when short trials cannot supply enough; per-trial training crops and candidate matching are also randomized.Source: S DATASET · p. 25

Related papers

Loading 2610.17507v1…