PaperlayerView exact revision on arXiv

Source-linked AI summary

Removing Timing Shortcuts Improves Non-Invasive Brain-to-Text

Dulhan Jayalath, Oiwi Parker Jones

arXiv:2609.40359v1cs.LGq-bio.NC

TL;DR

Non-invasive speech decoding seeks words from weak, spatially blurred brain signals, but overlapping word-aligned windows can expose timing cues that mimic neural information. The authors test this shortcut and decode words independently, making aggregation and linguistic priors more effective. Synthetic signals nearly match MEG accuracy, while SimpleB2T reaches 36.6% WER with five observations per word.

  • Problem

    Non-invasive speech decoding must recover words from weak, spatially blurred signals, and it is unclear how much joint-decoding gains reflect neural information versus timing leaked by overlapping windows.

    Source: §1 · pp. 1–2

  • Method

    The authors test overlap-preserving synthetic controls and decode each word independently, then combine predictions across observations and with an LLM prior.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    22.0% word accuracy on synthetic signals versus 22.3% on MEG; SimpleB2T reaches 36.6% WER with five observations per word.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Removing the timing shortcut makes combining observations and an explicit linguistic prior substantially more effective for perceived-speech decoding.

    Source: §1 · p. 2

    The evaluation uses perceived speech with known word onsets; attempted-speech decoding from paralyzed patients and segmentation without known onsets remain unevaluated.

    Source: §2.3 · p. 5, §4 · p. 10

Abstract

from arXiv · show

We find that major reported improvements in decoding words from non-invasive brain recordings are largely reproducible without any brain data. In the influential work of d'Ascoli et al. (2025), time series of brain activity from subjects perceiving continuous speech are segmented into fixed-length windows starting at each word. A neural network then generates predictions for all of the words in a sentence together. Neighbouring windows partially overlap, implicitly revealing the interval between words. Since these intervals indicate the duration of the words spoken, and different words tend to have different durations - for example, "the" is much shorter than "supercalifragilisticexpialidocious" - the neural network can improve its predictions of words without relying on the underlying brain activity. Consistent with this, the method reaches 22.0% balanced accuracy on synthetic signals containing no brain information, compared with 22.3% on real brain recordings. To prevent the network from learning this shortcut, we make a single, simple change. Instead of jointly encoding all windows in a sentence, we process each independently. As a result, the neural network achieves better performance by learning underlying word-specific information from brain recordings. This makes two existing strategies become much more effective than before. Both aggregating predictions from distinct neural responses to the same word and using a pretrained LLM as a linguistic prior now substantially improve results. On our perceived speech benchmark, this simple recipe (SimpleB2T) achieves a word error rate of 36.6% with five observations per word, approaching past invasive speech decoding performance, albeit under different conditions. The results in this work expose an important shortcut in brain-to-text decoding and show that removing it leads to a simple and considerably more effective strategy.

1 INTRODUCTION

The introduction identifies timing leakage in jointly decoded word-aligned windows as a source of apparent brain-to-text gains, then presents independent decoding as a simple fix that makes aggregation and linguistic priors more effective.

1 INTRODUCTION

  • The study targets word-aligned brain-to-text from perceived speech, a common test-bed toward the broader goal of decoding internal speech.Non-invasive MEG and EEG offer a route toward restoring communication, but measure weak and spatially blurred signals outside the brain.Source: §1 · p. 1
  • 22.3% word accuracy on MEG versus 22.0% on synthetic signals, while removing overlap lowers accuracy to 5.8%.The synthetic signals contain no brain information, showing that overlap can reproduce nearly the full joint-decoding gain.Source: Figure 1 · p. 2, §1 · p. 2
  • Jointly decoding all sentence words from word-onset windows improved classification by an average of 50% over independent decoding in d’Ascoli et al. (2025).The approach has been extended and incorporated into subsequent non-invasive brain-to-text studies and benchmarks.Source: §1 · p. 2
  • Overlapping windows expose word durations because shared samples appear at different relative positions, letting models use timing cues instead of brain activity.Typical word durations differ, so this timing information can narrow the plausible words; the paper identifies this as shortcut learning (Geirhos et al., 2020).Source: §1 · p. 2
  • Decoding each word independently prevents the network from learning this shortcut, making predictions more useful for aggregation and combination with an LLM prior.The paper reports that timing-driven predictions are poorly suited to these strategies, while predictions made without the shortcut can be combined effectively with both.Source: §1 · p. 2
  • 65.6% WER with one observation per word and 36.6% WER with five: SimpleB2T’s results on a core subset of clinically motivated sentences.SimpleB2T combines independent decoding with multiple observations and a linguistic prior.Source: §1 · pp. 2–3

2 WORD DURATION LEAKAGE IN WORD-ALIGNED DECODING

This section examines whether overlapping windows in word-aligned decoding reveal word-duration information that can explain the apparent gains from jointly decoding words.

2 WORD DURATION LEAKAGE IN WORD-ALIGNED DECODING

  • The section tests whether timing information exposed by overlapping windows can explain the gains attributed to jointly decoding words.The controls preserve or remove overlap to test whether overlap alone reproduces the gain and whether joint decoding remains useful without it.Source: §2 · p. 3, §2.3 · p. 4, §2.2 · p. 4

2.1 DECODING WORDS JOINTLY FROM NEURAL RESPONSES

  • The decoder uses known word onsets to extract three-second M/EEG windows, jointly maps sentence windows to T5-space embeddings, then retrieves nearest candidate words.The network is trained to make predicted embeddings similar to the target word embedding and dissimilar to embeddings of other batch words.Source: §2.1 · p. 3

2.2 OVERLAPPING WINDOWS REVEAL WORD DURATION

  • 99.9996% of adjacent word pairs overlap, sharing 90.6% of samples on average; their onset intervals correlate with word duration (r = 0.90).Matching shared samples between adjacent windows recovers the interval between word onsets.Source: §2.2 · pp. 3–4

2.3 SYNTHETIC SIGNALS REPRODUCE DECODING IMPROVEMENTS

  • 22.3% balanced word accuracy on MEG fell to 9.5% with isolated decoding, while synthetic overlapping windows reached 22.0% and removing overlap reduced accuracy to 5.8%.A model trained directly on word intervals reached 22.9%; the overlap effect was also reproduced across 32 subjects and additional datasets.Source: §2.3 · p. 5
  • Joint decoding is more accurate for words with consistent training durations and favors candidates whose typical durations match each test occurrence more than isolated decoding.Figure 2a plots word accuracy against duration variability; panel b compares duration mismatch before and after replacing the test occurrence’s duration with another occurrence of the same word.Source: Figure 2 · p. 4, §2.3 · p. 5

3 REMOVING TIMING SHORTCUTS IMPROVES SENTENCE RECONSTRUCTION

The section replaces joint sentence-level encoding with independent word decoding, then combines repeated neural responses and an LLM prior. On the clinical benchmark, LLM rescoring and additional observations each reach 36.6% WER at k = 5.

3.1 DECODING WORDS INDEPENDENTLY WITH A SIMPLE NEURAL NETWORK

  • The independent decoder retains the word-level encoder, semantic targets, and contrastive training, but maps each word-aligned window separately using about 20 million parameters rather than 200 million.Source: §3.1 · p. 5

3.2 COMBINING OBSERVATIONS AND LINGUISTIC PRIORS

  • SimpleB2T averages independently encoded responses, uses their agreement to adjust neural-score sharpness, and beam-searches sentence scores combined with a Qwen3-8B language model.Source: §3.2 · pp. 6–7

3.3 EXPERIMENTAL SETUP

  • The benchmark comprises 200 clinically motivated sentences in Core and Expanded subsets, with distinct non-overlapping recording events and up to five occurrences per sentence position.Source: §3.3 · p. 7

3.4 RESULTS

  • After independent decoding, aggregation and LLM rescoring become substantially more effective; with joint decoding, aggregation adds little and rescoring is close to or worse than the LLM-only baseline.Source: §3.4 · p. 7
  • At k = 5, LLM rescoring lowers WER from 74.3% to 36.6%, while increasing observations from one to five lowers WER from 65.6% to 36.6%.Source: §3.4 · p. 8
  • Sentence decoding improves steadily as more responses are combined without retraining, trading additional recording burden for more reliable predictions.Source: §3.4 · p. 8
  • At k = 5, the neural decoder and LLM alone yield 74.3% and 73.8% WER, respectively, while combining them reduces WER to 36.6%.Source: §3.4 · p. 8
  • Explicitly describing the patient communication setting improves decoding, but the current scoring rule can fail to rank good reconstructions first, leaving sentence selection a bottleneck.Source: §3.4 · p. 9

4 DISCUSSION

The discussion argues that overlapping word-aligned windows can expose timing cues that reproduce much of the gain attributed to joint decoding, while independent decoding removes that shortcut. Its conclusions are scoped to word-aligned perceived speech and a constrained communication setting, with imagined-speech decoding left as a future step.

4 DISCUSSION

  • Independent decoding removes the timing shortcut, making repeated-observation aggregation and linguistic priors more effective by improving use of word-specific neural signals.With timing-driven predictions, aggregation instead estimates typical word duration; without the shortcut, it estimates the word-specific neural signal.Source: §4 · p. 9
  • The shortcut matters when jointly decoded, overlapping windows have variable onset intervals; equal-duration reading protocols are unaffected, and joint decoding did not improve those datasets.Among nine datasets analysed by d’Ascoli et al. (2025), only LittlePrinceRead and Nieuwland used the same amount of time for each word.Source: §4 · pp. 9–10
  • The timing-leakage result applies when aligned, overlapping inputs jointly model informative timing; its importance depends on the task, and Brain2Qwerty timing explains little performance.The discussion notes that analogous shortcuts could occur in other settings with aligned, overlapping inputs.Source: §4 · p. 10
  • Evaluation assumes known word locations and repeated observations, unlike joint speech segmentation, though separate word attempts could make segmentation easier.SimpleB2T obtains large sentence-level gains from five observations per word, compared with 100 observations per phoneme in the 2025 PNPL competition.Source: §4 · p. 10
  • SimpleB2T approaches an invasive speech-decoding result under different conditions, but evaluates perceived speech in a constrained communication setting rather than decoding speech from paralysed patients.The comparison is to Moses et al. (2021), which reported median WER of 25.6% with a 50-word vocabulary and 50 test sentences.Source: §4 · p. 10
  • Future work includes improving beam-search candidate selection, reducing required observations, and extending evaluation to imagined speech for practical non-invasive interfaces.The discussion identifies internally generated speech as a key step toward practical non-invasive communication interfaces.Source: §4 · p. 10

AI USE STATEMENT

The paper reports its AI use and ethics boundaries, outlines reproducibility materials, and presents controls examining timing leakage across word-aligned windows and subjects.

AI USE STATEMENT

  • The authors report using generative AI to help construct the communication benchmark and draft and edit the manuscript, with generated sentences reviewed by the authors.AI also assisted with code generation and debugging; the authors say they inspected the code and verified outputs and scientific claims.Source: AI USE STATEMENT · p. 11

ETHICS STATEMENT

  • The work uses public LibriBrain100 recordings, and its communication benchmark is not clinically validated; the authors encourage user-centred evaluation before real-world use.The authors also state that the method decodes perceived, not imagined, speech and does not yet demonstrate a practical BCI.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The paper and appendices provide experimental protocols, preprocessing and model settings, controls, data splits, benchmark construction, and training seeds for reproducing experiments.Models are evaluated across up to five seeds.Source: REPRODUCIBILITY STATEMENT · p. 11

APPENDIX CONTENTS

  • Figure 5 relates annotated word duration to the interval from that word’s onset to the next; the paper reports a strong correlation (r = 0.90).The dashed line marks equality and the solid line a linear fit.Source: Figure 5 · p. 15, §A.1 · p. 15

A.1 RELATIONSHIP BETWEEN WORD TIMING AND DURATION

  • Across 122,120 adjacent word pairs in perceived speech data, the onset-to-onset interval strongly correlated with the annotated duration of the preceding word (r = 0.90); three-second windows overlapped for 99.9996% of adjacent word pairs.Source: §A.1 · p. 15

A.2 SYNTHETIC CONTROL DETAILS

  • The synthetic control compares shared continuous signals across word windows with independently sampled signals, isolating the overlap information while keeping signals unrelated to words.The shared signals use pulses smoothed at three timescales; the paper also describes a timing-only control that feeds onset intervals to a sentence-level transformer.Source: §A.2 · pp. 15–16

A.3 TIMING SHORTCUT REPRODUCTION ACROSS MULTIPLE SUBJECTS

  • The shared-synthetic decoder remained highly accurate across 32 other LibriBrain100 subjects and performed comparably to the MEG-trained decoder.The evaluation used a checkpoint trained on subject 0; the figure shows the mean and per-subject results, with standard deviations across five training seeds.Source: Figure 6 · p. 16, §A.3 · p. 16

A.4 TIMING SHORTCUT REPRODUCTION ACROSS DATASETS

The appendix reports that synthetic controls reproduce the joint-decoding performance pattern across two additional datasets, that joint predictions align more with duration-based predictions, and that retrained joint decoding without overlap does not outperform isolated decoding.

A.4 TIMING SHORTCUT REPRODUCTION ACROSS DATASETS

  • On the Armeni et al. (2022) and Le Petit Prince MEG datasets, the synthetic control performed on par with real MEG.The experiments used all subjects and specified session splits for Armeni et al., and the listening component with book-section splits for Le Petit Prince.Source: §A.4 · p. 16, Figure 7 · p. 17

A.5 AGREEMENT WITH DURATION-BASED WORD PREDICTIONS

  • Joint MEG and synthetic decoders showed larger divergence increases after shuffling than isolated decoding, indicating stronger alignment with duration-based predictions.The analysis compares decoder probabilities with a duration-based posterior; Figure 8 describes larger values as stronger alignment.Source: §A.5 · pp. 16–17, Figure 8 · p. 17

A.6 JOINTLY DECODING WORDS WITHOUT OVERLAPPING INPUTS

  • Retraining the joint architecture without overlap recovered some performance but still did not outperform isolated decoding, while the original model collapsed when tested without overlap.The non-overlapping sets preserved sentence word sequences using separate recordings, retaining 91% of test and 96% of training sentences.Source: Figure 9 · p. 18, §A.6 · p. 18

A.7 TIMING SHORTCUT CONTROL FOR BRAIN2QWERTY

A synthetic-input control tests whether aligned typing windows give Brain2Qwerty a timing shortcut. Synthetic performance is near chance compared with real MEG, so timing does not explain the strong reported performance.

A.7 TIMING SHORTCUT CONTROL FOR BRAIN2QWERTY

  • 6.5% validation and 6.8% test balanced character accuracy on synthetic inputs, versus 37.0% and 34.9% on real MEG, indicates timing explains little Brain2Qwerty performance.The synthetic condition preserves aligned-window structure; Figure 10 reports a 3.45% chance level.Source: §A.7 · pp. 18–19, Figure 10 · p. 19

B DATA AND EXPERIMENTAL DETAILS

The experiments define session-level data splits and use language-model, Gaussian-noise, and shuffled-representation controls to assess sentence decoding. The controls distinguish performance from linguistic priors alone from performance dependent on aligned brain evidence.

B.2 SENTENCE DECODING CONTROL DETAILS

  • The LM-only baseline removes neural evidence while retaining the frozen Qwen3-8B-Base model and the corresponding clinical prompt.Its deterministic result measures performance attributable to the language model under the benchmark's vocabulary and length constraints.Source: §B.2 · p. 19

B.1 DATA SPLITS

  • The split assigns recordings by session, with 121 training, 10 validation, and 31 test recordings; Core and Expanded use the test partition.Development sentences use the validation partition, and all recordings sharing a session label stay in the same split across tasks.Source: §B.1 · p. 19, Table 3 · p. 20

B.2 SENTENCE DECODING CONTROL DETAILS

  • The Gaussian-noise control matches MEG's sensor- and time-specific means and variances in expectation but contains no perceived-word information.The estimates use 10,000 training windows, with independent draws across sensors, time points, and occurrences.Source: §B.2 · pp. 19, 21
  • The shuffled control preserves the distribution and structure of neural representations but permutes them onto incorrect test occurrences.The permutation is selected independently of labels and without replacement.Source: §B.2 · p. 21

B.3 TRAINING AND DECODING DETAILS

Sentence decoding combines neural and language-model scores, with the language-model weight selected separately for each observation count. An analysis finds no consistent advantage when later words in a window are more predictive of its target.

B.3 TRAINING AND DECODING DETAILS

  • The language-model score sums autoregressive token log probabilities conditioned on a fixed clinical prompt and the current candidate-sentence prefix.Candidate words use a leading space, the first word is capitalised, “I” is uppercase, and token probabilities use the full tokenizer vocabulary.Source: §B.3 · pp. 21–22
  • Beam search expands each prefix with all 92 candidate words and adds the neural score to the language-model score weighted by λ.The known sentence length fixes the number of generated words; completed candidates are scored with sentence-ending punctuation before returning the highest-scoring one.Source: §B.3 · p. 22
  • As observations are aggregated, the development-selected language-model weight decreases and then stabilises.The selected λ is set separately for each observation count using the 50-sentence development set and fixed before test evaluation.Source: §B.3 · p. 22, Figure 11 · p. 22

C.1 PREDICTABILITY FROM SUBSEQUENT WORDS

  • Subsequent-context predictability is estimated from later words fully contained in each three-second window, using a frozen language model to score target-word candidates.Predictability ranges from 0.04% to 84.6%, with a median of 20.7%; positions are split into low and high groups with equal counts per target word.Source: §C.1 · p. 23
  • In word-matched positions, Brain accuracy was 24.0% for low-predictability and 27.1% for high-predictability contexts, while Brain + LM was 65.3% and 64.9%.The overall split showed no consistent high-predictability advantage, so SimpleB2T did not appear to significantly benefit from other words in each three-second window.Source: §C.1 · p. 23, Table 5 · p. 23

C.2 TRAIN–TEST MISMATCH CONTROL IN THE CLINICAL BENCHMARK

Retraining the joint decoder on non-overlapping sentence constructions tests whether its clinical-benchmark performance is limited by a training–evaluation mismatch. This adjustment does not materially improve sentence reconstruction.

C.2 TRAIN–TEST MISMATCH CONTROL IN THE CLINICAL BENCHMARK

  • The control addresses the mismatch between overlap-trained sentences and benchmark positions assembled from separately recorded occurrences.The benchmark therefore lacks coherent cross-word neural activity for the joint decoder to exploit.Source: §C.2 · pp. 23–24
  • The retraining constructs sentences with the original word sequences but substitutes each position's MEG window with a three-second window from another occurrence of that word.Assigned windows cannot overlap within a sentence; assignments are re-randomised between epochs, retaining 96.3% of eligible training occurrences.Source: §C.2 · p. 23
  • At k = 5 with LLM rescoring, the matched model had 79.2%, 85.5%, and 82.6% WER on Core, Expanded, and Full, versus 79.4%, 85.5%, and 82.7% for the original model.The control shows that matching the non-overlapping evaluation setting does not recover independent decoding's performance.Source: §C.2 · pp. 23–24, Table 6 · p. 24

C.3 WORD-LEVEL ANALYSIS

Word-level accuracy is most strongly associated with language-model predictability, while prompt specificity improves sentence reconstruction. The paper also reports normalized OVMI to aid comparisons across communication settings.

C.3 WORD-LEVEL ANALYSIS

  • LM predictability had the strongest word-accuracy association (r = 0.42), while duration variability was negatively associated (r = −0.26).Phonetic and semantic distinguishability, training frequency, and mean duration showed weak associations; the figure also reports accuracy across word bins.Source: §C.3 · p. 24, Figure 12 · p. 25

C.4 EFFECT OF THE LINGUISTIC PRIOR

  • Specifying the intended patient-communication setting improves decoding relative to more generic prompts.The prompt ablation was conducted after the main SimpleB2T experiments and did not select the prompt used in the main results.Source: §C.4 · p. 24, Table 7 · p. 26

C.5 OVMI SCORES

  • To aid comparisons across communication settings, the paper reports normalized OVMI under four reference distributions.The reported estimates use balanced word accuracies of 16.04/13.29/14.80% at k = 1 and 46.83/45.08/45.93% at k = 5 for Core/Expanded/Full, respectively.Source: §C.5 · p. 24, Table 8 · p. 26

D CLINICAL COMMUNICATION BENCHMARK

The clinical communication benchmark pairs 200 reviewed, clinically motivated sentences with word-aligned MEG windows assembled from held-out recordings. Each sentence position uses five distinct word occurrences, with no recording event reused.

D CLINICAL COMMUNICATION BENCHMARK

Loading 2609.40359v1…