PaperlayerView exact revision on arXiv

Source-linked AI summary

Brain-to-Language Decoding: Tasks, Signals, Methods, Evaluation, Practical Use and Beyond

Yiqian Yang, Yiqun Duan, Chenyu Liu, Yiqi Wang, Xinliang Zhou, Chin-Teng Lin, Yu Zhang

arXiv:2609.27650v2cs.CLcs.HCcs.NE

TL;DR

Brain-to-language decoding may support communication after speech loss and study how the brain represents language, but different tasks and neural measurements recover different information. This survey synthesises those task–signal–representation relationships and their methods, evaluation, and practical use; it identifies complementary representation routes and user-control needs while noting boundaries in the evidence.

  • Problem

    The survey asks which aspects of language different neural measurements can recover and how those representations support expression, across communication-restoration and language-representation aims.

    Source: §1 · pp. 1–2

  • Method

    The survey relates Articulated, Inner, and Perceived tasks to neural signals and decoder representations, comparing methods and reported results across invasive and non-invasive recordings.

    Source: §1 · pp. 2–3

  • Results

    The synthesis identifies complementary phonetic, acoustic, and semantic targets, shared representations for reuse across recording conditions and tasks, and interaction needs alongside decoding accuracy.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The synthesis points to transferable representations, self-generated expression, and interfaces that preserve the user’s control as research directions.

    Source: §1 · p. 3

    Current fUSI evidence does not establish language-content recovery, and motor-decoding results obtained through cranial windows cannot simply be assigned to transcranial acquisition.

    Source: §4.3.1 · pp. 18–19

Abstract

from arXiv · show

Brain-to-language decoding translates neural activity associated with language production, internal speech and perception into linguistic or expressive outputs. It offers a route to restoring communication after speech loss and a means of studying how the brain represents language. Advances in neural recording and representation learning have expanded the field from constrained recognition and acoustic reconstruction to text generation, streaming personalised speech and facial animation. This survey synthesises these developments across invasive and non-invasive measurements, drawing on a search without a lower year limit and source-led updates through September 2026. We connect Articulated, Inner and Perceived tasks to the neural populations they engage, the representations available to decoders and the outputs those representations can support. We examine model development, public resources and the evolution of evaluation, and compare published performance and communication costs within their reported protocols. The synthesis identifies complementary routes to progress: phonetic, acoustic and semantic targets preserve different aspects of a message; shared representations support reuse across recording conditions and tasks; and online communication increasingly depends on calibration, feedback and user control alongside decoding accuracy. Shared benchmarks enable algorithmic comparisons, while longitudinal studies reveal the demands of sustained use. We discuss these developments and their remaining limitations, then outline a prospective five-level trajectory from commands and language to meaning, scenarios and bidirectional cognitive exchange

1 Introduction

Brain-to-language decoding links neural activity to observable language outputs, with clinical aims of restoring communication and scientific aims of studying language representation. This survey relates tasks, signals, representations, methods, evaluation, and practical use across the field.

1 Introduction

  • The field seeks both communication after speech loss and computational insight into how sounds, words, and meanings are represented during speaking, inner speech, listening, and reading.These aims draw on the opportunity to learn language information from neural recordings and translate it into observable output.Source: §1 · p. 1
  • Research has advanced from auditory-cortex speech-feature reconstruction and phoneme-based recognition to sentence-level attempted-speech decoding, personalised voice, facial animation, streaming synthesis, and prolonged home use.Non-invasive EEG and MEG studies link neural activity to heard or read language, while haemodynamic recordings support semantic reconstruction over longer timescales.Source: §1 · p. 1
  • Figure 1 maps tasks, signals, methods, evaluation, and practical use, distinguishing generated outputs from evidence of reliable communication.Its sentence pair, event sequence, and uncertainty glyph are conceptual examples, not measured results.Source: Figure 1 · p. 2
  • Phoneme sequences support word composition, acoustic and articulatory representations offer routes to speech synthesis, and semantic representations permit wording changes while retaining aspects of meaning.Representation choices interact with the participant’s task and the timescale and coverage of the recording.Source: §1 · p. 2
  • The survey connects listening, reading, overt, attempted, and inner speech to different supervision needs, and examines how pretrained models connect neural encoders to text, speech, and visual expression.Listening and reading provide paired neural and language observations; overt speech supplies time-aligned acoustic targets, while attempted and inner speech require alternative labels, alignment, or transfer strategies.Source: §1 · p. 3
  • The synthesis compares datasets, supervision, decoding mechanisms, and published results in matched evaluation settings, then connects output quality, generalisation, and computational demands to interaction and sustained access.It motivates research on transferable representations, self-generated expression, and interfaces that preserve user control over a message.Source: §1 · p. 3

2 Review Method and Scope

The review defines a condition-based scope and compares prior perspectives through a documented search, selected evidence checks, and a linked registry. Its critical synthesis is broad but does not establish exhaustive retrieval or whole-paper risk-of-bias appraisal.

2.1 Scope and synthesis

  • The review covers neural language-content recovery in Articulated, Inner, and Perceived tasks, distinguishes content decoding from adjacent analyses, and compares results within experimental conditions rather than pooling scores.An experimental condition combines task, recording, supervision, output, tested content, controls, and use setting; missing controls or operating details remain unknown, not evidence for or against a capability.Source: §2.1 · pp. 3–4

2.2 Prior reviews and the synthesis developed here

  • The synthesis brings task, signal, and method perspectives into dialogue with evaluation and practical use, comparing how reviews treat supervision, output pathways, and operating requirements.It distinguishes linguistic target granularity from participant behaviour, and fluency, exact wording, and semantic fidelity as separate evidence of neural attribution.Source: §2.2 · pp. 4–5

2.3 Search and study selection

  • Searches without a lower-year limit combined Europe PMC and OpenAlex queries, citation tracking, and supplements; the original retrieval yielded 813 retained IDs and 795 canonical entries before source-led additions.The updated registry reached 920 report/resource entries; publication versions were linked, and companion releases and articles can describe the same acquisition.Source: §2.3 · pp. 6–7, Figure 2 · p. 7

2.4 Evidence extraction and source verification

  • Selected structured checks cover 246 entries, but field-level checking is not a whole-paper risk-of-bias appraisal; incomplete platform access and non-English coverage mean retrieval is not established as exhaustive.Native-platform searches did not yield exportable results for Scopus, Web of Science, IEEE Xplore, or ACM DL, and citation tracking is incomplete.Source: §2.4 · pp. 7–8

2.5 Corpus coverage

  • The registry contains 920 canonical report/resource entries, of which 246 have selected structured fields; its counts describe a selectively assembled index, not field prevalence or independent cohorts.It includes multi-label signal counts, and publication and resource entries may describe the same acquisition.Source: §2.5 · p. 8

3 Tasks

The section distinguishes executed articulation, internal speech simulation and externally presented language as task conditions, not labels for diseases or devices. These distinctions matter because content source changes available supervision and feedback, while communication needs shape task choices.

3.1 Intended users and communication needs

  • Communication needs include conversation, computer access and environmental control, while task groups distinguish articulation, internal simulation and externally presented language rather than disease or device classes.Preferences for active control and manageable operation also inform design; requirements vary across populations and available movements.Source: §3.1 · pp. 8–9, Table 2 · p. 10

3.2 Articulated: Overt, Mouthed and Limited

  • Overt supplies acoustic targets, Mouthed retains execution without audible speech, and Limited requires residual articulation; transfer must account for target availability and sensory feedback.Shared production structure offers a basis for transfer, but mode-specific timing and feedback make transfer a learning problem.Source: §3.2 · p. 11

3.3 Inner: Articulatory, Auditory and Mixed

  • Articulatory, Auditory and Mixed specify imagined speaking movement, imagined speech sounds, or an explicit combination; EMG or motion monitoring separately assesses compliance.Instructions define the task, while peripheral monitoring characterizes performance and potential contributions.Source: §3.3 · p. 11
  • Cued inner sentences supported self-paced intracortical communication with less reported physical effort and cross-task learning, while independently chosen messages require reference collection and decoder training.The cited study links shared but distinguishable attempted- and inner-speech representations to cross-task learning.Source: §3.3 · p. 11

3.4 Perceived: Auditory, Visual and Audiovisual

  • Perceived tasks label externally presented speech, written language or visible articulation; Visual names the input, not a restriction to visual brain regions.A written cue followed by an instructed imagery interval does not make that interval a Reading task.Source: §3.4 · p. 12
  • During silent lipreading, MEG carried information about unheard acoustic features beyond visible lip trajectories; listening-trained fMRI decoding was also tested on imagined storytelling using semantic agreement.The imagined-storytelling comparison used separately recorded accounts as references.Source: §3.4 · p. 12

3.5 Task instructions, cues and behavioural verification

  • Facial EMG distinguished phonetic classes during overt speech but not comparably at group level during instructed inner speech; increased activity alone was not specific to speech content.The analysis relates peripheral activity to the instruction and linguistic target.Source: §3.5 · p. 12
  • Valid transfer tests vary cue–target mapping and evaluate later intervals separately; self-generated expression also requires user-chosen content unavailable to the decoder and a check of intended meaning.Temporal separation should account for physiological response and cue carry-over; separating epochs alone does not establish independence.Source: §3.5 · pp. 12–13

4 Signals

Neural measurements provide complementary views of language, while task, sampled population, and decoding target determine which information an interface can recover. The section connects these choices to signal timing, coverage, access, and practical constraints.

4.1 Neural representations of language

  • Production, perception, reading, and inner speech expose task-dependent mixtures of phonetic, articulatory, acoustic, contextual, orthographic, and intention-related information across sampled populations.The population dimension associated with motor intention helped distinguish inner from attempted speech.Source: §4.1 · pp. 14–15
  • Acoustic targets support heard-speech reconstruction, phoneme sequences connect attempted speech to text, and contextual targets constrain generated descriptions; one recording can feed multiple outputs.Representation studies motivate targets, while held-out and online decoding test how effectively a system recovers them.Source: §4.1 · p. 15

4.2 Electrophysiological signals

  • Electrophysiological measurements range from scalp EEG and MEG to cortical-surface, depth, and intracortical recordings, with different coverage and signal access; sEEG also supports speech-state and intent-contact detection.A shared model architecture does not make the neural inputs equivalent.Source: §4.2 · p. 17

4.3 Haemodynamic signals

  • Haemodynamic methods measure vascular responses integrated over time; language applications include concept discrimination, gist reconstruction, cue-conditioned sentence generation, and stimulus-segment retrieval.Faster acquisition does not remove vascular-response delay, so target duration and alignment remain important.Source: §4.3 · p. 17

4.3.1 Functional ultrasound: mapping and decoding boundaries

  • Language-related vascular responses can be mapped with fUSI, whose fast image acquisition does not remove the underlying neurovascular response delay.Pulse echoes and Doppler processing yield a blood-volume-sensitive sequence; the signal is not a direct measurement of spikes.Source: §4.3.1 · p. 17, Figure 6 · p. 18
  • Human fUSI evidence separates language-response mapping from content decoding: window-based motor decoding and sustained access do not establish language-content recovery or transcranial decoding.Intra-operative language maps localize task responses but leave open whether they distinguish words or sentences.Source: §4.3.1 · pp. 18–19

4.4 Sensors, placement and recording configurations

  • 128- and 256-contact intra-operative µECoG improved non-word decoding in four participants; recurrent and Transformer sEEG reconstruction outperformed linear regression.Recording placement and coverage constrain accessible populations; trade-offs between configurations depend on the objective rather than establishing a global modality ranking.Source: §4.4 · pp. 19–20, Table 5 · p. 20

4.5 Signal quality and acquisition constraints

  • Acquisition controls must match the measurement: movement, gaze, vibration, superficial physiology, and recording hardware can complicate interpretation, while timing and supervision must match the decoder target.Auxiliary channels can test confounds or serve deliberately as communication inputs; those roles require different claims.Source: §4.5 · pp. 20–21

5 Methods

This section surveys brain-to-language decoding methods, covering how neural data, supervision, preprocessing, model priors, targets and test protocols shape outputs and what reported scores establish. It compares text-decoding approaches, reports a noninvasive MEG test, and frames clinical speech interfaces across compositional text, expressive communication and sustained use.

5.1.1 Experimental protocols and datasets

  • Dataset contrasts support distinct learning questions, while split units determine whether tests assess new trials, material, sessions or participants.Matched vocalised, mouthed and internal production supports cross-task learning; repeated observations add less linguistic diversity than new material, and matched tasks or languages can matter for transfer. Partitions should precede overlapping windows or learned preprocessing to prevent leakage.Source: §5.1.1 · pp. 21, 23, §5.1.4 · p. 25

5.1.2 Target labels and temporal alignment

  • Training-time iterative dynamic time warping can learn uncertain response onset and duration, whereas reference-dependent test alignment adds information and constrains claims of autonomous operation.Protocols should state which boundaries, alignments and transcripts are supplied at test time; haemodynamic models must also account for stimulus-to-response timing.Source: §5.1.2 · pp. 23–24

5.1.3 Preprocessing and artifact handling

  • Artifact correction can lower EEG classification accuracy when models exploit structured non-neural variation, so cleaning and performance require joint assessment.Reproducible comparisons specify temporal support, channels, rejection, auxiliary signals and normalization; learned transforms should be fit within training data, and artifact removal alone does not establish neural origin.Source: §5.1.3 · pp. 24–25

5.2 Output types and decoding objectives

  • Content outputs comprise candidate selection, text sequences, audible speech and facial animation, with intermediate targets including spectrograms, phoneme posteriors and semantic embeddings.Classification and retrieval select explicit alternatives, whereas text may target verbatim wording or semantic paraphrase; output, validation and use are separate axes.Source: §5.2 · p. 25

5.3 Neural features and learned representations

  • Text, voice and meaning routes map neural features respectively to linguistic-unit scores, articulatory or acoustic intermediates, and semantic embeddings for retrieval or gist generation.Pretrained speech and sequence models can reduce what must be learned from scarce paired neural data, while shared latent spaces address variation across neural populations and long recordings provide repeated observations within a person. The routes trade lexical inspectability, voice characteristics and wording flexibility against different error types.Source: §5.3 · pp. 25, 27

5.4 Classification and retrieval

  • 41% top-1 accuracy retrieved a correct 3-second segment among more than 1,000 candidates in the best MEG setting; word retrieval reached top-10 of 37% among 250 candidates against 4% chance.These retrieval scores depend on candidate granularity and observation budgets; inner-speech classification was participant- and condition-specific, and an MEG-to-audio attribution audit showed variable-length noise performance collapsed to chance under fixed-duration windows and stimulus-identity splits.Source: §5.4 · pp. 27, 29, 30

5.5 Text decoding

  • Text decoders either compose phoneme or character evidence with a language model or map recordings directly to sequences; controlled repeated-sentence studies report best WER of 3%.For Mandarin, a five-participant iEEG study decoded ten sentences at 21% WER with 93% tone classification, illustrating that tone accuracy and closed-inventory sentence recovery do not establish new-sentence composition.Source: §5.5 · p. 30
  • 5.10% WER is the reported BIT result, improving the cascaded ensemble’s 5.68% WER after cross-task and cross-species encoder pretraining.The shared Brain-to-Text ’24 test set separates algorithmic gains from differences in participant recordings; the retrospective attributes gains to complementary predictions, language-model rescoring, training and phonemic objectives.Source: §5.5 · pp. 30–31
  • BIT’s end-to-end ensemble reaches 10.22%, versus 24.69% for the earlier end-to-end system in its comparison, while cascaded language decoding remains advantageous.With the neural encoder held fixed, language-decoding changes improved WER across paired GRU and Transformer settings, locating gains at the interface between neural evidence and sequence inference rather than solely in a larger encoder (Feghhi et al., 2026).Source: §5.5 · pp. 31–32
  • Table 10 groups clinical speech interfaces from compositional text to audible, expressive, and sustained communication, with study-specific Ns and recurring cohorts that cannot be summed as independent.T15 recurs across Card and Wairagkar reports, and BRAVO3 across Metzger and Littlejohn.Source: Table 10 · p. 32
  • Phoneme-level sentence reconstruction from noninvasive Perceived–Auditory MEG yielded 62.2% phoneme error and 65.8% sentence CER.The reported reconstruction used ten hours per participant; a joint text-and-acoustic model proposes coupling the outputs.Source: §5.5 · p. 32

a Brain-to-Text ’24

Brain-to-text methods vary in how they connect neural activity to linguistic outputs and in what their benchmarks test. Reported gains span acoustic reconstruction, text overlap and semantic recovery, but depend on the decoding interface and evaluation conditions.

a Brain-to-Text ’24

  • Generative MEG-to-text connects neural encoders to generators through continuous neural features, discrete neural tokens, or predicted word- or sentence-level linguistic representations.These interfaces differ in timing needs and in whether generation is constrained mainly by lexical evidence or sentence meaning.Source: a Brain-to-Text ’24 · p. 33
  • On MEG-MASC, NeuGPT reports BLEU-1 12.92 and ROUGE-1F 13.06 versus NeuSpeech’s 5.49 and 8.43; Oxford reports BLEU-1 18 and CER 71% under story holdout.Oxford uses supplied word onsets and out-of-vocabulary positions in that comparison.Source: a Brain-to-Text ’24 · p. 33
  • On LibriBrain, Oxford’s BLEU-1 rises from 19 to 22 after beam rescoring and 24 after infilling, while CER falls from 77% to 76% and 69%.Pooling LibriBrain and Armeni training data retains BLEU-1 of 18 but changes CER from 71% to 72%.Source: a Brain-to-Text ’24 · pp. 33–34
  • On Brennan held-out participants, BrainECHO reaches BLEU-1 89.78 and WER 11.72%, versus NeuSpeech’s 85.31 and 16.97%.On shuffled Gwilliams MEG recording pairs, BrainECHO reaches BLEU-1 73.35 and WER 31.44%, versus NeuSpeech’s 50.49 and 71.17%.Source: a Brain-to-Text ’24 · pp. 34–35

5.5.1 Perceived–Visual: Reading EEG-to-text

  • In a 10,710-training-sample ZuCo comparison, BLEU-4 rises from 6.80 for EEG-to-Text to 8.22 for DeWave and 8.99 for E2T-PTR; reported results also reach 14.10 for multi-view EEG2Text and 17.95 for BELT-2.Training pools, test-prefix policies and output processing differ across these comparisons.Source: §5.5.1 · p. 36
  • Comparable EEG/noise performance in tested configurations shows why text overlap alone cannot establish neural contribution; COFETT and DELTA also report BLEU-1 gains with WER above 108% or at 110.03%.The COFETT values are 13.75, 15.55 and 14.36 BLEU-1 for BrainBART, BELT and DeWave, all with WER above 108%; DELTA raises BLEU-1 from 13.69 to 21.93 while WER remains 110.03%.Source: §5.5.1 · p. 36
  • In a unique-text split, EEG-to-Text, GLIM and SemKey score 11.90, 7.84 and 8.98 on original-reference BLEU-1, but 4.17%, 10.74% and 16.24% on generated-text 24-way identification.The strongest n-gram baseline is at chance on semantic identification.Source: §5.5.1 · p. 36
  • Brain-CLIPLM’s published comparison reports BLEU-1 of 17.0, 21.3 and 22.4 for EEG-to-Text, Guiding LLMs and Brain-CLIPLM, respectively.The corresponding ROUGE-1 F scores are 17.8, 20.3 and 22.9; Brain-CLIPLM uses ordered keyword anchors, language-model reasoning and a task-specific sentence corpus.Source: §5.5.1 · p. 36

a ZuCo · teacher-forced reports

The survey compares neural-to-language methods across distinct tasks, representations and evaluation protocols, emphasizing that output quality, transfer and computational cost must be interpreted within those conditions. It also examines how pretrained structure and adaptation support different forms of communication.

d ZuCo 1+2 · shared autoregressive reevaluation

  • EEG-to-text comparisons span separate teacher-forced and autoregressive settings; stronger tests hold generation conditions fixed and compare neural inputs with matched uninformative inputs.Participant or session holdout and controls for visual, ocular and task-related signals test whether the learned representation generalizes.Source: Figure 14 · p. 37, d ZuCo 1+2 · shared autoregressive… · p. 37

5.5.2 Semantic targets and generated text

  • MindSpeech uses HD-fNIRS from word-cloud-cued internal sentence production to prompt-tune Llama-2 for text-prefix-conditioned continuation, reporting BLEU-1 and BERT-precision improvements in three of four participants, while cues and the language prior constrain plausible sentences.Prompt-only and matched shuffled-signal comparisons are needed to measure the additional content supplied by the neural recording.Source: §5.5.2 · pp. 38–39
  • CogReader’s BLEU-1 rose from 25.4 to 36.2 across 20- to 60-TR windows, while PredFT rose from 24.3 to 26.4 and UniCoRN fell from 22.9 to 18.0.fMRI decoding reconstructed story gist and transferred to internally narrated stories and silent videos in three participants, but unrestricted self-generated communication remains unestablished.Source: §5.5.2 · p. 39, Figure 15 · p. 40

5.6 Speech synthesis

  • Interactive vowel control reached 85% first-round success in one participant, while Inner-speech synthesis remained unintelligible in Angrick et al.; NeuroTalk CER was 68.3% on twelve words and 83.1% on one held-out word.The Inner-speech results use different tasks and evaluation conditions, so they do not establish open-ended intelligible communication.Source: §5.6 · p. 40

5.7 Facial and body animation

  • In two participants, simultaneous real-time decoding reached 70% phrase accuracy and 66% gesture accuracy across ten phrases and ten gestures plus rest, triggering preset full-body avatar animations.Training on isolated and simultaneous attempts improved performance across contexts.Source: §5.7 · p. 41

5.6 Speech synthesis

  • 20.6% ASR WER for the LLM-rescored text cascade versus 105.4% for direct speech units and 47.2% for Brain2Speech-Net; direct units rendered fastest at RTF 0.060.These are shared-encoder implementations evaluated against synthetic reference speech; ASR WER and rendering speed do not establish listener intelligibility or interactive onset latency.Source: §5.6 · p. 41, Figure 16 · p. 42

5.8 Pretraining, transfer and personal calibration

  • On the trial split, CodeBrain reached 61.01% balanced accuracy, while recent encoder results ranged from 49.20% for BIOT to 56.35% for REVE; architecture, source data and tuning jointly shape adaptation.The comparison includes earlier scores quoted from prior reports under the trial split.Source: §5.8 · pp. 42–43
  • On subject-disjoint BCI-Speech, frozen-probe and fine-tuning rankings differed, and fine-tuning did not consistently improve performance across models.The comparison reports 20% chance; its two panels use separate datasets and adaptation procedures.Source: §5.8 · p. 43

a BCIC2020-3 · trial split

Transfer outcomes vary with the source and target task, and new-user, session and task mismatch require distinct adaptation strategies. Reported large-vocabulary Inner decoding concerns instructed production, not unrestricted message choice.

b BCIC2020-3 · trial split

  • Articulated EEG did not improve speaker-independent imagined-vowel classification in one study, while other work investigates combining overt, whispered and imagined modes.Transfer is useful to the extent that the target retains the learned representation without requiring an unavailable reference.Source: a BCIC2020-3 · trial split · p. 44
  • Attempted-speech pretraining plus up to an hour of Inner data supported cued online decoding of 50 words in three participants and 125,000 words in two, with large-vocabulary WERs of 26% and 54%.The full study enrolled four participants; these results concern instructed linguistic production, while spontaneous content and unrestricted message choice remain additional tests.Source: a BCIC2020-3 · trial split · p. 44
  • Shared latent dynamics, position-aware models, user-specific projections and session alignment address different recording mismatches, so transfer comparisons need explicit target-data budgets and output tasks.The surveyed strategies reuse different structures, including neural dynamics, phoneme information and session-level alignment.Source: a BCIC2020-3 · trial split · pp. 44–45

6 Evaluation

Evaluation now spans output quality, generalisation, communication costs and credibility, but these dimensions do not form a universal score. Results are interpretable when the tested task, available information, controls and intended use are explicit.

6.1 Quality

  • Candidate-set accuracy, retrieval rank and sequence error answer different questions; WER measures substitutions, deletions and insertions against reference words.Candidate vocabulary and class frequency affect classification results, while language-assisted transcripts should be distinguished from neural-unit decoding.Source: §6.1 · pp. 45–46
  • Overlap, semantic similarity, listener intelligibility and voice quality assess different outcomes, so the chosen measure should match the intended message or expression.Real/noise overlap scores remained similar in tested EEG-to-text configurations, while autonomous new-sentence generation is distinct from repeated-content recognition and embedding alignment.Source: §6.1 · pp. 46–47
  • A declared evaluation contract—held-out data, permitted inference information and scored output—makes shared-benchmark comparisons interpretable without ranking different tasks together.Benchmarks also need fixed targets, scoring implementation and training-data budgets; their protocols do not make distinct prediction settings equivalent.Source: §6 · p. 45, §6.1 · p. 47

6.2 Robustness

  • Holding out content, sessions or participants tests different forms of generalisation; participant separation alone can retain familiar sentences, and transfer comparisons depend on target-data budgets.These are complementary prediction settings, not successive levels on a single robustness scale.Source: §6.2 · pp. 47–48, Figure 20 · p. 47

6.3 Efficiency

  • Communication-rate measures and vocabulary coverage add context to accuracy: rapid output can still require correction, while open-vocabulary mutual information accounts for supported language coverage.Published rate and error pairs span different participants and protocols; they should not be read as a common ranking.Source: §6.3 · pp. 48–49
  • Calibration, feedback and finalisation contribute to communication time beyond raw decoding speed, and comparable end-to-end energy, memory and maintenance measures remain sparse.The reported incremental-feedback, finalisation and inference timings use distinct definitions and settings.Source: §6.3 · pp. 48–49

6.4.1 Information available during testing

  • Test-time inputs and controls determine what a decoder’s score supports; teacher forcing is relevant to autoregressive text, not a universal criterion for classifiers or synthesis.Behavioral and peripheral-channel comparisons, prior-only tests and signal replacements address different alternative explanations, and no single control resolves them all.Source: §6.4.1 · p. 49, §6.4.2 · pp. 50–51

6.4.3 Statistical support and replication

  • Participant-level variation and replication matter alongside peak accuracy: the Inner EEG audit averaged 36.1% statistically significant participants across tested datasets, while population-level reliability remains less characterised.That 36.1% is an average of dataset-level fractions, not a pooled participant proportion; repeated benchmark use tests algorithmic reproducibility rather than recruitment-level extension.Source: §6.4.3 · p. 51

7 Practical Use

A practical language interface combines decoding with feedback, correction, adaptation and sustained access. The interaction design determines how users initiate, review and repair messages, while support and deployment arrangements shape continued use.

7 Practical Use

  • A usable interface connects decoded content to user control, support and recording access so that a person can complete a communication goal.The communication loop may combine confirmation before release or provisional streaming, with different stopping and fallback controls.Source: §7 · p. 52, Figure 22 · p. 52

7.1.1 Speech onset, stopping and unintended activation

  • Automatic speech detection can support turn-taking, while deliberate gating separates communication from other verbal activity; false detections during sleep show why stopping controls matter.Card’s conversation mode finalises after six seconds without detected speech or an eye-tracker action; a later home-use report noted occasional nonsensical output during nominal sleep.Source: §7.1.1 · p. 53

7.1.2 Feedback, confirmation and error correction

  • Streaming feedback supports adjustment during production, while confirmation and correction can both repair the current message and provide labels for later adaptation.A constrained five-day syllable-imagery study found feedback-supported user learning with substantial individual differences, not conversational proficiency.Source: §7.1.2 · p. 53

7.1.3 Communication benefit and user experience

  • Communication benefit includes language, voice and expression choices, and should be assessed alongside workload, satisfaction and the user’s contribution to successful messages.Cued-copy tests more directly isolate word recovery than conversational ratings that include user-corrected output; patient-reported participation measures require validation for a particular aid.Source: §7.1.3 · p. 53, Table 12 · p. 54

7.2.1 Inference-time assistance

  • Language models and contextual cues constrain uncertain neural evidence, but names, negation and unusual expressions may require editing or confirmation when context conflicts with intent.The practical target is reduced correction effort while preserving intended distinctions.Source: §7.2.1 · p. 54

7.2.2 Long-term stability and recalibration

  • Calibration and continued adaptation use different sources of supervision, from prompted sentences to user-confirmed output, and impose costs in time, labels and assistance.The home system used background recalibration; a cued Inner-speech system combined attempted-speech pretraining with participant-specific online updates.Source: §7.2.2 · pp. 54–55

7.3.1 Setup, portability and maintenance

  • One participant used a wired, multi-computer system for more than 3,800 hours, with about 20 minutes of daily caregiver setup and self-directed use after setup.Setup assistance and independent message production describe complementary parts of the reported home-use arrangement.Source: §7.3.1 · p. 55

7.3.2 Privacy and user control

  • Activation, message release and data retention are separate privacy decisions; disabling logging changes the retained record rather than the decoder’s output policy.The cited examples include keyword gating, cooperation during reconstruction and a home-system privacy mode.Source: §7.3.2 · p. 55

8 Limitations

The limitations arise where richer representations, reusable training, personal fitting and interactive interfaces meet: task, signal, transfer and operating demands remain consequential.

8.1 Task coverage and self-generated expression

  • Self-generated expression remains an open test: representations learned from external language may not transfer, and user-chosen messages require judging intended meaning beyond prompt labels.Motor attempt, internal articulation and auditory imagery can share content structure while differing in intention and execution; confirmation and independent accounts of intended meaning become important.Source: §8.1 · pp. 56–57

8.2 Signal access and supervision

  • Reuse is constrained by mismatches between available supervision and intended use; source data help when they preserve distinctions needed by the target task.Target-data learning curves better characterize reuse than source hours alone, while new recording configurations also require content-specific information and practicable training labels.Source: §8.2 · p. 57

8.3 Transfer across content, people and time

  • Transfer across content, people and sessions remains unresolved in its reliability and target-supervision requirements.Familiar stimuli and output priors explain part of apparent transfer in tested settings; variation between participants, including unsuccessful users, affects how broadly a capability applies.Source: §8.3 · p. 57

8.4 Sustained communication and operating burden

  • Streaming, expressive outputs and prolonged home use move interfaces closer to everyday exchange but can add user or support-network work.Richer output may need additional control, continued learning depends on labels and recovery, and evaluation should include correction, unavailable time, setup and maintenance.Source: §8.4 · pp. 57–58

9 Beyond

The proposed five-level trajectory moves from selecting actions and formulating words to conveying meaning, sharing scenarios and receiving meaningful neural input. These are prospective functional targets, not grades of current systems.

9 Beyond

  • The five levels add progressively deeper exchange: action selection, formulated language, wording-independent meaning, an unfolding scenario, and neural receipt of meaningful content.The progression is not a ranking of existing systems; L3 remains a frontier and L4–L5 are prospective.Source: §9 · pp. 58, 60, Figure 23 · p. 58
  • L1 requires intentional selection among discrete commands, including the ability to withhold or cancel an action.Task-correlated neural activity alone is insufficient evidence of intentional command selection.Source: §9 · p. 58
  • L2 adds composition beyond a fixed command inventory, requiring new word combinations rather than learned command associations.Large-vocabulary speech neuroprostheses provide relevant foundations.Source: §9 · pp. 58–59
  • L3 requires preserving intended relations across different wording, including distinctions such as allowing a boat through versus blocking it.Paraphrase accuracy alone is insufficient; evidence should separate intended meaning from covertly formulated words and test transfer to new formulations.Source: §9 · p. 59
  • L4 externalises actors, goals, relations and changes in an unfolding situation, rather than only an abstract intention.Evidence should test new combinations of actors, relations and temporal changes beyond rehearsed narratives; voluntary scenario communication remains prospective.Source: §9 · p. 59
  • L5 adds reciprocal neural exchange: meaningful content informs understanding without audiovisual presentation, and the user can assess, reject and revise it.Learned stimulation codes are narrower precursors; a convincing transition would require untrained content and new judgements beyond memorised stimulation–response codes.Source: §9 · p. 59

10 Conclusion

The conclusion links recoverable language to task, sampled populations and measurement timescale, and treats accuracy, generalisation, calibration and interaction as complementary achievements. It calls for transferable, user-controlled communication while distinguishing fluent output from unrestricted access to thought.

10 Conclusion

  • Recoverable information depends jointly on behaviour, sampled neural populations and measurement timescale, so modality and decoding target should match the available information.Speaking and attempted speech reveal planning and articulatory activity; listening and reading provide sensory, lexical and contextual information, while inner speech retains task-dependent differences.Source: §10 · p. 60, B Supplementary Signal Mechanisms · p. 97
  • Phonetic and lexical targets support word sequences, acoustic and articulatory trajectories support voice, and contextual representations support meaning across wording changes.Pretraining and multimodal supervision connect these routes, while shared benchmarks make selected learning and inference choices comparable; performance matters under the task and use conditions.Source: §10 · p. 60

Impact Statement

  • More transferable self-chosen expression requires preserving intended distinctions, making errors recognisable and repairable, and showing user control at each richer exchange transition.Speech neuroprostheses can restore communication functions, but fluent output should not be read as unrestricted thought access; activation, message release and data retention should remain under explicit user authority.Source: §10 · p. 60, Impact Statement · p. 60

B Supplementary Signal Mechanisms

  • Acquisition comparisons must distinguish physiological origin, decoder-window constraints and recording access; nominal sampling speed alone does not establish equivalent information content.Electrical and magnetic measurements differ from vascular responses, and tissue access determines what can be sampled.Source: B Supplementary Signal Mechanisms · p. 97

C Search Strategy Summary

The search summary documents source-led updates, screening and structured extraction, while distinguishing registry records from independent cohorts and characterising recent reviews by their scope and checking depth.

C Search Strategy Summary

  • Search updates included recent-review discovery through 18 September and thematic follow-up searches on 19–21 September, with eligible reports added to the updated registry.Version consolidation used DOI, normalised title and explicit preprint–publication links while retaining source provenance.Source: C Search Strategy Summary · p. 99
  • The registry retained 938 IDs representing 920 canonical report/resource entries, with selected structured extraction for 246 entries.Extraction depth varies by claim, and entries are not independent participant cohorts.Source: C Search Strategy Summary · p. 99

D Recent Reviews and Their Scope

  • The recent-review supplement covers 2024–2026 publications and lists 30 directly relevant reviews plus 16 contextual reviews or perspectives.Scope checks range from abstract-level or targeted section checks to title/metadata only; these reports have different scopes, not independent replications.Source: D Recent Reviews and Their Scope · p. 99, Table 17 · p. 99

Contextual reviews and perspectives

  • Contextual reviews and perspectives are distinguished from directly relevant reviews in the review inventory, rather than treated as equivalent evidence.The inventory explicitly identifies preprints, short reviews, proceedings and online/issue-year differences.Source: D Recent Reviews and Their Scope · p. 99, Table 17 · p. 99

E.1 Coverage beyond the selected performance examples

The expanded registry adds directions beyond the selected performance examples, while distinguishing checked claims from pending leads and keeping adjacent evidence outside the speech-decoder comparison.

Articulated

  • The Articulated index spans phoneme classification, continuous decoding, synthesis, sparse electrodes, syntactic roles, speech envelopes and neural–audio matching; architectural gains alone do not establish transfer.Classification, waveform reconstruction and retrieval require separate comparisons, and transfer to a new speaker or unseen sentence is a distinct question.Source: §E.1 · pp. 101–102

Clinical attempted-speech boundaries

  • Clinical attempted-speech entries include bilingual and stable-control work, but task-level inclusion does not make pending records verified clinical milestones or directly comparable word-error results.Limited membership requires documented residual articulation; the broader clinical comparison retains source-labelled attempts outside that boundary.Source: §E.1 · p. 102

Cross-condition transfer

  • Cross-mode entries include overt-trained decoding and event-timing transfer, but event timing is not word identity and “silent” labels do not establish motionless neural decoding.Peripheral inputs, instructions and data splits require explicit inspection.Source: §E.1 · p. 102

Inner: datasets and multimodal measurement

  • Inner-speech additions broaden language and recording coverage through multimodal datasets, hybrid decoding and Arabic data, without themselves validating open-vocabulary decoding.Cross-trial features require checking for test-information sharing, and possible cohort overlap remains a verification task.Source: §E.1 · p. 102

Inner: methods, control and imagery design

  • Inner-speech methods span offline recognition and online control, while rhythm or state classification should not be pooled with content recognition.EEG-Translator reports a 6.2% improvement in class decoding accuracy over original imagined signals (Lee et al., 2025b); a Mandarin vowel classifier reports 49.47% four-class accuracy and 69.83% mean binary accuracy across ten participants (Wang et al., 2024d).Source: §E.1 · p. 102

Perceived: Auditory, Visual and Audiovisual conditions

  • Perceived–Auditory entries include mixed listening-and-response tasks, production-BCI feedback and distinct non-human-primate or audiovisual evidence strata.An Inner-to-Perceived–Auditory mapping proposal requires checking both its training paradigm and tested target; lip-reading preprint and formal versions count as one report.Source: §E.1 · p. 103

Functional ultrasound

  • Functional-ultrasound records inform measurement feasibility and design, but are not additional human speech-content decoders.Neural-versus-nonneural EEG contribution analysis is likewise methodological evidence rather than a communication milestone.Source: §E.1 · p. 103

Coverage and source checks

  • The synthesis distinguishes selected claim-level checks from study-wide verification: checked reading comparisons are not complete audits, and retained leads do not add verified systems to the main comparison.The performance examples are selective; the registry retains broader entries and pending work without treating their presence as positive decoding evidence.Source: §E.1 · pp. 101, 103

E.2 Additional published benchmark conditions

The additional benchmark conditions span word retrieval, sentence generation and text reconstruction, with different candidate sets, windows and input controls defining what each comparison tests.

Interpretation

  • Reported SD or SEM conventions vary by caption, so benchmark results should be read with their stated condition and uncertainty measure.The overview explicitly identifies parentheses as the source-reported SD or SEM specified in each caption.Source: §E.2 · p. 103

Word retrieval

  • Table 18 reports LibriBrain word-retrieval top-10 balanced accuracy for a 50-word candidate set, with SEM across three seeds shown in parentheses.The caption identifies the source table but supplies no model scores.Source: Table 18 · p. 103

Sentence generation

  • Table 19 compares LibriBrain sentence generation with matched noise inputs on 0–100 metrics, with source-reported SDs; all methods receive sentence boundaries.The d’Ascoli implementation also receives word timing, and baseline repeat counts are not enumerated, so differences are descriptive.Source: Table 19 · p. 104

Narratives reconstruction

  • Table 20 organizes Narratives text reconstruction by 20-, 40- and 60-TR windows, using the source 0–100 scale where higher is better.The caption supplies the evaluation dimensions, not the table’s model values.Source: Table 20 · p. 104

Prompted reconstruction

  • Table 21 keeps Huth reconstruction with a supplied text prompt separate from the no-prompt main comparison.The caption identifies the source table but gives no scores.Source: Table 21 · p. 104

F Selected Conditions and Source Checks

The selected checks pair reported results with their tested conditions and inference boundaries, separating benchmark performance, retrieval, and sustained-use evidence from broader claims of generalization or clinical access.

Imagined-speech benchmarks

  • The selected BCIC2020-3 and BCI-Speech tables specify different evaluation setups: official trial splits with paper-specific tuning versus subject-disjoint balanced accuracy.The BCI-Speech caption distinguishes frozen probing from full fine-tuning; these captions do not provide score values.Source: Table 22 · p. 105, Table 23 · p. 105

Longitudinal use

  • Over 3,800 h of T15 home BCI use includes idle time, while 99.2% word accuracy belongs to prompted copy rather than unrestricted conversation.The 92% conversational rating includes user-corrected sentences, and caregiver setup remains part of the use context.Source: F Selected Conditions and Source Checks · p. 107

MEG text generation

  • Table 4 reports Gwilliams BLEU-1 60.30 without test-time teacher forcing, while the audit does not directly evaluate every successor or MEG decoder.The inspected version and section locators specify the checked evidence.Source: F Selected Conditions and Source Checks · p. 107
  • Without test-time teacher forcing, BLEU-1 is 6.86 versus noise 3.87 and random sentence 5.86 in the reported MEG condition.The recording-pair holdout does not establish unseen-text generation.Source: F Selected Conditions and Source Checks · p. 107
  • BLEU-1 12.92 and CER 99.8 are reported for another MEG condition, but the test-time prefix policy is unstated and the random-sentence baseline is not a neural-input noise control.The WER 10.35 entry is audio ASR, not the neural decoding result.Source: F Selected Conditions and Source Checks · p. 107

Reading audit

  • Tested BART, Pegasus and T5 implementations showed similar EEG/noise performance in the reading audit.The audit uses unique sentences and an 80:10:10 split; retained participant N is not separately reported for Table 1.Source: F Selected Conditions and Source Checks · p. 107

Reading results

  • GLIM’s main-survey BLEU-1 7.84 and identification 10.74% are SemKey shared-benchmark results, not GLIM’s original Table 1.The original report describes text-disjoint partitions and generation without teacher forcing.Source: F Selected Conditions and Source Checks · p. 107
  • Brain-CLIPLM retrieval uses the task’s full sentence pool, making it corpus-assisted reconstruction rather than unrestricted unseen-text generation.Its Table 3 quotes earlier literature baselines rather than rerunning every method.Source: F Selected Conditions and Source Checks · p. 107

G Dataset and Resource Inventory

This inventory maps brain-language resources across production, imagery, attempted communication, reading, listening, perception and naturalistic interaction, spanning EEG, MEG, fMRI, intracranial, fNIRS and other non-invasive recordings. It describes variation in tasks, cohorts, access, data formats, recording durations and measurement conventions that qualifies how resource coverage and reported scale can be interpreted.

G Dataset and Resource Inventory

  • Resources cover controlled production and imagery, attempted communication, listening, reading and naturalistic interaction.The inventory includes 110 resource/version entries and distinguishes recruited, released and analysed cohorts where they differ.Source: G Dataset and Resource Inventory · p. 108
  • Resource scale is reported in distinct units—continuous hours, task-window hours, trials, runs or linguistic items—and shared recordings or participants may recur across releases.Continuous hours include within-run pauses, whereas task-window hours cover only stated epochs or stimulus intervals; derived files and simultaneous modalities are not independent additions.Source: G Dataset and Resource Inventory · p. 108, Table 27 · p. 108
  • The inventory includes multilingual word and letter resources, including overt and imagined Malayalam words and vocal and subvocal Telugu words.The listed Malayalam resource includes 26 words and overt English equivalents; the Telugu resource includes 100 words and English equivalents.Source: G Dataset and Resource Inventory · pp. 109–110
  • UBA natural dialogue EEG spans 168 h of continuous Dutch speech, while Chinese spoken-word production (Tan & Zhang, 2025) reports 51.26 h across 188 raw blocks.UBA is listed as public plus restricted, and the Chinese production resource includes Mandarin sentences with contrasting prosody.Source: G Dataset and Resource Inventory · p. 110

EEG resources

  • EEG resources span cued and cueless imagery, reading, listening, and covert or overt production, with access ranging from public data to withheld audio, author inquiry, or subscription.For example, ChineseEEG-2 is listed alongside resources whose imagined-word audio is withheld or whose raw data are available only by inquiry.Source: G Dataset and Resource Inventory · p. 110
  • The cited IIST BCI collections include Malayalam letters and dialect commands, Telugu vowels/consonants, and Telugu dialect commands; IEEE DataPort subscription access is noted for listed entries.Source: G Dataset and Resource Inventory · p. 111
  • The passages describe public OpenNeuro 2019/2021 cohorts sharing single-phoneme listening/repetition trials, native-Arabic Silent Speech/SSDC, a public Arabic–Hindi L2 word-reading release, and public COFETT/Chisco-2.0 files.SS-EEG shares inner speech only, not overt speech, and has a separate public tutorial sample; 3M-CPSEED is author-EULA with a teaser. COFETT/Chisco-2.0 reports 300 files, with 12 s of repeated coverage removed.Source: G Dataset and Resource Inventory · p. 111

MEG resources

  • The cited MEG resources include MEG-MASC audiobook listening (≈30 h; Donders data-use agreement), LibriBrain (≈49 h; 49 sessions; public, CC0), and LPP listening (175 runs; 22.30 h; public release and ANC data-use agreement).LPP MEG reading and LibriBrain100 have public raw data and annotations; LibriBrain100 is a separate reading collection, with no assumed cohort overlap with the listening release.Source: G Dataset and Resource Inventory · p. 112

Naturalistic perception resources

  • These resources include Riegel audiovisual matrix sentences, Wehbe Harry Potter data (160 runs, 39.99 h; public 7 T partial-brain release), Huth semantic maps, and StudyForrest audiovisual 3 T data with 84 story runs.The StudyForrest tutorial subset has 12 story runs per person and does not represent the entire raw cohort.Source: G Dataset and Resource Inventory · p. 112

G Dataset and Resource Inventory

  • Narrative Brain Dataset, Pereira linguistic-meaning data, and the Naturalistic Neuroimaging Database represent distinct fMRI resources.The Narrative Brain Dataset includes 3 T recordings from 15 participants in the original 20-person audio cohort, with shared recruitment but different acquisitions; Pereira offers processed fMRI and annotations rather than raw acquisition, while sentence experiments in the Naturalistic Neuroimaging Database have overlapping participant subsets.Source: G Dataset and Resource Inventory · p. 112
  • Narratives provides 264 raw BOLD runs totaling 161.20 h, with one film per participant; this is acquisition support, not 160.59 h of participant-weighted movie exposure.The distinction separates raw acquisition time from participant-weighted movie exposure.Source: G Dataset and Resource Inventory · p. 112
  • LPPC multilingual fMRI (Li et al., 2025a; 2022b) releases derived sentence-level data, including five training participants, five new evaluation participants, and four additional blocked-design participants.Its public data are processed sentence-level responses; the Brain-Score benchmark derivative differs from the full original cohort and conditions.Source: G Dataset and Resource Inventory · p. 113
  • The cited resources include T12 overt production of 100 Dutch words across 1,000 trials, public neural features and evaluation records, plus 1,718 copy-evaluation and 22,126 conversation records.Source: G Dataset and Resource Inventory · p. 113
  • Other production resources include overt, mimed, and imagined speech and natural conversational speech (MOUS; Schoffelen et al., 2019).MOUS releases word timing and model embeddings but withholds private utterance transcripts; its 17.51 h of continuous firing-rate time includes conversation pauses and listening.Source: G Dataset and Resource Inventory · p. 113
  • Access ranges from public raw or derived data to registration-, agreement-, or request-based releases, while exclusions and task-window definitions affect reported durations.Examples include incomplete MEG cases, fNIRS session exclusions, and imagery windows that exclude untimed sentence typing; some raw data are withheld for legal or ethical reasons.Source: G Dataset and Resource Inventory · pp. 113–114
  • Non-invasive resources span narrative listening, cued inner speech, audiobooks, and vocoded or noisy speech, with MEG, fMRI, EEG, and fNIRS recordings.For example, SMN4Lang (Wang et al., 2022) pairs 60 Mandarin stories with separate-session cued inner speech; modality durations differ, and one participant was excluded for poor EEG quality.Source: G Dataset and Resource Inventory · pp. 113–114
  • 72 fNIRS recordings (46 cochlear implant, 26 normal hearing) cover speech in quiet/noise, audiovisual speech, lipreading, and fat-talk/control conversations, with 2,646 sentence trials and 5.15 h of imagery windows.The inventory also reports 10.40 h of stimulus windows.Source: G Dataset and Resource Inventory · p. 114
  • Natural-conversation fNIRS resources include Hu et al. (2025) and 31 recordings totaling 13.87 h of continuous fNIRS.Source: G Dataset and Resource Inventory · p. 114
  • A public raw SNIRF archive on Dryad provides 40 stimulus windows per participant; appointment time is excluded.Source: G Dataset and Resource Inventory · p. 114
  • An OSF resource provides public raw and processed signals for 7.75 h of conversation windows, with only one partner wearing fNIRS.Source: G Dataset and Resource Inventory · p. 114

Related papers

Loading 2609.27650v2…