Source-linked AI summary
Accurate Decoding of Natural Sentences from Non-Invasive Brain Recordings
Mingfang Zhang, Jarod Lévy, Cedric Rommel, Jérémy Rapin, Corentin Bel, Julie Bonnaire, Daniel Nieto, Pierre Bourdillon, Svetlana Pinet, Stéphane d'Ascoli, Thomas Moreau, Jean-Rémi King
TL;DR
Non-invasive brain-to-text decoding remains less capable than invasive approaches and faces challenges for practical communication. Brain2Qwerty v2 decodes natural sentences from continuous MEG recordings, achieving 39% average WER and 22% for the best subject.
Problem
Non-invasive BCIs remain limited by signal quality, temporal resolution, and MEG performance lagging behind invasive recordings for communication.
Method
Brain2Qwerty v2 uses deep learning to asynchronously decode continuous MEG recordings into character-, word-, and sentence-level representations.
Results
39% average WER and 22% WER for the best subject, a twofold improvement over Brain2Qwerty v1’s 52% best-subject WER.
Takeaways & Limitations
The results significantly narrow the performance gap between non-invasive and invasive brain-to-text decoders.
Takeaways & Limitations
Autoregressive LLM decoding can produce fluent but entirely different sentences when MEG signal quality is insufficient, increasing CER.
Abstract
from arXiv · showhide
Restoring communication for people who have lost the ability to speak or move after a brain injury is a major challenge. While intracranial implants now enable high-performing brain-computer-interfaces, non-invasive alternatives are still lagging behind. Here, we present Brain2Qwerty v2, a model that can decode the production of natural sentences solely from real-time magnetoencephalography (MEG) recordings. By collecting 22,000 sentences typed by nine subjects, each recorded for 10 hours, our model leverages character, word and sentence-level representations to achieve an average word error rate (WER) of 39%. For our best participant, the model accurately decodes half of the sentences with one word error or less. Critically, decoding accuracy log-linearly improves with data volume, suggesting that the performance gap with intracranial approaches could be partially bridged through data scaling. We show that AI enables this performance in three main ways: the substitution of hand-crafted pipelines for event detection with deep learning, the finetuning of large language models to extract semantic representations, and the deployment of AI agents to iteratively refine our decoding pipeline via automated code development. Together, these results show that non-invasive brain-to-text decoding starts to operate at a level of accuracy previously thought exclusive to surgical implants, opening a path toward safe and efficient brain-computer-interfaces.
1 Introduction
Invasive BCIs have restored communication after severe speech or movement loss, while non-invasive systems remain constrained by signal quality, temporal resolution, and lower performance. Brain2Qwerty v2 addresses these challenges by decoding natural sentences from large-scale, non-invasive MEG recordings.
- Motivation: Invasive BCIs have enabled language production for patients with anarthria, locked-in syndrome, or amyotrophic lateral sclerosis using motor-cortex electrodes.Reported communication modes include attempted speech and handwriting.
- Motivation: Non-invasive BCIs face important challenges, including poor EEG signal-to-noise ratio and limited fMRI temporal resolution that restrict communication use.These constraints make EEG-based systems impractical for patients and limit fMRI communication applications.
- Motivation: 32% character error rate (CER) was achieved by prior MEG decoding of typed text using a classifier time-locked to each keystroke.MEG remains behind invasive recordings, despite its promise as a non-invasive alternative.
- Contribution: 22,000-sentence corpus was decoded with Brain2Qwerty v2 from non-invasive MEG recordings collected across nine healthy volunteers and 90 total recording sessions.Each trial involved listening to a sentence, a forced delay, and typing the corresponding text; the study focuses on neural activity during language production.
- Contribution: 10 times more data per subject than Lévy et al. (2025) and greater sentence diversity characterize the new dataset used by Brain2Qwerty v2.The dataset supports an approach that applies AI to address real-time decoding challenges.
2 Results
Brain2Qwerty v2 achieves competitive non-invasive sentence decoding through scalable asynchronous modeling, language-model integration, and automated pipeline refinement. Performance improves with data and sentence diversity, while semantic accuracy and fluent sentence recovery come with higher character-level error.
- Neural activity: Bilateral M1 and SMA activation appears at keystroke onset, with more spatially extended activation in the right hemisphere.Source reconstruction also detects residual activity surrounding keystrokes in the left dorso-lateral region.
- Asynchronous decoding: Asynchronous decoding uses CTC over continuous MEG windows, removing the need for externally supplied keystroke timings.The approach replaces synchronous event timing constraints with sequence prediction from continuous responses.
- Scaling and diversity: CER decreases log-linearly with recording hours, showing no saturation at approximately 90 pooled training hours.EnglishBCBL also improves synchronous decoding over SpanishBCBL as dataset size increases, with CER changing from 0.39 ± 0.02 to a lower value.
- Limitations: 0.31±0.03 CER is higher for Brain2Qwerty v2 than the Encoder alone at 0.28±0.03 and N-gram baseline at 0.26 ± 0.03.Autoregressive language modeling favors fluent sentences even when MEG token quality is insufficient, causing character-level divergence.
- Decoding quality: 28% of test sentences are perfect for the best subject, while 47% contain at most one word edit; the median subject reaches 15% perfect decodings.Typical errors are single substitutions or omissions, with meaning largely preserved and outputs remaining fluent.
3 Discussion
Brain2Qwerty v2 decodes sentences from continuous non-invasive MEG with 39% average WER and 22% WER for the best subject, while jointly using character-, word-, and sentence-level representations. The discussion highlights semantic decoding and autonomous AI optimization, alongside persistent variability, latency, scaling, and clinical hardware limitations.
- Summary: 39% average WER and 22% WER for the best subject demonstrate accurate sentence decoding from continuous non-invasive brain recordings.The model jointly trains three deep learning modules targeting character-, word-, and sentence-level representations.
- Semantic decoding: Removing ‘MEG tokens’ from the LLM input degrades WER by 16%, supporting the use of semantic representations in decoding.Brain2Qwerty v2 is described as the first architecture to jointly train character-, word-, and sentence-level decoding.
- Auto Research: Autonomous AI agents iteratively modified the code base to discover novel architectures, producing a substantial WER improvement over traditional parameter-limited Bayesian optimization.The agents were integrated into the Brain2Qwerty v2 development cycle to optimize neural decoding pipelines.
- Inter-individual variability: 17.1%–41.0% N-gram CER across subjects reveals substantial inter-individual variability, making encoder improvement through transfer or self-supervised pretraining a priority.Final model performance is highly correlated with upstream encoder quality.
- Scaling laws: 90 hours of aggregated data show no scaling-curve plateau, suggesting further data collection could improve performance despite a remaining gap with invasive BCIs.State-of-the-art invasive BCIs achieve below 2% WER for typing and below 6% CER for handwriting.
- A path towards non-invasive BCI: 25% and 50% of the MEG sensors preserve remarkably robust decoding, indicating that optimized sensor selection could reduce clinical hardware constraints.The current approach uses a 306-sensor cryogenic MEG system and remains challenging to adapt clinically.
4 Methods
The study used a controlled MEG typing protocol with nine proficient adult typists, standardized sentence stimuli, and leakage-free text splits. Signals were preprocessed, augmented, encoded hierarchically, and paired with lightweight LoRA-adapted language-model training and autonomous hyperparameter discovery.
- Participants: 9 healthy adult volunteers completed the study; all were right-handed, native English speakers, proficient typists, and met an 80% typing-accuracy threshold.Participants had a mean age of 34.6 years, with a range of 23–56.
- Task and stimuli: 2,560 unique English sentences were shared across participants, with each participant completing 10 sessions of 256 sentences.Stimuli were sampled from 20,000 Llama 4-generated sentences after filtering special characters and contractions.
- MEG preprocessing: MEG recordings were bandpass filtered at 0.5–45 Hz, notch filtered at 50 Hz, downsampled to 100 Hz, robust-scaled per channel, and clamped beyond ±5 robust standard deviations.No signal-space projection was applied.
- Data splits: 80/10/10 training, validation, and test splits were assigned by sentence text, ensuring that identical texts across subjects and sessions never crossed partitions.A deterministic hash-based splitter made assignments stable regardless of the included subjects.
- Data augmentation: Training augmentation combined baseline cropping, per-channel offsets, temporal and channel masking, and temporal stretching of sentence-aligned MEG segments.Segments extended from 400 ms before sentence onset through sentence offset plus a uniformly sampled 400–500 ms buffer; temporal masks lasted up to 50 frames with probability 0.2.
- Model and optimization: The model used a hierarchical encoder combining spatial channel merging, a convolutional feature extractor, and a Conformer sequence model, while LoRA-adapted language models were optimized for 30 epochs on one GPU.The LoRA regime supported rapid iteration over configurations, language-model sizes, and model-soup strategies.
Appendix · MEG source reconstruction over time
Figure S1 presents MEG source reconstruction over time relative to keystroke onset.
- MEG source reconstruction over time: Figure S1 shows MEG sources over time relative to keystroke onset.
Clustering analysis of embeddings from BrainModule and Conformer
tSNE analysis shows that training reorganizes key embeddings from participant-specific signatures in the BrainModule into key-class clusters in the Conformer, reflecting keyboard layout.
- BrainModule embeddings: Key representations after the BrainModule cluster by participant, retaining participant-specific signatures.Representations are extracted from time steps where CTC emits a key class, excluding blanks and spaces, with continuous emissions averaged.
- Conformer embeddings: Conformer representations clearly cluster by key class, reflecting the physical layout of the keyboard.tSNE is applied to key representations from test-set sentences.
- Training effect: The untrained Conformer clusters key representations by subject, indicating that key-class structure is learned during training.The comparison uses Conformer embeddings from both trained and untrained MEG Encoders.
Multi-subject training and cross-subject adaptation
The study evaluates how decoder performance depends on training-data composition by comparing subject-specific, joint multi-subject, and leave-one-out pretraining with target-subject finetuning.
- Cross-subject adaptation: Leave-one-out plus finetuning pretrains on the eight other subjects, then finetunes on the target subject with the Conformer frozen.
- Training-data composition: Three regimes compare training from scratch on the target subject, training jointly from scratch on all subjects, and pretraining on other subjects before finetuning.Evaluations cover the best, median, and worst subjects.
- Training-data composition: Per-subject training uses only the target subject’s data and runs for 4× the joint-training schedule to match optimisation steps.
Sensors ablation
Sensor ablation showed that Brain2Qwerty v2 decoding degrades smoothly and monotonically as MEG channels are removed. This suggests emerging 50–150-sensor OPM arrays may support sentence decoding, although reduced sensor counts incur increasing WER costs.
- Motivation: 306-channel MEG arrays are bulky, cryogenically cooled, and expensive, motivating room-temperature OPM alternatives in flexible helmets.OPM arrays typically contain 50–150 sensors.
- Results: +3.4 pp WER is the cost of reducing the array from 306 to 230 sensors, compared with the full-array baseline.The full-sensor configuration reached a WER of 0.433.
- Results: +2.3 pp WER and +5.7 pp WER were incurred by further reductions from 230 to 153 and from 153 to 76 sensors, respectively.WER standard errors were always ≤1 pp, and the trend was not driven by which channels survived subsampling.
Decoder behaviour across difficulty bands
Brain2Qwerty v2 performs best on easier sentences, where the strongest subject is often verbatim and errors are usually meaning-preserving substitutions. Performance degrades across difficulty bands, although common sentence templates can still be decoded perfectly by weaker subjects.
- Difficulty bands: 18 sentences are divided into best, median, and worst difficulty terciles using mean WER across all 9 subjects, with 6 randomly sampled per tercile.Figure S3 compares decoded examples for the best, median, and worst subjects across these bands.
- Best-decoded targets: 5 of 6 best-decoded targets are verbatim for the best subject, while the median subject differs by at most one or two words.Observed errors are almost always meaning-preserving single-word substitutions or grammatical insertions.
Contribution of the Neuro Tokens
Brain2Qwerty v2 uses MEG-derived Neuro Tokens to provide neural information to its LLM-based decoder, rather than merely correcting the encoder’s text output. Ablating these tokens conditions the LLM only on encoder predictions and worsens Character Error Rate.
- Neuro Tokens: Neuro Tokens are MEG-derived word embeddings produced by the CTC tokenizer and word projector.They are the additional neural input supplied to the LLM in Brain2Qwerty v2.
- Ablation: The ablation Brain2Qwerty + LLM conditions the LLM only on the encoder’s predictions.This configuration tests whether the LLM reads neural information or acts as a corrector of encoder text output.
- Results: 0.34 vs. 0.38 Character Error Rate is achieved by Brain2Qwerty v2 versus Brain2Qwerty + LLM.The two configurations share the same Qwen3-0.6B backbone.
Encoder importance for downstream accuracy
Downstream WER and SemER improve linearly as the MEG Encoder’s CER decreases. Encoder architecture choices jointly affect encoder quality and final sentence-decoding accuracy, particularly for low-signal MEG inputs.
- Encoder–decoder relationship: 0.78 and 0.68 are the Pearson correlations between encoder CER and downstream WER and SemER, respectively.Both downstream metrics scale linearly with encoder CER: better keystroke predictions correspond to better final sentence decoding.
- Architecture ablations: Three encoder architectures were compared on identical data: Temporal Patch Transformer, BrainModule + Transformer, and BrainModule + Conformer.The BrainModule + Conformer is the architecture used in this paper; the BrainModule + Transformer resembles Brain2Qwerty v1.
- Architecture ablations: Both the BrainModule and the Conformer contribute jointly to MEG Encoder output quality and final decoded-sentence quality.The architectural priors provided by the BrainModule are particularly important for low signal-to-noise MEG inputs.