Open Vocabulary Electroencephalography-To-Text Decoding and Zero-shot Sentiment Classification

1 Introduction

• Earlier natural-language decoding remains challenging, and prior brain-to-text and brain-to-speech approaches have limitations.The introduction also describes work on brain-signal and NLP connections, including using brain signals as an NLP modality and using NLP models to study brain language encoding.Source: §1 · pp. 1–2
• The paper reports an unintentional bug in the original evaluation code and warns that its results are not as expected.The paper directs readers to the repository README for details before using the codebase.Source: §1 · p. 1
• The paper extends decoding beyond small closed vocabularies to 50,265 tokens and uses non-invasive EEG data from various subjects and sources.The paper reports enlarging the vocabulary from several hundred tokens to 50,265.Source: §1 · p. 1
• The proposed framework uses pretrained language models for open-vocabulary EEG-to-text decoding and evaluates them on zero-shot sentence-level sentiment classification.It assumes the brain functions as a special text encoder and jointly fine-tunes the language model with additional projection layers.Source: §1 · p. 2
• Non-invasive EEG is easier to acquire and more publicly available than invasive-device data, but paired brain-text data remain insufficient for high-performance open-vocabulary decoding.The paper notes a growing availability of open-source EEG-integrated devices as a potential data source.Source: §1 · p. 2
• The paper introduces open-vocabulary EEG-to-text decoding and EEG-based sentence sentiment classification, and identifies its work as the first open-vocabulary brain-to-text study.The authors also describe pretrained language models as a first for EEG-to-text decoding.Source: §1 · p. 2

2 Task Definition

• Open-vocabulary EEG-to-text decoding maps word-level EEG feature sequences to text tokens, with test sentences unseen during training.Training EEG-text pairs may come from various subjects and reading-material categories.Source: Open vocabulary EEG-To-Text decoding · p. 2
• EEG-based sentence sentiment classification predicts the corresponding sentence's sentiment label from EEG features alone.The task description contrasts this with approaches that include text input alongside brain signals.Source: EEG-based sentence sentiment… · pp. 2–3
• Zero-shot sentiment classification requires no EEG-sentiment pairs; it uses EEG-text pairs and external text-sentiment pairs instead.The text-sentiment pairs can come from existing datasets such as Yelp and the Stanford Sentiment Treebank.Source: EEG-based sentence sentiment… · p. 3

3.1 EEG-To-Text Decoding

• The decoder treats EEG feature vectors as continuous input embeddings and trains an encoder to map them into pretrained BART representations for open-vocabulary text generation.The model formulates decoding as neural machine translation.Source: §3.1 · p. 3
• A six-layer, eight-head Multi-layer Transformer Encoder and feed-forward projection map EEG features before pretrained BART encodes and decodes token outputs.Source: §3.1 · p. 3
• Training minimizes text-reconstruction cross-entropy over the target sequence.Source: §3.1 · p. 3

3.2 Zero-shot Sentiment Classification Pipeline

• The zero-shot pipeline decodes EEG features into text, then classifies sentiment using a separately trained text classifier, requiring no EEG-sentiment pairs.The modular design allows the decoder and classifier to be upgraded independently.Source: §3.2 · p. 3
• Decoded text serves as an information filter for sentiment classification; the authors hypothesize that separating sentiment from text semantics is harder in natural-reading EEG.Source: §3.2 · p. 3

4.1 Dataset

• The ZuCo data cover normal and task-specific reading of movie reviews and Wikipedia; sentences with NaN values were omitted and test sentences were unseen.Each reading task was split by unique sentence into 80% train, 10% development, and 10% test.Source: §4.1 · pp. 3–4, Table 2 · p. 4

4.2 EEG-To-Text Decoding Evaluation

• BLEU-1 rose from 33.5% to 37.3% with diverse-source additions, versus 34.5% with uniform-source additions; adding a different subject half reached 37.4%.The best-performing model used data from 30 subjects reading movie reviews and Wikipedia articles.Source: Results · p. 4, Table 3 · p. 5
• Adding Task-Specific Reading data showed no visible improvement; the authors reason that its prompts may shift EEG-feature distributions from normal reading.In that task, readers answer a preliminary question while reading.Source: Results · p. 4
• When named-entity mentions were wrong, their entity types could still be correct; the evaluation compared entity-type sequences using LCS and multiset cosine similarity.The named-entity matching results are reported in Table 5.Source: Discussion · p. 5, Table 5 · p. 5
• The decoder sometimes generated semantically close words absent from training, but still struggled with fine-grained entities and grammatical errors.For example, it generated “bad” for the unseen typo “monsterous,” which resembles “monstrous.”Source: Discussion · pp. 5–6

4.3 EEG-based Sentiment Classification Evaluation

• The best zero-shot pipeline significantly outperformed all fully supervised sentiment baselines by over 20%.The pipeline combines the EEG-to-text decoder with a text-based classifier.Source: Pipeline · p. 6

5 Related Work

• Earlier brain-to-text approaches are grouped into motor-imagery, overt-speech, and inner-speech methods, using signals including EEG, ECoG, and fMRI.Source: §5 · p. 6
• Many high-performing brain-computer interfaces for language communication use invasive devices, while experiments often cover only ten to a few hundred unique words.The paper notes that ECoG is difficult to collect at large scale and impractical to extend to healthy people.Source: §5 · p. 7
• The proposed system decodes EEG features into text in a sequence manner over the whole English vocabulary and can handle unseen sentences and data from varied sources and subjects.Source: §5 · p. 7
• Prior text-elicited brain-signal sentiment studies often combined brain signals with text or image modalities, whereas this work uses EEG features alone and requires no EEG-sentiment labeled pairs.Source: §5 · p. 7

6 Conclusions and Future Work

• The paper introduces open-vocabulary EEG-to-text sequence decoding and EEG-based sentence sentiment classification with frameworks leveraging pretrained language models.Source: §6 · p. 7
• Future work includes collecting larger EEG-text datasets and extending the framework to multilingual settings.Source: §6 · p. 7
• The authors propose applying the model to open-vocabulary inner-speech decoding, for which a dedicated sentence-level dataset with larger word coverage is needed.Source: §6 · p. 7

A Implementation Details

• The EEG-to-text model uses 840-dimensional inputs, a 1024-dimensional pretrained BART hidden state, and a six-layer, eight-head TransformerEncoder with hidden dimension 2048.Source: A Implementation Details · p. 11
• One-step training at learning rate 5e-7 achieved equally good results with less training time than the two-step approach described in Lewis et al. 2020.Source: A Implementation Details · p. 11
• Zero-shot sentiment text classifiers were fine-tuned on Stanford Sentiment Treebank, while supervised EEG baselines were trained on SR v1.0 EEG-sentiment pairs.Source: A Implementation Details · p. 11

B Details About EEG Input Data

• Each word-level EEG vector concatenates features from eight frequency bands into 840 dimensions, with features aggregated on gaze duration in the experiments.Source: B Details About EEG Input Data · p. 11
• EEG and text sequence lengths may differ because a sentence word can have no fixation.Source: B Details About EEG Input Data · p. 11

• Typical English speech is 100-130 words per minute, versus 10-30 characters per minute for point-and-click interfaces and 115 characters per minute for imagined-handwriting communication.The passage also gives a typical speaking rate of 4 syllables per second.Source: C Details About Related Work · p. 11
• Overt-speech approaches capture low-level auditory features elicited by vocal-tract movements and therefore depend on particular languages and potentially accents (Herff et al. 2015; Anumanchipalli, Chartier, and Chang 2019; Makin, Moses, and Chang 2020; Moses et al. 2021).Source: C Details About Related Work · p. 11
• No sequence-to-sequence generation on unseen sentences had been explored; prior evaluations of unseen sentences or words used only pairwise matching.Source: C Details About Related Work · p. 11