PaperlayerView exact revision on arXiv

Source-linked AI summary

A Common Measure of Communication for Speech Brain-Computer Interfaces

Dulhan Jayalath, Benjamin Ballyk, Oiwi Parker Jones

arXiv:2609.02887v1cs.LGq-bio.NC

TL;DR

Speech BCIs lack a common progress measure because systems differ in experimental conditions, vocabularies, and communication distributions. The paper derives OVMI to evaluate decoded information against a shared reference distribution, showing that vocabulary coverage and decoding fidelity jointly determine communication capability. OVMI supports heterogeneous comparisons and vocabulary design, while its interpretation remains bounded by lexical and experimental-scope limitations.

  • Problem

    Speech BCI scores are rarely comparable because systems use different settings, vocabularies, datasets, and language distributions, while conventional metrics condition on supported words.

  • Method

    The paper derives open-vocabulary mutual information, weighting in-vocabulary information by the probability that the reference distribution assigns to supported words.

  • Results

    OVMI compares heterogeneous systems, reveals coverage–fidelity trade-offs and distribution dependence, and vocabulary selection improves accuracy across three speech domains.

  • Takeaways & Limitations

    OVMI provides a common communication scale for comparing speech BCIs and can guide vocabulary design when supported words are controllable.

  • Takeaways & Limitations

    The work treats out-of-vocabulary words as unsupported, evaluates lexical rather than contextual information, and retrospective comparisons often rely on a scalar estimator.

Abstract

from arXiv · show

Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.

1 Introduction

Speech BCI studies lack a common progress measure because they differ in settings, data, speech tasks, and vocabularies. The paper frames comparison around a reference word distribution and the information a decoder conveys from it.

  • Measurement gap: Current systems commonly use constrained vocabularies, making neither a universal vocabulary nor study-specific vocabularies satisfactory for comparison.
  • Measurement gap: Different recording methods, speech tasks, participant populations, and protocols often prevent shared-dataset evaluation across speech BCI systems.
  • Measurement gap: Benchmarks evaluate decoding on particular language distributions, but collecting neural datasets for every communication objective would be impractical.
  • Measurement gap: The paper identifies two questions: which word distribution users should communicate and how much information a speech BCI can convey from it.
  • Approach: OVMI evaluates lexical information relative to an explicit reference distribution, weighting in-vocabulary information by lexical coverage.
  • Approach: OVMI compares heterogeneous systems on a common communication scale and guides vocabulary selection across three speech domains.

2 Information-Theoretic Evaluation in BCIs

BCI information metrics traditionally quantify information within a predefined symbol set, often using assumptions suited to closed-symbol interfaces. These assumptions do not capture whether a vocabulary represents the words a user may wish to communicate.

  • Existing metrics: ITR and its refinements quantify information within a predefined set of symbols, fitting classical closed-symbol interfaces.
  • Existing metrics: Mutual information measures how much observing decoder output reduces uncertainty about the intended symbol.
  • Existing metrics: Wolpaw’s derivation assumes uniformly likely symbols and symmetric decoding errors, with correctness probability P.
  • Limitations of existing metrics: Natural-language words are non-uniform and Zipfian, so uniform-prior information measures are restrictive for speech.
  • Limitations of existing metrics: Existing formulations measure information within a predefined symbol set and do not account for whether that set represents all desired communication.

3 Open-Vocabulary Mutual Information

Conventional accuracy, WER, and in-vocabulary information can miss unsupported words in a user’s intended communication. OVMI addresses this by weighting decoded information by reference-distribution coverage, with estimators suited to available evaluation statistics.

  • Motivation: Accuracy, WER, and in-vocabulary mutual information measure reliable decoding only conditional on the intended word belonging to vocabulary S.
  • Definition: OVMI weights information conveyed among supported words by the probability that an intended word drawn from reference distribution p is supported.
  • Motivation: System A represents only 5% of a user’s possible words despite perfect in-vocabulary decoding, whereas System B supports every intended word but is less accurate.
  • Motivation: OVMI can rank System B above System A because it accounts for both representability and decoding fidelity.
  • Formal definition: OVMI is defined as I(X;Y|Z) = C(S) I(X;Y|X ∈S), combining lexical coverage with in-vocabulary mutual information.
  • Formal definition: OVMI excludes binary entropy about whether a word is in vocabulary because detecting support conveys no information about the word’s identity.
  • Relationship to existing metrics: When C(S) = 1, OVMI reduces to ordinary mutual information, and under uniform symbols with symmetric errors it reduces to Wolpaw’s per-trial information.
  • Estimation: The scalar estimator approximates equal per-word accuracy and uniformly distributed errors, while full confusion matrices support direct OVMI computation.

4 Common Metrics Overestimate Open-Vocabulary Performance

In-vocabulary metrics can substantially overestimate open-vocabulary communication because they ignore unsupported intended words and non-uniform word frequencies. OVMI corrects both issues by combining lexical coverage with in-vocabulary entropy.

  • OVMI measures a noiseless decoder as C(S)H(pS), while in-vocabulary entropy gives H(pS) and Wolpaw’s uniform-prior measure gives log2 V bits.The three quantities diverge substantially, especially for small vocabularies.
  • The uniform-prior quantity decomposes into OVMI, excess from ignoring lexical coverage, and excess from assuming supported words are uniformly distributed.These are the three terms represented in the Figure 2 decomposition.
  • At small V, incomplete lexical coverage causes in-vocabulary entropy and the uniform-prior measure to overestimate information transfer, even for perfectly accurate decoders.An intended word outside S cannot be communicated, so C(S) can remain far below 1.
  • As V increases, coverage approaches unity and OVMI approaches in-vocabulary entropy, but the gap to log2(V) continues growing because word frequencies are Zipfian.The cited systems use vocabularies of 50–250 Zipfian-distributed words, where both overestimation sources are substantial.
  • Accuracy and WER can overstate communication because they condition on supported words rather than measuring representation of an external reference distribution.Even perfect in-vocabulary accuracy can coexist with poor open-vocabulary communication when lexical coverage is low.

5 Results

The results use OVMI to compare heterogeneous speech BCIs against explicit communication distributions, revealing how lexical coverage, decoding fidelity, and intended domain shape measured performance. They also show that OVMI-guided vocabulary selection can improve accuracy across multiple speech domains.

  • 5.1 A Common Scale for Heterogeneous Speech BCIs: OVMI evaluates heterogeneous speech decoders against a shared reference distribution, including systems differing in vocabulary, dataset, task, and recording modality.SUBTLEX-UK is used as the default broad spoken-English reference distribution.
  • 5.1 A Common Scale for Heterogeneous Speech BCIs: 50-word invasive systems have high in-vocabulary information but limited broad-English coverage, whereas 125k-word invasive systems combine near-complete coverage with high decoding fidelity.Non-invasive systems generally obtain greater coverage but substantially lower in-vocabulary information.
  • 5.1 A Common Scale for Heterogeneous Speech BCIs: 93.7% is Card’s 2024 normalised OVMI, following 72.0% for Willett’s 125k-word system and 2.4% for the 50-word Moses system in 2021.The largest historical increase among invasive systems occurs when vocabulary expands after in-vocabulary fidelity has approached saturation.
  • 5.1 A Common Scale for Heterogeneous Speech BCIs: 2.4% is d’Ascoli’s LibriBrain100 result, compared with 1.8% for Armeni, 0.3% for MEG-XL on MEG-MASC, and 3.6% for Tang’s fMRI system.For perceived speech, OVMI represents lexical decoding capability relative to the speech distribution rather than communication.
  • 5.2 Information Transfer Depends on the Communication Distribution: 40.4% is the 50-word Willett system’s AAC OVMI, versus 6.4% under broad spoken English, showing that rankings depend on the reference distribution.Card remains comparatively stable across references at 93.7–97.5%, while smaller-vocabulary systems are more sensitive to the intended domain.
  • 5.3 OVMI for Vocabulary Selection: 15.4%, 16.3%, and 8.4% are the peak relative accuracy improvements over frequency selection for TIMIT, Podcasts, and Sherlock, respectively.OVMI matches or exceeds frequency, validation-accuracy, and random vocabulary selection, with diminishing advantage at larger vocabulary sizes.

6 Discussion

The discussion positions OVMI as a complement to benchmarks for comparing heterogeneous speech BCI capabilities against a shared communication objective. It also identifies scope limitations and cautions that higher OVMI does not necessarily indicate practical or clinical usefulness.

  • Discussion: The evaluation framework distinguishes benchmark comparisons, OVMI comparisons under a reference distribution, and user-centred assessments of practical usefulness.The table assigns medium cost to benchmarks, low cost to OVMI, and high cost to user-centred evaluation.
  • Discussion: OVMI comparisons use a common reference distribution to relate capabilities across heterogeneous experimental settings, complementing within-setting benchmarks.The paper recommends reporting OVMI alongside standard decoder metrics for easier comparison.
  • Discussion: OVMI does not erase differences in experimental setting, and higher OVMI does not necessarily imply a more practical or clinically useful paradigm.Perceived-speech decoding alone is given as an example unlikely to constitute a usable communication interface.
  • Discussion: Table 1 organizes the core variables of the open-vocabulary mutual-information formulation and points to their introducing sections, propositions, corollaries, and equations.Its references help readers locate where each symbol is first introduced.

B Derivation of Wolpaw’s Information Transfer Rate

The derivation starts from Wolpaw’s per-trial mutual information under a uniform symbol prior and symmetric decoder errors. It computes output and conditional entropies before substituting them into mutual information.

  • Assumptions: Wolpaw’s derivation assumes uniformly likely intended symbols and symmetric errors with correct-decoding probability P.Each incorrect symbol receives probability (1 −P)/(V −1).
  • Output entropy: Under the uniform prior, each decoder output is uniform over the V-symbol vocabulary, so H(Y ) = log2 V.One intended symbol contributes correct-output mass P, while the remaining V −1 symbols contribute error mass.
  • Conditional entropy: Conditioned on any intended symbol, the output distribution has one correct outcome with probability P and V −1 equally likely incorrect outcomes.Because this conditional distribution is independent of the intended symbol, its entropy is unchanged by averaging over symbols.
  • Mutual information: Substituting the output and conditional entropies into I(X; Y ) = H(Y ) −H(Y | X) yields Wolpaw’s in-vocabulary mutual-information expression.The resulting expression is reported as IWolpaw(X; Y ) = log2 V + P log2 P + (1 −P) log2.

C Proofs and Supplementary Derivations for OVMI

The supplementary derivations adopt the convention 0 log2 0 = 0.

  • The derivations define 0 log2 0 as 0.
  • Under the stated convention, the expression 0 log2 0 evaluates to zero.
  • This section uses 0 log2 0 = 0 as its explicit convention.

C.1 Proof of Proposition 1

The supplementary section proves Proposition 1 by decomposing mutual information through an in-vocabulary indicator and develops OVMI’s in-vocabulary channel formulation. It then specializes the formulation to symmetric and word-specific accuracy models.

  • C.1 Proof of Proposition 1: Proposition 1 introduces an indicator Z for whether the intended word is representable, equivalently whether the decoder emits a nonempty output.The proof uses Z as a deterministic function of both the intended input and decoder output.
  • C.1 Proof of Proposition 1: The chain rule decomposes mutual information into information about vocabulary support plus conditional information within the supported vocabulary.The resulting expression is I(X; Y ) = H2(C(S)) + C(S) I(X; Y | X ∈S).
  • C.2 Equivalent Entropy Derivation of Proposition 1: An equivalent entropy derivation expands H(Y ) by the support indicator and sets the entropy of the deterministic out-of-vocabulary output to zero.The conditional entropy H(Y | X) is likewise split according to whether X belongs to S.
  • C.3 Proposition 2: For a general in-vocabulary decoder, pS renormalizes the reference word distribution on S, while KS specifies the conditional output channel for supported words.The resulting mutual information is evaluated under pS and KS.
  • C.3 Proposition 2: The proof obtains the general OVMI expression by applying I(X; Y ) = H(Y ) −H(Y | X) under the conditional in-vocabulary distribution and multiplying by C(S).The channel entropy contribution is weighted by pS(x).
  • C.4–C.6 Corollaries: The symmetric-error corollary assigns each supported word a shared correct-decoding probability, whereas the word-specific model allows each word its own Pc(x).Both retain uniform allocation of error probability across incorrect in-vocabulary outputs.
  • C.6 Retaining Per-Word Accuracies in OVMI: The word-specific model relaxes the shared-accuracy assumption but retains the assumption that all incorrect outputs are equally likely conditional on an error.Its probability-mass expression combines correct decoding of a target with errors from other intended words.

C.7 Further Observations for the Homogeneous Symmetric Channel

The homogeneous symmetric-channel analysis shows how output distributions and entropy depend on vocabulary distribution and decoder performance. OVMI computation uses these assumptions, typically excluding sparsely observed words for reliable macro-accuracy estimates.

  • If pS is uniform, then qS is uniform.
  • At chance performance, P = 1/V, qS is uniform for any pS.
  • When P ≠ 1/V, qS is non-uniform if and only if pS is non-uniform.
  • For informative decoders with P > 1/V, non-uniformity in pS propagates to qS and lowers output entropy relative to the uniform distribution.
  • OVMI can be computed with Python pseudocode under homogeneous accuracy and symmetric-error assumptions, excluding words with fewer than five instances when estimating Pmacro.

E Table of Results

The appendix explains how OVMI is estimated from published summary statistics and how reference distributions preserve intended-word coverage outside a decoder vocabulary. It also defines the cross-study reporting conventions and reference corpora used for comparison.

  • OVMI retrospective estimates approximate the decoder channel when studies report vocabulary size and accuracy or WER instead of complete confusion matrices.
  • The approximation assumes equal correct-decoding probability across supported words and uniformly distributed errors among the remaining V − 1 outputs.
  • Macro accuracy is preferred when available because frequency is already represented by the external reference distribution p.
  • 0.744, 0.909, 0.762, and 0.975 are the reported P values obtained from WER-derived results for Moses, Willett’s systems, and Card, respectively.
  • Words outside decoder vocabulary S remain in the reference distribution and contribute uncovered mass 1 − C(S), so p is not renormalised to S.
  • The reference distributions include SUBTLEX-UK, Switchboard, Universal Core Vocabulary, and Sherlock, with entropies of 9.77, 8.27, 4.30, and 8.44 bits, respectively.
  • OVMI is reported in bits per intended word and can be normalised by H(p) to express the percentage of available lexical information conveyed.

G Vocabulary Optimisation Experiment Details

The vocabulary-optimisation experiment evaluates a shared candidate pool across matched speech domains and decoder seeds. It estimates coverage from training text, uses full or approximate channels depending on validation data, and tests OVMI against frequency-based selection.

  • Matched-domain results use subject 0 from LibriBrain100 across TIMIT, Podcasts, and Sherlock, with domain-specific training, validation, and test splits.
  • The shared candidate pool contains up to 250 words occurring at least five times in pooled neural training data, ordered by decreasing training count.
  • Five independently initialised contrastive brain-to-T5-embedding decoder instances cache cosine scores for all 250 candidates, allowing vocabulary evaluation by restricting the cached matrix.
  • Domain reference probabilities are estimated from full training-text distributions rather than renormalised over the candidate pool, with no validation or test text included.
  • At V = 150, the symmetric-error OVMI fallback was used for all TIMIT seeds and Sherlock seeds 0, 3, and 5; other combinations used full validation channels.
  • OVMI and Frequency are compared separately at each vocabulary size using 100,000 Monte Carlo permutations and Holm correction across nine sizes per domain.
  • Figure 7 presents OVMI computation for a candidate vocabulary subset using a single accuracy scalar.
Loading 2609.02887v1…