A Common Measure of Communication for Speech Brain-Computer Interfaces
Abstract
Speech brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common communication scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
Methodology
The paper proposes Open-Vocabulary Mutual Information (OVMI), a metric designed to standardize the evaluation of speech Brain-Computer Interfaces (BCIs). The core methodological contribution is the decomposition of mutual information into a component that accounts for lexical coverage (the probability that an intended word is in the decoder's vocabulary) and in-vocabulary decoding fidelity. The authors derive a scalar estimator based on macro-accuracy and vocabulary size, allowing for retrospective evaluation of existing systems that do not report full confusion matrices. The mathematical derivation is rigorous, clearly distinguishing OVMI from Wolpaw's Information Transfer Rate (ITR) by explicitly modeling the out-of-vocabulary probability mass. The approach is theoretically sound, leveraging information theory to address a practical benchmarking gap.
Results & evaluation
The experimental section is robust, applying OVMI to a diverse set of heterogeneous systems, including invasive intracortical decoders (Moses, Willett, Card) and non-invasive MEG/EEG decoders (d'Ascoli, MEG-XL). The authors demonstrate that conventional metrics like accuracy can overstate communication capability by ignoring lexical coverage. A key strength is the sensitivity analysis, showing how OVMI scores change when evaluated against different reference distributions (broad spoken English, conversation, AAC, narrative). The vocabulary selection experiment provides strong empirical evidence of utility, showing that optimizing vocabulary size and composition via OVMI yields up to 16.3% relative improvement in accuracy over frequency-based selection.
Reproducibility
The paper provides high reproducibility. The authors release a Python package and an interactive OVMI explorer. The appendix contains detailed derivations, pseudocode for the scalar estimator, and specific details on how reference distributions were constructed (e.g., using SUBTLEX-UK, Switchboard, UCV). The experimental setup for the vocabulary optimization is clearly described, including data splits, model architecture (contrastive brain-to-T5-embedding), and training details.
Limitations
The primary limitation is the reliance on the scalar estimator for retrospective comparisons, which assumes symmetric error distributions and uniform per-word accuracy (macro-accuracy). This may not hold for all decoders, particularly those with complex error structures. Additionally, the metric evaluates lexical information only, ignoring contextual or semantic information that might be conveyed through paraphrasing or sentence-level structure. The choice of reference distribution $p$ significantly impacts the score, and while the authors provide multiple examples, the "correct" distribution for a specific clinical application remains an open question.
Broader impact
This work has significant potential to standardize progress tracking in the speech BCI field. By providing a common scale for heterogeneous systems, it facilitates more meaningful comparisons between invasive and non-invasive approaches and across different experimental paradigms. The ability to use OVMI for vocabulary design offers a practical tool for system developers to optimize their interfaces for specific communication goals. It addresses a critical gap in the field where progress was previously difficult to quantify across different studies. The paper introduces Open-Vocabulary Mutual Information (OVMI), a principled information-theoretic metric that enables fair comparison of heterogeneous speech brain-computer interfaces by accounting for both lexical coverage and decoding fidelity relative to a reference communication distribution. By demonstrating that conventional metrics overstate communication capability and showing that OVMI-guided vocabulary selection improves system accuracy, the work provides a crucial tool for standardizing evaluation and guiding the development of more effective speech restoration technologies.
Author of this paper? Send a correction.