SENSE: Semantic Neural Speech Synthesis from Brain Dynamics via Spatial Graph Encoding
Abstract
Reconstructing speech from non-invasive brain signals offers a promising pathway for restoring communication in individuals who are cognitively intact but unable to speak. Existing EEG-to-speech approaches formulate this task as acoustic reconstruction, optimizing waveform fidelity while ignoring whether the generated speech preserves high-level semantic content. In this work, we revisit this formulation and argue that EEG signals carry not only acoustic but also semantic information. We identify two key limitations of prior methods: (1) the neglect of spatial relationships between EEG electrodes, and (2) the failure to exploit the semantic structure of the N400 paradigm, where congruent and incongruent trials reflect distinct semantic processing. We propose SENSE(Semantic-EEG Neural Speech SynthEsis), which combines a graph-based EEG encoder over electrode geometry with EEG Semantic Conditioning (ESC), aligning EEG to a pretrained semantic space using only congruent trials. On the N400 dataset, SENSE consistently outperforms prior methods on both acoustic and semantic metrics, and model-internal channel attribution suggests distributed reliance on auditory, sensorimotor, and centro-parietal regions, consistent with known speech-perception neuroscience. In the unseen-subject setting, SENSE trained on only two subjects already surpasses the strongest baseline trained on all eighteen subjects in word error rate, and matches it on acoustic metrics with as few as eight subjects.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SENSE reformulates EEG-to-speech synthesis by integrating spatial graph encoding with paradigm-aware semantic conditioning, demonstrating that leveraging the N400 semantic structure significantly improves both acoustic fidelity and semantic consistency, particularly in low-data cross-subject scenarios.
Methodology
The paper proposes SENSE, a framework for EEG-to-speech synthesis that addresses two specific gaps in prior work: the lack of spatial modeling of EEG electrodes and the neglect of semantic structure in the N400 paradigm. The methodology is technically sound and well-motivated. The use of a graph-based encoder with a fixed anatomy-derived adjacency matrix is a logical choice to capture spatial correlations without overfitting on the limited dataset size (7,200 pairs). The integration of EEG Semantic Conditioning (ESC) is the core novelty; by aligning EEG representations with a frozen CLIP space using only congruent trials, the authors effectively leverage the known neurophysiological properties of the N400 component. The training strategy, which uses a bridge mechanism to expose the decoder to EEG-derived conditioning vectors during training, is a clever solution to the distribution shift problem at inference time. The use of S4 blocks for temporal encoding is consistent with recent state-of-the-art baselines like FESDE, ensuring a fair comparison.
Results & evaluation
The experiments are conducted on the standard N400 dataset. The paper claims consistent outperformance on both acoustic (MCD, Mel-Correlation) and semantic (WER) metrics. A particularly strong result is the generalization capability: the model trained on only two subjects surpasses the strongest baseline trained on all 18 subjects in Word Error Rate. This suggests that the semantic conditioning effectively captures invariant features across subjects, which is a significant practical advantage for BCI applications where per-subject calibration is burdensome. The ablation studies likely confirm the contribution of the graph encoder and the ESC module, though the truncated text prevents a full review of the ablation tables. The channel attribution analysis linking model reliance to auditory and sensorimotor regions adds biological plausibility to the technical results.
Reproducibility
The authors provide a link to a project page with code and audio samples. The methodology section provides sufficient detail on the graph construction (BioSemi montage, Gaussian kernel), the encoder architecture (S4 blocks, Conformer for phonemes), and the loss functions. The specific hyperparameters for the graph convolution and the initialization of the channel gates are mentioned. The use of standard components (VITS, CLIP ViT-B/32, HiFi-GAN) further aids reproducibility.
Limitations
The primary limitation is the reliance on the N400 paradigm, which is a passive listening task. It is unclear how well this approach generalizes to active speech production or different speech contexts. The use of a fixed adjacency matrix, while justified by data scarcity, may limit the model's ability to adapt to individual variations in brain topology. Additionally, the semantic alignment is restricted to congruent trials, which means the model may not effectively handle or distinguish incongruent semantic processing in a generative context, although this is arguably outside the scope of standard speech synthesis.
Broader impact
This work has significant implications for non-invasive BCIs aimed at restoring communication for individuals with conditions like ALS or locked-in syndrome. By demonstrating that semantic information can be effectively extracted from EEG and used to condition speech synthesis, the paper moves the field beyond mere acoustic reconstruction towards meaningful communication. The strong cross-subject generalization results are particularly impactful, as they suggest that large-scale per-subject training may not be necessary, making the technology more accessible and practical for real-world deployment. SENSE reformulates EEG-to-speech synthesis by integrating spatial graph encoding with paradigm-aware semantic conditioning, demonstrating that leveraging the N400 semantic structure significantly improves both acoustic fidelity and semantic consistency, particularly in low-data cross-subject scenarios.
Author of this paper? Send a correction.