NeuroCodec: EEG-Guided Latent-Domain Generative Model for High-Fidelity Target Speaker Extraction
Abstract
Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
Methodology
The paper proposes NeuroCodec, a generative framework for EEG-guided Target Speaker Extraction (TSE) that operates in the latent space of a frozen neural audio codec (DAC) rather than predicting time-frequency masks. The methodology is technically sound and innovative in its combination of components: it uses Chebyshev graph convolutions for spatial EEG modeling, Mamba (selective state-space models) for efficient temporal modeling, and cross-attention for modality fusion. The introduction of a six-phase training curriculum to stabilize the adversarial and spectral losses is a practical contribution to the training stability of generative speech models. The causal streaming extension (OnlineNeuroCodec) introduces an "auditory attractor" mechanism using an Exponential Moving Average (EMA) of previous target latents to maintain speaker identity across hops, which is a clever, parameter-free alternative to recurrent speaker encoders.
Results & evaluation
The experiments are conducted on two standard datasets (Cocktail Party and KUL) under a Subject-Independent (SI) setting, which is rigorous. The paper reports improvements in STOI, ESTOI, and PESQ over strong baselines like M3ANet and NeuroSpex+. A significant strength is the inclusion of a subjective MUSHRA listening test with 20 participants, which validates the perceptual quality claims (82.24 vs 49.63 for M3ANet). The ablation studies effectively isolate the contributions of the Mamba backbone, EEG encoder components, and loss functions. The streaming evaluation demonstrates a significant speed advantage (RTF) over the NeuroHeed baseline while maintaining higher accuracy.
Reproducibility
The paper states that source code will be released upon acceptance, but no code or demo links are currently provided in the text. The implementation details are reasonably specific (e.g., DAC downsampling factors, window sizes, EMA rates), which aids reproducibility. However, the reliance on specific pretrained DAC models and the complex multi-phase training schedule may make exact reproduction difficult without the authors' code.
Limitations
The primary limitation is the computational cost of the generative approach, although the RTF analysis suggests it is viable for streaming. The subjective study size (n=20) is modest. The paper acknowledges that the system is not yet ready for embedded hearing-aid hardware. The comparison with some baselines (NeuroHeed+, MSFNet) is limited to reported numbers from other papers due to lack of public code, which weakens the comparative analysis slightly.
Broader impact
This work has significant potential for assistive hearing technologies by enabling high-fidelity, wideband speech extraction guided by neural attention. The shift from masking to generative latent synthesis could reduce artifacts in overlapping speech regions. The use of Mamba for efficient temporal modeling in audio-EEG fusion is a trend that may influence future multimodal audio architectures. The paper presents a novel generative approach to EEG-guided target speaker extraction by operating in the latent space of a neural audio codec, effectively addressing the fidelity limitations of traditional time-frequency masking. The combination of Mamba-based temporal modeling, graph-based EEG encoding, and a stable multi-phase training curriculum yields strong objective and subjective performance improvements, particularly in wideband reconstruction and real-time streaming efficiency.
Author of this paper? Send a correction.