Breaking Shortcut Learning for Cross-Trial EEG-Guided Target Speech Extraction via Two-Stage Training
Abstract
Recent end-to-end models for EEG-guided target speech extraction report impressive results, underscoring potential for neuro-steered hearing technologies. However, our analysis reveals that high within-trial performance can be driven by trial-specific EEG structure that acts as shortcuts for target selection, leading to poor generalization on unseen trials. To overcome this gap, we propose TRUST-TSE, a two-stage framework to mitigate shortcut learning. By introducing contrastive pretraining with attended-speaker negative sampling, we encourage the EEG encoder to capture fine-grained EEG--speech alignment while suppressing trial-identity cues. We also employ a confidence-weighted extraction objective based on EEG--source similarity to guide extraction using the learned representations. Experiments on KUL and DTU datasets show that TRUST-TSE outperforms end-to-end baselines under strict cross-trial protocols, addressing a key reliability bottleneck of existing approaches.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a critical analysis of shortcut learning in EEG-guided speech extraction and proposes a robust two-stage training framework (TRUST-TSE) that significantly improves cross-trial generalization, addressing a major reliability bottleneck in neuro-steered audio technologies.
Methodology
The paper proposes TRUST-TSE, a two-stage framework designed to mitigate shortcut learning in EEG-guided target speech extraction (TSE). The core methodological contribution lies in the diagnosis that end-to-end models exploit trial-specific EEG artifacts (trial identity) rather than genuine attention signals. To counter this, Stage 1 employs contrastive pretraining with a novel "attended-speaker negative sampling" strategy. This forces the EEG encoder to align with specific speech segments within the same trial, thereby suppressing trial-level shortcuts. Stage 2 uses a confidence-weighted SI-SDR objective, where the weight is derived from the similarity between the frozen EEG embedding and the audio embeddings of the attended vs. ignored sources. This allows the extractor to handle ambiguous or contradictory guidance segments by weighting gradients accordingly. The approach is theoretically sound and addresses a critical flaw in current evaluation protocols for neuro-steered audio systems.
Results & evaluation
The authors conduct rigorous experiments on two public datasets, KUL and DTU, under strict cross-trial protocols. They demonstrate that standard end-to-end baselines (NeuroHeed, M3ANet) suffer significant performance drops when evaluated on unseen trials compared to within-trial evaluations, confirming the shortcut hypothesis. TRUST-TSE consistently outperforms these baselines in cross-trial selection accuracy and separation quality (SI-SDR). The paper includes extensive ablation studies validating the components: the specific negative sampling strategy, the confidence weighting mechanism, and the superiority of contrastive embeddings over envelope decoding. Stress tests (EEG shuffling, trial-wise permutation) further confirm that TRUST-TSE relies on meaningful EEG-audio alignment rather than shortcuts. The results are robust across different window lengths and show generalization to unseen subjects.
Reproducibility
The paper provides detailed descriptions of the model architectures, training hyperparameters, and data preprocessing steps. The authors explicitly state that the source code is publicly available on GitHub, which significantly enhances reproducibility. The evaluation protocols are clearly defined, including the specific fold constructions to prevent data leakage. The inclusion of supplementary material with additional metrics (PESQ, STOI) and unseen-subject results adds to the transparency.
Limitations
The primary limitation is the reliance on public datasets which, while standard, are relatively small in scale and diversity compared to large-scale consumer audio datasets. The performance gains, while statistically significant and methodologically important, are modest in absolute terms (e.g., ~15% accuracy gain on KUL). The method assumes a known-subject setting in the main experiments, although unseen-subject results are provided. The confidence weighting mechanism, while effective, introduces a dependency on the quality of the frozen EEG encoder; if Stage 1 fails to capture attention, Stage 2 may struggle.
Broader impact
This work has significant implications for the development of reliable neuro-steered hearing aids and brain-computer interfaces. By highlighting the fragility of current end-to-end models and providing a robust alternative, it pushes the field towards more rigorous evaluation standards and more reliable real-world deployment. It also contributes to the broader understanding of shortcut learning in multimodal representation learning, offering a template for ensuring that models learn task-relevant features rather than spurious correlations. This paper presents a critical analysis of shortcut learning in EEG-guided speech extraction and proposes a robust two-stage training framework (TRUST-TSE) that significantly improves cross-trial generalization, addressing a major reliability bottleneck in neuro-steered audio technologies.
Author of this paper? Send a correction.