XingChen-UniFE: A Unified Generative Speech Front-End via Complementary Acoustic-Semantic Modeling
Abstract
This paper presents XingChen-UniFE, a unified large audio language model (LALM) for general speech restoration (GSR) and target speaker extraction (TSE). The proposed model employs complementary semantic and acoustic representations through a dual-stream encoder, where a self-supervised learning model provides hierarchical semantic features and a learnable acoustic branch captures fine-grained structures from complex spectra. A bidirectional cross-attention module further facilitates information exchange between the two representations. The fused representation conditions the LALM for discrete speech token generation with task and speaker information. Experiments demonstrate strong performance on both GSR and TSE and robust generalization to unseen degradation conditions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.
Methodology
The paper proposes XingChen-UniFE, a unified Large Audio Language Model (LALM) for General Speech Restoration (GSR) and Target Speaker Extraction (TSE). The core methodological contribution is a dual-stream encoder that fuses semantic representations from a frozen self-supervised learning (SSL) model (WavLM) with acoustic representations from a learnable branch processing complex STFT spectrograms. A bidirectional cross-attention module is introduced to facilitate explicit information exchange between these two streams before conditioning the LALM backbone. The LALM generates interleaved acoustic and semantic tokens autoregressively, which are then decoded by a neural audio codec (H-Codec). The approach addresses the limitation of SSL features lacking fine-grained acoustic details (like phase) by explicitly modeling complex spectra, which is a reasonable and technically sound design choice for restoration tasks.
Results & evaluation
The experiments are conducted on the Libri2Mix dataset for TSE and various simulated degradation conditions for GSR. The model is compared against strong baselines like LLaSE-G1, UniSE, and UniTok. The proposed model achieves state-of-the-art or competitive results across PESQ, STOI, SpeechBERTScore (SBS), and WER metrics. Notably, the paper demonstrates robust generalization to unseen degradation types (codec distortion, wind noise) that were excluded from training, which is a significant strength. Ablation studies confirm the complementary roles of the semantic and acoustic branches and the importance of the bidirectional fusion mechanism.
Reproducibility
The paper provides sufficient implementation details, including model architecture dimensions (28 layers, 1024 hidden dim), training hyperparameters (400k steps, batch size 64, AdamW), and data composition (approx. 10k hours target speech, 405k hours noise). However, the specific codebase and pre-trained checkpoints are not linked in the provided text, and the reliance on specific internal or less common datasets (e.g., Emilia, CN-Celeb) may pose minor barriers for full reproduction without access to those resources.
Limitations
The model is computationally expensive due to the large LALM backbone (502M trainable parameters) and the dual-stream encoder, which may limit real-time deployment on edge devices. The evaluation is primarily limited to English speech (Libri2Mix, LibriSpeech); performance on multilingual or non-English speech is not assessed. Additionally, the paper lacks subjective listening tests (MOS), relying solely on objective metrics, which may not fully capture perceptual quality nuances.
Broader impact
This work contributes to the trend of unifying speech enhancement tasks under generative frameworks. By leveraging LALMs, it opens avenues for more complex speech processing tasks that require semantic understanding alongside acoustic restoration. The demonstrated generalization to unseen degradations suggests potential for robust deployment in diverse real-world environments. The paper presents a unified LALM-based framework for speech restoration and speaker extraction that effectively combines semantic and acoustic features via bidirectional cross-attention. The technical contribution is solid, with strong empirical results demonstrating improved performance over existing generative baselines and good generalization capabilities, though the lack of subjective evaluation and multilingual testing limits the comprehensiveness of the validation.
Author of this paper? Send a correction.