Discriminative-Generative Target Speaker Extraction with Decoder-Only Language Models
Abstract
Target speaker extraction (TSE) aims to recover the speech signal of a desired speaker from a mixed audio recording, given a short enrollment utterance. Most existing TSE approaches are based on discriminative modeling paradigms. Although effective at suppressing interfering speakers, these methods often struggle to produce speech with high perceptual quality and naturalness. To address this limitation, we first propose LauraTSE, a generative TSE model built upon an auto-regressive decoder-only language model. However, purely generative approaches may suffer from hallucinations, content drift, and limited controllability, which may undermine their reliability in complex acoustic scenarios. To overcome these challenges, we further introduce a discriminative-generative TSE framework. In this framework, a discriminative front-end is employed to robustly extract the target speaker's speech, yielding stable and controllable intermediate representations. A generative back-end then operates in the neural audio codec representation space to reconstruct fine-grained speech details and enhance perceptual quality. This two-stage design effectively combines the robustness and controllability of discriminative models with the superior naturalness and quality enhancement capabilities of generative models. Moreover, we systematically investigate collaborative training strategies for the proposed framework, including freezing or fine-tuning the front-end, incorporating an auxiliary SI-SDR loss, and exploring both auto-regressive and non-auto-regressive inference mechanisms. Experimental results demonstrate that the proposed framework achieves a more favorable trade-off among speech quality, intelligibility, and speaker consistency.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a discriminative-generative framework for target speaker extraction that effectively combines the strengths of both modeling paradigms, demonstrating improved speech quality and intelligibility. The comprehensive evaluation and systematic exploration of training strategies highlight its potential impact on the field of audio processing.
Methodology
The paper introduces a novel discriminative-generative framework for target speaker extraction (TSE) that combines the strengths of both modeling paradigms. The proposed architecture consists of a discriminative front-end for robust extraction and a generative back-end for high-quality reconstruction. This two-stage approach is well-justified, addressing the limitations of existing methods that struggle with perceptual quality and naturalness. The integration of collaborative training strategies and the systematic exploration of both auto-regressive and non-auto-regressive inference mechanisms further enhance the methodology's robustness. However, the paper could benefit from clearer explanations of the model's training dynamics and the rationale behind specific design choices.
Results & evaluation
The experimental validation is comprehensive, utilizing a well-defined dataset (LibriMix) and employing various evaluation metrics that are appropriate for assessing speech quality, intelligibility, and speaker consistency. The results demonstrate a favorable trade-off among these metrics, showcasing the effectiveness of the proposed framework. However, the paper could improve by providing more detailed comparisons with other state-of-the-art methods and discussing the implications of the results in a broader context.
Reproducibility
While the paper provides a thorough description of the model architecture and training procedures, it lacks specific implementation details such as hyperparameter settings, training duration, and the computational resources used. Including this information would enhance reproducibility and allow other researchers to replicate the findings more easily.
Limitations
One limitation of the proposed approach is its reliance on a two-stage framework, which may introduce additional complexity and computational overhead. Additionally, the performance of the generative model, LauraTSE, is shown to be sensitive to the design and discretization of input representations, which could impact its reliability in diverse acoustic scenarios. The paper also raises questions about the scalability of the generative model with respect to training data size and its comparative performance against discriminative models in terms of semantic consistency.
Broader impact
The proposed framework has significant potential applications in real-world scenarios such as voice recognition, assistive technologies, and audio processing systems where target speaker extraction is critical. By improving speech quality and intelligibility in mixed environments, this research could enhance communication technologies and contribute to advancements in human-computer interaction. The paper presents a discriminative-generative framework for target speaker extraction that effectively combines the strengths of both modeling paradigms, demonstrating improved speech quality and intelligibility. The comprehensive evaluation and systematic exploration of training strategies highlight its potential impact on the field of audio processing.
Author of this paper? Send a correction.