GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model
Abstract
Language Model (LM)-based generative modeling has emerged as a promising direction for TSE, offering potential for improved generalization and high-fidelity speech. We present GenTSE, a two-stage decoder-only generative LM approach for TSE: Stage-1 predicts coarse semantic tokens, and Stage-2 generates fine acoustic tokens. Separating semantics and acoustics stabilizes decoding and yields more faithful, content-aligned target speech. Both stages use continuous SSL or codec embeddings, offering richer context than discretized-prompt methods. To reduce exposure bias, we employ a Frozen-LM Conditioning training strategy that conditions the LMs on predicted tokens from earlier checkpoints to reduce the gap between teacher-forcing training and autoregressive inference. We further employ DPO to better align outputs with human perceptual preferences. Experiments on Libri2Mix show that GenTSE surpasses previous LM-based systems in speech quality, intelligibility, and speaker consistency.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of GenTSE, a novel two-stage generative language model for target speaker extraction that effectively separates semantic and acoustic generation, improving speech quality and intelligibility. This work represents a meaningful advancement in the field of audio processing and generative modeling, addressing critical challenges in speaker extraction tasks.
Methodology
The paper introduces a two-stage generative language model architecture for target speaker extraction (TSE), which is innovative in its separation of semantic and acoustic token generation. The use of Frozen-LM Conditioning (FLC) to mitigate exposure bias and Direct Preference Optimization (DPO) to align outputs with human perceptual preferences are significant methodological advancements. The hierarchical modeling approach effectively reduces the complexity of direct acoustic modeling, which is a notable strength.
Results & evaluation
The experiments are well-structured, utilizing the Libri2Mix dataset, a standard benchmark for TSE. The results demonstrate that GenTSE outperforms existing models in terms of speech quality, intelligibility, and speaker consistency, which is quantitatively supported by various metrics. The ablation studies provide insights into the importance of each component of the model.
Reproducibility
The implementation details are provided, including model architectures, training configurations, and evaluation metrics. However, the absence of a public code repository or demo limits the reproducibility of the results. The paper could benefit from providing access to code or models to facilitate further research.
Limitations
While the paper presents a strong approach, it does not address potential scalability issues or the computational cost associated with training the two-stage model. Additionally, the reliance on specific datasets may limit the generalizability of the findings to other contexts or languages.
Broader impact
The advancements in TSE have significant implications for applications in speech processing, including voice recognition, telecommunications, and assistive technologies. The ability to extract a target speaker's voice from a mixture could enhance user experiences in various audio applications. The main contribution of this paper is the introduction of GenTSE, a novel two-stage generative language model for target speaker extraction that effectively separates semantic and acoustic generation, improving speech quality and intelligibility. This work represents a meaningful advancement in the field of audio processing and generative modeling, addressing critical challenges in speaker extraction tasks.
Author of this paper? Send a correction.