A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech
Abstract
Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by formulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similarity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured representations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high-quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the viability of this embedding-to-reasoning pipeline. Our work establishes speaker drift as a standalone research problem and bridges geometric signal analysis with LLM-based perceptual reasoning in modern TTS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a pioneering framework for automatic speaker drift detection in synthesized speech, leveraging cosine similarity and LLMs to enhance the coherence of TTS systems. The methodology is innovative, and the experimental results demonstrate substantial technical impact, making it a valuable contribution to the field of machine learning and speech synthesis.
Methodology
The proposed methodology effectively addresses the issue of speaker drift in synthesized speech by formulating it as a binary classification task. The use of cosine similarity to assess speaker consistency is both innovative and theoretically justified, providing a solid foundation for the proposed framework. The integration of large language models (LLMs) for reasoning based on structured representations of similarity scores is a novel approach that bridges low-level acoustic features with high-level cognitive evaluation. The construction of a synthetic benchmark dataset with human-validated annotations is a significant contribution, allowing for systematic evaluation of the proposed method.
Results & evaluation
The experiments conducted are robust, with a clear evaluation strategy involving multiple state-of-the-art LLMs. The results demonstrate the effectiveness of the proposed framework in detecting speaker drift, outperforming baseline methods. The use of F1 scores and accuracy as evaluation metrics is appropriate, and the ablation studies provide insights into the impact of different design choices on performance. However, the dataset size could be larger to enhance the generalizability of the findings.
Reproducibility
The paper provides sufficient details regarding the methodology and experimental setup, allowing for reproducibility. However, the lack of a publicly accessible dataset or code repository limits the ease with which other researchers can replicate the results. Providing a demo or project URL would enhance reproducibility.
Limitations
One limitation is the reliance on synthetic data, which may not fully capture the complexities of real-world speaker drift scenarios. Additionally, while the theoretical guarantees for cosine similarity are compelling, the practical implications of these guarantees in diverse acoustic environments remain to be explored. The dataset's size and diversity may also restrict the generalization of the findings.
Broader impact
The implications of this research are significant for applications in TTS systems, particularly in enhancing user experience in interactive and long-form speech applications. By addressing speaker drift, the framework can improve the coherence and naturalness of synthesized speech, which is crucial for virtual assistants, audiobooks, and other multimedia applications. The work also opens avenues for further research in speaker consistency and the integration of LLMs in audio processing tasks. This paper introduces a pioneering framework for automatic speaker drift detection in synthesized speech, leveraging cosine similarity and LLMs to enhance the coherence of TTS systems. The methodology is innovative, and the experimental results demonstrate substantial technical impact, making it a valuable contribution to the field of machine learning and speech synthesis.
Author of this paper? Send a correction.