GLM-TTS Technical Report
Abstract
This work proposes GLM-TTS, a production-level TTS system designed for efficiency, controllability, and high-fidelity speech generation. GLM-TTS follows a two-stage architecture, consisting of a text-to-token autoregressive model and a token-to-waveform diffusion model. With only 100k hours of training data, GLM-TTS achieves state-of-the-art performance on multiple open-source benchmarks. To meet production requirements, GLM-TTS improves speech quality through an optimized speech tokenizer with fundamental frequency constraints and a GRPO-based multi-reward reinforcement learning framework that jointly optimizes pronunciation, speaker similarity, and expressive prosody. In parallel, the system enables efficient and controllable deployment via parameter-efficient LoRA-based voice customization and a hybrid phoneme-text input scheme that provides precise pronunciation control. Our code is available at https://github.com/zai-org/GLM-TTS. Real-time speech synthesis demos are provided via Z.ai (audio.z.ai), the Zhipu Qingyan app/web (chatglm.cn).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
GLM-TTS presents a robust framework for efficient and high-quality text-to-speech synthesis, effectively addressing critical challenges in the field. The innovative use of reinforcement learning and hybrid input mechanisms positions it as a significant contribution to advancing TTS technology, particularly for languages with complex phonetic structures.
Methodology
The methodology of GLM-TTS is well-structured, utilizing a two-stage architecture that effectively combines autoregressive and diffusion models for TTS. The introduction of a multi-reward reinforcement learning framework is particularly innovative, addressing common challenges in TTS systems such as pronunciation accuracy and emotional expressiveness. The use of a hybrid phoneme-text input scheme and optimized speech tokenizer enhances the system's controllability and adaptability, especially for languages with complex phonetic structures like Chinese. The detailed data processing pipeline and the enhancements made to the speech tokenizer demonstrate a thorough understanding of the underlying challenges in TTS.
Results & evaluation
The experiments conducted are comprehensive, comparing GLM-TTS against state-of-the-art models across various benchmarks. The results indicate that GLM-TTS achieves competitive performance with significantly less training data, which is a notable achievement. The evaluation metrics used, including CER, WER, and SIM, provide a clear picture of the system's capabilities. However, the paper could benefit from more detailed ablation studies to further validate the contributions of individual components.
Reproducibility
The paper provides a link to the code repository and demo, which is a positive aspect for reproducibility. However, the details regarding the training process, hyperparameters, and specific datasets used are somewhat limited. More explicit information on the experimental setup would enhance reproducibility.
Limitations
One limitation is the reliance on proprietary datasets, which may hinder the generalizability of the results. Additionally, while the system shows promise in emotional expressiveness, the paper acknowledges that the performance may vary across different emotional contexts, indicating potential areas for improvement. The complexity of the model may also pose challenges for deployment in resource-constrained environments.
Broader impact
The GLM-TTS system has significant implications for various applications, including virtual assistants, educational tools, and content creation. Its ability to generate high-fidelity, expressive speech with reduced training data makes it accessible for low-resource scenarios, potentially democratizing TTS technology. The focus on controllability and customization also opens avenues for personalized applications in diverse linguistic contexts. GLM-TTS presents a robust framework for efficient and high-quality text-to-speech synthesis, effectively addressing critical challenges in the field. The innovative use of reinforcement learning and hybrid input mechanisms positions it as a significant contribution to advancing TTS technology, particularly for languages with complex phonetic structures.
Author of this paper? Send a correction.