DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance
Abstract
Controllable text-to-speech (TTS) systems face significant challenges in achieving independent manipulation of speaker timbre and speaking style, often suffering from entanglement between these attributes. We present DMP-TTS, a latent Diffusion Transformer (DiT) framework with explicit disentanglement and multi-modal prompting. A CLAP-based style encoder (Style-CLAP) aligns cues from reference audio and descriptive text in a shared space and is trained with contrastive learning plus multi-task supervision on style attributes. For fine-grained control during inference, we introduce chained classifier-free guidance (cCFG) trained with hierarchical condition dropout, enabling independent adjustment of content, timbre, and style guidance strengths. Additionally, we employ Representation Alignment (REPA) to distill acoustic-semantic features from a pretrained Whisper model into intermediate DiT representations, stabilizing training and accelerating convergence. Experiments show that DMP-TTS delivers stronger style controllability than open-source baselines while maintaining competitive intelligibility and naturalness. Code and demos will be available at https://y61329697.github.io/DMP-TTS/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents DMP-TTS, a controllable TTS framework that effectively disentangles speaker timbre and speaking style through innovative methodologies. The contributions are significant, addressing key challenges in the field and providing a solid foundation for future research and applications in controllable speech synthesis.
Methodology
The paper introduces DMP-TTS, a novel framework for controllable TTS that utilizes a latent Diffusion Transformer architecture, which is innovative in its approach to disentangling speaker timbre and speaking style. The use of a CLAP-based style encoder (Style-CLAP) for aligning audio and text cues is a significant methodological advancement. Additionally, the introduction of chained classifier-free guidance (cCFG) allows for independent control of multiple attributes, which is a notable improvement over existing methods. The methodology is well-structured, with clear explanations of the components and their interactions, although some technical details could benefit from further elaboration.
Results & evaluation
The experiments conducted are thorough, utilizing a high-quality dataset of approximately 300 hours of Chinese speech. The evaluation metrics are appropriate, including both objective measures (WER, speaker similarity) and subjective measures (NMOS, QMOS). The results demonstrate that DMP-TTS outperforms existing baselines in terms of style controllability while maintaining competitive intelligibility and naturalness. However, the paper could have benefited from a more detailed discussion of the statistical significance of the results.
Reproducibility
The implementation details are provided, including the architecture, training procedure, and hyperparameters, which enhance reproducibility. The authors mention that code and demos will be available, which is essential for the community to validate and build upon their work. However, the absence of a direct link to a code repository limits immediate access to the implementation.
Limitations
While the paper presents a strong framework, it does not address potential limitations in terms of scalability to larger datasets or multilingual capabilities. Additionally, the reliance on a specific pre-trained model (Whisper) for representation alignment may limit the generalizability of the approach. The paper also notes that speaker similarity metrics are lower than some baselines, suggesting room for improvement in that area.
Broader impact
The advancements in controllable TTS have significant implications for applications in virtual assistants, audiobooks, and any domain requiring personalized speech synthesis. The ability to independently manipulate style and timbre enhances user experience and could lead to more engaging human-computer interactions. The work may also inspire further research into disentangled representations in other domains of machine learning. The paper presents DMP-TTS, a controllable TTS framework that effectively disentangles speaker timbre and speaking style through innovative methodologies. The contributions are significant, addressing key challenges in the field and providing a solid foundation for future research and applications in controllable speech synthesis.
Author of this paper? Send a correction.