ReStyle-TTS: Relative and Continuous Style Control for Zero-Shot Speech Synthesis
Abstract
Zero-shot text-to-speech models can clone a speaker's timbre from a short reference audio, but they also strongly inherit the speaking style present in the reference. As a result, synthesizing speech with a desired style often requires carefully selecting reference audio, which is impractical when only limited or mismatched references are available. While recent controllable TTS methods attempt to address this issue, they typically rely on absolute style targets and discrete textual prompts, and therefore do not support continuous and reference-relative style control. We propose ReStyle-TTS, a framework that enables continuous and reference-relative style control in zero-shot TTS. Our key insight is that effective style control requires first reducing the model's implicit dependence on reference style before introducing explicit control mechanisms. To this end, we introduce Decoupled Classifier-Free Guidance (DCFG), which independently controls text and reference guidance, reducing reliance on reference style while preserving text fidelity. On top of this, we apply style-specific LoRAs together with Orthogonal LoRA Fusion to enable continuous and disentangled multi-attribute control, and introduce a Timbre Consistency Optimization module to mitigate timbre drift caused by weakened reference guidance. Experiments show that ReStyle-TTS enables user-friendly, continuous, and relative control over pitch, energy, and multiple emotions while maintaining intelligibility and speaker timbre, and performs robustly in challenging mismatched reference-target style scenarios.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents ReStyle-TTS, a novel framework for zero-shot speech synthesis that enables continuous and relative style control while preserving speaker timbre. This work contributes significantly to the field of TTS by addressing limitations in existing methods and providing a robust solution for expressive speech synthesis.
Methodology
The methodology proposed in ReStyle-TTS is innovative, particularly with the introduction of Decoupled Classifier-Free Guidance (DCFG) and Orthogonal LoRA Fusion (OLoRA). DCFG effectively separates the influences of text and reference audio, allowing for more flexible style control, which is a significant advancement over existing methods that rely on absolute style targets. The use of style-specific LoRAs enables continuous control over multiple attributes, which is a notable improvement in the field of TTS. The Timbre Consistency Optimization (TCO) module is also a valuable addition, addressing the common issue of timbre drift when reducing reference dependency. Overall, the proposed methods are well-structured and demonstrate a clear progression from existing techniques.
Results & evaluation
The experiments conducted are comprehensive, utilizing a variety of datasets and evaluation metrics, including Word Error Rate (WER) and timbre similarity. The subjective evaluations through Mean Opinion Score for Style Accuracy (MOS-SA) add depth to the assessment of the model's performance. The experiments effectively demonstrate the model's capabilities in handling contradictory-style generation and continuous control over multiple attributes, showcasing its robustness in challenging scenarios. However, the paper could benefit from a more extensive comparison with a broader range of existing methods to highlight its advantages more clearly.
Reproducibility
The paper provides sufficient details on the experimental setup, including the training process, datasets used, and evaluation metrics. However, the lack of a publicly available code repository or demo URL limits the reproducibility of the results. Future work should consider releasing the code and models to facilitate further research and validation of the findings.
Limitations
A significant limitation mentioned is the scalability of the model to new attributes, which requires additional dataset collection and fine-tuning of LoRAs. This could hinder the practical application of the model in real-world scenarios where rapid adaptation to new styles is necessary. Additionally, while the model performs well in controlled settings, its performance in more diverse and unpredictable real-world conditions remains to be evaluated.
Broader impact
The advancements presented in ReStyle-TTS have the potential to significantly impact the field of speech synthesis, particularly in applications requiring expressive and controllable speech, such as virtual assistants, audiobooks, and gaming. The ability to manipulate speech styles continuously and relative to reference audio could enhance user experiences across various domains, making TTS systems more versatile and user-friendly. The paper presents ReStyle-TTS, a novel framework for zero-shot speech synthesis that enables continuous and relative style control while preserving speaker timbre. This work contributes significantly to the field of TTS by addressing limitations in existing methods and providing a robust solution for expressive speech synthesis.
Author of this paper? Send a correction.