DeepASMR: LLM-Based Zero-Shot ASMR Speech Generation for Anyone of Any Voice
Abstract
While modern Text-to-Speech (TTS) systems achieve high fidelity for read-style speech, they struggle to generate Autonomous Sensory Meridian Response (ASMR), a specialized, low-intensity speech style essential for relaxation. The inherent challenges include ASMR's subtle, often unvoiced characteristics and the demand for zero-shot speaker adaptation. In this paper, we introduce DeepASMR, the first framework designed for zero-shot ASMR generation. We demonstrate that a single short snippet of a speaker's ordinary, read-style speech is sufficient to synthesize high-fidelity ASMR in their voice, eliminating the need for whispered training data from the target speaker. Methodologically, we first identify that discrete speech tokens provide a soft factorization of ASMR style from speaker timbre. Leveraging this insight, we propose a two-stage pipeline incorporating a Large Language Model (LLM) for content-style encoding and a flow-matching acoustic decoder for timbre reconstruction. Furthermore, we contribute DeepASMR-DB, a comprehensive 670-hour English-Chinese multi-speaker ASMR speech corpus, and introduce a novel evaluation protocol integrating objective metrics, human listening tests, LLM-based scoring and unvoiced speech analysis. Extensive experiments confirm that DeepASMR achieves state-of-the-art naturalness and style fidelity in ASMR generation for anyone of any voice, while maintaining competitive performance on normal speech synthesis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DeepASMR, a pioneering framework for zero-shot ASMR speech generation that leverages a two-stage architecture to synthesize high-quality ASMR speech from any speaker's voice without requiring whispered training data. This work significantly advances the state of the art in speech synthesis, particularly in generating nuanced, low-intensity speech styles essential for relaxation.
Methodology
The methodology is innovative, introducing a two-stage framework that effectively separates content and style in ASMR speech synthesis. The use of a large language model (LLM) for content-style encoding and a flow-matching acoustic decoder for timbre reconstruction is a significant advancement. The identification of a latent factorization within the token space is a novel insight that enhances the model's ability to generate ASMR speech without requiring whispered training data from the target speaker. The proposed task prompt selection via a virtual speaker pool is also a clever solution to mitigate timbre leakage and ensure speaker identity preservation.
Results & evaluation
The experiments are extensive and well-structured, demonstrating the effectiveness of DeepASMR across various tasks, including intra-style and cross-style synthesis. The introduction of DeepASMR-DB, a large bilingual ASMR dataset, is a valuable contribution to the field. The evaluation metrics are comprehensive, combining objective, subjective, and LLM-based assessments, which provide a holistic view of the model's performance. The results indicate that DeepASMR outperforms existing methods in terms of naturalness and style fidelity, particularly in the challenging Normal-to-ASMR task.
Reproducibility
The paper provides detailed implementation details, including training configurations and evaluation protocols, which enhance reproducibility. The availability of the DeepASMR-DB dataset and the demo URL further supports the reproducibility of the research. However, the reliance on proprietary models and specific architectures may pose challenges for full replication without access to the same resources.
Limitations
While the paper presents a significant advancement in ASMR synthesis, it does not address potential limitations in terms of the generalizability of the model across diverse languages and cultural contexts beyond English and Chinese. Additionally, the reliance on a large dataset may limit accessibility for researchers with fewer resources. The trade-offs observed in the iterative inference refinement process, where style intensity may compromise speaker identity, also highlight a potential area for improvement.
Broader impact
The ability to generate high-fidelity ASMR speech has implications for various applications, including mental health, relaxation, and entertainment. This research could lead to advancements in personalized audio experiences, enhancing user engagement and satisfaction. The release of the DeepASMR-DB dataset may also stimulate further research in ASMR synthesis and related fields. The main contribution of this paper is the introduction of DeepASMR, a pioneering framework for zero-shot ASMR speech generation that leverages a two-stage architecture to synthesize high-quality ASMR speech from any speaker's voice without requiring whispered training data. This work significantly advances the state of the art in speech synthesis, particularly in generating nuanced, low-intensity speech styles essential for relaxation.
Author of this paper? Send a correction.