InstructAudio: Unified speech and music generation with natural language instruction
Abstract
Text-to-speech (TTS) and text-to-music (TTM) models face significant limitations in instruction-based control. TTS systems usually depend on reference audio for timbre, offer only limited text-level attribute control, and rarely support dialogue generation. TTM systems are constrained by input conditioning requirements that depend on expert knowledge annotations. The high heterogeneity of these input control conditions makes them difficult to joint modeling with speech synthesis. Despite sharing common acoustic modeling characteristics, these two tasks have long been developed independently, leaving open the challenge of achieving unified modeling through natural language instructions. We introduce InstructAudio, a unified framework that enables instruction-based (natural language descriptions) control of acoustic attributes including timbre (gender, age), paralinguistic (emotion, style, accent), and musical (genre, instrument, rhythm, atmosphere). It supports expressive speech, music, and dialogue generation in English and Chinese. The model employs joint and single diffusion transformer layers with a standardized instruction-phoneme input format, trained on 50K hours of speech and 20K hours of music data, enabling multi-task learning and cross-modal alignment. Fig. 1 visualizes performance comparisons with mainstream TTS and TTM models, demonstrating that InstructAudio achieves optimal results on most metrics. To our best knowledge, InstructAudio represents the first instruction-controlled framework unifying speech and music generation. Audio samples are available at: https://qiangchunyu.github.io/InstructAudio/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
InstructAudio represents a significant advancement in unified audio generation, combining speech and music synthesis under a single instruction-controlled framework. This innovative approach not only enhances the flexibility of audio generation but also sets a foundation for future research in multimodal AI systems.
Methodology
The methodology presented in InstructAudio is robust, employing a multimodal diffusion transformer architecture that effectively integrates both speech and music generation tasks. The authors introduce a standardized instruction-phoneme input format that allows for unified control over various acoustic attributes through natural language descriptions. This approach is innovative, as it addresses the limitations of existing models that require reference audio for timbre control, thus enabling a more flexible and user-friendly interaction with the model. The use of joint and single diffusion transformer layers is well-justified, and the training on a substantial dataset of 50K hours of speech and 20K hours of music enhances the model's capacity for multi-task learning and cross-modal alignment.
Results & evaluation
The experimental evaluation is thorough, comparing InstructAudio against state-of-the-art models in both TTS and TTM tasks. The authors provide a comprehensive set of metrics, including objective measures like Word Error Rate (WER) and subjective evaluations such as Mean Opinion Scores (MOS). The results demonstrate that InstructAudio achieves superior performance in instruction-based TTS tasks while maintaining competitive capabilities in music generation. However, the paper could benefit from additional clarity in the presentation of results, particularly in the tables and figures, to enhance the reader's understanding of the comparative performance.
Reproducibility
The paper provides a detailed account of the architecture, training process, and datasets used, which supports reproducibility. However, the absence of a public code repository limits the ease with which other researchers can replicate the results. The authors mention a significant dataset and specific training configurations, but sharing the code and model weights would greatly enhance reproducibility.
Limitations
The paper acknowledges some limitations, such as the inherent information loss associated with the text-only control mechanism, which can lead to one-to-many mapping ambiguities and potentially lower audio quality compared to reference audio-based methods. Additionally, the constraint of generating short audio clips for music may limit the model's applicability in scenarios requiring longer compositions. These limitations are important to consider for future work.
Broader impact
The potential applications of InstructAudio are significant, spanning various domains such as entertainment, education, and accessibility. By enabling unified control over speech and music generation through natural language instructions, this framework could facilitate more intuitive interactions with AI systems in creative fields. The ability to generate expressive speech and music could also enhance user experiences in virtual environments and assistive technologies. InstructAudio represents a significant advancement in unified audio generation, combining speech and music synthesis under a single instruction-controlled framework. This innovative approach not only enhances the flexibility of audio generation but also sets a foundation for future research in multimodal AI systems.
Author of this paper? Send a correction.