DiffWave: A Versatile Diffusion Model for Audio Synthesis
Abstract
In this work, we propose DiffWave, a versatile diffusion probabilistic model for conditional and unconditional waveform generation. The model is non-autoregressive, and converts the white noise signal into structured waveform through a Markov chain with a constant number of steps at synthesis. It is efficiently trained by optimizing a variant of variational bound on the data likelihood. DiffWave produces high-fidelity audios in different waveform generation tasks, including neural vocoding conditioned on mel spectrogram, class-conditional generation, and unconditional generation. We demonstrate that DiffWave matches a strong WaveNet vocoder in terms of speech quality (MOS: 4.44 versus 4.43), while synthesizing orders of magnitude faster. In particular, it significantly outperforms autoregressive and GAN-based waveform models in the challenging unconditional generation task in terms of audio quality and sample diversity from various automatic and human evaluations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DiffWave, a versatile diffusion model for audio synthesis that achieves high fidelity and speed in generating audio waveforms. This work significantly advances the state of the art in audio synthesis by addressing key challenges in speed and quality while leveraging the strengths of diffusion models.
Methodology
The paper presents DiffWave, a novel diffusion probabilistic model for audio synthesis that operates in a non-autoregressive manner. It utilizes a Markov chain to convert white noise into structured waveforms, optimizing a variant of the variational lower bound on data likelihood. The architecture employs a feed-forward and bidirectional dilated convolution approach, which is innovative in its ability to synthesize high-dimensional audio in parallel without the constraints of autoregressive models. The proposed method is well-grounded in existing literature on diffusion models and builds on their strengths while addressing limitations of previous approaches.
Results & evaluation
The experiments are comprehensive, comparing DiffWave against state-of-the-art models like WaveNet and GAN-based vocoders across multiple tasks, including neural vocoding and unconditional generation. The results demonstrate that DiffWave achieves comparable or superior audio quality (as measured by MOS) while significantly improving synthesis speed. The use of various automatic and human evaluation metrics strengthens the findings, showcasing the model's versatility and effectiveness in different audio generation contexts.
Reproducibility
The paper provides sufficient details regarding the model architecture, training procedures, and evaluation metrics, which aids in reproducibility. However, the absence of a public code repository limits the ease of reproducing results. The authors mention training on specific hardware (Nvidia GPUs) and provide hyperparameter settings, which are helpful for replication.
Limitations
While DiffWave shows promise, it is noted to be slower than some flow-based models, indicating potential for further optimization in inference speed. Additionally, the paper does not explore the model's performance on a wider variety of audio tasks beyond those tested, which could limit its applicability in broader contexts.
Broader impact
The development of DiffWave has significant implications for real-time audio synthesis applications, including text-to-speech systems, music generation, and other interactive audio applications. Its ability to generate high-quality audio efficiently could enhance user experiences in various domains, from entertainment to assistive technologies. The main contribution of this paper is the introduction of DiffWave, a versatile diffusion model for audio synthesis that achieves high fidelity and speed in generating audio waveforms. This work significantly advances the state of the art in audio synthesis by addressing key challenges in speed and quality while leveraging the strengths of diffusion models.
Curator note
Kong et al.; diffusion for waveform synthesis; vocoder + unconditional generation; launched audio diffusion
Author of this paper? Send a correction.