SemanticVocoder: Bridging Audio Generation and Audio Understanding via Semantic Latents
Abstract
Recent audio generation models typically rely on Variational Autoencoders (VAEs) and perform generation within the VAE latent space. Although VAEs excel at compression and reconstruction, their latents inherently encode low-level acoustic details rather than semantically discriminative information, leading to entangled event semantics and complicating the training of generative models. To address these issues, we discard VAE acoustic latents and introduce semantic encoder latents, thereby proposing SemanticVocoder, a generative vocoder that directly synthesizes waveforms from semantic latents. Equipped with SemanticVocoder, our text-to-audio generation model achieves a Frechet Distance of 12.823 and a Frechet Audio Distance of 1.709 on the AudioCaps test set, as the introduced semantic latents exhibit superior discriminability compared to acoustic VAE latents. Beyond improved generation performance, it also serves as a promising attempt towards unifying audio understanding and generation within a shared semantic space. Generated samples are available at https://zeyuxie29.github.io/SemanticVocoder/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SemanticVocoder, a generative vocoder that synthesizes audio waveforms from semantic latents, overcoming the limitations of traditional VAE-based approaches and demonstrating superior performance in both audio generation and understanding tasks. This work represents a significant step forward in bridging the gap between audio generation and understanding, with implications for various applications in the audio processing domain.
Methodology
The paper introduces SemanticVocoder, a novel approach that replaces traditional VAE acoustic latents with semantic latents for audio generation. The methodology is well-structured, leveraging a flow-matching approach to synthesize waveforms directly from semantic representations, thus addressing the limitations of conventional VAE-based systems. The use of a pretrained MAE encoder for extracting semantic latents is a significant innovation, enabling the model to focus on high-level semantic information rather than low-level acoustic details. The proposed architecture effectively balances the optimization difficulty across the text-to-latent and latent-to-waveform stages, which is a notable advancement in the field.
Results & evaluation
The experiments are comprehensive, utilizing multiple datasets (AudioCaps, AudioSet, and WavCaps) to evaluate the performance of SemanticVocoder against existing models. The reported results demonstrate superior performance in terms of Fréchet Distance and Fréchet Audio Distance, indicating that the model generates audio closer to real distributions. Additionally, the paper includes evaluations on audio understanding tasks, showcasing the discriminative power of semantic latents. However, the lack of subjective evaluation metrics is a minor drawback.
Reproducibility
The paper provides detailed implementation specifics, including model architecture, training parameters, and datasets used, which enhances reproducibility. However, the absence of a publicly available code repository limits the ease of reproduction for external researchers.
Limitations
The paper acknowledges limitations such as dependency on the pretrained semantic encoder's performance, constraints on audio length generation, and the need for subjective evaluations to complement objective metrics. These factors could impact the model's applicability in real-world scenarios.
Broader impact
SemanticVocoder has the potential to significantly advance the field of audio generation and understanding by providing a unified framework that leverages semantic information. This could lead to improved applications in areas such as content creation, audio synthesis, and interactive media, thereby enhancing user experiences in various domains. The main contribution of this paper is the introduction of SemanticVocoder, a generative vocoder that synthesizes audio waveforms from semantic latents, overcoming the limitations of traditional VAE-based approaches and demonstrating superior performance in both audio generation and understanding tasks. This work represents a significant step forward in bridging the gap between audio generation and understanding, with implications for various applications in the audio processing domain.
Author of this paper? Send a correction.