DashengTokenizer: One layer is enough for unified audio understanding and generation
Abstract
This paper introduces DashengTokenizer, a continuous audio tokenizer engineered for joint use in both understanding and generation tasks. Unlike conventional approaches, which train acoustic tokenizers and subsequently integrate frozen semantic knowledge, our method inverts this paradigm: we leverage frozen semantic features and inject acoustic information. In linear evaluation across 22 diverse tasks, our method outperforms previous audio codec and audio encoder baselines by a significant margin while maintaining competitive audio reconstruction quality. Notably, we demonstrate that this acoustic injection improves performance for tasks such as speech emotion recognition, music understanding, and acoustic scene classification. We further evaluate the tokenizer's generative performance on text-to-audio (TTA), text-to-music (TTM), and speech enhancement (SE). Our approach surpasses standard variational autoencoder (VAE)-based methods on TTA and TTM tasks, while its effectiveness on SE underscores its capabilities as a general-purpose audio encoder. Finally, our results challenge the prevailing assumption that VAE-based architectures are a prerequisite for audio synthesis. Checkpoints are available at https://huggingface.co/mispeech/dashengtokenizer.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DashengTokenizer, a unified audio tokenizer that enhances both understanding and generation tasks by inverting the traditional paradigm of acoustic tokenization. This innovative approach, alongside its competitive performance across various benchmarks, positions it as a significant advancement in the field of audio machine learning.
Methodology
The methodology presented in the paper is innovative, as it proposes the DashengTokenizer, which inverts the conventional approach of audio tokenization by leveraging frozen semantic features to inject acoustic information. The simplicity of using a linear projection for acoustic injection is a notable strength, making the method efficient and accessible. However, the paper could benefit from a more detailed explanation of the training process and the selection of hyperparameters.
Results & evaluation
The experimental evaluation is robust, covering a wide range of tasks across understanding and generation domains, with comparisons to existing state-of-the-art methods. The use of diverse datasets and benchmarks strengthens the findings, and the results indicate significant improvements over traditional methods, particularly in understanding tasks. However, more detailed statistical analysis of the results could enhance the credibility of the claims.
Reproducibility
The paper provides a clear overview of the architecture and training setup, which aids reproducibility. The availability of checkpoints on Hugging Face is a positive aspect, although the paper lacks specific implementation details that could help other researchers replicate the experiments more easily.
Limitations
One limitation is the potential overfitting to the specific datasets used for training and evaluation, which may not generalize to all audio tasks. Additionally, the reliance on a frozen semantic encoder may limit adaptability to new domains without retraining.
Broader impact
The DashengTokenizer has the potential to significantly impact audio understanding and generation tasks, making it a valuable tool for applications in speech recognition, music analysis, and environmental sound classification. Its efficiency and performance could lead to broader adoption in real-world applications, particularly in areas requiring high-fidelity audio processing. The main contribution of this paper is the introduction of DashengTokenizer, a unified audio tokenizer that enhances both understanding and generation tasks by inverting the traditional paradigm of acoustic tokenization. This innovative approach, alongside its competitive performance across various benchmarks, positions it as a significant advancement in the field of audio machine learning.
Author of this paper? Send a correction.