Scaling Open Discrete Audio Foundation Models with Interleaved Semantic, Acoustic, and Text Tokens
Abstract
Current audio language models are predominantly text-first, either extending pre-trained text LLM backbones or relying on semantic-only audio tokens, limiting general audio modeling. This paper presents a systematic empirical study of native audio foundation models that apply next-token prediction to audio at scale, jointly modeling semantic content, acoustic details, and text to support both general audio generation and cross-modal capabilities. We provide comprehensive empirical insights for building such models: (1) We systematically investigate design choices -- data sources, text mixture ratios, and token composition -- establishing a validated training recipe. (2) We conduct the first scaling law study for discrete audio models via IsoFLOP analysis on 64 models spanning $3{\times}10^{18}$ to $3{\times}10^{20}$ FLOPs, finding that optimal data grows 1.6$\times$ faster than optimal model size. (3) We apply these lessons to train SODA (Scaling Open Discrete Audio), a suite of models from 135M to 4B parameters on 500B tokens, comparing against our scaling predictions and existing models. SODA serves as a flexible backbone for diverse audio/text tasks -- we demonstrate this by fine-tuning for voice-preserving speech-to-speech translation, using the same unified architecture.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SODA, a scalable audio foundation model that effectively integrates semantic, acoustic, and text tokens, providing a comprehensive framework for advancing audio modeling. This work significantly enhances the understanding of scaling laws in audio models and sets a foundation for future innovations in the field.
Methodology
The methodology presented in the paper is robust and systematic, focusing on the design choices that influence the performance of audio foundation models. The authors thoroughly investigate various aspects, including data sources, text mixture ratios, and token composition, which are critical for optimizing model performance. The introduction of the SODA model, which integrates semantic, acoustic, and text tokens, represents a significant advancement in audio modeling. The use of next-token prediction at scale is a novel approach that extends the capabilities of existing models.
Results & evaluation
The paper includes a comprehensive empirical evaluation, particularly through the IsoFLOP analysis that examines scaling laws for discrete audio models. The authors provide extensive experimentation across 64 models, which is a commendable effort to validate their findings. The results indicate that optimal data grows faster than model size, which is a valuable insight for future research in this area. However, the paper could benefit from more detailed comparisons with existing models beyond the scaling predictions.
Reproducibility
While the authors mention establishing a validated training recipe, the paper lacks specific implementation details that would facilitate reproducibility. Providing access to code or detailed hyperparameter settings would enhance the paper's contribution to the community and allow for independent verification of results.
Limitations
One limitation is the reliance on a specific architecture for the SODA model, which may not generalize well to all audio tasks. Additionally, the paper does not address potential biases in the training data or the implications of using large-scale models in real-world applications. The scaling law findings, while insightful, may also be context-dependent and require further validation across diverse datasets.
Broader impact
The implications of this research are significant, as it opens up new avenues for audio generation and cross-modal tasks, such as speech-to-speech translation. The ability to model semantic content alongside acoustic details can enhance applications in various domains, including entertainment, accessibility, and communication technologies. The findings could influence future research directions and encourage the development of more sophisticated audio models. The main contribution of this paper is the introduction of SODA, a scalable audio foundation model that effectively integrates semantic, acoustic, and text tokens, providing a comprehensive framework for advancing audio modeling. This work significantly enhances the understanding of scaling laws in audio models and sets a foundation for future innovations in the field.
Author of this paper? Send a correction.