MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models
Abstract
Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents MOSS-Audio-Tokenizer, a novel end-to-end audio tokenizer that significantly improves audio processing capabilities for autoregressive models. Its comprehensive methodology and robust experimental validation establish it as a noteworthy contribution to the field of machine learning and audio processing.
Methodology
The paper introduces the Causal Audio Tokenizer (CAT), a novel architecture that employs a fully end-to-end approach to audio tokenization using a homogeneous stack of causal Transformer blocks. This design minimizes fixed inductive biases, allowing for high-fidelity audio reconstruction across diverse domains. The architecture's simplicity and scalability are emphasized, with joint optimization of the encoder, quantizer, decoder, and discriminator, which is a significant departure from existing methods that often rely on pretrained components or complex architectures. The methodology is well-structured, with clear explanations of the training objectives and the integration of semantic modeling through audio-to-text tasks.
Results & evaluation
The authors conduct extensive experiments to evaluate the performance of MOSS-Audio-Tokenizer against existing audio tokenizers across various bitrate regimes. The results demonstrate state-of-the-art reconstruction quality in speech, sound, and music, with a clear advantage in low-bitrate scenarios. The use of both objective and subjective evaluation metrics strengthens the findings, providing a comprehensive assessment of the model's capabilities. The experiments are well-designed, showcasing the effectiveness of the proposed Progressive Sequence Dropout training strategy and the model's robustness across different conditions.
Reproducibility
The paper provides detailed implementation information, including architecture specifications, training schedules, and optimization strategies. However, it lacks a publicly accessible code repository or demo URL, which could hinder reproducibility. The absence of shared code or datasets limits the ability for other researchers to validate the findings independently.
Limitations
While the paper presents a strong technical contribution, it does not sufficiently address potential limitations, such as the computational resources required for training the large-scale model and the generalizability of the results to real-world applications. Additionally, the reliance on a large dataset for training may not be feasible for all researchers.
Broader impact
The development of MOSS-Audio-Tokenizer has significant implications for the field of audio processing and generation, particularly in enhancing the capabilities of autoregressive models. Its ability to provide high-fidelity audio reconstruction and support various downstream tasks like text-to-speech and automatic speech recognition positions it as a valuable tool for future audio foundation models. The research could lead to advancements in applications such as virtual assistants, content creation, and accessibility technologies. The paper presents MOSS-Audio-Tokenizer, a novel end-to-end audio tokenizer that significantly improves audio processing capabilities for autoregressive models. Its comprehensive methodology and robust experimental validation establish it as a noteworthy contribution to the field of machine learning and audio processing.
Author of this paper? Send a correction.