MiMo-Audio: Audio Language Models are Few-Shot Learners
Abstract
Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks (MMSU, MMAU, MMAR, MMAU-Pro), spoken dialogue benchmarks (Big Bench Audio, MultiChallenge Audio) and instruct-TTS evaluations, approaching or surpassing closed-source models. Model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-Audio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MiMo-Audio, an audio language model that demonstrates few-shot learning capabilities across diverse audio tasks, achieving state-of-the-art performance while challenging traditional task-specific fine-tuning approaches. This work significantly advances the field of audio processing by showcasing the potential of large-scale pretraining and generalization in audio language models.
Methodology
The methodology presented in MiMo-Audio is innovative, leveraging a large-scale pretraining approach similar to that of GPT-3 but applied to the audio domain. The authors systematically evaluate few-shot learning capabilities across various audio tasks, which is a significant departure from traditional task-specific fine-tuning methods. The introduction of instruction-tuning and thinking mechanisms enhances the model's adaptability and performance, making the methodology robust and well-justified.
Results & evaluation
The experiments are comprehensive, covering a wide range of audio tasks and benchmarks. The paper reports state-of-the-art results on multiple open-source benchmarks, demonstrating the effectiveness of the proposed models. The evaluation metrics are appropriate, and the results are presented clearly, allowing for easy comparison with existing models. However, the paper could benefit from more detailed ablation studies to further validate the contributions of different components.
Reproducibility
The authors provide access to model checkpoints and a full evaluation suite, which is essential for reproducibility. However, the paper lacks detailed implementation specifics, such as hyperparameter settings and training configurations, which are crucial for other researchers to replicate the results accurately.
Limitations
The paper acknowledges limitations in terms of the model's performance on certain niche audio tasks and the potential for overfitting on specific datasets. Additionally, the reliance on large-scale pretraining may not be feasible for all research groups due to resource constraints.
Broader impact
The implications of this research are significant, as it opens new avenues for audio processing and understanding through few-shot learning. The ability to generalize across tasks with minimal examples could revolutionize applications in speech recognition, audio generation, and interactive systems, making advanced audio technologies more accessible. The main contribution of this paper is the introduction of MiMo-Audio, an audio language model that demonstrates few-shot learning capabilities across diverse audio tasks, achieving state-of-the-art performance while challenging traditional task-specific fine-tuning approaches. This work significantly advances the field of audio processing by showcasing the potential of large-scale pretraining and generalization in audio language models.
Author of this paper? Send a correction.