AudioLM: a Language Modeling Approach to Audio Generation
Abstract
We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen speakers. Furthermore, we demonstrate how our approach extends beyond speech by generating coherent piano music continuations, despite being trained without any symbolic representation of music.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AudioLM, a novel framework for high-quality audio generation that effectively combines semantic and acoustic tokenization strategies to produce coherent and contextually relevant audio continuations. This work represents a significant advancement in the field of audio generation, addressing key challenges and opening avenues for future research in multimodal audio systems.
Methodology
The methodology proposed in AudioLM is innovative, leveraging a hybrid tokenization scheme that combines semantic and acoustic tokens to achieve high-quality audio generation with long-term coherence. The use of a multi-stage Transformer-based language model to generate audio from these tokens is a significant advancement in the field, addressing the challenges of both audio quality and structural consistency. The approach is well-justified and supported by a thorough exploration of the trade-offs between different tokenization strategies, showcasing a strong understanding of the underlying audio representation challenges.
Results & evaluation
The experiments conducted are comprehensive, covering both speech and piano music generation. The authors provide detailed evaluations using both subjective and objective metrics, such as the ViSQOL score for reconstruction quality and ABX error rates for phonetic discriminability. The results demonstrate the model's ability to generate coherent continuations while maintaining speaker identity and prosody, which is a notable achievement. The subjective evaluation further supports the effectiveness of the proposed method, with high preference rates for generated audio.
Reproducibility
The paper provides sufficient details regarding the model architecture, training procedures, and datasets used, which enhances the reproducibility of the results. However, the absence of a publicly available code repository limits the ease of reproduction for other researchers. The authors do mention the training setup and hyperparameters, which is helpful for those looking to replicate the study.
Limitations
One limitation is the potential for bias in the generated audio, particularly in terms of speaker identity and accent representation, which could lead to ethical concerns. Additionally, while the model shows promising results, the subjective evaluation indicates that there is still room for improvement in distinguishing between original and synthesized audio, suggesting that the model may not be foolproof in all scenarios.
Broader impact
The ability to generate high-quality audio with long-term coherence has significant implications for various applications, including speech synthesis for individuals with speech impairments and music composition. However, the potential misuse of such technology for impersonation or generating misleading audio content raises ethical concerns that must be addressed through responsible AI practices. The authors acknowledge these risks and propose a detection mechanism for synthesized speech, which is a positive step towards mitigating misuse. The main contribution of this paper is the introduction of AudioLM, a novel framework for high-quality audio generation that effectively combines semantic and acoustic tokenization strategies to produce coherent and contextually relevant audio continuations. This work represents a significant advancement in the field of audio generation, addressing key challenges and opening avenues for future research in multimodal audio systems.
Curator note
Borsos et al., Google; hierarchical language model over SoundStream tokens; coherent long-form audio
Discussion
Talks & videos
Related later work
Author of this paper? Send a correction.