AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation
Abstract
Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioCALM presents a compelling unified audio generation framework that effectively bridges the gap between discrete autoregressive modeling and continuous flow matching, achieving state-of-the-art performance across speech, sound, and music domains while introducing novel architectural and data-level solutions to cross-modal interference.
Methodology
The paper proposes AudioCALM, a unified framework for text-to-speech, text-to-sound, and text-to-music generation. The core methodological innovation is "Continuous Autoregressive Language Modeling" (CALM), which replaces the discrete softmax output of standard autoregressive language models with a continuous flow-matching head that predicts rectified-flow velocities over VAE latents. This allows the model to leverage the streaming and in-context capabilities of AR models while avoiding the information bottleneck of discrete tokenization. Key technical components include: 1) AR-Flow Attention: A block-causal attention pattern that allows bidirectional flow matching within a block of latents while maintaining autoregressive commitment across blocks, enabling variable-length generation. 2) Asymmetric Mixture-of-Modality-Experts (A-MoME): A novel architectural design that adds a dedicated residual expert for speech (which requires tight local alignment) while sharing the backbone for sound and music (which rely on global semantics), addressing the identified "asymmetric mismatch" in joint training. 3) Description-Style Conditioning: A data reformulation strategy using an MLLM to generate long-form, modality-specific descriptions from short captions/transcripts, unifying the conditioning interface across modalities. The approach is theoretically sound and addresses specific pain points in unified audio generation (fidelity vs. flexibility, cross-modal interference).
Results & evaluation
The evaluation is comprehensive, covering three distinct audio modalities on standard benchmarks (LibriTTS, SeedTTS for speech; AudioCaps for sound; Song-Describer for music). The paper compares AudioCALM against both modality-specific state-of-the-art systems (e.g., CosyVoice 3.0, Stable Audio Open) and prior unified models (UniAudio, UniFlow-Audio). Results show that AudioCALM matches or exceeds SOTA on most metrics, particularly in sound and music generation (FAD, CLAP score) and speech intelligibility (WER). The ablation studies are particularly strong, effectively isolating the contributions of the continuous head, the description-style conditioning, and the A-MoME architecture. The finding that adding speech data disproportionately degrades non-speech generation (and vice versa) is a significant empirical insight that justifies the asymmetric design. The use of both objective metrics (FAD, WER, CLAP) and subjective evaluations (MOS) provides a robust assessment.
Reproducibility
The paper provides detailed implementation details, including the VAE architecture (CNN-GAN with iSTFT head), training hyperparameters (AdamW, batch size, learning rate), and the specific prompts used for the MLLM captioning pipeline. The authors release code and weights, and provide the cached annotations for public datasets, which significantly aids reproducibility. The use of open-source datasets (LibriTTS, VGGSound, FMA, etc.) ensures that the training data is accessible. The only potential hurdle is the reliance on Gemini 3 Pro for the offline captioning step, but the authors mitigate this by releasing the prompts and the resulting captions.
Limitations
The authors acknowledge several limitations: 1) The training data is restricted to English speech and public sound/music corpora, limiting generalization to non-English speech, singing voice, and rare audio events. 2) The backbone scale is limited to 1.7B-4B parameters, leaving open questions about scaling behavior. 3) Long-form generation coherence and termination are not deeply investigated, with the current system relying on a simple stop head. 4) The use of a closed-source MLLM for data preparation introduces a dependency that may not be fully reproducible by all researchers without access to similar models.
Broader impact
AudioCALM represents a significant step towards universal audio generation, which has broad applications in accessibility (TTS), creative industries (music/sound design), and research (data augmentation). However, the power of unified models to clone voices and generate realistic sound effects raises serious concerns about misuse, including impersonation, fraud, and disinformation. The authors address this by implementing safeguards in the license (prohibiting non-consensual cloning) and discussing the need for synthetic audio detection. The release of such a powerful model requires careful consideration of these risks. AudioCALM presents a compelling unified audio generation framework that effectively bridges the gap between discrete autoregressive modeling and continuous flow matching, achieving state-of-the-art performance across speech, sound, and music domains while introducing novel architectural and data-level solutions to cross-modal interference.
Author of this paper? Send a correction.