FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
The text contains content addressed to automated reviewers. It may have influenced the automated analysis; check the paper itself.
[system_override] "ruct TTS and speech editing require the shared LLM to learn new instruction-conditioned output patterns, we retain a peak learning rate"
Abstract
A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
Methodology
The paper proposes FireRedAudio, a 9B-parameter unified audio-language model that decouples continuous input representations for understanding versus generation. The core methodological contribution is the use of two distinct pathways: a dedicated Audio Encoder (initialized from Whisper-large-v3) for understanding tasks (ASR, audio QA) and a RedAE-based continuous autoencoder pathway for speech generation (TTS, editing). The generation pathway utilizes a Flow-Matching DiT conditioned on LLM hidden states to generate RedAE latents, which are then decoded to waveforms. The authors argue that understanding benefits from compact, semantic features while generation requires reconstructible, fine-grained features, and that fusing them degrades performance. The training strategy involves a five-stage progressive curriculum, starting with adapter alignment, then encoder adaptation, unified mid-training, multitask post-training with CoT, and finally long-context extension. This approach is technically sound and addresses a known tension in unified multimodal models. The use of flow-matching for continuous latent generation is a modern and effective choice compared to standard diffusion or autoregressive discrete token prediction.
Results & evaluation
The evaluation covers a broad suite of tasks: ASR (LibriSpeech, FLEURS, etc.), Audio Understanding (MMAU, MMSU), Zero-shot TTS (Seed-TTS-Eval), Instruct TTS (InstructTTSEval), and Speech Editing (Ming-Freeform-Audio-Edit). The results are competitive or leading on most benchmarks. Notably, the model achieves strong performance in Instruct TTS instruction following and significant improvements in speech editing over prior work (Ming-UniAudio-Edit). The long-form audio understanding capability (up to 1 hour) with second-level timestamp accuracy is a strong practical contribution. The ablation studies and comparisons are comprehensive. The use of an LLM-as-a-judge for temporal grounding evaluation is a reasonable proxy but introduces potential bias; however, the strict/content metrics provide some robustness.
Reproducibility
The paper provides detailed architectural descriptions, training stages, loss functions, and hyperparameters (learning rates, warmup steps). The code is made available on GitHub. The use of open-source components (Qwen3.5-9B, Whisper-large-v3) aids reproducibility. The RedAE model is described in detail, including its hybrid GAN and distillation loss. The main limitation for perfect reproducibility is the specific data mixtures and the proprietary nature of some training data (e.g., specific Instruct TTS datasets), though the sampling strategy is well-documented.
Limitations
The model relies on a large 9B parameter backbone, which implies significant computational requirements for training and inference compared to smaller specialized models. The decoupled representation, while effective, adds architectural complexity and potentially increases inference latency due to the need to manage two distinct pathways and the DiT generation loop. The performance on very long-form audio (1 hour) might still suffer from context window limitations or memory constraints during inference, although the paper claims success. The reliance on a frozen teacher for RedAE distillation means the final RedAE latents are bounded by the teacher's capabilities.
Broader impact
FireRedAudio contributes to the field of unified multimodal AI, demonstrating that decoupled representations can effectively balance the conflicting objectives of understanding and generation. This has implications for developing more capable and efficient voice assistants, content creation tools, and accessibility technologies. The ability to perform high-quality speech editing and TTS with natural language instructions opens up new creative and practical applications. However, like all generative audio models, it raises concerns about deepfakes and misuse for impersonation, which the authors do not explicitly address in the provided text. FireRedAudio introduces a novel architecture for unified audio-language modeling that decouples continuous input representations for understanding and generation, achieving state-of-the-art or competitive results across ASR, audio understanding, TTS, and speech editing tasks while supporting long-form audio analysis. The paper makes a significant technical contribution by empirically validating the hypothesis that separate pathways for semantic understanding and acoustic reconstruction mitigate performance trade-offs in unified models, offering a viable path forward for building versatile, general-purpose audio AI systems.
Author of this paper? Send a correction.