Unified Audio Intelligence Without Regressing on Text Intelligence
Abstract
Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents Nemotron-Labs-Audex, a unified audio-text LLM that successfully integrates audio understanding and generation into a strong text-only MoE backbone, achieving state-of-the-art audio capabilities while preserving text reasoning through a novel multi-stage SFT and text-only RL training strategy.
Methodology
The paper proposes Audex, a unified audio-text Large Language Model built on the Nemotron-Cascade-2-30B-A3B backbone. The core architectural contribution is a simple, unified Transformer decoder that ingests audio via an AF-Whisper encoder projected into the text embedding space, while treating text and quantized audio tokens uniformly during autoregressive generation. A key methodological innovation is the "multi-stage supervised fine-tuning" (SFT) strategy designed to prevent catastrophic forgetting of text reasoning capabilities, which is a common failure mode in unified multimodal models. The authors compare this against a single-stage consolidated SFT, finding that the multi-stage approach preserves long-context attention mechanisms better. The model utilizes separate codecs for speech (X-Codec2) and non-speech audio (X-Codec) to optimize tokenization efficiency and quality. The training pipeline includes text-only Cascade RL and multi-domain on-policy distillation, demonstrating that audio capabilities can be preserved without regression on text benchmarks when the RL phase is kept text-only.
Results & evaluation
The evaluation is comprehensive, covering text-only benchmarks (reasoning, knowledge, alignment, long-context, agentic) and audio-specific tasks (ASR, AST, audio understanding, TTS, TTA, S2S). Audex-30B-A3B achieves state-of-the-art or highly competitive results on audio understanding and ASR benchmarks, matching proprietary models like Qwen3.5-Omni-Flash. Crucially, it shows marginal or no regression on text benchmarks compared to its text-only backbone, outperforming competitors like Qwen3-Omni which show significant degradation. The paper provides detailed ablation studies on SFT strategies, inference hyperparameters (CFG values, temperature), and codec choices. The use of Fréchet Distance (FAD) for text-to-audio generation and WER/Similarity for TTS provides standard, rigorous metrics.
Reproducibility
The authors release model checkpoints on Hugging Face. The paper provides detailed descriptions of the architecture, training data curation (157.4B audio tokens, 320.5B text tokens), and training infrastructure (Megatron-LM, Transformer Engine). However, the exact composition of the 157.4B audio tokens is not fully open-sourced as a single dataset, relying on a mix of public and proprietary curation. The code for the specific training recipes and data processing pipelines is not explicitly linked to a public repository, though the Hugging Face links are provided. The detailed hyperparameters and loss functions are described, aiding reproducibility.
Limitations
The model is limited to 10-second fixed-length outputs for text-to-audio generation, which restricts its utility for long-form audio synthesis. The text-to-speech similarity scores are lower than specialized TTS models, likely due to the codec choice and unified training objective not optimizing for speaker identity preservation. The reliance on a single-stage or multi-stage SFT without audio-text RL (as noted in the paper) means audio capabilities are not further enhanced through reinforcement learning, leaving room for improvement. The model's performance on multi-lingual ASR/AST, while good, is not explicitly broken down by language in the main text, only averaged.
Broader impact
This work contributes significantly to the field of multimodal AI by demonstrating that unified audio-text models can achieve strong performance without sacrificing the core reasoning and agentic capabilities of their text-only predecessors. This addresses a critical bottleneck in AGI development where multimodal integration often degrades general intelligence. The release of the model facilitates open research in audio generation and understanding. However, the ability to generate high-quality audio and speech raises concerns about deepfakes and misinformation, necessitating responsible use policies. This paper presents Nemotron-Labs-Audex, a unified audio-text LLM that successfully integrates audio understanding and generation into a strong text-only MoE backbone, achieving state-of-the-art audio capabilities while preserving text reasoning through a novel multi-stage SFT and text-only RL training strategy.
Author of this paper? Send a correction.