Qwen-Audio-3.0-Gen-Preview Technical Report
Abstract
Existing single-domain and multi-task audio systems remain limited in directly organizing speech, music, sound effects, ambience, and multiple roles into long-form temporal scenes. We present Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive framework that uses a Diffusion Transformer (DiT) and a shared variational autoencoder (VAE) to generate the complete mixed waveform. Prompt enhancement converts free-form requests into structured temporal records that are rendered as textual conditions, while a two-stage data curriculum and semantic conditional views train the proposed model to use these conditions across domains. A shared continuous VAE compresses 48kHz stereo waveforms into 25Hz latent sequences and incorporates semantic supervision, providing one representation for speech, music, sound effects, and their mixtures. On Seed-TTS-Eval, speaker similarity is the proposed model's clearest strength across all three subsets, and on the multi-speaker benchmark, the proposed model shows higher cross-turn consistency than Seed-Audio-1.0 in both languages. On AudioCaps, its advantages are concentrated in evaluations using large audio-language models and AudioBox. Relative to Seed-Audio-1.0, it achieves stronger temporal localization. Using approximately 10% music data of a dedicated in-house model, the proposed model remains close across all seven SongBench components and leads in three while retaining speech and general-audio capabilities. These results demonstrate the potential of unified generation for temporally structured, multi-domain audio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
Methodology
The paper proposes Qwen-Audio-3.0-Gen-Preview, a unified non-autoregressive (NAR) audio generation framework based on a Diffusion Transformer (DiT) and a shared continuous Variational Autoencoder (VAE). The core technical contribution lies in the data and conditioning pipeline rather than a radical architectural shift. The authors introduce a sophisticated "Prompt Enhancement" module that converts free-form text into structured temporal records (global scene, source profiles, time-localized events). This structured data is then rendered into textual conditions for the DiT. A key methodological component is the "Semantic Conditional View" training strategy, which uses conditional dropout on specific semantic units (e.g., hiding dialogue to force the model to rely on scene context) to improve robustness and controllability. The shared VAE is trained with auxiliary semantic supervision using a frozen LLM, aiming to create a latent space that is both acoustically high-fidelity and semantically rich. While the use of DiTs and continuous latents is established in recent audio generation literature (e.g., AudioGen, MusicGen), the specific integration of hierarchical temporal annotations and the two-stage curriculum for long-form scene generation represents a meaningful engineering contribution to the field of unified audio models.
Results & evaluation
The evaluation is comprehensive, covering Speech (Seed-TTS-Eval), General Audio (AudioCaps), and Music (SongBench). The paper demonstrates strong performance in speaker similarity and cross-turn consistency for multi-speaker dialogue, which is a significant challenge for unified models. On AudioCaps, the model leads in Large Audio-Language Model (LALM) based evaluations, suggesting better semantic alignment than competitors, although it does not lead in standard CLAP scores. The introduction of an in-house benchmark for temporal localization (mIoU) provides valuable insight into the model's ability to follow timing instructions, showing improvement over Seed-Audio-1.0. The comparison with a dedicated in-house music model highlights the efficiency of the unified approach, achieving competitive results with significantly less music data. However, the reliance on LALMs for evaluation introduces potential bias, as the judges are from the same ecosystem (Qwen/Gemini), and the lack of human MOS scores for the general audio/music tasks limits the perceptual validation.
Reproducibility
The paper provides detailed descriptions of the data annotation pipeline, synthetic data construction (using Scaper-like recipes), and the training curriculum. The architecture of the VAE and DiT is described, including the latent dimensions (128-dim, 25Hz) and the semantic supervision objective. However, specific hyperparameters for the DiT training, the exact composition of the 200,000 hours of training data, and the code for the Prompt Enhancement module are not fully open-sourced or detailed to the extent required for exact replication. The use of proprietary in-house models for evaluation (Seed-Audio-1.0 baseline, in-house music model) also complicates direct comparison.
Limitations
The paper acknowledges limitations regarding the synthetic data, noting residual context in source clips and mismatches in simulated acoustics. The reliance on LLMs for prompt enhancement and evaluation introduces a dependency on the capabilities of those specific models. The model's performance on music composition (melody, structure) lags behind dedicated autoregressive models, indicating that the NAR DiT architecture may struggle with long-range musical dependencies compared to sequential planners. The evaluation metrics, particularly the LALM-based ones, may not fully capture human perceptual quality or nuance.
Broader impact
This work advances the state of unified audio generation, moving towards systems that can produce complex, temporally structured scenes with multiple audio types. This has significant implications for content creation in gaming, film, and virtual reality, where coherent audio scenes are essential. By demonstrating that a single model can handle speech, music, and sound effects with temporal control, it reduces the complexity of audio production pipelines. However, the potential for misuse in generating realistic deepfakes or misleading audio content remains a concern, necessitating robust safety measures. This paper presents a significant engineering advancement in unified audio generation by introducing a structured, temporally-aware conditioning framework and a shared continuous latent space, demonstrating strong capabilities in multi-speaker dialogue and scene-level audio organization while maintaining competitive performance across diverse audio domains.
Author of this paper? Send a correction.