Qwen-Audio-3.1-ASR Technical Report
Abstract
We present Qwen-Audio-3.1-ASR, a family of audio-language models that builds upon its predecesor Qwen-Audio-3.0-ASR with improved speech recognition and expanded capabilities in speaker-diarized transcription and general audio understanding. The family retains support for 30 languages and 16 Chinese dialects, together with instruction-controlled capabilities including historical-context conditioning, hierarchical hotword customization, and native transcription polishing. Speaker-diarized ASR produces structured outputs containing speaker labels, timestamps, and recognized text, maintaining consistent speaker labels across turns and the representation of overlapping speech. The general audio understanding capabilities cover audio-, speech-, and music-captioning, temporal grounding and sound event detection, as well as audio question answering and multiple-choice reasoning. These tasks broaden the application scope from recognizing spoken words to interpreting speakers, acoustic events, and their temporal relations. To accommodate different deployment requirements, the family includes variants for asynchronous file transcription, low-latency streaming, interactive voice input, and non-streaming recognition, with speaker-diarized transcription and general audio understanding available in certain variants. Evaluations span conventional ASR, multilingual and dialectal speech, speaker-diarized transcription, and general audio understanding, demonstrating the capabilities of the Qwen-Audio-3.1-ASR family across transcription and broader audio analysis tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Qwen-Audio-3.1-ASR, a unified audio-language model family that integrates high-accuracy multilingual speech recognition, speaker-diarized transcription, and general audio understanding through a novel training pipeline featuring On-Policy Distillation and asynchronous reinforcement learning.
Methodology
The paper proposes Qwen-Audio-3.1-ASR, a family of audio-language models that integrates speech recognition, speaker diarization, and general audio understanding into a unified framework. The methodology is robust, leveraging a Mixture-of-Experts (MoE) LLM backbone initialized from Qwen3, paired with a pretrained audio encoder (BEST-RQ + AED). A significant methodological contribution is the "On-Policy Distillation" (OPD) strategy, which uses student-generated responses to provide token-level feedback from specialized teachers, addressing the distribution mismatch in conventional distillation. Additionally, the introduction of "FunVerl," an asynchronous reinforcement learning framework, allows for complex reward modeling (combining WER, DER, and temporal consistency) without the computational bottleneck of synchronous rollouts. The serialization of speaker-diarized output as a single autoregressive sequence with specific loss reweighting for speaker IDs is a practical and effective design choice.
Results & evaluation
The evaluation is comprehensive, covering multilingual ASR (30 languages), Chinese dialects (16 varieties), speaker-diarized transcription, and general audio understanding (captioning, QA, temporal grounding). The paper reports improvements over the previous generation (Qwen-Audio-3.0-ASR) and compares against strong baselines like Whisper and FireRedAudio. The use of specific metrics like cpWER (concatenated minimum-permutation word error rate) and DER for diarization is appropriate. However, the paper is a technical report, so some experimental details (e.g., specific dataset splits for the "general audio understanding" benchmarks) are less transparent than in a peer-reviewed journal article. The results demonstrate strong performance, particularly in the integrated capabilities of the "Next" variant.
Reproducibility
Reproducibility is moderate. While the architecture and training stages are described in detail, the specific hyperparameters for the MoE layers, the exact composition of the 20 million hours of pretraining data, and the implementation details of the "FunVerl" framework are not fully disclosed. The code and models are likely available via the provided URL, which aids in practical reproduction, but the scientific reproducibility of the training pipeline from scratch is limited by the proprietary nature of the data and infrastructure.
Limitations
The paper does not extensively discuss the computational cost of the MoE architecture during inference, which is a critical factor for deployment. The "general audio understanding" capabilities are limited to the "Next" variant, creating a fragmentation in the model family. Additionally, the reliance on synthetic data for certain tasks (like domain-entity augmentation) may introduce biases that are not fully analyzed. The lack of ablation studies on the specific contribution of the On-Policy Distillation versus standard RL is a minor gap.
Broader impact
This work has significant impact on the field of speech technology by demonstrating that a single model can handle not just transcription, but also speaker identification and semantic understanding of audio. This unification simplifies system architectures for applications like meeting assistants, media analysis, and accessibility tools. The open-sourcing of the model family (implied by the GitHub URL) will likely accelerate research in multimodal audio understanding. The paper presents Qwen-Audio-3.1-ASR, a unified audio-language model family that integrates high-accuracy multilingual speech recognition, speaker-diarized transcription, and general audio understanding through a novel training pipeline featuring On-Policy Distillation and asynchronous reinforcement learning.
Author of this paper? Send a correction.