MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching
Abstract
Generating coherent audio scenes that simultaneously blend speech, music, and sound effects remains a significant challenge. Current approaches typically rely on a disjointed pipeline where a frozen, decoupled text encoder feeds a separate audio decoder, limiting cross-modal optimization and leading to poor speech intelligibility. To overcome these limitations, we introduce MiDashengLM-Gen, an end-to-end framework that couples a pre-trained Large Language Model (LLM) with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. MiDashengLM-Gen represents a first approach for general text-to-audio generation with one end-to-end trained model. Empirical evaluations demonstrate that MiDashengLM-Gen drastically improves speech intelligibility over existing unified models. On the Seed-TTS benchmark, English Word Error Rate (WER) drops from 12.15% to 2.79%, approaching the performance of dedicated Text-to-Speech (TTS) systems (1.24%). Furthermore, the framework extends effectively to multilingual settings, yielding highly competitive multilingual WERs compared to existing baselines. Lastly, the model maintains competitive mixed-audio generation quality on the MECAT benchmark. Code and checkpoints are available at https://github.com/xiaomi-research/midashenglm-gen and https://huggingface.co/mispeech/midashenglm-gen, and the demo page is available at https://xingws.github.io/midashenglm-gen-demo/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MiDashengLM-Gen introduces an end-to-end LLM-based autoregressive framework with per-token flow matching for unified mixed-audio scene generation, achieving state-of-the-art speech intelligibility and competitive mixed-audio quality. The paper presents a significant step forward in unified audio generation by effectively integrating large language models with continuous flow matching. The key insight regarding the DiT width constraint provides valuable theoretical and practical guidance for training such hybrid architectures. The substantial improvement in speech intelligibility addresses a major bottleneck in previous unified models, making the generated audio more usable for practical applications. While the approach builds on existing components (LLMs, Flow Matching, Tokenizers), the specific integration for variable-length mixed-audio generation is novel and well-validated.
Methodology
The paper proposes MiDashengLM-Gen, an end-to-end framework for unified audio scene generation (speech, music, sound effects). The core architectural innovation is replacing the frozen text encoder and non-autoregressive diffusion backbone of its predecessor (Dasheng AudioGen) with a pre-trained Large Language Model (Qwen3-1.7B) coupled with per-token conditional flow matching. The method utilizes a structured multi-view captioning approach to decompose audio scenes into semantic views (global, transcript, SFX, music, env, speaker). A key technical contribution is the identification of a convergence prerequisite: the DiT decoder width must strictly exceed the audio latent dimensionality. The approach integrates an audio-text alignment stage to map high-dimensional audio latents into the LLM's token space. While the combination of LLMs and flow matching is not entirely new, applying it to *unified mixed-audio scene generation* with *per-token* flow matching for variable-length output is a distinct methodological shift from previous fixed-length or disjointed pipeline approaches.
Results & evaluation
The evaluation is comprehensive, covering single-type (AudioCaps, MusicCaps) and mixed-type (MECAT) generation, as well as speech intelligibility (Seed-TTS, multilingual WER/CER). The results show significant improvements in speech intelligibility (WER drop from 12.15% to 2.79% on Seed-TTS) compared to the previous Dasheng AudioGen model. The model maintains competitive performance on mixed-audio metrics (FAD, FD, KL) on MECAT, although it trails dedicated sound-effect models like TangoFlux on pure sound effects, which is acknowledged. The ablation studies effectively validate the necessity of the audio-text alignment stage and the DiT width constraint. The use of standard benchmarks (Seed-TTS, MECAT) ensures fair comparison.
Reproducibility
The paper provides detailed implementation details, including model sizes (Qwen3-1.7B, DiT 16 layers), training hyperparameters (batch size, LR, epochs), and data sources. Code and checkpoints are made publicly available via GitHub and Hugging Face, significantly enhancing reproducibility. The description of the structured captioning and the specific flow matching objective is sufficiently detailed for replication.
Limitations
The authors acknowledge several limitations: variable-length generation is bounded by training data distribution (1-20 seconds); speech intelligibility still trails dedicated TTS systems (especially in low-resource languages); and the model supports only coarse speaker-style control without explicit voice cloning or fine-grained temporal control. The trade-off between mixed-audio coordination and single-source acoustic fidelity (e.g., trailing TangoFlux on sound effects) is also a noted limitation.
Broader impact
This work advances the field of generative audio by demonstrating that LLM-based autoregressive frameworks can effectively handle complex, mixed-audio scene generation with high speech intelligibility. This has significant implications for immersive media, gaming, and film production where coherent multi-source audio is required. The open-source release contributes to the community's ability to build upon unified audio generation models. MiDashengLM-Gen introduces an end-to-end LLM-based autoregressive framework with per-token flow matching for unified mixed-audio scene generation, achieving state-of-the-art speech intelligibility and competitive mixed-audio quality. The paper presents a significant step forward in unified audio generation by effectively integrating large language models with continuous flow matching. The key insight regarding the DiT width constraint provides valuable theoretical and practical guidance for training such hybrid architectures. The substantial improvement in speech intelligibility addresses a major bottleneck in previous unified models, making the generated audio more usable for practical applications. While the approach builds on existing components (LLMs, Flow Matching, Tokenizers), the specific integration for variable-length mixed-audio generation is novel and well-validated.
Author of this paper? Send a correction.