MFlowAudio: Efficient Text-to-Audio Synthesis via Mamba-based Stateful Flow Matching
Abstract
Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Methodology
MFlowAudio introduces a novel latent audio generation framework that combines Flow Matching with a custom-designed TCMamba backbone. The core methodological innovation lies in the TCMamba block, which addresses the limitations of Mamba for 2D spectrograms by proposing a dual-scan mechanism: TimeMamba for long-range causal dependencies along the temporal axis (linear complexity) and ChannelMamba for cross-channel correlations like harmonic structures. This is a clever adaptation of SSMs to the audio domain, specifically designed for 2D latent representations. The formulation of Stateful Flow Matching (SFM) is another significant contribution, enabling chunk-wise training and streaming generation with O(1) caching complexity. This directly tackles the memory and computational bottlenecks of traditional diffusion/flow models for long-form audio. The theoretical analysis provided in the appendix, demonstrating the equivalence of chunked and one-time integration under certain causal assumptions, provides a strong foundation for SFM. Furthermore, the State-Decoupled Conditional Guidance (SDCG) mechanism is a novel approach to handle dynamic prompt switching, aiming to neutralize vector field collisions during on-the-fly semantic transitions. This mechanism, including the Orthogonal Score Projection, is well-motivated to balance prompt adherence and acoustic continuity. The overall architecture is clearly described, leveraging standard components like VAE for audio encoding and T5 for text encoding, integrated with the TCMamba blocks. The use of FiLM and cross-attention for conditioning is standard but well-integrated.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Author of this paper? Send a correction.