AuraSE: Low-Hallucination Generative Speech Enhancement via Multimodal Flow Matching and Inference Policy Optimization
Abstract
Generative speech enhancement models can produce cleaner and more natural-sounding speech than conventional discriminative approaches, but may hallucinate by changing speech content or speaker identity, even with transcript conditioning. We present AuraSE, a flow-matching framework that addresses hallucination through complementary modality and inference designs. First, a double-stream-to-single-stream multimodal Diffusion Transformer (MMDiT) allows transcript and acoustic representations to interact while preserving a dedicated pathway for the degraded input. Second, we find that the best decoder configuration, governed by guidance scale, sampling temperature, and step count, varies substantially across utterances. This observation motivates Inference Policy Optimization (IPO), an online, on-policy preference optimization method. IPO generates multiple candidates from the current model under different inference configurations, ranks them with a multi-objective reward, and learns from their relative preferences. AuraSE-IPO ranks first on 11 of 12 metrics across the synthetic test sets and obtains the highest DNSMOS and blind-listening scores among the evaluated systems on the real DNS blind test set. At deployment, it uses a fixed $10$-step ODE decoder without classifier-free guidance (CFG) or per-utterance configuration search.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AuraSE introduces a multimodal flow-matching framework with Inference Policy Optimization to reduce hallucination in speech enhancement. The paper presents a rigorous and effective method for aligning generative speech models with human preferences for content fidelity and speaker identity, achieving state-of-the-art results on both synthetic and real-world benchmarks.
Methodology
The paper proposes AuraSE, a flow-matching framework for speech enhancement that integrates a double-stream-to-single-stream Multimodal Diffusion Transformer (MMDiT) with a novel Inference Policy Optimization (IPO) stage. The MMDiT design is sound, allowing transcript and acoustic representations to interact via joint attention while preserving separate pathways, which effectively mitigates hallucination by anchoring content to text without overwriting acoustic identity. The IPO method is the primary technical contribution; it treats inference hyperparameters (CFG scale, temperature, step count) as a policy space. By generating candidates under different policies, ranking them with a multi-objective reward, and performing on-policy preference optimization (similar to DPO but with refreshed buffers), the model distills the benefits of per-utterance optimal inference into a single fixed decoder. This is a creative and effective approach to handling the sample-dependence of diffusion/flow-matching samplers.
Results & evaluation
The experimental evaluation is comprehensive. The authors compare against a wide range of baselines, including discriminative (VoiceFixer), autoregressive (LLaSE-G1), masked generative (AnyEnhance), and flow-matching (FlowSE, SGMSE, StoRM) systems. They evaluate on both synthetic (VCTK + WHAM!/DEMAND) and real-world (DNS Challenge blind) datasets. The metrics cover perceptual quality (DNSMOS), content fidelity (WER, BERTScore), and speaker identity (SIM). The results show AuraSE-IPO achieving state-of-the-art performance on 11/12 synthetic metrics and the best blind-listening scores on real data. The ablation studies convincingly demonstrate the contribution of the double-stream architecture and the IPO training procedure over standard DPO/GRPO.
Reproducibility
The paper provides detailed architectural specifications (hidden sizes, block counts, attention heads) and training hyperparameters in the supplementary material. The use of standard datasets (Emilia, LibriTTS, DNS Challenge) and open-source components (Whisper, Vocos, WavLM) enhances reproducibility. However, the specific implementation of the IPO buffer management and reward normalization might require careful tuning, which is partially detailed.
Limitations
The method relies on an external ASR model (Whisper) for transcript conditioning, which introduces a dependency on ASR accuracy. If the ASR fails, the text conditioning may be unhelpful or harmful, though the double-stream design mitigates this. The IPO training process is computationally expensive due to the need for multiple rollouts and reward evaluations per input. The paper focuses on English speech; generalization to other languages is not tested.
Broader impact
This work has significant implications for the field of generative speech enhancement. It demonstrates that inference-time diversity can be leveraged as a training signal, a concept that could be extended to other diffusion-based audio tasks (TTS, voice conversion). The focus on hallucination reduction is critical for real-world deployment of generative enhancers, where content integrity is paramount. The integration of multimodal conditioning (text + audio) in a flow-matching framework sets a new standard for robust speech restoration. AuraSE introduces a multimodal flow-matching framework with Inference Policy Optimization to reduce hallucination in speech enhancement. The paper presents a rigorous and effective method for aligning generative speech models with human preferences for content fidelity and speaker identity, achieving state-of-the-art results on both synthetic and real-world benchmarks.
Author of this paper? Send a correction.