DiT-Flow: Speech Enhancement Robust to Multiple Distortions based on Flow Matching in Latent Space and Diffusion Transformers
Abstract
Recent advances in generative models, such as diffusion and flow matching, have shown strong performance in audio tasks. However, speech enhancement (SE) models are typically trained on limited datasets and evaluated under narrow conditions, limiting real-world applicability. To address this, we propose DiT-Flow, a flow matching-based SE framework built on the latent Diffusion Transformer (DiT) backbone and trained for robustness across diverse distortions, including noise, reverberation, and compression. DiT-Flow operates on compact variational auto-encoders (VAEs)-derived latent features. We validated our approach on StillSonicSet, a synthetic yet acoustically realistic dataset composed of LibriSpeech, FSD50K, FMA, and 90 Matterport3D scenes. Experiments show that DiT-Flow consistently outperforms state-of-the-art generative SE models, demonstrating the effectiveness of flow matching in multi-condition speech enhancement. Despite ongoing efforts to expand synthetic data realism, a persistent bottleneck in SE is the inevitable mismatch between training and deployment conditions. By integrating LoRA with the MoE framework, we achieve both parameter-efficient and high-performance training for DiT-Flow robust to multiple distortions with using 4.9% percentage of the total parameters to obtain a better performance on five unseen distortions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of DiT-Flow, a novel speech enhancement framework that effectively utilizes flow matching and latent representations to improve robustness against multiple distortions. This work represents a significant step forward in the field of audio processing, addressing common challenges faced in real-world applications and demonstrating the potential for future advancements in speech enhancement technologies.
Methodology
The methodology of DiT-Flow is robust, leveraging flow matching and latent Diffusion Transformers to enhance speech under multiple distortions. The integration of LoRA with the Mixture-of-Experts framework is particularly innovative, allowing for parameter-efficient adaptation to varying acoustic conditions. The use of a synthetic dataset, StillSonicSet, designed to simulate realistic conditions, further strengthens the approach. However, the paper could benefit from clearer descriptions of hyperparameter choices and training procedures.
Results & evaluation
The experiments are comprehensive, validating DiT-Flow against state-of-the-art models across various conditions. The use of multiple evaluation metrics, including PESQ, ESTOI, and DNSMOS, provides a well-rounded assessment of performance. The results demonstrate significant improvements over baseline models, particularly in challenging scenarios, indicating the effectiveness of the proposed methods. However, the paper lacks detailed comparisons with a broader range of existing methods, which could provide more context for its contributions.
Reproducibility
The paper includes sufficient detail regarding the model architecture and training process, but lacks a clear link to code or datasets, which hampers reproducibility. Providing access to the StillSonicSet dataset and the trained models would enhance reproducibility and facilitate further research.
Limitations
One limitation is the reliance on synthetic data, which may not fully capture the complexities of real-world audio environments. Additionally, while the model shows robustness to multiple distortions, its performance in extreme or novel conditions remains to be tested. The computational efficiency of the model in real-time applications also needs further exploration.
Broader impact
The advancements in speech enhancement presented in this paper have significant implications for real-world applications, particularly in telecommunication, virtual meetings, and assistive technologies. The ability to enhance speech quality in diverse acoustic environments can improve communication clarity and accessibility for users in various settings. The main contribution of this paper is the development of DiT-Flow, a novel speech enhancement framework that effectively utilizes flow matching and latent representations to improve robustness against multiple distortions. This work represents a significant step forward in the field of audio processing, addressing common challenges faced in real-world applications and demonstrating the potential for future advancements in speech enhancement technologies.
Author of this paper? Send a correction.