PromptReverb: Multimodal Room Impulse Response Generation Through Latent Rectified Flow Matching
Abstract
Room impulse response (RIR) generation remains a critical challenge for creating immersive virtual acoustic environments. Current methods suffer from two fundamental limitations: the scarcity of full-band RIR datasets and the inability of existing models to generate acoustically accurate responses from diverse input modalities. We present PromptReverb, a two-stage generative framework that addresses these challenges. Our approach combines a variational autoencoder that upsamples band-limited RIRs to full-band quality (48 kHz), and a conditional diffusion transformer model based on rectified flow matching that generates RIRs from descriptions in natural language. Empirical evaluation demonstrates that PromptReverb produces RIRs with superior perceptual quality and acoustic accuracy compared to existing methods, achieving 8.8% mean RT60 error compared to -37% for widely used baselines and yielding more realistic room-acoustic parameters. Our method enables practical applications in virtual reality, architectural acoustics, and audio production where flexible, high-quality RIR synthesis is essential.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of PromptReverb, a novel two-stage generative framework for room impulse response generation that leverages multimodal inputs and advanced machine learning techniques to overcome existing limitations in the field. This work significantly advances the state of the art in audio synthesis and has the potential to impact various practical applications in immersive audio environments.
Methodology
The methodology presented in PromptReverb is innovative, employing a two-stage generative framework that integrates a variational autoencoder (VAE) for upsampling band-limited RIRs and a conditional diffusion transformer model based on rectified flow matching for generating RIRs from natural language descriptions. The use of a caption-then-rewrite pipeline to generate diverse textual prompts from visual inputs is a notable strength, allowing for a more intuitive and creative interaction with the model. The architectural decoupling of the VAE and the diffusion transformer is a clever design choice that addresses the limitations of existing models effectively.
Results & evaluation
The empirical evaluation is robust, with a comprehensive dataset of 145,976 samples that enhances the model's training and validation. The paper presents both objective metrics (mean RT60 error) and subjective evaluations (human listener ratings), providing a well-rounded assessment of the model's performance. The results indicate significant improvements over existing methods, particularly in perceptual quality and acoustic accuracy, which are critical for practical applications in audio synthesis.
Reproducibility
The paper provides a detailed description of the model architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of a publicly available code repository or demo URL limits the ease with which other researchers can replicate the results.
Limitations
One limitation is the reliance on the quality of the initial band-limited datasets, which could affect the overall performance of the model. Additionally, while the subjective evaluations show improvements over baselines, the small sample size and high variance in listener ratings suggest that further validation with larger participant groups is necessary to substantiate claims of superiority.
Broader impact
The potential applications of PromptReverb are significant, particularly in fields such as virtual reality, architectural acoustics, and audio production. By enabling intuitive natural language control over acoustic properties, the framework could democratize access to high-quality audio synthesis, making it more accessible to non-experts and enhancing creative workflows in various domains. The main contribution of this paper is the introduction of PromptReverb, a novel two-stage generative framework for room impulse response generation that leverages multimodal inputs and advanced machine learning techniques to overcome existing limitations in the field. This work significantly advances the state of the art in audio synthesis and has the potential to impact various practical applications in immersive audio environments.
Author of this paper? Send a correction.