Smule Renaissance Small: Efficient General-Purpose Vocal Restoration
Abstract
Vocal recordings on consumer devices commonly suffer from multiple concurrent degradations: noise, reverberation, band-limiting, and clipping. We present Smule Renaissance Small (SRS), a compact single-stage model that performs end-to-end vocal restoration directly in the complex STFT domain. By incorporating phase-aware losses, SRS enables large analysis windows for improved frequency resolution while achieving 10.5x real-time inference on iPhone 12 CPU at 48 kHz. On the DNS 5 Challenge blind set, despite no speech training, SRS outperforms a strong GAN baseline and closely matches a computationally expensive flow-matching system. To enable evaluation under realistic multi-degradation scenarios, we introduce the Extreme Degradation Bench (EDB): 87 singing and speech recordings captured under severe acoustic conditions. On EDB, SRS surpasses all open-source baselines on singing and matches commercial systems, while remaining competitive on speech despite no speech-specific training. We release both SRS and EDB under the MIT License.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Smule Renaissance Small (SRS), a novel single-stage model for efficient vocal restoration that operates directly in the complex STFT domain, achieving competitive performance against more complex systems while being optimized for real-time inference on consumer devices. This work represents a significant advancement in the field of audio processing, particularly for applications requiring robust restoration under challenging acoustic conditions.
Methodology
The methodology presented in the paper is innovative, leveraging a compact single-stage model that operates directly in the complex STFT domain, which is a departure from traditional two-stage approaches. The incorporation of phase-aware losses and a band-split generator allows for improved frequency resolution and efficiency, making the model suitable for real-time applications on consumer devices. The design choices, such as the general-purpose corruption module and the use of a temporal-convolutional backbone, demonstrate a thoughtful approach to addressing the challenges of multi-degradation vocal restoration.
Results & evaluation
The experimental evaluation is robust, utilizing both objective metrics and subjective assessments to compare the proposed SRS model against existing systems. The introduction of the Extreme Degradation Bench (EDB) provides a valuable resource for evaluating vocal restoration under realistic conditions. The results indicate that SRS outperforms several strong baselines, which is a significant achievement given its single-stage architecture and lack of speech-specific training.
Reproducibility
The authors have committed to releasing both the SRS model and the EDB dataset under the MIT License, which enhances reproducibility. However, the paper could benefit from more detailed implementation specifics, such as hyperparameter settings and training procedures, to facilitate independent replication of results.
Limitations
One limitation is that while SRS performs well on singing restoration, its performance on speech restoration is comparatively weaker, particularly given that it was not trained specifically on speech data. Additionally, the reliance on consumer devices for inference may limit the model's applicability in more demanding professional settings.
Broader impact
The potential applications of SRS are significant, particularly in the context of mobile audio processing, where real-time vocal restoration can enhance user experiences in various consumer applications, such as music production and podcasting. The release of the EDB dataset also encourages further research in the domain of audio restoration, paving the way for advancements in this area. The main contribution of this paper is the introduction of Smule Renaissance Small (SRS), a novel single-stage model for efficient vocal restoration that operates directly in the complex STFT domain, achieving competitive performance against more complex systems while being optimized for real-time inference on consumer devices. This work represents a significant advancement in the field of audio processing, particularly for applications requiring robust restoration under challenging acoustic conditions.
Author of this paper? Send a correction.