RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models
Abstract
Modern voice cloning (VC) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, and diverse downstream processing, which can significantly hurt robustness. Despite rapid progress in VC driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive benchmark that evaluates Robustness in VC across the full generation pipeline, including input variation, generation challenges, output post-processing, and adversarial perturbations, covering 10 robustness tasks, 225 speakers, 14,370 utterances, and 11 representative modern VC models. Our evaluation uncovers substantial robustness gaps in VC: performance can deteriorate sharply under common input shifts and post-processing; long-context and cross-lingual scenarios further expose stability limitations; and both passive noise and proactive perturbation influence generation robustness. Collectively, these findings provide a unified picture of how current VC models fail in practice and introduce a standardized, open-source testbed to support the development of more robust and deployable VC models. We open-source our project at https://github.com/Nanboy-Ronan/RVCBench.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of RVCBench, a comprehensive benchmark for evaluating the robustness of voice cloning models under realistic conditions. This work significantly advances the understanding of the limitations of current voice cloning technologies and provides a valuable resource for future research aimed at improving their robustness and applicability.
Methodology
The paper introduces RVCBench, a benchmark designed to evaluate the robustness of voice cloning models across various challenges. The methodology is comprehensive, covering a wide range of robustness tasks and including a significant dataset of 225 speakers and over 14,000 utterances. The authors systematically assess the performance of 11 modern voice cloning models under different conditions, which is a valuable approach to understanding the limitations of current technology. However, the paper could benefit from a more detailed explanation of how the robustness tasks were selected and the specific metrics used for evaluation.
Results & evaluation
The experiments are well-structured, with a clear focus on identifying performance gaps in voice cloning models under realistic conditions. The inclusion of various input variations and adversarial perturbations is a strong point, as it reflects real-world challenges. The results highlight significant robustness issues, which are crucial for advancing the field. However, the paper lacks a comparative analysis with existing benchmarks, which would strengthen its contributions.
Reproducibility
The paper mentions that the project is open-sourced, which is a positive aspect for reproducibility. However, it lacks detailed implementation instructions or specific configurations used during experiments, which could hinder other researchers from replicating the results effectively.
Limitations
One limitation is the potential bias in the selection of speakers and utterances, which may not represent the full diversity of voice characteristics in the real world. Additionally, while the benchmark covers various robustness tasks, it may not encompass all possible deployment scenarios that could affect voice cloning performance.
Broader impact
The findings of this paper have significant implications for the development of more robust voice cloning technologies, which could enhance applications in personalized speech interfaces and dubbing. By identifying and addressing robustness gaps, the research can contribute to safer and more reliable deployment of voice cloning systems in real-world applications. The main contribution of this paper is the introduction of RVCBench, a comprehensive benchmark for evaluating the robustness of voice cloning models under realistic conditions. This work significantly advances the understanding of the limitations of current voice cloning technologies and provides a valuable resource for future research aimed at improving their robustness and applicability.
Author of this paper? Send a correction.