MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows
Abstract
Zero-shot voice conversion (VC) aims to transfer timbre from a source speaker to any unseen target speaker while preserving linguistic content. Growing application scenarios demand models with streaming inference capabilities. This has created a pressing need for models that are simultaneously fast, lightweight, and high-fidelity. However, existing streaming methods typically rely on either autoregressive (AR) or non-autoregressive (NAR) frameworks, which either require large parameter sizes to achieve strong performance or struggle to generalize to unseen speakers. In this study, we propose MeanVC, a lightweight and streaming zero-shot VC approach. MeanVC introduces a diffusion transformer with a chunk-wise autoregressive denoising strategy, combining the strengths of both AR and NAR paradigms for efficient streaming processing. By introducing mean flows, MeanVC regresses the average velocity field during training, enabling zero-shot VC with superior speech quality and speaker similarity in a single sampling step by directly mapping from the start to the endpoint of the flow trajectory. Additionally, we incorporate diffusion adversarial post-training to mitigate over-smoothing and further enhance speech quality. Experimental results demonstrate that MeanVC significantly outperforms existing zero-shot streaming VC systems, achieving superior conversion quality with higher efficiency and significantly fewer parameters. Audio demos and code are publicly available at https://aslp-lab.github.io/MeanVC.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MeanVC, a lightweight and efficient framework for streaming zero-shot voice conversion that significantly improves audio quality and computational efficiency. The work is a meaningful addition to the field, addressing critical challenges in voice conversion technology while paving the way for practical applications in real-time scenarios.
Methodology
The proposed MeanVC framework effectively combines autoregressive and non-autoregressive paradigms through a chunk-wise autoregressive denoising strategy and mean flows for efficient spectrogram synthesis. The introduction of diffusion adversarial post-training is a notable enhancement aimed at addressing over-smoothing artifacts, which is a common issue in generative models. The methodology is well-structured, leveraging existing architectures while innovatively addressing their limitations, particularly in terms of efficiency and quality in zero-shot voice conversion.
Results & evaluation
The experiments are comprehensive, utilizing a substantial dataset (10,000 hours of Mandarin data) and a well-defined evaluation setup that includes both subjective and objective metrics. The results demonstrate that MeanVC outperforms existing models in terms of audio quality, efficiency, and parameter count, which is critical for real-time applications. The comparisons with baseline models are thorough, providing a clear picture of MeanVC's advantages.
Reproducibility
The paper provides sufficient detail regarding the architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of a public code repository limits the ease of reproduction, despite the demo URL being available.
Limitations
While the paper presents significant advancements, it acknowledges that MeanVC's performance in DNSMOS is lower than that of Seed-VC, which could be attributed to its smaller parameter size. Additionally, the reliance on chunk sizes may introduce challenges in maintaining contextual integrity, particularly in real-time applications.
Broader impact
The advancements in zero-shot voice conversion have implications for various applications, including personalized voice assistants, dubbing in media, and privacy-preserving technologies. The lightweight nature of MeanVC makes it suitable for real-time applications, potentially broadening its adoption in commercial products. The main contribution of this paper is the introduction of MeanVC, a lightweight and efficient framework for streaming zero-shot voice conversion that significantly improves audio quality and computational efficiency. The work is a meaningful addition to the field, addressing critical challenges in voice conversion technology while paving the way for practical applications in real-time scenarios.
Author of this paper? Send a correction.