MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Abstract
Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streaming zero-shot VC, but it has several limitations: its chunk-wise autoregressive denoising doubles the effective training sequence length, conversion quality degrades under small-chunk settings, and its timbre encoder directly relies on reference mel-spectrograms, making it sensitive to reference audio quality. To address these limitations we propose MeanVC 2. We introduce future-receptive chunking (FRC), which explicitly schedules past and future receptive fields across diffusion transformer decoder layers and removes clean-chunk teacher forcing. By incorporating bounded future context, FRC enables stable conversion with a 40 ms chunk size. We further introduce a universal timbre token encoder, which constructs a timbre representation from a global speaker embedding and retrieves fine-grained timbre cues via cross-attention, improving robustness to low-quality references and enhancing zero-shot speaker similarity. Experimental results show that MeanVC 2 significantly outperforms MeanVC, while reducing latency from 211 ms to 110 ms. Audio samples are publicly available. The source code will be publicly released.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MeanVC 2, a robust low-latency streaming zero-shot voice conversion system that significantly improves upon its predecessor by addressing key limitations through innovative methodologies. The technical contributions, particularly in the areas of training efficiency and robustness to low-quality audio, position this work as a valuable advancement in the field of audio processing and machine learning.
Methodology
The paper introduces two significant innovations: Future-Receptive Chunking (FRC) and the Universal Timbre Token Encoder (UTTE). FRC enhances the training efficiency and stability of voice conversion by optimizing the attention mechanism across chunks, allowing for effective processing with smaller chunk sizes. UTTE decouples timbre extraction from the reliance on high-quality reference audio, improving robustness against low-quality inputs. This dual approach addresses critical limitations of the previous MeanVC model, making the methodology both innovative and practical for real-time applications.
Results & evaluation
The authors conducted extensive experiments comparing MeanVC 2 with its predecessor and other baseline systems. The results demonstrate significant improvements in audio quality, speaker similarity, and reduced latency, with MeanVC 2 achieving a first-packet latency of 110 ms compared to 211 ms for MeanVC. The use of subjective metrics (NMOS, SMOS) and objective metrics (CER, DNSMOS) provides a comprehensive evaluation of performance, reinforcing the effectiveness of the proposed methods.
Reproducibility
The paper mentions that the source code will be publicly released, which is a positive indicator for reproducibility. However, detailed implementation specifics, such as hyperparameters and training configurations, are not exhaustively covered, which could pose challenges for independent replication of the results.
Limitations
While the proposed methods show substantial improvements, the paper does not address potential limitations in terms of the generalizability of the model across diverse languages or accents. Additionally, the reliance on a specific dataset (Emilia corpus) may limit the applicability of the findings to other datasets or real-world scenarios.
Broader impact
The advancements in low-latency voice conversion have significant implications for various applications, including real-time communication, dubbing, and assistive technologies for individuals with speech impairments. The ability to perform zero-shot voice conversion with improved robustness could enhance user experiences in virtual environments and multimedia content creation. The main contribution of this paper is the introduction of MeanVC 2, a robust low-latency streaming zero-shot voice conversion system that significantly improves upon its predecessor by addressing key limitations through innovative methodologies. The technical contributions, particularly in the areas of training efficiency and robustness to low-quality audio, position this work as a valuable advancement in the field of audio processing and machine learning.
Author of this paper? Send a correction.