MeanVoiceFlow2: Joint Optimization of Mean Flow and Content Encoder for Fast One-Step Zero-Shot Voice Conversion
Abstract
Flow-matching approaches to voice conversion (VC) have gained attention owing to their high speech quality and strong speaker similarity. Among them, one-step models such as MeanVoiceFlow are particularly attractive because they enable efficient inference; however, their reliance on a computationally intensive content encoder remains a bottleneck. We therefore propose MeanVoiceFlow2, a framework that jointly optimizes a flow-based conversion module and a computationally efficient content encoder. The model is trained through conversion distillation using MeanVoiceFlow and the reconstruction of real data. We further incorporate diffusion-GAN training with sample mixing and teacher-guided conditioning augmentation to enhance realism and disentanglement. Experiments on zero-shot VC showed that MeanVoiceFlow2 achieved higher perceptual quality and approximately $9\times$ faster inference than MeanVoiceFlow while maintaining comparable speaker similarity. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/meanvoiceflow2/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MeanVoiceFlow2 introduces a joint optimization framework for flow-based voice conversion that eliminates the computational bottleneck of heavy content encoders. By combining conversion distillation, real-data reconstruction, diffusion-GAN training, and teacher-guided conditioning augmentation, the model achieves a 9x speedup over its teacher while maintaining or improving perceptual quality and speaker similarity, offering a practical solution for real-time zero-shot voice conversion.
Methodology
The paper proposes MeanVoiceFlow2, a framework that addresses the computational bottleneck of content encoders in one-step flow-matching voice conversion models. The core contribution is the joint optimization of a lightweight, trainable content encoder and a student flow network, guided by a teacher model (MeanVoiceFlow). The training strategy is multi-faceted: (1) Conversion distillation to mimic the teacher's input-output behavior; (2) Real-data reconstruction to ensure fidelity to actual speech distributions; (3) Diffusion-GAN training with sample mixing to enhance perceptual realism and stabilize optimization without relying on external pretrained vocoders; and (4) Teacher-guided conditioning augmentation to improve content-speaker disentanglement. The method is technically sound, leveraging established concepts (distillation, GANs, flow matching) in a novel combination specifically tailored to the constraints of efficient VC. The removal of the heavy pretrained content encoder is a significant architectural simplification.
Results & evaluation
The experiments are comprehensive, evaluating the model on both VCTK and LibriTTS datasets for zero-shot VC. The evaluation includes objective metrics (UTMOS, DNSMOS, CER, SECS) and subjective MOS tests (naturalness and speaker similarity) with 12 participants. Ablation studies are provided for each component of the training strategy (distillation, reconstruction, adversarial training, conditioning augmentation), clearly demonstrating the contribution of each part. The comparison with the teacher model (MeanVoiceFlow) and a competing method (FasterVoiceGrad) shows that MeanVoiceFlow2 achieves comparable or better quality with a ~9x speedup in inference. The use of multiple datasets and metrics strengthens the validity of the results.
Reproducibility
The paper provides detailed descriptions of the network architectures (U-Net, convolutional layers), training hyperparameters (learning rate, batch size, epochs), and loss functions. The code and audio samples are available on the project page. The specific implementation details of the "adaptively weighted loss" and the "sample mixing" strategy are described, aiding reproducibility. However, some details regarding the specific initialization of the student model and the exact implementation of the discriminator might require further clarification for full reproduction.
Limitations
The model is evaluated primarily on English speech (VCTK, LibriTTS). Performance on other languages or multilingual settings is not discussed. The reliance on a teacher model (MeanVoiceFlow) for distillation means that the quality of the student is upper-bounded by the teacher, although the paper argues that the student can surpass the teacher in perceptual quality due to the GAN component. The computational cost of training (500 epochs for teacher, 250 for student) is significant, though this is a one-time cost. The paper does not discuss the impact of the lightweight content encoder on the robustness of the model to noisy inputs or different speaking styles compared to the heavy encoder.
Broader impact
The proposed method has significant implications for real-time voice conversion applications, such as live dubbing, accessibility tools, and voice cloning. By reducing the inference time by ~9x while maintaining quality, it makes high-quality VC feasible on edge devices or in latency-sensitive scenarios. The joint optimization approach could be extended to other generative tasks where heavy encoders are bottlenecks. The use of diffusion-GANs without external vocoders offers a more self-contained training pipeline, which is beneficial for research and deployment. MeanVoiceFlow2 introduces a joint optimization framework for flow-based voice conversion that eliminates the computational bottleneck of heavy content encoders. By combining conversion distillation, real-data reconstruction, diffusion-GAN training, and teacher-guided conditioning augmentation, the model achieves a 9x speedup over its teacher while maintaining or improving perceptual quality and speaker similarity, offering a practical solution for real-time zero-shot voice conversion.
Author of this paper? Send a correction.