StyleStream 2.0: Fast and Controllable Streaming Voice Style Conversion
Abstract
Voice style conversion (VSC) aims to transform a source utterance to match the timbre, accent, and emotion of a target voice while preserving linguistic content. StyleStream 1.0 introduced the first streamable zero-shot VSC system with state-of-the-art conversion quality, but remained limited in two ways: approximately 1s end-to-end latency on consumer hardware, and the lack of a text-based interface for designing or editing target voices. We present StyleStream 2.0, a fast and controllable streaming VSC framework that addresses these limitations. To reduce latency, we train a few-step pixel MeanFlow model and further fine-tune it with autoregressive feedback inspired by Self Forcing, improving robustness under streaming inference. This reduces end-to-end latency to 520~ms on a consumer GPU, a 2x speedup over StyleStream 1.0, while maintaining conversion quality. To enable controllable style manipulation, we remove the mel-spectrogram context and instead route target style information through a compact style embedding. In this embedding space, we train a unified text-conditioned flow matching model that supports both text-based voice design, which maps natural language prompts to voice styles, and instruction-based voice editing, which modifies a specific attribute of a source utterance while preserving the rest. Experiments show that StyleStream 2.0 achieves the strongest target style fidelity on VSC, voice design, and voice editing, while remaining competitive on intelligibility and non-edited attribute preservation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
StyleStream 2.0 advances the field of streaming voice style conversion by achieving a 2x latency reduction through few-step pixel MeanFlow and autoregressive fine-tuning, while introducing a unified text-conditioned framework for controllable voice design and editing. The paper demonstrates that holistic style conversion can be made both fast and controllable, bridging the gap between high-quality offline conversion and real-time interactive applications.
Methodology
The paper proposes a two-stage framework for streaming Voice Style Conversion (VSC) that effectively decouples content from style. The core methodological contribution is the replacement of the multi-step diffusion stylizer with a few-step pixel MeanFlow (pMF) model, which is further fine-tuned using an autoregressive feedback mechanism inspired by Self Forcing. This addresses the critical train-test mismatch in streaming inference where the model must condition on its own imperfect past outputs. Additionally, the authors introduce a unified text-conditioned DiT "Voice Controller" that operates in a compact style-embedding space, enabling both zero-shot voice design from text prompts and attribute-specific editing (age, gender, accent, emotion). The architectural choice to remove mel-spectrogram context and rely solely on a perceiver-resampler encoded style embedding is a significant design decision that facilitates the text-based control interface.
Results & evaluation
The experimental evaluation is comprehensive, covering reference-conditioned VSC, text-based voice design, and voice editing. The authors compare against strong baselines including CosyVoice 2.0/3.0, Vevo 2.0, and their own StyleStream 1.0. Results demonstrate a 2x latency reduction (520ms vs ~1s) while maintaining or improving style fidelity metrics (S-SIM, A-SIM, E-SIM). The subjective evaluation (MOS) is well-conducted with 40 participants, showing competitive naturalness and improved style consistency. The ablation studies effectively isolate the contributions of the autoregressive fine-tuning and the new style encoder, confirming their necessity for streaming robustness and style accuracy.
Reproducibility
The paper provides substantial detail on the training data construction (CapSpeech-derived corpus), the specific models used for evaluation (VoxProfile, Whisper), and the inference settings (NFE, CFG scale, chunk sizes). The inclusion of the demo URL and detailed appendices on data generation and evaluation protocols enhances reproducibility. However, the specific hyperparameters for the DiT and MeanFlow training are not fully detailed in the main text, though referenced in appendices.
Limitations
The system is evaluated primarily on English speech, and the generalization to other languages is not discussed. The text-based control is limited to four specific attributes (age, gender, accent, emotion), which may not cover the full spectrum of vocal qualities users might wish to control (e.g., pitch, speaking rate, specific vocal textures). The reliance on a frozen HuBERT encoder for content extraction may limit the system's ability to handle highly noisy or non-speech audio inputs.
Broader impact
This work has significant implications for real-time interactive speech systems, such as live dubbing, accessibility tools, and voice assistants. The ability to control voice style via natural language prompts lowers the barrier to entry for non-expert users and enables more intuitive interaction with speech synthesis systems. The reduction in latency makes streaming VSC practical for consumer hardware, potentially enabling new applications in gaming, virtual reality, and remote communication. StyleStream 2.0 advances the field of streaming voice style conversion by achieving a 2x latency reduction through few-step pixel MeanFlow and autoregressive fine-tuning, while introducing a unified text-conditioned framework for controllable voice design and editing. The paper demonstrates that holistic style conversion can be made both fast and controllable, bridging the gap between high-quality offline conversion and real-time interactive applications.
Author of this paper? Send a correction.