UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
Abstract
Large language models (LLMs) have demonstrated promising performance in both automatic speech recognition (ASR) and text-to-speech (TTS) systems, gradually becoming the mainstream approach. However, most current approaches address these tasks separately rather than through a unified framework. This work aims to integrate these two tasks into one unified model. Although discrete speech tokenization enables joint modeling, its inherent information loss limits performance in both recognition and generation. In this work, we present UniVoice, a unified LLM framework through continuous representations that seamlessly integrates speech recognition and synthesis within a single model. Our approach combines the strengths of autoregressive modeling for speech recognition with flow matching for high-quality generation. To mitigate the inherent divergence between autoregressive and flow-matching models, we further design a dual attention mechanism, which switches between a causal mask for recognition and a bidirectional attention mask for synthesis. Furthermore, the proposed text-prefix-conditioned speech infilling method enables high-fidelity zero-shot voice cloning. Experimental results demonstrate that our method can achieve or exceed current single-task modeling methods in both ASR and zero-shot TTS tasks. This work explores new possibilities for end-to-end speech understanding and generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents UniVoice, a unified transformer framework that integrates autoregressive speech recognition with flow-matching-based speech synthesis. This work is significant as it explores a novel approach to joint modeling in speech processing, addressing critical limitations in current methodologies and demonstrating robust performance across multiple tasks.
Methodology
The methodology presented in this paper is innovative, as it proposes a unified framework that integrates autoregressive speech recognition with flow-matching-based synthesis. The dual attention mechanism and text-prefix-guided speech infilling method are significant contributions that address the limitations of existing models that treat ASR and TTS as separate tasks. The continuous representation approach is a notable departure from traditional discrete tokenization methods, which often suffer from information loss. The paper also provides a clear description of the model architecture, training objectives, and attention mask design, which enhances the understanding of the proposed methods.
Results & evaluation
The experimental evaluation is thorough, utilizing the LibriHeavy dataset for both ASR and TTS tasks. The results demonstrate that UniVoice achieves competitive performance compared to state-of-the-art models in both domains, with specific metrics provided for robustness, similarity, and quality. The ablation studies effectively showcase the advantages of the proposed methods over baseline models, although the paper acknowledges trade-offs in performance when compared to specialized models.
Reproducibility
The paper provides sufficient implementation details, including model architecture, training procedures, and evaluation metrics, which supports reproducibility. The authors also mention plans to open-source the code and checkpoints, which is a positive step towards enabling other researchers to replicate and build upon their work.
Limitations
The paper identifies several limitations, including the focus on only ASR and TTS tasks, the relatively small dataset and model size, and the underutilization of the conversational capabilities of LLMs. These limitations suggest that while the work is a significant step forward, there is potential for further development and exploration in future research.
Broader impact
The unified framework proposed in this paper has the potential to advance the field of speech processing by enabling more efficient and effective models that can handle both recognition and synthesis tasks. This could lead to improvements in applications such as virtual assistants, automated transcription services, and voice cloning technologies, ultimately enhancing user experience and accessibility in various domains. The paper presents UniVoice, a unified transformer framework that integrates autoregressive speech recognition with flow-matching-based speech synthesis. This work is significant as it explores a novel approach to joint modeling in speech processing, addressing critical limitations in current methodologies and demonstrating robust performance across multiple tasks.
Author of this paper? Send a correction.