Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers
Abstract
Traditional speech systems typically rely on separate, task-specific models for text-to-speech (TTS), automatic speech recognition (ASR), and voice conversion (VC), resulting in fragmented pipelines that limit scalability, efficiency, and cross-task generalization. In this paper, we present General-Purpose Audio (GPA), a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture. GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly perform TTS, ASR, and VC without architectural modifications. This unified design combines a fully autoregressive formulation over discrete speech tokens, joint multi-task training across speech domains, and a scalable inference pipeline that achieves high concurrency and throughput. The resulting model family supports efficient multi-scale deployment, including a lightweight 0.3B-parameter variant optimized for edge and resource-constrained environments. Together, these design choices demonstrate that a unified autoregressive architecture can achieve competitive performance across diverse speech tasks while remaining viable for low-latency, practical deployment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents GPA, a unified autoregressive framework for speech tasks that consolidates TTS, ASR, and VC into a single architecture, demonstrating competitive performance and efficiency. The innovative methodology and comprehensive evaluation contribute meaningfully to the advancement of speech processing technologies, with potential for significant real-world applications.
Methodology
The paper introduces the General-Purpose Audio (GPA) model, which unifies TTS, ASR, and VC tasks within a single autoregressive framework. The methodology is innovative in its use of a shared discrete audio token space and instruction-driven task induction, allowing seamless switching between tasks without architectural modifications. The dual-tokenizer scheme enhances the model's ability to represent both semantic and acoustic information effectively. The joint multi-task training approach is well-justified, leveraging shared representations to improve performance across tasks. However, the reliance on a single autoregressive architecture may limit the model's ability to achieve peak performance in highly specialized tasks.
Results & evaluation
The empirical evaluation is comprehensive, covering training dynamics, performance metrics, and scalability across different model sizes. The authors provide quantitative results on benchmarks for TTS and ASR, demonstrating competitive performance relative to existing models. The evaluation of latency and throughput under streaming conditions is particularly relevant for real-world applications, showcasing the model's efficiency. However, the lack of separate evaluation metrics for voice conversion may limit the understanding of its performance in that domain.
Reproducibility
The authors emphasize reproducibility by providing complete training code and configuration files on a public GitHub repository. This transparency is commendable and facilitates further research and validation of their results. The detailed description of the training protocol, data composition, and sample construction strategies enhances the reproducibility of the study.
Limitations
The paper acknowledges several limitations, including the potential bottleneck introduced by the unified architecture, which may hinder performance on specialized tasks. Additionally, the inference cost and latency scaling with sequence length pose challenges for certain applications. The ASR performance of the lightweight model variant is noted to be comparatively weaker, indicating that further optimization and scaling could yield improvements.
Broader impact
The GPA model has significant implications for the field of speech processing, as it addresses the fragmentation of existing systems by providing a unified framework for multiple speech tasks. This could lead to more efficient deployment in real-time applications, particularly in resource-constrained environments. The model's ability to generalize across tasks may also foster advancements in cross-task learning and knowledge transfer in speech technologies. The paper presents GPA, a unified autoregressive framework for speech tasks that consolidates TTS, ASR, and VC into a single architecture, demonstrating competitive performance and efficiency. The innovative methodology and comprehensive evaluation contribute meaningfully to the advancement of speech processing technologies, with potential for significant real-world applications.
Author of this paper? Send a correction.