UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
Abstract
Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive evaluations show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baseline tokenizers on both understanding and generation tasks, effectively serving as a unified audio interface. We publicly release all our code, including training and inference scripts, together with the model checkpoints at https://github.com/Tencent/Universal_Audio_Tokenizer.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents UniAudio-Token, a framework that enhances semantic speech tokenizers with general audio perception, effectively bridging the gap between linguistic and acoustic understanding in audio-LLMs. The technical contributions are substantial, with innovative methodologies and rigorous evaluations demonstrating the framework's effectiveness across diverse audio tasks.
Methodology
The proposed methodology introduces two innovative components: Semantic-Acoustic Primitives (SAP) for structured supervision and Semantic-Acoustic Equilibrium (SAE) for adaptive feature fusion. This dual approach effectively addresses the limitations of existing semantic tokenizers by disentangling linguistic content from acoustic details, thereby enhancing the model's ability to process both speech and general audio. The methodology is well-structured, with clear explanations of the components and their interactions, supported by thorough theoretical backing.
Results & evaluation
The experimental evaluation is comprehensive, utilizing a variety of benchmarks and metrics to assess the performance of UniAudio-Token against several baseline models. Results demonstrate significant improvements in both understanding and generation tasks across diverse audio types, validated through quantitative metrics (e.g., WER, MOS) and qualitative analyses (e.g., t-SNE visualizations). The inclusion of ablation studies further strengthens the findings by isolating the contributions of the SAP and SAE mechanisms.
Reproducibility
The authors have provided a detailed account of their training process, including hyperparameters and datasets used, which enhances reproducibility. They also released code and model checkpoints, facilitating further experimentation by other researchers. However, the reliance on specific pre-trained models (e.g., Whisper) may limit the ease of replication for those without access to these resources.
Limitations
The paper acknowledges limitations in waveform-level reconstruction quality for complex non-speech audio compared to specialized codecs, and the current training primarily covers English and Chinese, which may restrict its applicability to other languages. Additionally, the compact design may introduce trade-offs in fidelity for certain audio types.
Broader impact
The advancements presented in this work have the potential to significantly enhance audio understanding and generation in various applications, including speech synthesis, environmental sound classification, and multimodal interaction systems. The framework's ability to integrate linguistic and acoustic features could lead to more robust audio-LLMs, improving user experiences in voice assistants, content creation, and accessibility technologies. The paper presents UniAudio-Token, a framework that enhances semantic speech tokenizers with general audio perception, effectively bridging the gap between linguistic and acoustic understanding in audio-LLMs. The technical contributions are substantial, with innovative methodologies and rigorous evaluations demonstrating the framework's effectiveness across diverse audio tasks.
Author of this paper? Send a correction.