DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion
Abstract
Speech tokenizers serve as the cornerstone of discrete Speech Large Language Models (Speech LLMs). Existing tokenizers either prioritize semantic encoding, fuse semantic content with acoustic style inseparably, or achieve incomplete semantic-acoustic disentanglement. To achieve better disentanglement, we propose DSA-Tokenizer, which explicitly disentangles speech into discrete semantic and acoustic tokens via distinct optimization constraints. Specifically, semantic tokens are supervised by ASR to capture linguistic content, while acoustic tokens focus on mel-spectrograms restoration to encode style. To eliminate rigid length constraints between the two sequences, we introduce a hierarchical Flow-Matching decoder that further improve speech generation quality. Furthermore, We employ a joint reconstruction-recombination training strategy to enforce this separation. DSA-Tokenizer enables high fidelity reconstruction and flexible recombination through robust disentanglement, facilitating controllable generation in speech LLMs. Our analysis highlights disentangled tokenization as a pivotal paradigm for future speech modeling. Audio samples are avaialble at https://anonymous.4open.science/w/DSA_Tokenizer_demo/. The code and model will be made publicly available after the paper has been accepted.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The DSA-Tokenizer presents a novel approach to speech tokenization by achieving effective semantic-acoustic disentanglement, which significantly enhances the controllability and quality of speech generation in large language models. This work not only advances the state-of-the-art in speech processing but also opens avenues for future research in audio modeling and ethical considerations surrounding voice synthesis technologies.
Methodology
The DSA-Tokenizer introduces a dual-stream architecture that effectively disentangles semantic and acoustic tokens through distinct optimization constraints. This methodology is innovative as it combines ASR supervision for semantic tokens and mel-spectrogram restoration for acoustic tokens, addressing the limitations of existing tokenizers that either mix these attributes or fail to achieve complete disentanglement. The introduction of a hierarchical Flow-Matching decoder and a joint reconstruction-recombination training strategy further enhances the model's capabilities, allowing for flexible recombination of speech attributes.
Results & evaluation
The experiments conducted are comprehensive, utilizing a variety of datasets and evaluation metrics to assess both reconstruction fidelity and cross-utterance recombination capabilities. The results demonstrate that DSA-Tokenizer outperforms existing models in key metrics such as UTMOS scores, WER, and speaker similarity, indicating its effectiveness in achieving high-quality speech generation while maintaining semantic integrity.
Reproducibility
The paper provides detailed training procedures, model architectures, and evaluation setups, which enhance reproducibility. However, the actual code and model are not yet publicly available, which could hinder independent verification of results until they are released.
Limitations
The paper acknowledges limitations such as inference latency due to the deep architecture of the decoder, which may not be suitable for real-time applications. Additionally, the model's applicability to other audio modalities beyond speech has not been explored, suggesting a narrow focus that could limit broader applicability.
Broader impact
The DSA-Tokenizer has significant implications for the development of controllable speech generation systems, particularly in applications like voice cloning and speech synthesis. However, the potential for misuse in generating deepfakes raises ethical concerns that necessitate the development of detection and mitigation strategies. The DSA-Tokenizer presents a novel approach to speech tokenization by achieving effective semantic-acoustic disentanglement, which significantly enhances the controllability and quality of speech generation in large language models. This work not only advances the state-of-the-art in speech processing but also opens avenues for future research in audio modeling and ethical considerations surrounding voice synthesis technologies.
Author of this paper? Send a correction.