TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
Abstract
We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of TagSpeech, a unified framework for multi-speaker ASR and diarization that effectively addresses fine-grained speaker-content alignment through innovative methodologies. This paper makes a substantial technical contribution to the field of audio processing, particularly in enhancing the accuracy and efficiency of multi-speaker systems.
Methodology
The methodology presented in TagSpeech is innovative, particularly in its use of Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The decoupled semantic and speaker streams, along with the Serialized Output Training (SOT), provide a structured approach to learning turn-taking dynamics. The interleaved time anchor mechanism is a significant contribution that enhances the system's ability to synchronize semantic understanding with speaker tracking, addressing a critical gap in existing methods. The end-to-end nature of the framework, which explicitly models speaker-content alignment, is a notable advancement over previous works that have largely focused on either ASR or implicit diarization.
Results & evaluation
The experiments conducted on the AMI and AliMeeting benchmarks are robust, demonstrating consistent improvements in Diarization Error Rate (DER) compared to strong baselines like Qwen-Omni and Gemini. The focus on complex speech overlaps is particularly relevant, as this is a common challenge in multi-speaker environments. The results indicate that TagSpeech not only achieves better performance but does so with a parameter-efficient training paradigm, which is a critical consideration for practical applications.
Reproducibility
The paper provides a GitHub repository link, which is essential for reproducibility. However, the paper should ideally include more detailed implementation instructions, hyperparameter settings, and any specific configurations used during training and evaluation to facilitate easier reproduction of results by other researchers.
Limitations
While the framework shows promise, the paper does not sufficiently address potential limitations, such as the scalability of the approach to larger datasets or more diverse speaker profiles. Additionally, the reliance on a frozen LLM backbone may limit the adaptability of the model to new domains or languages without further fine-tuning.
Broader impact
The implications of TagSpeech are significant, particularly in applications requiring accurate transcription and speaker identification in multi-speaker scenarios, such as meetings, interviews, and media production. The advancements in handling speech overlaps could lead to better accessibility and usability of ASR technologies in real-world applications. The main contribution of this work is the introduction of TagSpeech, a unified framework for multi-speaker ASR and diarization that effectively addresses fine-grained speaker-content alignment through innovative methodologies. This paper makes a substantial technical contribution to the field of audio processing, particularly in enhancing the accuracy and efficiency of multi-speaker systems.
Author of this paper? Send a correction.