Train Short, Infer Long: Speech-LLM Enables Zero-Shot Streamable Joint ASR and Diarization on Long Audio
Abstract
Joint automatic speech recognition (ASR) and speaker diarization aim to answer the question "who spoke what" in multi-speaker scenarios. In this paper, we present an end-to-end speech large language model (Speech-LLM) for Joint strEamable DIarization and aSr (JEDIS-LLM). The model is trained only on short audio under 20s but is capable of streamable inference on long-form audio without additional training. This is achieved by introducing a Speaker Prompt Cache (SPC) with an on-the-fly update mechanism during chunk-wise streaming inference, inspired by the autoregressive nature of LLMs. The SPC also allows the seamless use of pre-enrolled speaker profiles which is common in many scenarios like meeting transcription. To further enhance diarization capability, we incorporate word-level speaker supervision into the speech encoder during training. Experimental results demonstrate that our system outperforms strong baselines, including Sortformer and Meta-Cat in the local setting on audio up to 20s, and DiarizationLM on long-form audio, despite being fully end-to-end and streamable while DiarizationLM follows a cascaded offline pipeline. To the best of our knowledge, this is the first work enabling zero-shot streamable joint ASR and diarization on long audio using a Speech-LLM trained only on short audio, achieving state-of-the-art performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a pioneering approach to joint ASR and diarization using a Speech-LLM trained on short audio, achieving state-of-the-art performance in long audio scenarios. The innovative methodology and strong experimental results position this work as a meaningful contribution to the field of audio processing and machine learning.
Methodology
The proposed methodology introduces a novel end-to-end Speech-LLM that leverages a Speaker Prompt Cache (SPC) for streamable inference on long audio. The SPC mechanism is innovative, allowing for real-time updates and the integration of pre-enrolled speaker profiles, which is a significant advancement in ASR and diarization tasks. The incorporation of word-level speaker supervision into the speech encoder is also a noteworthy methodological enhancement that likely improves diarization accuracy.
Results & evaluation
The experiments are well-structured, demonstrating the model's performance against established baselines like Sortformer, Meta-Cat, and DiarizationLM. The results indicate a strong performance in both short and long audio scenarios, showcasing the model's versatility and effectiveness. However, the paper could benefit from more detailed statistical analysis and comparisons across different datasets to strengthen the claims of superiority.
Reproducibility
The paper lacks sufficient detail regarding the implementation specifics, such as the architecture of the Speech-LLM, training procedures, and dataset descriptions. This omission raises concerns about the reproducibility of the results. Providing access to code or a detailed methodology section would significantly enhance reproducibility.
Limitations
One limitation is the reliance on short audio training data, which may affect the model's generalizability to diverse audio environments. Additionally, the performance metrics could be expanded to include more comprehensive evaluations across various speaker configurations and acoustic conditions.
Broader impact
The implications of this work are significant, particularly in applications such as meeting transcription, customer service, and any domain requiring accurate speaker identification in multi-speaker environments. The ability to perform zero-shot streamable ASR and diarization could lead to more efficient and user-friendly audio processing systems. The paper presents a pioneering approach to joint ASR and diarization using a Speech-LLM trained on short audio, achieving state-of-the-art performance in long audio scenarios. The innovative methodology and strong experimental results position this work as a meaningful contribution to the field of audio processing and machine learning.
Author of this paper? Send a correction.