CLAR: CIF-Localized Alignment for Retrieval-Augmented Speech LLM-Based Contextual ASR
Abstract
Speech LLM-based ASR often struggles with named entities and long-tail words due to strong internal language-model priors. Retrieval-augmented biasing can help, but its effectiveness depends on accurate hotword localization in full-utterance speech under weak supervision. We propose CLAR, a dual-encoder speech-text retriever that uses Continuous Integrate-and-Fire (CIF) to learn monotonic token-level alignments without timestamps. With length-aware localized matching, CLAR anchors short-entity acoustic cues and reduces representation dilution and attention drift. The retriever is trained with a multi-granularity objective combining global and local segment-level contrastive losses and a CIF quantity constraint. At inference, top-ranked hotwords are injected as contextual prompts for the Speech LLM, improving recognition without shallow fusion. Experiments show that CLAR significantly improves hotword retrieval and reduces both CER and B-WER against strong contextual ASR baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents CLAR, a novel dual-encoder retrieval system that enhances contextual ASR by effectively localizing hotwords without timestamp supervision. The innovative methodology and significant experimental results position CLAR as a valuable contribution to the field of speech recognition and retrieval-augmented systems.
Methodology
The paper introduces CLAR, a dual-encoder speech-text retriever that utilizes a Continuous Integrate-and-Fire (CIF) mechanism for monotonic token-level alignment without timestamps. The methodology is innovative, particularly in its approach to localized matching and the multi-granularity training objective that combines various contrastive losses. The use of CIF for hotword retrieval under weak supervision is a significant advancement in addressing the challenges of named entity recognition in ASR systems.
Results & evaluation
The experiments are well-structured, utilizing relevant datasets (AISHELL-1 and AISHELL-2) and metrics (CER, B-WER, recall, F1 score) to evaluate the performance of CLAR against strong baselines. The results demonstrate significant improvements in hotword retrieval and ASR accuracy, validating the effectiveness of the proposed method. However, the paper could benefit from additional comparative analyses with more recent state-of-the-art methods.
Reproducibility
The paper provides sufficient details regarding the architecture, training procedures, and evaluation metrics, which would allow for reproducibility. However, the lack of a publicly available code repository or demo limits accessibility for further validation by the research community.
Limitations
The paper does not address potential limitations in terms of scalability to larger datasets or multilingual settings, which could affect the generalizability of the findings. Additionally, the reliance on weak supervision may introduce challenges in alignment accuracy, particularly in noisy environments.
Broader impact
The advancements presented in this paper have significant implications for improving ASR systems in real-world applications, particularly in domains requiring accurate recognition of low-frequency words and named entities. The modular nature of CLAR allows for integration with various Speech LLMs, potentially enhancing user interactions in conversational AI systems. The paper presents CLAR, a novel dual-encoder retrieval system that enhances contextual ASR by effectively localizing hotwords without timestamp supervision. The innovative methodology and significant experimental results position CLAR as a valuable contribution to the field of speech recognition and retrieval-augmented systems.
Author of this paper? Send a correction.