Speaker Verification with Speech-Aware LLMs: Evaluation and Augmentation
Abstract
Speech-aware large language models (LLMs) can accept speech inputs, yet their training objectives largely emphasize linguistic content or specific fields such as emotions or the speaker's gender, leaving it unclear whether they encode speaker identity. First, we propose a model-agnostic scoring protocol that produces continuous verification scores for both API-only and open-weight models, using confidence scores or log-likelihood ratios from the Yes/No token probabilities. Using this protocol, we benchmark recent speech-aware LLMs and observe weak speaker discrimination (EERs above 20% on VoxCeleb1). Second, we introduce a lightweight augmentation that equips an LLM with ASV capability by injecting frozen ECAPA-TDNN speaker embeddings through a learned projection and training only LoRA adapters. On TinyLLaMA-1.1B, the resulting ECAPA-LLM achieves 1.03% EER on VoxCeleb1-E, approaching a dedicated speaker verification system while preserving a natural-language interface.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the development of a novel scoring protocol and augmentation technique that significantly enhances the speaker verification capabilities of speech-aware LLMs. This research addresses a critical gap in the ability of LLMs to process and discriminate speaker identity, which is essential for advancing applications in voice recognition and personalized AI systems.
Methodology
The paper introduces a model-agnostic scoring protocol for evaluating speaker verification capabilities in speech-aware LLMs, which is a significant advancement in the field. The methodology is sound, utilizing both confidence scoring and log-likelihood ratios to derive continuous verification scores. The augmentation of LLMs with ECAPA-TDNN speaker embeddings through LoRA adapters is innovative, allowing for the integration of speaker verification capabilities while maintaining the natural language interface of LLMs. The approach is well-structured and addresses a critical gap in existing LLM capabilities regarding speaker identity discrimination.
Results & evaluation
The experiments are comprehensive, benchmarking various speech-aware LLMs against the VoxCeleb1 dataset. The results demonstrate the weak performance of off-the-shelf models in speaker discrimination, with EERs exceeding 20%. The introduction of the ECAPA-LLM shows a significant improvement, achieving an EER of 1.03%, which is commendable. The evaluation metrics are appropriate, and the use of multiple trials and datasets enhances the robustness of the findings. However, the paper could benefit from more extensive comparisons with state-of-the-art ASV systems.
Reproducibility
The paper provides sufficient details regarding the datasets used, the training procedures, and the evaluation metrics, which facilitates reproducibility. However, the absence of a publicly accessible code repository limits the ability for others to replicate the results fully. Including a GitHub link or similar would enhance reproducibility significantly.
Limitations
The paper acknowledges limitations in the scoring methods, particularly the coarse nature of confidence-based scoring for closed systems and the high failure rates observed in some models. Additionally, the performance of larger models like Ministral3-3B was unexpectedly poor, suggesting potential issues with embedding space alignment that require further investigation. The reliance on specific datasets may also limit the generalizability of the findings.
Broader impact
The implications of this research are substantial, as it paves the way for integrating speaker verification capabilities into general-purpose LLMs. This could enhance applications in biometric authentication, personalized assistants, and dialogue analysis. The findings suggest a promising direction for future research in multimodal AI systems, where both linguistic and speaker identity information can be processed jointly. The main contribution of this work is the development of a novel scoring protocol and augmentation technique that significantly enhances the speaker verification capabilities of speech-aware LLMs. This research addresses a critical gap in the ability of LLMs to process and discriminate speaker identity, which is essential for advancing applications in voice recognition and personalized AI systems.
Author of this paper? Send a correction.