Text-Independent Speaker Verification Using Discrete Audio Tokens
Abstract
Neural audio codecs (NACs) enable efficient audio compression and have achieved success in downstream tasks such as speech synthesis. However, their discrete representations consistently underperform traditional spectral features in automatic speaker verification (ASV). We empirically demonstrate that speaker cues are implicitly preserved in discrete tokens but remain underutilized by conventional ASV training paradigms. To address this, we propose a Cross-Feature Knowledge Distillation (CFKD) framework. By guiding the codec-based student to mimic the embedding space of a strong Fbank-based teacher, CFKD provides structured supervision for effective utilization of speaker information in tokens. Experiments on the VoxCeleb benchmarks show that CFKD substantially improves the ASV performance of codec-based systems, allowing them to approach the accuracy of Fbank-based teacher models and highlighting the potential of discrete audio tokens for diverse speech tasks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
Methodology
The paper proposes Cross-Feature Knowledge Distillation (CFKD) to address the performance gap between discrete neural audio codec tokens and continuous spectral features (Fbanks) in Automatic Speaker Verification (ASV). The core methodology involves training a student model (using EnCodec tokens) to mimic the embedding space of a teacher model (using Fbanks) via cosine similarity loss at the embedding level. The authors also conduct an ablation study on backbone architectures (1D vs 2D), concluding that 1D architectures (ECAPA-TDNN) are more suitable for discrete tokens due to the lack of spatial adjacency in the token latent space. The approach is conceptually sound and addresses a clear problem: discrete tokens from generative codecs are underutilized in discriminative tasks. However, the novelty is moderate as knowledge distillation is a standard technique, and applying it across different feature modalities (cross-feature) is a logical but not radically new extension. The specific insight regarding the inductive bias of 1D vs 2D networks for discrete tokens is a valuable empirical contribution.
Results & evaluation
The experiments are conducted on standard ASV benchmarks (VoxCeleb1 and VoxCeleb2). The results show that CFKD significantly improves the performance of token-based systems, narrowing the gap with Fbank-based systems. For instance, on Vox1-O with ECAPA-TDNN, the EER drops from 3.38% (baseline) to 2.25% (CFKD), approaching the teacher's 2.21%. The paper includes error analysis showing complementary error sets between teacher and student, which supports the claim that tokens preserve distinct speaker cues. The evaluation is thorough, covering different bitrates and backbones. However, the comparison is primarily against the Fbank baseline and a prior work (Codec-ASV). The improvement is substantial but the absolute performance of the token-based system still lags slightly behind the Fbank teacher in some settings (e.g., minDCF 0.231 vs 0.236 is very close, but in other configs, the teacher is better). The results are convincing but not revolutionary.
Reproducibility
The paper provides detailed descriptions of the experimental setup, including the EnCodec configuration (24kHz, 32 quantizers), backbone architectures, training hyperparameters (learning rate, batch size, optimizer), and data augmentation (MUSAN, RIR). The use of standard frameworks (Wespeaker) and datasets (VoxCeleb) enhances reproducibility. The specific distillation weight ($\alpha=40$) is noted as unusually high, which is a specific hyperparameter choice that might require tuning for other datasets. Overall, the method is reproducible.
Limitations
The primary limitation is that the token-based system, even with CFKD, does not consistently outperform the Fbank teacher; it merely approaches it. This suggests that while the gap is closed, the discrete representation may still have inherent limitations in capturing fine-grained speaker details compared to continuous spectral features. Additionally, the method relies on a pre-trained Fbank teacher, which adds complexity to the training pipeline (two models instead of one). The paper does not explore the impact of different codec types or bitrates on the distillation efficacy beyond the standard EnCodec configuration. The claim that tokens "preserve sufficient speaker-discriminative information" is supported, but the mechanism of *how* the distillation extracts this is somewhat opaque (black-box embedding alignment).
Broader impact
This work has significant implications for the field of speech processing, particularly in bridging the gap between generative audio models (which use discrete tokens) and discriminative tasks (like ASV). It suggests that discrete representations can be effectively used for identity verification, which is crucial for applications requiring low-bitrate transmission or integration with large language models that operate on discrete tokens. It challenges the dominance of spectral features in ASV and opens avenues for unified audio models that can handle both generation and recognition tasks efficiently. The paper presents a practical and effective method for leveraging discrete neural audio tokens in Automatic Speaker Verification through cross-feature knowledge distillation, demonstrating that discrete representations can achieve performance comparable to traditional spectral features.
Author of this paper? Send a correction.