VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning
Abstract
Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
Methodology
The paper introduces VIPBench, a large-scale benchmark (124,876 judgments) designed to evaluate how well speech representation models align with human perception of voice identity, rather than just metadata-based speaker labels. The methodology is rigorous, involving a stratified sampling of 100 public figures across sociophonetic groups, genders, and ages. The stimuli include real recordings, state-of-the-art voice clones (Cartesia Sonic 3), and continuous morphs. The evaluation protocol defines four distinct tasks: continuous prediction of perceived sameness, Platt-calibrated verification, speaker-level representational similarity, and real-to-synthetic calibration transfer. The use of cross-validation over speakers and the inclusion of a noise ceiling based on inter-rater reliability demonstrates a sophisticated statistical approach to benchmarking.
Results & evaluation
The experiments evaluate 11 publicly available models, including supervised (x-vector, ECAPA-TDNN, Resemblyzer) and self-supervised (wav2vec 2.0, HuBERT, WavLM, XLS-R) models, as well as Whisper. The results reveal that while supervised models generally outperform self-supervised ones in aligning with human perception, none reach the noise ceiling. A key finding is the significant divergence between metadata labels and human perception for voice clones, where listeners often perceive clones as different speakers. The paper provides detailed ablations on layer selection for SSL models and robustness checks using unfamiliar trials and different TTS systems (CosyVoice 2). The inclusion of demographic fairness analysis is a strong addition, showing consistent performance across sociophonetic groups for most models.
Reproducibility
The paper is highly reproducible. The dataset, code, and embeddings are released on Hugging Face. The authors provide detailed information on stimulus generation, including the specific TTS settings and random seeds used for morphing. The analysis notebook is provided, allowing users to reproduce the results without needing to re-extract embeddings. The clear documentation of the IRB protocol and data collection process further enhances trust and reproducibility.
Limitations
The benchmark is limited to American English speakers and public figures, which may not generalize to other languages or private individuals. The reliance on a single commercial TTS system (Cartesia) for the main set of clones and morphs could introduce system-specific biases, although a small extension with CosyVoice 2 mitigates this partially. The lack of test-retest reliability for individual listeners is a minor limitation, though the high inter-rater reliability compensates for this.
Broader impact
This work has significant implications for the development of voice biometric systems, cybersecurity, and intellectual property protection. By highlighting the gap between metadata-based speaker verification and human perception, the paper motivates the training of speaker models directly on human perceptual judgments. This could lead to more robust and human-aligned voice authentication systems that are better equipped to handle voice cloning attacks. The benchmark also provides a valuable resource for researchers studying human voice perception and the alignment of machine representations with human cognition. The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
Author of this paper? Send a correction.