VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio
Abstract
General-purpose audio representations aim to map acoustically variable instances of the same event to nearby points, resolving content identity in a zero-shot setting. Unlike supervised classification benchmarks that measure adaptability via parameter updates, we introduce VocSim, a training-free benchmark probing the intrinsic geometric alignment of frozen embeddings. VocSim aggregates 125k single-source clips from 19 corpora spanning human speech, animal vocalizations, and environmental sounds. By restricting to single-source audio, we isolate content representation from the confound of source separation. We evaluate embeddings using Precision@k for local purity and the Global Separation Rate (GSR) for point-wise class separation. To calibrate GSR, we report lift over an empirical permutation baseline. Across diverse foundation models, a simple pipeline, frozen Whisper encoder features, time-frequency pooling, and label-free PCA, yields strong zero-shot performance. However, VocSim also uncovers a consistent generalization gap. On blind, low-resource speech, local retrieval drops sharply. While performance remains statistically distinguishable from chance, the absolute geometric structure collapses, indicating a failure to generalize to unseen phonotactics. As external validation, our top embeddings predict avian perceptual similarity, improve bioacoustic classification, and achieve state-of-the-art results on the HEAR benchmark. We posit that the intrinsic geometric quality measured here proxies utility in unlisted downstream applications. We release data, code, and a public leaderboard to standardize the evaluation of intrinsic audio geometry.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces VocSim, a training-free benchmark for evaluating zero-shot content identity in audio representations, significantly contributing to the field by providing a rigorous framework for assessing the intrinsic quality of audio embeddings. The comprehensive methodology and experimental validation enhance its relevance and potential impact on future research in audio processing and machine learning.
Methodology
The authors present a novel benchmark, VocSim, designed to evaluate the intrinsic geometric alignment of audio embeddings in a zero-shot setting. The methodology is robust, employing a large dataset of 125k single-source audio clips aggregated from diverse corpora, which allows for rigorous testing of generalization capabilities. The use of training-free metrics, such as Precision@k and Global Separation Rate (GSR), is innovative and addresses the limitations of existing benchmarks that rely on supervised learning paradigms. The transductive PCA approach to mitigate anisotropy in embedding spaces is a thoughtful addition that enhances the evaluation process.
Results & evaluation
The experiments are comprehensive, evaluating multiple foundation models and providing a detailed analysis of their performance across different audio domains. The results reveal significant insights into the generalization capabilities of these models, particularly highlighting a notable performance drop on blind, low-resource speech datasets. The external validation of the embeddings' utility in predicting avian perceptual similarity and achieving state-of-the-art results on the HEAR benchmark further underscores the practical implications of the findings.
Reproducibility
The paper includes a clear description of the experimental setup, data preprocessing, and evaluation metrics, which facilitates reproducibility. The authors have made their code and dataset publicly available, enhancing the transparency and accessibility of their research. However, the reliance on transductive PCA may complicate strict adherence to zero-shot evaluation protocols, which could be a point of contention for some researchers.
Limitations
The primary limitation noted is the generalization gap observed in low-resource speech, indicating that current models may not generalize well across different phonotactics. Additionally, the benchmark's focus on single-source audio excludes polyphonic scenarios, which may limit its applicability in real-world contexts. The ethical considerations regarding data sovereignty and the handling of indigenous language data are also acknowledged, which is commendable but reflects the complexities involved in such research.
Broader impact
The implications of this research are significant, particularly in addressing biases in low-resource languages and the potential perpetuation of digital divides in audio processing technologies. By highlighting the performance disparities of state-of-the-art models on underrepresented languages, the authors advocate for more equitable advancements in machine learning applications. The benchmark also serves as a tool for future research to improve audio representation models, fostering advancements in bioacoustics and environmental sound classification. The paper introduces VocSim, a training-free benchmark for evaluating zero-shot content identity in audio representations, significantly contributing to the field by providing a rigorous framework for assessing the intrinsic quality of audio embeddings. The comprehensive methodology and experimental validation enhance its relevance and potential impact on future research in audio processing and machine learning.
Author of this paper? Send a correction.