Multi-Axis Speech Similarity via Factor-Partitioned Embeddings
Abstract
Speech encodes multiple simultaneous attributes--linguistic content, speaker identity, dialect, gender--that conventional single-vector embeddings conflate. We present a factor-partitioned embedding framework that maps each utterance into a single vector whose subspaces correspond to distinct axes of variation. A shared acoustic encoder feeds per-axis linear projection heads, each trained via distillation from a specialist teacher or a contrastive objective over shared-label pairs. The resulting embeddings support attribute-conditioned retrieval: similarity is computed as a signed weighted sum over per-axis cosine scores, allowing retrieval that jointly considers what was said and how --or explicitly suppresses one attribute to surface another. We evaluate on cross-corpus retrieval over corpora sharing the Harvard sentence prompts, demonstrating that signed axis weighting can suppress same-speaker bias and surface semantically matched utterances across recording conditions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a novel factor-partitioned embedding framework for speech that allows for controllable multi-axis similarity searches. This work represents a meaningful advancement in the field of speech representation learning, addressing the complexities of speech attributes and providing a robust solution for attribute-conditioned retrieval tasks.
Methodology
The paper introduces a factor-partitioned embedding framework that effectively separates various attributes of speech into distinct subspaces, allowing for nuanced similarity searches. The methodology is well-structured, employing a shared acoustic encoder and per-axis linear projection heads trained through distillation and contrastive objectives. This approach is innovative in its use of signed axis weighting to control retrieval outcomes, providing a significant advancement over traditional single-vector embeddings that conflate multiple attributes.
Results & evaluation
The experiments conducted are thorough, utilizing cross-corpus retrieval tasks with well-defined metrics such as Precision@k and preference-flip tests. The evaluation on datasets sharing the Harvard sentence prompts is appropriate, demonstrating the framework's ability to suppress same-speaker bias and surface semantically matched utterances. The results are compelling, showing that the proposed method outperforms baseline models significantly.
Reproducibility
The paper provides a clear description of the architecture, data sources, and training processes, which enhances reproducibility. However, the absence of a publicly available implementation or code repository limits the ability for other researchers to replicate the results fully.
Limitations
One limitation identified is the reliance on specific datasets, which may not generalize well to other speech domains. Additionally, the paper notes challenges in separating the gender axis due to its correlation with speaker identity, suggesting that further exploration is needed to improve this aspect. The potential for mode collapse in models trained without auxiliary tasks is also a concern.
Broader impact
The proposed framework has significant implications for applications in speech retrieval, speaker recognition, and potentially in areas such as voice conversion and personalized speech synthesis. By enabling controllable retrieval based on multiple axes, it opens avenues for more sophisticated user interactions with speech technologies. The main contribution of this paper is the introduction of a novel factor-partitioned embedding framework for speech that allows for controllable multi-axis similarity searches. This work represents a meaningful advancement in the field of speech representation learning, addressing the complexities of speech attributes and providing a robust solution for attribute-conditioned retrieval tasks.
Author of this paper? Send a correction.