WhAM: Towards A Translative Model of Sperm Whale Vocalization
Abstract
Sperm whales communicate in short sequences of clicks known as codas. We present WhAM (Whale Acoustics Model), the first transformer-based model capable of generating synthetic sperm whale codas from any audio prompt. WhAM is built by finetuning VampNet, a masked acoustic token model pretrained on musical audio, using 10k coda recordings collected over the past two decades. Through iterative masked token prediction, WhAM generates high-fidelity synthetic codas that preserve key acoustic features of the source recordings. We evaluate WhAM's synthetic codas using Fréchet Audio Distance and through perceptual studies with expert marine biologists. On downstream classification tasks including rhythm, social unit, and vowel classification, WhAM's learned representations achieve strong performance, despite being trained for generation rather than classification. Our code is available at https://github.com/Project-CETI/wham
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of WhAM, a novel transformer-based model for synthesizing sperm whale vocalizations, which demonstrates significant advancements in acoustic translation, generation, and classification tasks. The comprehensive evaluation of the model's performance, combined with its innovative methodology, positions this work as a meaningful contribution to both machine learning and bioacoustics research.
Methodology
The paper introduces WhAM, a transformer-based model that synthesizes sperm whale vocalizations through a novel approach of finetuning VampNet, a masked acoustic token model. The methodology is well-structured, utilizing a two-phase training process that includes domain adaptation and species-specific finetuning. The use of Fréchet Audio Distance (FAD) for evaluating acoustic similarity is appropriate, and the perceptual study with marine biologists adds a strong qualitative dimension to the evaluation of the model's outputs. The iterative masked token prediction approach is innovative and aligns well with the goals of generating high-fidelity synthetic codas.
Results & evaluation
The experiments are comprehensive, involving quantitative metrics (FAD) and qualitative assessments through expert perceptual studies. The results indicate that WhAM can effectively translate various audio inputs into sperm whale codas, achieving FAD-indistinguishability from natural codas for several input types. The downstream classification tasks demonstrate that the model's learned representations are useful, even though the primary focus is on generation. The combination of quantitative and qualitative evaluations provides a robust assessment of the model's capabilities.
Reproducibility
The paper provides sufficient details on the training and testing protocols, including the datasets used and the computational resources required. The authors commit to releasing the model and code, which enhances reproducibility. However, the availability of the datasets may be limited due to collaboration constraints, which could affect full reproducibility.
Limitations
The paper acknowledges several limitations, including the fixed architecture of the audio codec, which may restrict the model's ability to capture nuanced acoustic features. Additionally, the generated codas exhibit some unnatural characteristics, such as inconsistent background noise and timing issues. The authors also highlight the challenges associated with dataset quality and the need for more robust evaluation protocols.
Broader impact
The research has significant implications for bioacoustics, potentially advancing our understanding of sperm whale communication and contributing to conservation efforts. However, the authors should consider the ethical implications of generating synthetic vocalizations, particularly regarding their use in behavioral studies and the potential for misinterpretation of animal communication. The main contribution of this paper is the introduction of WhAM, a novel transformer-based model for synthesizing sperm whale vocalizations, which demonstrates significant advancements in acoustic translation, generation, and classification tasks. The comprehensive evaluation of the model's performance, combined with its innovative methodology, positions this work as a meaningful contribution to both machine learning and bioacoustics research.
Author of this paper? Send a correction.