Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups
Abstract
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
Methodology
The paper proposes a rigorous methodology for constructing a physically verified benchmark for vocal attribution. The core innovation is the use of an on-body accelerometer-derived vibration signal to ground-truth caller identity, which is then withheld from the benchmark models to prevent shortcut learning. This is a sophisticated design choice that distinguishes the work from standard active speaker detection datasets that rely on perceptual annotation or close-talk references. The pipeline is detailed and auditable, involving synchronized multimodal data (5-channel audio, 3-camera video, 3D pose, radio telemetry) and a multi-stage verification process including manual review and geometric filtering. The evaluation protocol is particularly strong, defining three distinct regimes (session-disjoint, held-out-experiment, leave-one-individual-out) to explicitly separate in-domain recognition from transferable spatial reasoning.
Results & evaluation
The experimental evaluation is comprehensive, testing a wide array of baselines ranging from classical physics-based methods (SRP) to deep learning architectures (SELDnet++, EINV2, CST-former). The results clearly demonstrate the paper's central thesis: models that rely on familiar voice identity perform well in-domain but fail on unseen individuals, while spatially-aware models transfer better. The inclusion of modality intervention experiments (zeroing audio, shuffling geometry) provides strong causal evidence for what drives performance. The "on-body reference" ceiling confirms the consistency of the labels. The analysis of "close birds" and localization diagnostics adds depth to the evaluation.
Reproducibility
Reproducibility is high. The authors release the full dataset, code for all evaluated methods, and fixed evaluation protocols. The paper provides extensive details on the data acquisition pipeline, camera calibration, and label construction in the appendices. The use of standard open-source tools (YOLOX, OC-SORT, Whisper) for preprocessing further aids reproducibility. The clear separation of the verification channel from the benchmark inputs ensures that future researchers can fairly evaluate new methods without access to the ground-truth sensor.
Limitations
The primary limitation is the scale and diversity of the data. The dataset is limited to 10 zebra finches across 3 experiments in a single installation. This limits the generalizability of the findings to other species or larger, more dynamic groups. The radio telemetry features are limited due to carrier drift, which the authors acknowledge. The benchmark currently only covers single-vocalizer events, leaving multi-caller scenarios for the unevaluated V33DA++ release.
Broader impact
This work has significant impact on the fields of bioacoustics, multimodal learning, and behavioral analysis. By providing a physically verified benchmark, it enables the development of more robust and generalizable models for animal communication. The insights into shortcut learning and the importance of spatial reasoning are applicable to other multimodal attribution tasks, such as human active speaker detection. The release of the dataset and code will likely spur further research in this niche but important area. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
Author of this paper? Send a correction.