CASTELLA: Long Audio Dataset with Captions and Temporal Boundaries
Abstract
We introduce CASTELLA, a human-annotated audio benchmark for the task of audio moment retrieval (AMR). Although AMR has various useful potential applications, there is still no established benchmark with real-world data. The early study of AMR trained the model with solely synthetic datasets. Moreover, the evaluation is based on annotated dataset of fewer than 100 samples. This resulted in less reliable reported performance. To ensure performance for applications in real-world environments, we present CASTELLA, a large-scale manually annotated AMR dataset. CASTELLA consists of 1,009, 213, and 640 audio recordings for train, valid, and test split, respectively, which is 24 times larger than the previous dataset. We also establish a baseline model for AMR using CASTELLA. Our experiments demonstrate that a model fine-tuned on CASTELLA after pre-training on the synthetic data outperformed a model trained solely on the synthetic data by 10.4 points in [email protected]. CASTELLA is publicly available in https://h-munakata.github.io/CASTELLA-demo/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the CASTELLA dataset, a large-scale, human-annotated benchmark for audio moment retrieval that addresses the limitations of previous datasets. This work not only enhances the reliability of AMR evaluations but also sets a foundation for future research in the field, particularly in the development of models that can effectively handle real-world audio data.
Methodology
The methodology employed in the creation of the CASTELLA dataset is robust, utilizing crowd-sourcing for annotation and ensuring quality through a multi-step review process. The dataset's design addresses significant gaps in existing audio moment retrieval (AMR) benchmarks, particularly by focusing on real-world audio recordings and providing both global and local captions with temporal boundaries. The use of a large-scale dataset (over 1,000 audio recordings) is a notable improvement over previous efforts, which were limited in size and scope.
Results & evaluation
The experiments conducted using the CASTELLA dataset are well-structured, demonstrating the effectiveness of fine-tuning models pre-trained on synthetic datasets. The results indicate a significant performance improvement when using CASTELLA, showcasing the dataset's utility in advancing AMR tasks. The paper provides clear metrics (Recall1 and mAP) to assess model performance, which is essential for evaluating the impact of the dataset on AMR research.
Reproducibility
The paper outlines the experimental setup and model configurations in detail, which aids in reproducibility. However, the lack of a publicly accessible code repository limits the ability for others to replicate the exact experiments. Including a GitHub repository or similar would enhance reproducibility significantly.
Limitations
One limitation identified is the reliance on crowd-sourced annotations, which, while beneficial for scale, may introduce variability in quality. Additionally, the dataset's focus on English captions may limit its applicability in multilingual contexts. The paper also notes that current models struggle with short audio moments, indicating a potential area for further research and improvement.
Broader impact
The CASTELLA dataset has the potential to significantly impact various applications, including audio indexing, content retrieval in media, and enhancing accessibility for audio content. By providing a comprehensive benchmark for AMR, it opens avenues for improved user experiences in audio search and retrieval systems. The main contribution of this paper is the introduction of the CASTELLA dataset, a large-scale, human-annotated benchmark for audio moment retrieval that addresses the limitations of previous datasets. This work not only enhances the reliability of AMR evaluations but also sets a foundation for future research in the field, particularly in the development of models that can effectively handle real-world audio data.
Author of this paper? Send a correction.