MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations
Abstract
Humans rely on multisensory integration to perceive spatial environments, where auditory cues enable sound source localization in three-dimensional space. Despite the critical role of spatial audio in immersive technologies such as VR/AR, most existing multimodal datasets provide only monaural audio, which limits the development of spatial audio generation and understanding. To address these challenges, we introduce MRSAudio, a large-scale multimodal spatial audio dataset designed to advance research in spatial audio understanding and generation. MRSAudio spans four distinct components: MRSLife, MRSSpeech, MRSMusic, and MRSSing, covering diverse real-world scenarios. The dataset includes synchronized binaural and ambisonic audio, exocentric and egocentric video, motion trajectories, and fine-grained annotations such as transcripts, phoneme boundaries, lyrics, scores, and prompts. To demonstrate the utility and versatility of MRSAudio, we establish five foundational tasks: audio spatialization, and spatial text to speech, spatial singing voice synthesis, spatial music generation and sound event localization and detection. Results show that MRSAudio enables high-quality spatial modeling and supports a broad range of spatial audio research. Demos and dataset access are available at https://mrsaudio.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MRSAudio, a large-scale multimodal spatial audio dataset designed to advance research in spatial audio understanding and generation. This dataset addresses a critical gap in existing resources, enabling a variety of foundational tasks that can significantly impact the development of immersive audio technologies.
Methodology
The methodology for creating the MRSAudio dataset is robust, involving a well-structured approach to data collection across various real-world scenarios. The inclusion of synchronized binaural and ambisonic audio, alongside exocentric and egocentric video, provides a comprehensive framework for spatial audio research. The refined annotations, including phoneme boundaries and transcripts, enhance the dataset's utility for multiple tasks. The establishment of five foundational tasks demonstrates a clear application of the dataset, although the paper could benefit from a more detailed explanation of the methodology used for data collection and annotation processes.
Results & evaluation
The experimental evaluation is well-defined, showcasing the dataset's capabilities through the proposed foundational tasks. The results indicate that MRSAudio supports high-quality spatial modeling, which is crucial for advancements in spatial audio technologies. However, the paper lacks quantitative metrics or comparisons with existing datasets to substantiate the claims of superiority or uniqueness. More detailed experimental results, including performance benchmarks, would strengthen the findings.
Reproducibility
The paper provides a link to the dataset and demos, which is a positive aspect for reproducibility. However, it does not include sufficient implementation details or code repositories to allow for complete reproducibility of the experiments. Future work should consider providing code or detailed guidelines for researchers to replicate the experiments.
Limitations
One limitation is the lack of comparative analysis with existing multimodal datasets, which would help contextualize the contributions of MRSAudio. Additionally, the paper does not address potential biases in the dataset or the limitations of the tasks defined. The scope of the dataset may also be limited to specific environments, which could affect generalizability.
Broader impact
The MRSAudio dataset has significant potential applications in immersive technologies such as VR/AR, gaming, and assistive technologies. By providing a rich resource for spatial audio research, it can facilitate advancements in sound source localization, audio rendering, and multimodal interaction. The implications for enhancing user experiences in virtual environments are substantial, making this dataset a valuable contribution to the field. The main contribution of this paper is the introduction of MRSAudio, a large-scale multimodal spatial audio dataset designed to advance research in spatial audio understanding and generation. This dataset addresses a critical gap in existing resources, enabling a variety of foundational tasks that can significantly impact the development of immersive audio technologies.
Author of this paper? Send a correction.