MT3: Multi-Task Multitrack Music Transcription
Abstract
Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in automatic music transcription by introducing a unified Transformer-based model capable of multi-task learning across diverse datasets. This work not only improves transcription accuracy for low-resource instruments but also sets a new standard for evaluation metrics in the field, thus contributing meaningfully to the ongoing development of music understanding technologies.
Methodology
The paper presents a robust methodology for Multi-Task Multitrack Music Transcription (MT3) by leveraging a Transformer architecture to handle multiple instruments in a unified framework. The authors introduce a novel tokenization scheme that allows the model to output MIDI-like events, effectively addressing the challenges of simultaneous transcription of various instruments. The approach is innovative in its use of a single model trained across multiple datasets, which is a departure from traditional task-specific architectures. The proposed method also includes a new evaluation metric that accounts for instrument identification alongside note transcription, which enhances the rigor of performance assessment.
Results & evaluation
The experiments are comprehensive, utilizing six diverse datasets that span a range of musical styles and instrumentation. The results demonstrate significant improvements over state-of-the-art (SOTA) models, particularly for low-resource instruments, indicating the effectiveness of the proposed multi-task learning framework. The paper provides thorough comparisons with baseline models, including both machine learning and professional DSP software, which strengthens the validity of the findings. The leave-one-dataset-out experiments further showcase the model's generalization capabilities, although the paper could benefit from more extensive ablation studies to dissect the contributions of individual components.
Reproducibility
The authors commit to reproducibility by providing access to their code and detailed descriptions of their experimental setup, including dataset splits and training procedures. However, the reliance on external datasets and potential variations in their annotations may pose challenges for complete reproducibility. The paper does well to outline the specifics of the datasets used, but additional details on the training environment and hyperparameter settings could enhance clarity.
Limitations
One notable limitation is the model's focus on Western music traditions, which excludes non-Western musical forms that may not fit the discrete note framework employed. Additionally, the paper acknowledges potential labeling issues in some datasets, particularly regarding timing accuracy, which could impact the model's performance. The model's reliance on a single architecture may also limit its adaptability to future advancements in music transcription technology.
Broader impact
The implications of this work extend beyond academic research, as improved music transcription systems can enhance music education, facilitate music analysis, and support the development of music generation tools. The introduction of a unified framework for multi-task transcription could lead to more accessible music technology applications, benefiting musicians and researchers alike. The work also highlights the need for better dataset alignment and evaluation metrics in the field, which could foster further advancements in music understanding. The paper presents a significant advancement in automatic music transcription by introducing a unified Transformer-based model capable of multi-task learning across diverse datasets. This work not only improves transcription accuracy for low-resource instruments but also sets a new standard for evaluation metrics in the field, thus contributing meaningfully to the ongoing development of music understanding technologies.
Curator note
Gardner et al., Google Magenta; ICLR 2022; T5 sequence-to-sequence multi-instrument transcription across datasets
Author of this paper? Send a correction.