MuScriptor: An Open Model for Multi-Instrument Music Transcription
Abstract
Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
Methodology
The paper proposes a practical and effective pipeline for multi-instrument automatic music transcription (AMT) using a decoder-only Transformer architecture. The core methodological contribution lies in the training strategy rather than architectural novelty: it combines large-scale synthetic pre-training (1.45M MIDIs) with fine-tuning on a newly collected dataset of 170k real-world recordings, followed by Reinforcement Learning (RL) post-training using a GRPO-like algorithm. The introduction of instrument presence conditioning at inference time is a valuable engineering choice that stabilizes predictions. The application of RL (specifically GRPO without KL penalties) to align transcription outputs with high-quality human annotations is a novel adaptation of techniques typically seen in LLMs to the audio-symbolic domain. However, the base architecture is a standard adaptation of existing sequence-to-sequence models (like MT3), limiting its theoretical novelty.
Results & evaluation
The experimental evaluation is comprehensive and robust. The authors conduct extensive ablation studies demonstrating the necessity of each training stage (synthetic pre-training, real fine-tuning, RL post-training). They show significant improvements over state-of-the-art baselines (YourMT3+, MT3) across multiple metrics (F1, Onset F1, Offset F1, Multi F1) on a diverse test set. The analysis of synthetic data effectiveness is particularly strong, providing clear evidence that synthetic data helps when real data is scarce but is insufficient on its own. The inclusion of a large, diverse real-world dataset (11k hours) addresses a critical gap in the field. The results are consistent and the improvements are substantial, particularly in generalization to real-world mixes.
Reproducibility
The paper provides significant details for reproducibility. The authors release the model weights and inference code via GitHub. They describe the data collection process, including the use of audio-symbolic synchronization and filtering criteria. The hyperparameters for training and RL are specified. However, the exact source and licensing of the 170k real-world recordings are not fully detailed in the text (referred to as "internal dataset"), which may pose challenges for full independent replication of the data pipeline, although the model weights are available. The RL implementation details are clear enough for implementation.
Limitations
The model struggles with overlapping notes of the same instrument and pitch, a common issue in dense music, which the authors acknowledge requires different tokenization. The performance on onset/offset precision for certain genres (like choral music) remains lower, indicating domain-specific challenges. The reliance on a specific instrument taxonomy (36 subgroups) limits the granularity of transcription. The RL post-training is performed on a small subset (300 tracks), which may limit the generalization of the RL benefits compared to the broader fine-tuning data.
Broader impact
MuScriptor represents a significant step forward for open-source music information retrieval tools. By providing a high-quality, open-weight model that works on real-world music, it lowers the barrier for researchers and practitioners to use AMT in downstream applications like music generation, education, and archival. The release of the large-scale dataset also contributes to the community, although the licensing details are crucial for broader adoption. The work highlights the importance of real-world data and alignment techniques in bridging the gap between synthetic benchmarks and practical applications. The paper presents MuScriptor, an open-weight multi-instrument music transcription model that achieves state-of-the-art performance on real-world music by combining synthetic pre-training, large-scale real-data fine-tuning, and reinforcement learning alignment, addressing the critical domain shift problem in automatic music transcription.
Author of this paper? Send a correction.