Rubato: Transcribing Piano Music with Timestamps
Abstract
We consider the conversion of musical recordings into human-readable sheet music annotated with timestamps. Such output lets a listener clearly visualize rubato (temporally expressive playing), a learner diagnose ensemble precision and timing choices against the written music, and a musicology scholar compare performance styles across recordings of the same work. We introduce (1) a prompt-conditioned encoder-decoder model, named Rubato, trained to output (2) a new textual representation for polyphonic music, named InterMo, which we designed for compatibility with sequence-to-sequence training. Our experiments demonstrate that Rubato produces timestamped piano sheet music from audio with higher notational accuracy than the best existing approaches, which are based on cascades. We find that even if the cascade is given ground-truth MIDI instead of audio, Rubato performs better, suggesting that the ceiling of existing approaches is primarily representational, not acoustic. Further, because Rubato is trained on several related tasks (with prompts), it competes with or outperforms the best single-task systems on related but simpler tasks like MIDI note grounding and beat/downbeat detection. A demo is available at https://nctamer.github.io/rubato-transcription .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents Rubato, a comprehensive system for transcribing piano music into time-aligned scores, significantly advancing the field of automatic music transcription by integrating audio input with structured musical output. The innovative approach and strong experimental results position this work as a meaningful contribution to both machine learning and music technology.
Methodology
The paper introduces a novel prompt-conditioned encoder-decoder model, Rubato, which integrates audio-to-score transcription with temporal alignment in a single autoregressive pass. The use of a new textual representation, InterMo, for polyphonic music is innovative, as it allows for a more coherent and structured output that maintains both musical and temporal integrity. The methodology effectively addresses the limitations of existing cascade approaches by eliminating the need for intermediate representations, thus streamlining the transcription process. The multitask training approach leveraging various dialects enhances the model's robustness and versatility.
Results & evaluation
The experiments are thorough, comparing Rubato against multiple state-of-the-art systems across different tasks, including MIDI note grounding and beat detection. The results demonstrate that Rubato consistently outperforms existing methods in terms of notational accuracy and temporal alignment, even under conditions where oracle inputs are provided to baselines. The use of multiple datasets for evaluation strengthens the validity of the findings.
Reproducibility
The paper provides sufficient details about the training data, model architecture, and evaluation metrics, which supports reproducibility. However, the lack of a public code repository limits the ease with which other researchers can replicate the results. The authors do release score excerpts and synthesized utterances for reproducibility, which is a positive aspect.
Limitations
One limitation is the potential overfitting to the training datasets, particularly given the complexity of the model and the variety of tasks it is trained on. Additionally, the reliance on synthesized audio for some training data may not fully capture the nuances of real-world performances. The paper does not address how the model might perform with non-piano instruments or in more complex musical contexts.
Broader impact
The implications of this research are significant for music education, performance analysis, and musicology, as it allows for a more nuanced understanding of expressive timing in musical performances. The ability to generate accurate, timestamped sheet music can aid learners in diagnosing timing choices and enhance the comparative analysis of performance styles. This work could also inspire further advancements in automatic music transcription and multimodal music analysis systems. The paper presents Rubato, a comprehensive system for transcribing piano music into time-aligned scores, significantly advancing the field of automatic music transcription by integrating audio input with structured musical output. The innovative approach and strong experimental results position this work as a meaningful contribution to both machine learning and music technology.
Author of this paper? Send a correction.