SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
Abstract
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarce, and accurate local predictions can still produce inconsistent musical sequences. We present SheetSage2, a unified music transcription framework that combines synthetic data, task-specific structured decoding, and autoregressive distillation. Automatically annotated MIDI, rendered into audio, provides scalable supervision across music understanding tasks. Task-specific structured decoders integrate complementary musical cues and their temporal dependencies to produce musically coherent scores. Autoregressive distillation further retains transcription accuracy without task-specific dynamic programming at inference. Across eight benchmark collections, a single SheetSage2-AR model exceeds the listed prior systems on 12 of 15 benchmark--metric pairs in our evaluation, substantially improving over SheetSage1 and surpassing task-specific models on several benchmarks. Model weights and inference code are publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SheetSage2 introduces a unified framework for lead-sheet transcription that effectively combines synthetic data augmentation, structured decoding for coherence, and autoregressive distillation to achieve state-of-the-art performance across multiple music understanding benchmarks. The methodology is rigorous, addressing the critical issues of data scarcity and musical consistency, and the open-source release ensures high reproducibility and impact on the field.
Methodology
The paper proposes a sophisticated three-stage pipeline: (1) Synthetic data generation via MIDI rendering and pseudo-label bootstrapping to overcome data scarcity; (2) A "Prober" model using frame-wise prediction with task-specific structured decoders (HMM/CRF-like) to handle noisy/partial annotations; and (3) Autoregressive (AR) distillation where an AR student learns from the Prober's structured outputs. The structured decoding is particularly strong, explicitly modeling tempo anchors, meter, and pitch context to ensure musical coherence, which is a significant improvement over independent task prediction. The use of synthetic supervision is a clever workaround for the lack of high-quality lead-sheet annotations.
Results & evaluation
The evaluation is extensive, covering 8 benchmark collections and 15 benchmark-metric pairs. The model outperforms prior systems on 12 of these pairs. The comparison against SheetSage1 and task-specific models (madmom, etc.) is relevant. However, the reliance on "listed prior systems" without a full table of all baselines in the main text (truncated) makes it slightly harder to verify the breadth of comparison, though the abstract claims are strong. The use of synthetic data for training and real data for testing is a valid and important experimental design choice.
Reproducibility
High. The authors provide model weights and inference code on HuggingFace. The methodology is detailed, including specific decoding steps, loss functions, and data pipeline descriptions. The use of standard MIDI rendering (FluidSynth) ensures the synthetic data generation is reproducible.
Limitations
The paper relies heavily on the quality of the synthetic data and the initial pseudo-labels. If the symbolic analysis of MIDI is flawed, the synthetic supervision will propagate errors. The AR model, while removing the need for dynamic programming at inference, may still struggle with very long-range dependencies compared to the structured teacher, though the paper claims it retains consistency. The "Unknown" institution status in the prompt prevents a definitive institutional credit, though the technical quality suggests a top-tier lab.
Broader impact
This work significantly advances the state of the art in automatic music transcription, moving from isolated tasks to coherent lead-sheet generation. The synthetic data pipeline is a transferable technique for other audio understanding tasks where labeled data is scarce. The open-sourcing of weights and code will accelerate research in MIR. SheetSage2 introduces a unified framework for lead-sheet transcription that effectively combines synthetic data augmentation, structured decoding for coherence, and autoregressive distillation to achieve state-of-the-art performance across multiple music understanding benchmarks. The methodology is rigorous, addressing the critical issues of data scarcity and musical consistency, and the open-source release ensures high reproducibility and impact on the field.
Author of this paper? Send a correction.