MuSViT: A Foundation Vision Model for Sheet Music Representation
Abstract
Foundation models have transformed vision and language processing by providing rich, reusable representations that transfer across diverse tasks. Sheet music, as a visual encoding of musical language, lacks such a strong domain-specific backbone. We introduce MuSViT (Music Score Vision Transformer): the first foundation vision model for sheet music representation -- a ViT encoder pre-trained via Masked Autoencoders on 9.7 million pages from the IMSLP. To handle the complexity of real-world scores, we adopt a two-stage curriculum: a synthetic warm-up on typeset scores followed by large-scale training on the full IMSLP corpus. We evaluate MuSViT on four downstream tasks -- full-page and staff-level music score recognition, music symbol detection, and score difficulty classification -- under two scenarios: linear probing (frozen encoder) and fine-tuning. Under linear probing, MuSViT consistently outperforms modern vision encoders, revealing that general-purpose representations, regardless of scale, fall systematically short on the structured symbolic properties of musical notation. Under fine-tuning, MuSViT generally improves upon task-specific state-of-the-art methods. An additional embedding-transcription consistency analysis reveals that MuSViT encodes symbolic musical structure directly in its representation space -- unlike other encoders, whose embeddings do not correlate with music notation content. These results establish MuSViT as a foundation backbone for sheet music understanding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MuSViT is a significant contribution to domain-specific foundation models, demonstrating that large-scale self-supervised pre-training on structured symbolic data yields representations that are semantically aligned with the domain's logic, outperforming general-purpose vision encoders and enabling state-of-the-art performance on key music document analysis tasks.
Methodology
The paper proposes MuSViT, a Vision Transformer (ViT) pre-trained via Masked Autoencoders (MAE) on 9.7 million sheet music pages from the IMSLP. The methodology is sound and follows established self-supervised learning paradigms (MAE) adapted for the specific constraints of sheet music (fine-grained patches, 2D positional encodings). The introduction of a two-stage curriculum learning strategy—starting with synthetic data to prevent dimensional collapse before moving to real-world scans—is a significant methodological contribution. This addresses a specific failure mode (average patch prediction) observed when training directly on heterogeneous real-world data. The architecture choices (ViT-B/S variants, 2D PE) are well-justified for the structured, symbolic nature of the domain. However, the core algorithmic novelty is incremental; it applies known SOTA vision techniques (MAE, ViT) to a new, under-explored domain rather than proposing a fundamentally new architectural or learning mechanism.
Results & evaluation
The evaluation is comprehensive and rigorous. The authors assess MuSViT on four diverse downstream tasks: full-page recognition, staff-level recognition, symbol detection, and difficulty classification. They employ two standard protocols for foundation models: linear probing (to test representation quality) and fine-tuning (to test adaptability). The results show that MuSViT consistently outperforms general-purpose vision encoders (DINOv3, Qwen3-VL, PaliGemma, Kosmos-2.5) in linear probing, highlighting the inadequacy of generic visual features for symbolic music. In fine-tuning, it matches or exceeds task-specific state-of-the-art methods. The inclusion of an "embedding-transcription consistency analysis" is a strong point, providing qualitative and quantitative evidence that the learned representations align with symbolic musical structure, unlike general-purpose models. The use of large-scale, real-world data (IMSLP) adds significant weight to the empirical claims.
Reproducibility
The paper provides substantial detail for reproduction. The dataset source (IMSLP) is public, and the authors release code, pre-training scripts, and evaluation scripts via a project URL. The architecture details, hyperparameters, and training protocols are described in the main text and supplementary material. The two-stage curriculum and specific masking ratios are clearly defined. The use of standard libraries (Hugging Face for baselines) and clear evaluation metrics (SER, mAP, Accuracy) ensures that the results can be verified.
Limitations
The primary limitation is that the model is a vision-only encoder. While effective for OMR-related tasks, it does not inherently capture the temporal or harmonic structure of music without downstream symbolic processing (e.g., via a transcription head). The reliance on IMSLP data, while large, introduces biases towards public domain and historically preserved scores, potentially affecting performance on contemporary or highly stylized modern scores not well-represented in the corpus. Additionally, the "synthetic warm-up" stage relies on generated data, which may not fully capture the noise and degradation of real-world scans, although the authors argue this is necessary for stability. The paper does not explore multimodal extensions (e.g., audio alignment) which could further enhance the utility of the representations.
Broader impact
This work establishes a foundational backbone for sheet music understanding, addressing a significant gap in the intersection of computer vision and music information retrieval. By providing a strong, reusable representation, it lowers the barrier for developing new applications in musicology, education, and archival digitization. The finding that general-purpose vision models fail to capture symbolic structure has broader implications for other structured visual domains (e.g., chemical structures, mathematical notation). The public release of the model and code accelerates research in this niche but culturally significant area. MuSViT is a significant contribution to domain-specific foundation models, demonstrating that large-scale self-supervised pre-training on structured symbolic data yields representations that are semantically aligned with the domain's logic, outperforming general-purpose vision encoders and enabling state-of-the-art performance on key music document analysis tasks.
Author of this paper? Send a correction.