SongFormer: Scaling Music Structure Analysis with Heterogeneous Supervision
Abstract
Music structure analysis (MSA) underpins music understanding and controllable generation, yet progress has been limited by small, inconsistent corpora. We present SongFormer, a scalable framework that learns from heterogeneous supervision. SongFormer (i) fuses short- and long-window self-supervised audio representations to capture both fine-grained and long-range dependencies, and (ii) introduces a learned source embedding to enable training with partial, noisy, and schema-mismatched labels. To support scaling and fair evaluation, we release SongFormDB, the largest MSA corpus to date (over 10k tracks spanning languages and genres), and SongFormBench, a 300-song expert-verified benchmark. On SongFormBench, SongFormer sets a new state of the art in strict boundary detection (HR.5F) and achieves the highest functional label accuracy, while remaining computationally efficient; it surpasses strong baselines and Gemini 2.5 Pro on these metrics and remains competitive under relaxed tolerance (HR3F). Code, datasets, and model are publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SongFormer presents a scalable framework for music structure analysis that effectively combines multi-resolution self-supervised learning and heterogeneous supervision. This work addresses critical limitations in existing MSA methods and sets a new standard for performance in the field, making substantial contributions to both methodology and dataset resources.
Methodology
The methodology presented in SongFormer is robust and innovative, integrating multi-resolution self-supervised learning (SSL) representations to capture both fine-grained and long-range dependencies in music structure analysis (MSA). The introduction of a learned source embedding to handle heterogeneous supervision is a significant contribution, allowing the model to train effectively on datasets with varying quality and annotation schemes. The architecture leverages a Transformer encoder, which is well-suited for sequence modeling, and the detailed description of the feature fusion and downsampling processes demonstrates a thoughtful approach to balancing computational efficiency with performance.
Results & evaluation
The experimental evaluation is thorough, utilizing the newly introduced SongFormDB and SongFormBench datasets, which are substantial contributions to the field. The results show that SongFormer outperforms existing baselines, including the Gemini 2.5 Pro model, in both strict boundary detection and functional label accuracy. The use of multiple evaluation metrics (HR.5F, HR3F, ACC) provides a comprehensive view of the model's performance. The ablation studies further validate the importance of each component in the architecture, showcasing the effectiveness of the proposed methods.
Reproducibility
The authors have made significant efforts to ensure reproducibility by providing access to the datasets, code, and model. The detailed descriptions of the experimental setup, including training parameters and evaluation metrics, enhance the likelihood that other researchers can replicate the results. However, the paper could benefit from clearer instructions on how to set up the environment and run the experiments.
Limitations
One limitation is the potential trade-off between the robustness of label predictions and boundary sharpness when incorporating additional datasets with lower annotation quality. Additionally, while the paper claims computational efficiency, the actual resource requirements for training and inference are not explicitly stated, which could be a concern for practical applications. The reliance on large-scale datasets may also limit applicability in scenarios where such data is not available.
Broader impact
The advancements made by SongFormer have significant implications for music understanding and generation, potentially enhancing applications in music information retrieval, automated music composition, and interactive music systems. By addressing the challenges of data scarcity and annotation inconsistency in MSA, this work paves the way for more sophisticated music analysis tools that could benefit both researchers and practitioners in the field. SongFormer presents a scalable framework for music structure analysis that effectively combines multi-resolution self-supervised learning and heterogeneous supervision. This work addresses critical limitations in existing MSA methods and sets a new standard for performance in the field, making substantial contributions to both methodology and dataset resources.
Author of this paper? Send a correction.