Can LLMs understand LilyPond? A benchmark for symbolic music generation and understanding
Abstract
Symbolic music evaluation for large language models remains fragmented across representations, datasets, and metrics. We introduce LilyBench, a LilyPond-based benchmark that jointly evaluates symbolic music generation and music understanding on the same family of open-weight LLMs. The benchmark includes a 200-prompt generation suite and ten understanding tasks adapted from ABC-Eval, covering syntax, metadata prediction, structural sequencing, and music recognition. Generation quality is evaluated using compile rate, MusPy descriptor distributions via Jensen-Shannon similarity, and LilyBERT-based Fréchet Music Distance (FMD). Experiments on four open-weight models show that executable LilyPond generation is achievable in zero-shot settings, while structural understanding tasks remain challenging despite strong performance on composer and genre recognition. Our experiments also reveal systematic disagreements between descriptor-based and embedding-based metrics, suggesting that symbolic music evaluation benefits from metric triangulation rather than single-score ranking. We release the benchmark, prompt bank, and evaluation code to support future research in symbolic music generation and understanding at https://github.com/CSCPadova/lilybench
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents LilyBench, the first benchmark for evaluating symbolic music generation and understanding using LLMs, significantly advancing the field by providing a comprehensive evaluation framework and revealing critical insights into model performance.
Methodology
The paper introduces a novel benchmark, LilyBench, which evaluates both symbolic music generation and understanding using large language models (LLMs) in a unified framework. The methodology is well-structured, utilizing a combination of generation prompts and understanding tasks adapted from existing benchmarks. The use of multiple evaluation metrics (compile rate, Jensen-Shannon similarity, and Fréchet Music Distance) provides a comprehensive approach to assess model performance, revealing the strengths and weaknesses of different evaluation strategies.
Results & evaluation
The experiments are rigorous, involving four open-weight models and a well-defined reference corpus of Baroque works. The results demonstrate that while LLMs can generate executable LilyPond in zero-shot settings, they struggle with structural understanding tasks. The systematic comparison of metrics highlights the importance of using multiple evaluation approaches to capture model performance accurately.
Reproducibility
The authors provide a link to the GitHub repository containing the benchmark, prompt bank, and evaluation code, enhancing reproducibility. However, the paper could benefit from more detailed descriptions of the experimental setups and configurations used for the models.
Limitations
The paper acknowledges that structural understanding tasks remain challenging, indicating that while generation is feasible, deeper musical reasoning is not yet fully achievable. Additionally, the reliance on specific datasets may limit the generalizability of the findings.
Broader impact
This work has significant implications for the field of symbolic music generation and understanding, providing a standardized evaluation framework that can facilitate future research and development in this area. It opens avenues for improving LLM capabilities in music-related tasks, potentially influencing applications in music education, composition, and automated music analysis. The paper presents LilyBench, the first benchmark for evaluating symbolic music generation and understanding using LLMs, significantly advancing the field by providing a comprehensive evaluation framework and revealing critical insights into model performance.
Author of this paper? Send a correction.