Libretto: Giving LLM Agents a Sense of Musical Structure
Abstract
Generative music systems can now produce impressive audio from text prompts, but audio outputs are difficult to inspect, edit, and diagnose as musical structure. We introduce Libretto, an agent-facing framework for symbolic music generation and revision. Libretto uses an LLM-native grammar with explicit onset slots, voices, and bar-level organization, then evaluates each piece in a corpus-calibrated statistical space over rhythm, harmony, melody, texture, form, and variation. The same structural axes support retrieval, diagnosis, copy-risk control, and iterative self-revision. Across gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation, Libretto turns symbolic music from a raw token sequence into a measurable and editable object for language-model agents.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Libretto presents a structured, agent-centric framework for symbolic music generation that leverages corpus-calibrated structural metrics to enable interpretable diagnosis and iterative self-revision, offering a significant methodological contribution to controllable AI music composition despite lacking perceptual validation.
Methodology
The paper introduces "Libretto," a framework designed to bridge the gap between generative audio models and symbolic music representation by creating an LLM-native grammar for symbolic music. The core methodological contribution is not a new neural architecture for generation, but rather a structured representation system and an evaluation loop. The grammar explicitly defines onset slots, voices, and bar-level organization, making the symbolic output directly editable and interpretable by an LLM agent. The evaluation mechanism relies on a "corpus-calibrated statistical cloud" comprising 29 structural axes (rhythm, harmony, melody, texture, form, variation) computed from the symbolic representation. These axes are mapped to percentiles against a reference corpus, allowing the agent to diagnose structural deviations (e.g., "too sparse," "harmonically unstable") rather than relying on black-box aesthetic scores. The agent loop involves generation, measurement against these axes, and iterative self-revision based on musician-readable feedback. This approach shifts the focus from end-to-end differentiable generation to a retrieval-augmented, self-correcting agentic workflow.
Results & evaluation
The authors evaluate the framework across four tasks: gap filling, reference-guided full-piece generation, gradual morphing, and educational music generation. They use a corpus of 314 MIDI files from the Lakh MIDI Dataset. The experiments demonstrate that the structural axes can distinguish between genres (e.g., Jazz vs. Folk on harmonic complexity) and that the agent loop improves pass rates for structural validity (e.g., gap-filling pass rate increased from 12% to significantly higher levels with the loop). The paper provides qualitative examples and quantitative metrics for copy-risk and structural degeneracy. However, the evaluation is largely internal and self-referential; it measures how well the generated pieces fit the *defined* structural axes and avoid copying, rather than assessing musical quality via human listening tests or comparison to state-of-the-art audio generation models (like Suno or Udio) in terms of perceptual quality. The dataset size (314 songs) is small for corpus calibration, though sufficient for the descriptive statistics used.
Reproducibility
The paper provides a clear description of the grammar, the 29 axes, and the evaluation gates. The code and project website are linked, which enhances reproducibility. The reliance on a specific LLM (Claude Code with Opus 4.8) for the agent loop is noted, which allows other researchers to replicate the agentic behavior, though the specific prompts and retrieval mechanisms would need to be carefully reconstructed from the text and code. The definition of the structural axes is mathematically precise in the appendix.
Limitations
A significant limitation is the lack of perceptual evaluation. The system optimizes for structural properties defined by the authors, but there is no evidence that these properties correlate with human judgments of musical quality or "goodness." The small reference corpus (314 songs) limits the generalizability of the statistical cloud, potentially biasing the "idiomatic" norms towards the specific genres present in that small set. The abstraction of the grammar (ignoring velocity, timbre, micro-timing) means it cannot capture expressive performance nuances, limiting its applicability to purely structural composition tasks. Furthermore, the reliance on a proprietary LLM (Claude) for the agent loop raises questions about the accessibility and cost of the method for broader research communities.
Broader impact
Libretto offers a novel perspective on AI-assisted music creation by treating symbolic music as a structured, editable object for LLM agents. This could empower musicians and educators by providing tools for targeted theory practice, gap-filling, and style exploration. It contributes to the broader field of AI creativity by demonstrating how structured representations can enhance the controllability and interpretability of generative models. However, the potential for generating high-quality, commercially viable music is limited by the lack of audio fidelity and expressive nuance in the current symbolic representation. Libretto presents a structured, agent-centric framework for symbolic music generation that leverages corpus-calibrated structural metrics to enable interpretable diagnosis and iterative self-revision, offering a significant methodological contribution to controllable AI music composition despite lacking perceptual validation.
Author of this paper? Send a correction.