RIME: Enabling Large-Scale Agentic Post-Production
Abstract
Almost every piece of recorded music you have ever heard was modified before it reached you; commercial releases rarely spring fully-formed from the mind of a musician. Despite the promise of music generation models for one-shot output, such fine-grained iterative refinement workflows are a complementary problem, and largely out of their reach. There is also a gap for musicians: while they can express what they want to hear, not all have the facility with studio production tools to implement the complex set of actions needed to realize these intuitions. We formalize this task as agentic post-production, wherein individual aspects of a song are targeted, refined, and combined into a final track. We argue the bottleneck is data: existing corpora do not reflect how realistic post-production chains map onto the vocabulary musicians and engineers actually use. We argue there is a language for modifying recorded music that is dense, consistent, and learnable. We introduce the Rule-based Instructions for Music Editing (RIME) framework, which generates realistic paired edit-instruction data from any baseline music dataset grounded in canonical methods, design patterns, and constraints derived from real production workflows. RIME leverages POEMS, a new toolkit that combines stem separation, mixing, and common studio effects for use by multimodal agents. We use RIME and POEMS to generate 3,000 pairs of edit instructions and ground truth audio, and use this data to evaluate existing multimodal LLMs as agents on this task, showing persistent challenges in current models' post-production capabilities. We also demonstrate RIME's ability to improve post-production agent performance via supervised fine-tuning. We see RIME as an early step toward iterative musical agents, collaborative systems that could transform music production much as interactive coding agents have reshaped software engineering.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
Methodology
The paper introduces RIME, a framework for generating synthetic edit-instruction data for music post-production, and POEMS, a toolkit exposing audio processing tools via the Model Context Protocol (MCP). The methodology is sound and addresses a specific gap: the lack of structured, iterative post-production data compared to one-shot generation data. The use of rule-based recipes, motifs, and parameter priors to generate grounded (input, output, instruction) triples is a clever approach to synthesizing training data. However, the novelty is somewhat limited by the reliance on existing, well-understood signal processing libraries (Pedalboard, Demucs) and standard LLM tool-calling architectures. The core contribution is the *system design* and *benchmark creation* rather than a new algorithmic breakthrough in audio synthesis or model architecture. The abstraction layer generation for prompts is a nice touch for simulating real-world ambiguity.
Results & evaluation
The experiments evaluate zero-shot and fine-tuned multimodal LLMs on the generated RIME benchmark. The results clearly demonstrate that current models struggle with agentic post-production, particularly in parameter inference and chain assembly. The supervised fine-tuning results show improvement, validating the utility of the synthetic data. The evaluation metrics (FAD, KAD, Edit Similarity, Graph F1) are appropriate for the task. The inclusion of an artifact-removal ("poisoning") split adds depth. However, the dataset size (3,000 pairs) is relatively small for modern LLM benchmarks, though reasonable for this specific, high-fidelity audio task. The reliance on MERT embeddings for audio similarity is standard but has known limitations in capturing perceptual quality compared to human listening tests, which are absent here.
Reproducibility
The paper provides detailed descriptions of the POEMS toolkit, recipe schemas, and agent architecture. The use of open-weight models for fine-tuning and standard libraries for audio processing aids reproducibility. However, the specific implementation details of the "prompt templates" and the exact "parameter priors" are referenced but not fully enumerated in the main text (though likely in the appendix, which is partially provided). The lack of a public code repository URL in the provided text is a minor hindrance, but the description is sufficient for a competent researcher to reconstruct the system.
Limitations
The primary limitation is the dependence on the quality of the underlying stem separation model (Demucs), which can introduce artifacts. The rule-based nature of RIME means it can only generate edits present in the recipe catalog, limiting the diversity of edits compared to fully generative approaches. The evaluation relies heavily on embedding-based metrics, which may not correlate perfectly with human preference for audio quality. The dataset size is small, potentially limiting the generalizability of the fine-tuned models. The focus on specific instrument stems (vocals, drums, bass, etc.) via Demucs limits the scope to tracks where these stems can be reliably separated.
Broader impact
This work has significant potential impact on the field of AI-assisted music production. By formalizing agentic post-production and providing a benchmark, it opens up a new avenue for research beyond one-shot generation. It highlights the challenges of iterative, tool-using agents in the audio domain. The framework could empower musicians with less technical expertise to achieve professional-quality mixes through natural language interaction. It also serves as a testbed for improving multimodal LLMs' reasoning and tool-use capabilities in a complex, structured domain. This paper presents RIME and POEMS, a framework and toolkit for generating synthetic data and benchmarking agentic post-production agents, addressing the gap between one-shot music generation and iterative studio workflows.
Author of this paper? Send a correction.