ProCraft: Instruction-Driven Music Production from Free-Form Text
Abstract
Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoising and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Methodology
The paper proposes ProCraft, a 0.5B parameter autoregressive codec language model for music editing. The methodology is sound, leveraging the InspireMusic backbone and WavTokenizer. The introduction of a "persistent edit head" with gated cross-attention to maintain instruction relevance over long sequences is a logical architectural choice for autoregressive models. The "On-the-fly Editing-pair Synthesis" is a clever data generation strategy using a tool-calling LLM to create compound instruction-audio pairs, addressing a significant data scarcity issue in this niche. The alignment of intermediate layers with MERT and pitch/instrument targets is a strong technical contribution to preserve unedited content.
Results & evaluation
The evaluation is comprehensive, covering seven operations across eight instruments. The use of both synthetic (Slakh) and real (MoisesDB) data for out-of-domain testing is rigorous. The metrics are well-chosen, combining general audio metrics (FAD, CLAP) with task-specific ones (SI-SDR, edit rate, preservation metrics). The user study, while small (16 listeners), provides necessary subjective validation. The model outperforms baselines on compound instructions, which is the core claim.
Reproducibility
The paper promises to release code and weights. The training procedure, including the two-pass training for the edit head, is described in detail. The data synthesis pipeline is explained, though the specific prompts and tool-calling logic for the LLM are likely in the appendix. The use of standard components (WavTokenizer, MERT) aids reproducibility.
Limitations
The model is limited to 30-second excerpts, which is short for full music production workflows. The user study is small. The model is trained primarily on synthetic data (Slakh) and tested on real data, which may limit generalization to complex, real-world mixes with non-standard instruments or production styles. The 0.5B parameter size, while efficient, may limit the complexity of creative edits compared to larger models.
Broader impact
This work contributes to the field of controllable audio generation by demonstrating that a single lightweight model can handle diverse and compound editing tasks. It bridges the gap between task-specific models and general-purpose text-guided editors. The data synthesis technique is applicable to other audio editing tasks. ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Author of this paper? Send a correction.