Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping
Abstract
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. In this paper, we introduce a cross-modal framework that learns implicit music styles from raw audio and applies them to symbolic music generation. Inspired by BLIP-2, our model leverages a Querying Transformer (Q-Former) to extract style representations from a large, pre-trained audio language model (LM), and further applies them to condition a symbolic LM for generating piano arrangements. We adopt a two-stage training strategy: contrastive learning to align auditory style with symbolic expression, followed by generative modeling for music arrangement. Our model generates piano performances jointly conditioned on a lead sheet (content) and a reference audio example (style), enabling controllable and stylistically faithful arrangement. Experiments demonstrate the effectiveness of our approach in piano cover generation, style transfer, and audio-to-MIDI retrieval, achieving substantial improvements in style-aware alignment and music quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Methodology
The paper proposes a cross-modal framework for piano arrangement by leveraging a Querying Transformer (Q-Former) to bridge audio and symbolic domains. The core innovation lies in using the Q-Former to extract implicit style representations from a frozen audio language model (MusicGen) and conditioning a symbolic language model (MuseCoco) with these representations, alongside a lead sheet for content. The two-stage training process—contrastive/matching learning for style alignment followed by generative modeling—is a logical extension of vision-language bootstrapping methods (like BLIP-2) to the audio-symbolic domain. The approach effectively disentangles content (lead sheet) from style (audio reference), addressing a significant gap in current music generation systems that often rely on text prompts or explicit symbolic controls. The use of LoRA adapters to condition the frozen symbolic LM is a practical and efficient design choice.
Results & evaluation
The experimental evaluation is comprehensive, covering objective metrics (MCA, CA, GPC, VCC, TA) and subjective listening tests. The inclusion of out-of-distribution datasets (Ballroom, GTZAN) strengthens the claim of generalization. The results show competitive performance against baselines like PiCoGen2 and A2M, particularly in style coherence metrics. The audio-to-MIDI retrieval task further validates the quality of the learned cross-modal representations. However, the lower content preservation scores on the in-distribution POP909 set compared to PCG2 are a notable weakness, attributed to errors in the upstream Sheetsage transcription. The subjective evaluation, while well-designed with ANOVA analysis, relies on a small sample size (21 participants), which limits the statistical power of the findings.
Reproducibility
The paper provides detailed model configurations, including parameter counts, training epochs, batch sizes, and optimizer settings. The use of open-source or widely available backbones (MusicGen, MuseCoco, Sheetsage) enhances reproducibility. The data pairing strategy and specific metric definitions are clearly described. The lack of a public code repository is a minor drawback, but the detailed appendix and demo page mitigate this to some extent.
Limitations
The authors acknowledge that the style representation is segment-level (4-bar), which may not capture long-term stylistic evolution or transitions. The reliance on Sheetsage for lead sheet extraction introduces potential error propagation. The method is currently limited to piano arrangement, and generalizing to multi-track or more complex instrumental arrangements is identified as future work. The subjective evaluation sample size is small.
Broader impact
This work contributes to the field of AI-assisted music creation by providing a tool for expressive piano arrangement. It enables musicians to reinterpret existing audio with specific stylistic nuances while maintaining harmonic and melodic integrity. The cross-modal representation learning technique has broader applications in audio-symbolic retrieval and style transfer. The potential for misuse in generating copyrighted music arrangements without permission is a consideration, though the focus on arrangement rather than direct cloning mitigates some risks. The paper presents a novel and technically sound approach to cross-modal music generation, effectively applying vision-language bootstrapping techniques to the audio-symbolic domain for piano arrangement.
Author of this paper? Send a correction.