SyMuPe: Affective and Controllable Symbolic Music Performance
Abstract
Emotions are fundamental to the creation and perception of music performances. However, achieving human-like expression and emotion through machine learning models for performance rendering remains a challenging task. In this work, we present SyMuPe, a novel framework for developing and training affective and controllable symbolic piano performance models. Our flagship model, PianoFlow, uses conditional flow matching trained to solve diverse multi-mask performance inpainting tasks. By design, it supports both unconditional generation and infilling of music performance features. For training, we use a curated, cleaned dataset of 2,968 hours of aligned musical scores and expressive MIDI performances. For text and emotion control, we integrate a piano performance emotion classifier and tune PianoFlow with the emotion-weighted Flan-T5 text embeddings provided as conditional inputs. Objective and subjective evaluations against transformer-based baselines and existing models show that PianoFlow not only outperforms other approaches, but also achieves performance quality comparable to that of human-recorded and transcribed MIDI samples. For emotion control, we present and analyze samples generated under different text conditioning scenarios. The developed model can be integrated into interactive applications, contributing to the creation of more accessible and engaging music performance systems.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant advancement in the field of machine learning for music performance, introducing a novel framework and model that effectively captures the nuances of expressive piano playing while allowing for intuitive control through emotional and textual inputs. The comprehensive evaluation and innovative methodology position this work as a valuable contribution to both the academic and practical realms of music technology.
Methodology
The methodology presented in this paper is robust and innovative, particularly the introduction of the PianoFlow model which utilizes conditional flow matching for expressive music performance rendering. The integration of emotion classification and text embeddings for control over performance adds a significant layer of sophistication. The authors also provide a comprehensive framework (SyMuPe) for tokenizing and modeling symbolic music, which is a notable advancement in the field. The use of a curated dataset of 2,968 hours of aligned musical scores and expressive MIDI performances is commendable, as it addresses the limitations of existing datasets.
Results & evaluation
The experimental evaluation is thorough, employing both objective metrics and subjective listening tests to validate the performance of the PianoFlow model against transformer-based baselines and external models. The results indicate that PianoFlow achieves superior performance in terms of expressiveness and quality, with a high win rate in listener preference tests. The use of diverse evaluation methods strengthens the findings and demonstrates the model's capabilities effectively.
Reproducibility
The paper provides detailed descriptions of the model architecture, training configurations, and evaluation methodologies, which are essential for reproducibility. However, the absence of a public code repository limits the ability for other researchers to replicate the experiments fully. The authors mention the use of a supercomputer for their experiments, which may not be accessible to all researchers, potentially hindering reproducibility.
Limitations
The authors acknowledge several limitations, including the approximation of pedal effects and the handling of trills, which could impact the realism of the generated performances. Additionally, the emotion classifier's limitations and the potential for arbitrary text inputs to produce undesirable results are noted. The lack of ablation studies to assess the impact of different components on performance is also a shortcoming.
Broader impact
The potential applications of this research are significant, as it contributes to the development of more intelligent and controllable music performance systems. The ability to generate expressive music performances that can be controlled through text and emotion inputs may enhance interactive applications in music education, entertainment, and therapy. This work could pave the way for more accessible music composition tools, fostering creativity among users with varying musical backgrounds. This paper presents a significant advancement in the field of machine learning for music performance, introducing a novel framework and model that effectively captures the nuances of expressive piano playing while allowing for intuitive control through emotional and textual inputs. The comprehensive evaluation and innovative methodology position this work as a valuable contribution to both the academic and practical realms of music technology.
Author of this paper? Send a correction.