The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity
Abstract
While generative models for music composition are increasingly capable, their adoption by musicians is hindered by text-prompting, an asynchronous workflow disconnected from the embodied, responsive nature of instrumental performance. To address this, we introduce Aria-Duet, an interactive system facilitating a real-time musical duet between a human pianist and Aria, a state-of-the-art generative model, using a Yamaha Disklavier as a shared physical interface. The framework enables a turn-taking collaboration: the user performs, signals a handover, and the model generates a coherent continuation performed acoustically on the piano. Beyond describing the technical architecture enabling this low-latency interaction, we analyze the system's output from a musicological perspective, finding the model can maintain stylistic semantics and develop coherent phrasal ideas, demonstrating that such embodied systems can engage in musically sophisticated dialogue and open a promising new path for human-AI co-creation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Aria-Duet, a novel system that facilitates real-time musical collaboration between humans and AI, addressing critical interaction challenges and demonstrating the potential for sophisticated musical dialogue. This work significantly advances the field of AI in music by integrating state-of-the-art generative models with embodied performance, paving the way for future explorations in human-AI co-creation.
Methodology
The paper presents a well-structured methodology that integrates a generative model with a real-time interactive system for musical co-creation. The use of a Yamaha Disklavier as a physical interface is innovative, allowing for a more embodied interaction between the human performer and the AI. The authors address critical engineering challenges such as latency and coherence in musical transitions, demonstrating a deep understanding of both the technical and artistic aspects of music performance. The continuous prefill strategy and the custom playback adjustments are particularly noteworthy as they enhance the user experience significantly.
Results & evaluation
The paper includes a demonstration of the system's capabilities through a video showcasing various musical prompts. The analysis of the system's output from a musicological perspective is thorough, providing insights into how well the model maintains stylistic semantics and coherence in its continuations. However, the paper lacks quantitative metrics or formal evaluations of the system's performance, which would strengthen the claims made about its effectiveness.
Reproducibility
The paper provides a GitHub repository link for the Aria-Duet system, which is a positive aspect for reproducibility. However, detailed implementation specifics regarding the model training and the real-time engine setup are somewhat limited, which may pose challenges for other researchers attempting to replicate the work.
Limitations
The paper acknowledges the need for further research to assess the broader impact of the Aria-Duet system on creativity and co-creation. Additionally, while the system shows promise, the authors note that it may not always generate highly inventive outputs, particularly with novel compositions. The reliance on a specific hardware setup (Disklavier) could also limit accessibility for some users.
Broader impact
The potential applications of this work are significant, as it opens up new avenues for human-AI collaboration in music composition. By addressing the interaction challenges faced by musicians when using AI tools, this research could help foster greater engagement with AI in creative processes. The implications extend beyond music, suggesting that similar approaches could be applied in other artistic domains where real-time interaction is crucial. The main contribution of this paper is the introduction of Aria-Duet, a novel system that facilitates real-time musical collaboration between humans and AI, addressing critical interaction challenges and demonstrating the potential for sophisticated musical dialogue. This work significantly advances the field of AI in music by integrating state-of-the-art generative models with embodied performance, paving the way for future explorations in human-AI co-creation.
Author of this paper? Send a correction.