Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Abstract
Most applications of generative AI involve a sequential interaction in which a person inputs a prompt and waits for a response, and where reaction time and adaptivity are not important factors. In contrast, live jamming is a collaborative interaction that requires real-time coordination and adaptation without access to the other player's future moves, while preserving diversity to sustain a creative flow. Reinforcement learning post-training enables effective adaptation through on-policy interaction, yet it often reduces output diversity by exploiting coherence-based rewards. This collapse, known as ``reward hacking'', affects many RL post-training pipelines, but is especially harmful in live jamming, where musical creativity relies on dynamic variation and mutual responsiveness. In this paper, we propose a novel adversarial training method on policy-generated trajectories to mitigate reward hacking in RL post-training for melody-to-chord accompaniment. A co-evolving discriminator separates policy trajectories from the data distribution, while the policy maximizes the discriminator output in addition to coherence rewards to prevent collapse to trivial outputs. We evaluate accompaniment quality and output diversity in simulation with both fixed test melodies and learned melody agents, and we conduct a user study with the model deployed in a real-time interactive system with expert musicians. Quantitative evaluation and user feedback demonstrate improved output diversity, harmonic coherence, adaptation speed and user agency. Our results demonstrate a simple yet effective method to mitigate reward hacking in RL post-training of generative sequence models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel adversarial training method that effectively mitigates reward hacking in reinforcement learning for generative music systems. This work significantly advances the field by improving the adaptability and creativity of AI in real-time music collaboration, showcasing the potential for more dynamic and responsive human-AI interactions.
Methodology
The paper introduces Generative Adversarial Post-Training (GAPT), a novel approach that combines reinforcement learning with adversarial training to mitigate reward hacking in real-time music accompaniment. The methodology is well-structured, employing a two-phase training schedule for the discriminator to stabilize learning and enhance output diversity. The integration of adversarial rewards alongside coherence rewards is a significant innovation, addressing the limitations of traditional RL post-training methods.
Results & evaluation
The experiments are comprehensive, involving both simulated and real-time interactions with expert musicians. The use of multiple datasets and the detailed evaluation metrics (note-in-chord ratio and Vendi Score) provide a robust framework for assessing the model's performance. The user study adds qualitative insights, demonstrating the model's effectiveness in real-world scenarios.
Reproducibility
The paper provides sufficient implementation details, including model architectures, training procedures, and hyperparameters. The release of code and datasets enhances reproducibility, allowing other researchers to validate the findings and build upon the work.
Limitations
While the paper addresses reward hacking effectively, it does not explore the potential computational costs associated with the adversarial training approach. Additionally, the user study is limited to a small sample size of 12 musicians, which may not fully capture the diversity of user experiences in live jamming scenarios.
Broader impact
The proposed method has significant implications for the field of generative music systems, particularly in enhancing human-AI collaboration in creative contexts. By improving adaptability and user agency, this research could lead to more engaging and interactive musical experiences, potentially influencing the design of future AI systems in creative domains. The paper presents a novel adversarial training method that effectively mitigates reward hacking in reinforcement learning for generative music systems. This work significantly advances the field by improving the adaptability and creativity of AI in real-time music collaboration, showcasing the potential for more dynamic and responsive human-AI interactions.
Author of this paper? Send a correction.