SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton
Abstract
Generating symphonic music requires simultaneously managing high-level structural form and dense, multi-track orchestration. Existing symbolic models often struggle with a "complexity-control imbalance", in which scaling bottlenecks limit long-term granular steerability. We present SymphonyGen, a 3D hierarchical framework for contemporary cinematic orchestration. SymphonyGen employs a cascading decoder architecture that decomposes the Bar, Track, and Event axes, improving computational efficiency and scalability over conventional 1D or 2D models. We introduce "short-score" conditioning via a beat-quantized multi-voice harmony skeleton, enabling outline control while preserving textural diversity. The model is further refined using Group Relative Policy Optimization (GRPO) with a cross-modal audio-perceptual reward, aligning symbolic output with modern acoustic expectations. Additionally, we implement a dissonance-averse sampling algorithm to suppress unintended tonal clashes during inference. Objective evaluations show that both reinforcement learning and dissonance-averse sampling effectively enhance harmonic cleanliness while maintaining melodic expression. Subjective evaluations demonstrate that SymphonyGen outperforms baselines in musicality and preference for orchestral music generation. Demo page: https://symphonygen.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of SymphonyGen, a novel 3D hierarchical framework for orchestral music generation that effectively addresses the complexities of high-level structural form and dense orchestration. This work represents a substantial advancement in the field of AI music generation, combining innovative methodologies with rigorous evaluation to produce a system that aligns closely with modern acoustic expectations.
Methodology
The paper introduces a 3D hierarchical architecture that effectively manages the complexities of orchestral music generation by decomposing the task into Bar, Track, and Event levels. This cascading decoder architecture enhances computational efficiency and scalability, which is a significant improvement over conventional models. The introduction of a "short-score" conditioning via a beat-quantized multi-voice harmony skeleton is innovative, allowing for greater control over the generated music while maintaining textural diversity. The use of Group Relative Policy Optimization (GRPO) with a cross-modal audio-perceptual reward is a novel approach that aligns the generated symbolic output with acoustic expectations, addressing the limitations of previous models. The dissonance-averse sampling algorithm further refines the output by suppressing unintended tonal clashes, showcasing a thoughtful integration of music theory into the generative process.
Results & evaluation
The experimental design is robust, featuring both objective and subjective evaluations. The use of a large dataset (SymphonyNet) for training and validation ensures that the model is well-tested across various orchestral styles. Objective metrics such as harmony precision, recall, and dissonance scores provide quantitative assessments of the model's performance, while subjective evaluations involving listener preferences add qualitative insights. The results indicate that SymphonyGen outperforms baseline models in terms of musicality and preference, particularly among general listeners, which is a strong endorsement of its effectiveness.
Reproducibility
The paper provides detailed implementation information, including architecture specifications, training procedures, and evaluation metrics. However, the absence of a publicly available code repository limits reproducibility. The authors mention that implementation details will be available in their codebase, but without immediate access, it is challenging to fully assess reproducibility.
Limitations
The paper acknowledges some limitations, such as the potential for "strange" harmonies or "noisy" segments in the generated music, which may stem from errors in harmony skeleton generation. Additionally, the subjective evaluations indicate that while the model performs well, it may still produce overly full orchestrations at times, suggesting room for improvement in balancing orchestration richness with clarity.
Broader impact
SymphonyGen has significant implications for the field of AI-assisted music composition, particularly in cinematic orchestration. By providing a controllable framework for composers, it enhances the collaborative potential between human creativity and AI-generated music. The model's ability to produce high-quality orchestral compositions could benefit various applications, including film scoring, video game music, and other multimedia projects, ultimately enriching the landscape of contemporary music creation. The main contribution of this paper is the development of SymphonyGen, a novel 3D hierarchical framework for orchestral music generation that effectively addresses the complexities of high-level structural form and dense orchestration. This work represents a substantial advancement in the field of AI music generation, combining innovative methodologies with rigorous evaluation to produce a system that aligns closely with modern acoustic expectations.
Author of this paper? Send a correction.