YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality
Abstract
Symbolic models make melody, harmony, rhythm, and form explicit but typically stop before a finished recording; audio models produce complete songs while leaving composition implicit. We introduce YuE2, which unifies symbolic and audio music generation at frontier quality through symbolic planning. A single AR-NAR Mixture-of-Transformers (MoT) first writes a readable score specifying melody and harmony, expands it into semantic music tokens, and realizes it as full-song audio. In comparisons using the same checkpoint, experts prefer symbolic planning for overall quality and musicality, with 49.3% of overall preferences versus 34.6% without planning. Experts also favor the unified model over a separate language model and diffusion Transformer. On WildSongBench, YuE2 scores 6.73 on SongBench Global Avg, exceeding all evaluated public baselines. Selecting from eight candidates (best-of-8), YuE2 reaches 6.96, the highest observed mean among all evaluated systems. Expert listening further establishes its competitiveness with proprietary song generators, favoring best-of-8 over Suno v4.5 and yielding nearly balanced preferences against Suno v5. To learn this generation process from recordings without aligned scores, we introduce MERT2 and SheetSage2 to supply semantic and symbolic supervision. MERT2 sets a new state of the art in music representation learning, surpassing previous best results on 14 of 15 MARBLE metrics; SheetSage2 leads 12 of 15 benchmark-metric pairs in our lead-sheet transcription comparison. The same checkpoint follows score edits while largely preserving unedited musical content and generates zero-shot covers without cover-specific training. Its readable score also enables agentic music editing, with external language models translating user feedback into revisions of the composition.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
YuE2 introduces a unified AR-NAR Mixture-of-Transformers architecture that unifies symbolic and audio music generation through a symbolic planning stage, achieving frontier quality and enabling editable, controllable music creation. The paper demonstrates that explicitly modeling composition (score) before audio realization improves perceived quality and musicality, supported by strong benchmark results and expert preferences. The introduction of MERT2 and SheetSage2 provides robust semantic and symbolic supervision, setting new standards in music representation learning and transcription. This work represents a significant step towards interpretable and controllable generative music systems, bridging the gap between symbolic composition and audio production.
Methodology
The paper proposes a unified architecture (YuE2) that bridges symbolic and audio music generation using an AR-NAR Mixture-of-Transformers (MoT). The core methodological contribution is the "symbolic planning" stage, where the model first generates a readable score (melody, harmony, form) before expanding into semantic tokens and finally acoustic latents via flow matching. This hierarchical approach allows for explicit control over composition. Additionally, the paper introduces two auxiliary models: MERT2 for semantic representation learning and SheetSage2 for symbolic supervision (lead-sheet transcription), which are used to train the main model on unaligned audio data. The integration of these components into a single checkpoint that supports editing and cover generation is a significant architectural advancement.
Results & evaluation
The evaluation is extensive, utilizing both objective benchmarks (WildSongBench, SongBench, MARBLE) and subjective expert listening tests. The results show YuE2 outperforming public baselines and being competitive with proprietary systems like Suno v4.5 and v5. The ablation study comparing generation with and without symbolic planning demonstrates a clear preference for the planned approach (49.3% vs 34.6%). The introduction of MERT2 and SheetSage2 is supported by state-of-the-art results on their respective benchmarks (MARBLE and transcription metrics). The inclusion of zero-shot cover generation and agentic editing case studies further validates the utility of the unified framework.
Reproducibility
As a technical report from an industry lab (HKGAI), the paper likely lacks full open-source code and model weights at the time of release, which limits immediate reproducibility. However, the detailed description of the architecture (AR-NAR MoT, flow matching) and the specific benchmarks used provides a clear roadmap for replication. The reliance on proprietary datasets for training the auxiliary models may pose a barrier for independent researchers.
Limitations
The paper relies heavily on expert listening tests, which can be subjective and difficult to scale. The comparison with proprietary systems (Suno) is limited to specific versions and may not reflect the full capability of those closed-source models. The "best-of-8" selection strategy, while effective for benchmarking, introduces a computational overhead that may not be practical for real-time applications. Additionally, the paper does not extensively discuss the computational cost of the multi-stage generation process compared to direct audio generation.
Broader impact
This work has significant implications for the music production industry by providing a tool that allows for both high-quality audio generation and explicit compositional control. The ability to edit scores and generate covers zero-shot opens up new possibilities for music creation, education, and collaboration. The unification of symbolic and audio domains could lead to more interpretable and controllable generative music systems, potentially influencing how AI is integrated into professional music workflows. YuE2 introduces a unified AR-NAR Mixture-of-Transformers architecture that unifies symbolic and audio music generation through a symbolic planning stage, achieving frontier quality and enabling editable, controllable music creation. The paper demonstrates that explicitly modeling composition (score) before audio realization improves perceived quality and musicality, supported by strong benchmark results and expert preferences. The introduction of MERT2 and SheetSage2 provides robust semantic and symbolic supervision, setting new standards in music representation learning and transcription. This work represents a significant step towards interpretable and controllable generative music systems, bridging the gap between symbolic composition and audio production.
Author of this paper? Send a correction.