SPHERE: Automatic Music Upmixing via Audio Language Model Post-Training with Spatial Heuristic Rewards
Abstract
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations from existing ALMs, which encode both music semantics and mixing knowledge. Specifically, we propose a post-training recipe that first employs rejection sampling SFT, followed by reinforcement learning (RL) with verifiable rewards (RLVR) via GRPO. We propose Sphere (Spatial Heuristic Rewards), a deterministic reward suite inspired by music mixing conventions, to guide our post-training. It consists of 6 perceptually-motivated sub-rewards and encourages the output mix to be centered, balanced and spacious. More broadly, our results suggest that expert domain knowledge can be encoded as verifiable rewards and distilled into language models, without task-specific architectures.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
Methodology
The paper proposes a novel post-training recipe for Audio Language Models (ALMs) to perform automatic music upmixing. The core method involves two stages: Rejection Sampling Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR) using Group Relative Policy Optimization (GRPO). The key innovation is the "Sphere" reward function, a deterministic suite of six perceptually-motivated sub-rewards (centering, spread, level balance, spotlight prominence, per-role consistency) derived from music mixing conventions. The authors demonstrate that encoding expert domain knowledge as verifiable rewards allows a small student model (Qwen2.5-Omni 3B) to surpass larger frontier models (Gemini 2.5 Pro) in both objective reward scores and subjective human preference. The methodology is sound, leveraging the semantic understanding of ALMs rather than task-specific encoders, and effectively addresses the lack of paired spatial mixing data by using a teacher model to generate high-quality synthetic data filtered by the reward function.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors use the MedleyDB dataset, segmenting it into 30-second windows to fit context limits. They conduct a subjective A/B listening test with 44 participants, showing a 76.8% preference for high-reward mixes, validating the alignment of the Sphere reward with human perception. Objective evaluation on 618 held-out test points shows the post-trained model achieving a total reward of 4.981, significantly outperforming the zero-shot teacher (4.548) and other baselines. Ablation studies are thorough, demonstrating that data quality (stricter filtering) is more important than quantity for SFT, and that removing specific sub-rewards leads to reward hacking (e.g., collapsing spatial spread). The inclusion of a multi-stimulus preference test with expert listeners further strengthens the results, showing consistent improvements from SFT to SFT+RL.
Reproducibility
The paper provides detailed descriptions of the training setup, including hyperparameters for LoRA, learning rates, and the RL process (GRPO with PPO-style clipping). The prompt template is fully disclosed in the appendix, which is crucial for reproducibility given the reliance on prompt engineering. However, the use of proprietary models (Gemini 2.5 Pro as teacher) and a proprietary HRTF spatializer limits full reproducibility. The code and data are not explicitly linked in the provided text, though a project demo website is mentioned. The detailed ablation on reward shaping (Gaussian vs. Sigmoid vs. Step) provides valuable insights for practitioners.
Limitations
The primary limitation is the context window constraint, which limits the number of stems (max 29) and segment length (30 seconds), excluding 23.5% of the dataset. This may not capture long-term musical dynamics. Additionally, the reliance on a proprietary teacher model for data generation introduces potential biases. The subjective tests, while statistically significant, involve a relatively small number of participants (44 for A/B, 10 for multi-stimulus). The reward function, while aligned with human preference, is heuristic-based and may not capture all aspects of musical aesthetics.
Broader impact
This work demonstrates a practical recipe for distilling expert domain knowledge into language models via verifiable rewards, applicable beyond music mixing to other data-limited domains. It highlights the potential of ALMs for complex audio tasks that require semantic understanding, moving beyond simple feature extraction. The approach could inspire similar post-training strategies for other audio generation or processing tasks where ground truth data is scarce but expert heuristics are available. The paper introduces a novel post-training framework for Audio Language Models to perform automatic music upmixing, leveraging a deterministic reward suite (Sphere) inspired by mixing conventions to guide SFT and RL, resulting in a small model that outperforms frontier LLMs in both objective and subjective evaluations.
Author of this paper? Send a correction.