Remix the Timbre: Diffusion-Based Style Transfer Across Polyphonic Stems
Abstract
Timbre transfer aims to modify the timbral identity of a musical recording while preserving the original melody and rhythm. While single-instrument timbre transfer has made substantial progress, existing approaches to multi-instrument settings rely on separate-then-transfer pipelines that propagate source separation artifacts and produce incoherent synthesized timbres across stems. This paper proposes MixtureTT, to the best of our knowledge the first system for flexible per-stem timbre transfer directly from a polyphonic mixture. Given a mixture and a separate timbre reference for each target voice, MixtureTT jointly transfers all stems to the specified instruments through a shared diffusion process. Modeling the dependencies across the per-stem content and cross-stem harmonic, the proposed joint stem diffusion transformer eliminates cascaded separation error, reduces inference cost by a factor equal to the number of stems, and yields more coherent multi-stem outputs. Despite operating under a strictly harder input condition, evaluations on the SATB choral dataset show that MixtureTT outperforms single-instrument baselines on both objective and subjective metrics demonstrating the necessity of dedicated multi-instrument timbre transfer over the naive separate-then-transfer pipelines. As a result, this work confirms that the cross-stem modeling is essential for mixture-level timbre transfer as the proposed joint setting consistently exceeds an equivalent single-stem ablation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents MixtureTT, a novel approach for per-stem timbre transfer directly from polyphonic mixtures, significantly advancing the field of music audio processing. The technical contributions, particularly the joint diffusion model and its evaluation, mark a meaningful step towards more coherent and efficient audio manipulation techniques.
Methodology
The proposed methodology, MixtureTT, innovatively addresses the challenge of timbre transfer in polyphonic music by employing a joint stem diffusion transformer that operates directly on polyphonic mixtures without requiring explicit source separation. This approach is a significant departure from traditional separate-then-transfer pipelines, which are prone to artifacts and inconsistencies. The architecture effectively balances per-stem independence with cross-stem coordination, allowing for coherent audio generation. The use of a shared diffusion process and the introduction of disentanglement losses further enhance the model's ability to maintain timbral fidelity while preserving content integrity.
Results & evaluation
The experimental evaluation is robust, utilizing both objective metrics (e.g., Fréchet Audio Distance, Jaccard Distance, Chroma Cosine Similarity) and subjective assessments through a listening test with human participants. The results consistently demonstrate that MixtureTT outperforms single-instrument baselines, even when those baselines are provided with isolated stems. This is a strong validation of the proposed method's effectiveness. The use of the SATB choral dataset is appropriate, though it may limit generalizability to other musical contexts.
Reproducibility
While the paper details the training process and architecture, the lack of specific implementation details, such as hyperparameters and the exact training environment, may hinder reproducibility. The authors mention a demo URL, which could provide additional insights into the model's performance, but the absence of a public code repository is a drawback.
Limitations
One limitation is the reliance on a specific dataset (CocoChorales), which may not fully represent the diversity of musical styles and genres. Additionally, while the model shows promise, the scalability to larger ensembles or more complex musical structures remains untested. The paper also does not address potential computational costs associated with training and inference, which could be a barrier for broader adoption.
Broader impact
The implications of this work are significant for music production and audio engineering, as it enables more flexible and coherent manipulation of musical recordings. This could streamline workflows for musicians and producers, allowing for innovative creative possibilities in music composition and arrangement. Furthermore, the findings encourage further exploration of mixture-level modeling in generative music tasks, potentially influencing future research directions. This paper presents MixtureTT, a novel approach for per-stem timbre transfer directly from polyphonic mixtures, significantly advancing the field of music audio processing. The technical contributions, particularly the joint diffusion model and its evaluation, mark a meaningful step towards more coherent and efficient audio manipulation techniques.
Author of this paper? Send a correction.