Polyphonia: Zero-Shot Timbre Transfer in Polyphonic Music with Acoustic-Informed Attention Calibration
Abstract
The advancement of diffusion-based text-to-music generation has opened new avenues for zero-shot music editing. However, existing methods fail to achieve stem-specific timbre transfer, which requires altering specific stems while strictly preserving the background accompaniment. This limitation severely hinders practical application, since real-world production necessitates precise manipulation of components within dense mixtures. Our key finding is that, while vanilla cross-attention captures semantic features of stems, it lacks the spectral resolution to strictly localize targets in dense mixtures, leading to boundary leakage. To resolve this dilemma, we propose Polyphonia, a zero-shot editing framework with Acoustic-Informed Attention Calibration. Rather than relying solely on diffuse semantic attention, Polyphonia leverages a probabilistic acoustic prior to establish coarse boundaries, enabling non-target stems preserved precise semantic synthesis. For evaluation, we propose PolyEvalPrompts, a standardized prompt set with 1,170 timbre transfer tasks in polyphonic music. Specifically, Polyphonia achieves an increase of 15.5% in target alignment compared to baselines, while maintaining competitive music fidelity and non-target integrity.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this work is the introduction of Polyphonia, a framework that advances controllable music generation by enabling precise intra-stem editing in polyphonic music. This paper significantly enhances the state-of-the-art in audio processing by providing a novel methodology that effectively tackles the challenges of timbre transfer while maintaining the integrity of complex audio mixtures.
Methodology
The methodology presented in Polyphonia is innovative, addressing the critical challenge of stem-specific timbre transfer in polyphonic music through a novel framework that combines Acoustic-Informed Attention Calibration with a probabilistic acoustic prior. This dual-path mechanism effectively resolves semantic-acoustic misalignment, allowing for precise editing while maintaining the integrity of non-target stems. The use of Ideal Ratio Mask (IRM) as an acoustic prior is particularly noteworthy, as it enhances the model's ability to discern and manipulate specific audio components within complex mixtures. The paper also introduces a comprehensive evaluation framework, PolyEvalPrompts, which is essential for assessing the performance of the proposed method across a wide range of timbre transfer tasks.
Results & evaluation
The experiments conducted are robust, utilizing well-established datasets (MUSDB18-HQ and MusicDelta) and a diverse set of state-of-the-art baselines for comparison. The results demonstrate significant improvements in target alignment and non-target integrity, with quantitative metrics such as CLAP, LPAPS, and CQT1-PCC providing a thorough assessment of the model's performance. The inclusion of subjective evaluations through Mean Opinion Scores (MOS) adds depth to the experimental analysis, confirming the effectiveness of the proposed method in real-world applications.
Reproducibility
The paper provides detailed implementation information, including the architecture used (AudioLDM 2) and the specifics of the acoustic prior extraction process. However, the absence of a public code repository limits full reproducibility. While the methodology is well-documented, access to the actual implementation would enhance the ability of other researchers to replicate the results.
Limitations
One limitation noted is the reliance on pre-trained Blind Source Separation (BSS) models, which may introduce biases based on the training data used, potentially affecting the generalizability of the approach to non-Western music or less common instruments. Additionally, the method's performance may degrade in extremely dense mixtures, although it shows graceful degradation rather than catastrophic failure.
Broader impact
The potential applications of Polyphonia are significant, particularly in the realm of music production and editing, where precise control over individual audio stems is crucial. However, ethical considerations regarding intellectual property and the potential for misuse in unauthorized remixes must be addressed. The authors advocate for the development of provenance tracking and audio watermarking technologies to safeguard artistic integrity. The main contribution of this work is the introduction of Polyphonia, a framework that advances controllable music generation by enabling precise intra-stem editing in polyphonic music. This paper significantly enhances the state-of-the-art in audio processing by providing a novel methodology that effectively tackles the challenges of timbre transfer while maintaining the integrity of complex audio mixtures.
Author of this paper? Send a correction.