Mix2Morph: Learning Sound Morphing from Noisy Mixes
Abstract
We introduce Mix2Morph, a text-to-audio diffusion model fine-tuned to perform sound morphing without a dedicated dataset of morphs. By finetuning on noisy surrogate mixes at higher diffusion timesteps, Mix2Morph yields stable, perceptually coherent morphs that convincingly integrate qualities of both sources. We specifically target sound infusions, a practically and perceptually motivated subclass of morphing in which one sound acts as the dominant primary source, providing overall temporal and structural behavior, while a secondary sound is infused throughout, enriching its timbral and textural qualities. Objective evaluations and listening tests show that Mix2Morph outperforms prior baselines and produces high-quality sound infusions across diverse categories, representing a step toward more controllable and concept-driven tools for sound design. Sound examples are available at https://anniejchu.github.io/mix2morph .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Mix2Morph represents a substantial advancement in sound morphing techniques, leveraging innovative training strategies and augmentation methods to produce high-quality audio infusions. The comprehensive evaluation of its performance against existing models highlights its potential to enhance sound design practices significantly.
Methodology
The paper presents a novel approach to sound morphing through the Mix2Morph model, which utilizes a finetuning strategy on noisy surrogate mixes. This method allows the model to learn morphing without the need for a dedicated morph dataset, addressing a significant limitation in the field. The use of higher diffusion timesteps to focus on capturing high-level morphing concepts while suppressing low-level artifacts is particularly innovative. The augmentation techniques, including temporal and spectral alignment, are well-justified and enhance the model's ability to produce coherent morphs. However, the methodology could benefit from a more detailed discussion on the choice of augmentation modes and their impact on the results.
Results & evaluation
The experiments are comprehensive, evaluating Mix2Morph against several baselines through both objective metrics and subjective listening tests. The paper provides a clear rationale for the selection of sound pairs and the design of the evaluation metrics, including the Latent Compressibility Score (LCS) and directionality measures. The results demonstrate that Mix2Morph consistently outperforms existing methods, showcasing its effectiveness in generating high-quality sound infusions. The statistical analysis of the subjective evaluations adds robustness to the findings.
Reproducibility
The paper includes sufficient details regarding the model architecture and training procedures, allowing for reproducibility. However, the lack of a publicly available code repository limits the ease with which other researchers can replicate the results. Providing access to the training data or a similar dataset would further enhance reproducibility.
Limitations
One limitation is the reliance on noisy surrogate mixes, which may not fully capture the complexity of high-quality morphs. Additionally, while the model shows improvements over baselines, there may still be cases where perceptual coherence is not fully achieved, particularly with more complex sound pairs. The subjective evaluation is limited to a small sample size, which may not represent the broader community's perceptions.
Broader impact
The Mix2Morph model has significant implications for sound design, particularly in fields such as film, gaming, and virtual reality, where high-quality sound morphing is essential for creating immersive experiences. The ability to generate sound infusions without extensive datasets opens new avenues for creativity and exploration in audio production. Mix2Morph represents a substantial advancement in sound morphing techniques, leveraging innovative training strategies and augmentation methods to produce high-quality audio infusions. The comprehensive evaluation of its performance against existing models highlights its potential to enhance sound design practices significantly.
Author of this paper? Send a correction.