Music Inpainting With Similarity-Guided Latent Diffusion
Abstract
We propose a music inpainting technique combining similarity matching and a diffusion-based generative model to replace large segments of audio recordings with suitable alternatives. The proposed hybrid method, Similarity-Guided Diffusion Posterior Sampling (SimDPS), retrieves candidate replacement segments from the wider signal based on contextual similarity and uses them to guide the diffusion process toward coherent reconstructions. Being model-agnostic, the framework is compatible with any pre-trained diffusion model operating in either the waveform or latent domain. Objective evaluations demonstrate that SimDPS outperforms widely adopted RePaint and DPS methods across acoustic features and neural-embedding distance metrics. Subjective evaluations on multi-genre music datasets confirm that SimDPS enhances perceptual plausibility compared to previous diffusion-based methods and demonstrates robustness to poorly matched similarity candidates. These findings pave the way for improved music restoration and context-aware audio editing.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[The paper introduces SimDPS, a training-free, model-agnostic framework that enhances long-gap music inpainting by guiding diffusion posterior sampling with similarity-retrieved candidates, effectively combining the structural reliability of retrieval with the generative flexibility of diffusion models to improve perceptual plausibility and musical coherence.].
Methodology
The paper proposes SimDPS, a hybrid framework for long-gap music inpainting that integrates similarity-based retrieval with diffusion posterior sampling (DPS). The core innovation lies in the "Similarity-Guided" mechanism, where retrieved candidate segments from the source signal act as an auxiliary noisy observation in the missing region. This allows the diffusion model to be guided by structural priors from the track itself, mitigating the "identity drift" common in pure generative approaches for long gaps. The method is model-agnostic, supporting both latent (ACE-Step) and waveform (MR-CQTdiff) domains. The introduction of a multi-candidate ensemble (SimDPS-Multi) with a spectral loss function to handle conflicting gradients is a thoughtful technical addition, although its effectiveness is context-dependent.
Results & evaluation
The evaluation is robust, covering both objective metrics (MFCC, Chroma, Rhythm, Boundary Error, MusicFM, CLAP) and subjective listening tests (MUSHRA). The use of MUSDB18-HQ for general music and MAESTRO for piano provides a good breadth of genres. The results clearly show SimDPS outperforming baselines like RePaint, DPS, and pure similarity search, particularly in preserving musical coherence over 5-second gaps. The subjective tests confirm that the method improves perceptual plausibility, especially when the retrieved similarity match is imperfect.
Reproducibility
The paper provides high reproducibility. It specifies the exact models used (ACE-Step 1.5, MR-CQTdiff), hyperparameters (noise schedules, guidance weights, context lengths), and datasets. Code and audio examples are available via the provided URL. The detailed description of the similarity search procedure and the modified likelihood formulation allows for faithful re-implementation.
Limitations
The method relies on the availability of a source corpus (the rest of the track) for similarity search, which limits its applicability to scenarios where the full track is not available or where the missing segment is unique (no repetition). The subjective evaluation sample size (23 and 17 participants) is modest. The performance of SimDPS-Multi is inconsistent across genres, sometimes performing worse than SimDPS-Solo or even baselines when the retrieved matches are poor or conflicting.
Broader impact
This work contributes to the field of audio restoration and editing by providing a practical, training-free method for long-gap inpainting. It bridges the gap between deterministic retrieval methods and stochastic generative models, offering a path toward more robust audio editing tools. The model-agnostic nature of the framework makes it applicable to future diffusion-based audio models. [The paper introduces SimDPS, a training-free, model-agnostic framework that enhances long-gap music inpainting by guiding diffusion posterior sampling with similarity-retrieved candidates, effectively combining the structural reliability of retrieval with the generative flexibility of diffusion models to improve perceptual plausibility and musical coherence.].
Author of this paper? Send a correction.