AnyAccomp: Generalizable Accompaniment Generation via Quantized Melodic Bottleneck
Abstract
Singing Accompaniment Generation (SAG) is the process of generating instrumental music for a given clean vocal input. However, existing SAG techniques use source-separated vocals as input and overfit to separation artifacts. This creates a critical train-test mismatch, leading to failure on clean, real-world vocal inputs. We introduce AnyAccomp, a framework that resolves this by decoupling accompaniment generation from source-dependent artifacts. AnyAccomp first employs a quantized melodic bottleneck, using a chromagram and a VQ-VAE to extract a discrete and timbre-invariant representation of the core melody. A subsequent flow-matching model then generates the accompaniment conditioned on these robust codes. Experiments show AnyAccomp achieves competitive performance on separated-vocal benchmarks while significantly outperforming baselines on generalization test sets of clean studio vocals and, notably, solo instrumental tracks. This demonstrates a qualitative leap in generalization, enabling robust accompaniment for instruments - a task where existing models completely fail - and paving the way for more versatile music co-creation tools. Demo audio and code: https://anyaccomp.github.io
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of AnyAccomp, a framework that effectively resolves the train-test mismatch in singing accompaniment generation by utilizing a quantized melodic bottleneck to enhance generalization capabilities. This work represents a significant advancement in the field of audio machine learning, addressing critical challenges and paving the way for more versatile music co-creation tools.
Methodology
The paper introduces a novel two-stage framework, AnyAccomp, which effectively decouples accompaniment generation from source-dependent artifacts through a quantized melodic bottleneck using VQ-VAE and a flow-matching model. This approach is innovative as it addresses the critical train-test mismatch prevalent in existing SAG models, which typically overfit to artifacts from source-separated vocals. The use of a chromagram for timbre-invariant representation is a significant methodological advancement, allowing the model to focus on the core melody rather than irrelevant acoustic details. The combination of a robust representation and a flow-matching transformer for accompaniment generation is well-conceived and demonstrates a clear understanding of the challenges in the field.
Results & evaluation
The experiments are comprehensive, utilizing a substantial dataset of 8k hours of paired singing voice and accompaniment data. The evaluation metrics are well-defined, including both objective measures (FAD, APA) and subjective assessments (MOS tests), which provide a holistic view of the model's performance. The results convincingly demonstrate that AnyAccomp outperforms existing models, particularly in generalization to clean vocals and instrumental tracks, validating the effectiveness of the proposed methodology. However, the paper could benefit from more detailed comparisons with additional baseline models to further substantiate its claims.
Reproducibility
The implementation details are clear, including model architectures, training parameters, and data preparation processes, which facilitate reproducibility. The authors provide sufficient information about the training setup, including the number of parameters and optimization strategies. However, the absence of a publicly available code repository limits the ease of reproduction for other researchers, which is a significant consideration in machine learning research.
Limitations
While the proposed method shows promising results, the paper does not address potential limitations in terms of computational efficiency and scalability. The reliance on a large dataset for training may also pose challenges for users with limited resources. Additionally, the model's performance on highly diverse or unconventional vocal inputs has not been thoroughly tested, which could affect its applicability in real-world scenarios.
Broader impact
The framework has the potential to significantly enhance music creation tools, enabling artists and producers to generate high-quality instrumental accompaniments from vocal inputs. This could democratize music production, making it more accessible to amateurs and non-professionals. Furthermore, the advancements in generalization could lead to more robust AI systems in creative fields, fostering innovation in music technology and related domains. The main contribution of this paper is the introduction of AnyAccomp, a framework that effectively resolves the train-test mismatch in singing accompaniment generation by utilizing a quantized melodic bottleneck to enhance generalization capabilities. This work represents a significant advancement in the field of audio machine learning, addressing critical challenges and paving the way for more versatile music co-creation tools.
Author of this paper? Send a correction.