DUO-TOK: Dual-Track Semantic Music Tokenizer for Vocal-Accompaniment Generation
Abstract
Duo-Tok is a source-aware dual-codebook tokenizer for vocal-accompaniment music that targets the growing tension between reconstruction quality and language-model (LM) learnability in modern lyrics-to-song systems. Existing codecs either prioritize high-fidelity reconstruction with difficult-to-model acoustic tokens or compress aggressively into semantic tokens that are LM-friendly but lossy, and they rarely make the tokenizer itself aware of dual-track structure. Duo-Tok follows a four-stage, SSL-centered pipeline: we first pretrain a BEST-RQ-style encoder on large-scale audio, then stabilize and factorize the representation with Gaussian replacement noise and multi-task supervision, before freezing the encoder to learn SimVQ-based dual codebooks with hard routing for vocals and accompaniment, and finally training latent diffusion decoders on top of the discrete tokens. Duo-Tok at 0.75 kbps shifts the empirical reconstruction-generation Pareto frontier, achieving the best music-tagging AP and the lowest vocabulary-normalized LM perplexity among compared codecs while maintaining reconstruction quality comparable to state-of-the-art music tokenizers.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Duo-Tok, a novel dual-track semantic music tokenizer that effectively balances reconstruction quality and language model learnability for vocal-accompaniment generation. The proposed methodology and experimental results demonstrate a meaningful advancement in the field, although further work is needed to enhance reproducibility and explore broader applications.
Methodology
The methodology presented in the paper is innovative, utilizing a dual-codebook approach that addresses the limitations of existing music tokenization methods. The four-stage pipeline is well-structured, beginning with pretraining on large-scale audio data, followed by representation stabilization and factorization, which enhances the model's robustness. The use of SimVQ-based dual codebooks for vocals and accompaniment is particularly noteworthy, as it allows for better semantic representation while maintaining high reconstruction quality. However, the paper could benefit from a more detailed explanation of the multi-task supervision and Gaussian replacement noise techniques, as these are critical to understanding the effectiveness of the proposed method.
Results & evaluation
The experiments conducted are comprehensive, comparing Duo-Tok against state-of-the-art codecs in terms of music tagging and language model perplexity. The results indicate a significant improvement in the empirical reconstruction-generation Pareto frontier, showcasing the effectiveness of the proposed approach. However, the paper lacks detailed descriptions of the datasets used, which could affect the reproducibility and generalizability of the results. Additionally, more extensive ablation studies could strengthen the claims regarding the advantages of the dual-codebook approach.
Reproducibility
The paper does not provide sufficient implementation details or access to code repositories, which raises concerns about reproducibility. While the methodology is well-defined, the absence of a clear implementation guide or publicly available code makes it challenging for other researchers to replicate the results or build upon the work.
Limitations
One limitation of the study is the lack of a thorough exploration of the trade-offs between reconstruction quality and learnability in different contexts. Additionally, the paper does not address potential scalability issues when applying the method to larger datasets or more complex music genres. The reliance on specific training data may also limit the applicability of the findings to diverse musical styles.
Broader impact
The implications of this research are significant for the fields of music generation and audio processing. By improving the efficiency and quality of vocal-accompaniment generation, Duo-Tok has the potential to enhance various applications, including music production, AI-assisted songwriting, and interactive music systems. The advancements in tokenization methods could also influence future research in related areas, such as audio synthesis and machine learning for creative tasks. The main contribution of this paper is the introduction of Duo-Tok, a novel dual-track semantic music tokenizer that effectively balances reconstruction quality and language model learnability for vocal-accompaniment generation. The proposed methodology and experimental results demonstrate a meaningful advancement in the field, although further work is needed to enhance reproducibility and explore broader applications.
Author of this paper? Send a correction.