SALAD-VAE: Semantic Audio Compression with Language-Audio Distillation
Abstract
Modern generative and multimodal models increasingly rely on compact latent representations that trade and balance semantic richness with high-fidelity reconstruction. We introduce SALAD-VAE, a continuous and highly compact semantic Audio Variational Autoencoder, which operates in the frequency domain and achieves state-of-the-art compression with very low latent frame rate (7.8 Hz) while surfacing semantic structure and producing high audio quality. We enhance the standard VAE semantic losses and augmentation, specifically contrastive learning and CLAP-based embedding distillation, enabling it to generalize across diverse audio domains. With a significantly less computational complex architecture than comparable state-of-the-art VAEs, SALAD-VAE matches their reconstruction quality while it consistently outperforms them on a wide range of classification benchmarks. Furthermore, the proposed additional loss function provides a trained CLAP projection layer, which can be used zero-shot audio captioning and classification matching pretrained CLAP audio-text embeddings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of SALAD-VAE, a novel audio VAE that achieves high-quality audio compression while enabling semantic audio processing capabilities through innovative training techniques. This work significantly advances the state of the art in audio representation, providing a practical solution for integrating audio and language models.
Methodology
The methodology presented in SALAD-VAE is innovative, leveraging a continuous frequency-domain VAE architecture that incorporates advanced techniques such as contrastive learning and CLAP-based embedding distillation. The use of polyphonic data augmentation and a denoising autoencoder principle enhances generalization across diverse audio domains. The proposed contrastive loss and CLAP loss contribute significantly to semantic representation, enabling zero-shot classification and caption generation, which are notable advancements in the field of audio processing.
Results & evaluation
The experimental evaluation is robust, utilizing a comprehensive set of metrics for both reconstruction quality and latent space representation. The authors compare their model against strong baselines, demonstrating superior performance in latent space probing and competitive reconstruction quality. The use of diverse datasets like AudioSet and thorough evaluation of zero-shot capabilities adds to the credibility of the results. However, the paper could benefit from more extensive ablation studies to quantify the impact of each proposed loss function more clearly.
Reproducibility
The paper provides detailed implementation details, including architecture specifications, training data, and loss functions, which facilitate reproducibility. However, the absence of a publicly accessible code repository limits the ability for independent verification of results. Future work should consider releasing the model and code to enhance reproducibility.
Limitations
One limitation is the potential trade-off between reconstruction quality and latent space representation when combining multiple loss functions, as indicated in the results. Additionally, while the model performs well across various audio types, its performance on more complex audio tasks or real-world applications remains to be thoroughly validated. The reliance on specific datasets may also limit the generalizability of the findings.
Broader impact
The advancements made by SALAD-VAE have significant implications for audio processing applications, particularly in areas requiring efficient audio representation, such as speech recognition, music generation, and audio classification. The ability to perform zero-shot classification and generate captions opens new avenues for multimodal applications, enhancing accessibility and usability in various domains. The main contribution of this paper is the introduction of SALAD-VAE, a novel audio VAE that achieves high-quality audio compression while enabling semantic audio processing capabilities through innovative training techniques. This work significantly advances the state of the art in audio representation, providing a practical solution for integrating audio and language models.
Author of this paper? Send a correction.