SAGE: Semantic Audio Generative Encoder
Abstract
Audio autoencoders compress waveforms into compact latent representations that serve as the interface between raw audio and downstream models. Current systems navigate a three-way trade-off between reconstruction quality, semantic structure of the latent space, and inference speed, typically favoring one or two of these at the expense of the others. This paper introduces SAGE, Semantic Audio Generative Encoder: a compact variational autoencoder, trained solely on publicly available music, that shapes its latent by distilling embeddings from a pretrained audio-text model. This 105M-parameter model runs at the inference cost of Stable Audio Open and reaches the listening-test quality of SAME-L, an autoencoder 8x larger and 4x slower, while surpassing both on objective perceptual and distributional metrics of reconstruction. Furthermore, it sets the state of the art on all nineteen probing tasks of latent semantics, in domain and out of domain. These results establish SAGE as a lightweight audio autoencoder that strikes the best balance of the three-way trade-off among those we evaluate, combining high reconstruction fidelity, state-of-the-art semantic structure, and fast inference.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SAGE introduces a compact, spectrogram-native VAE with semantic distillation that achieves state-of-the-art reconstruction and latent semantics at low inference cost. The paper demonstrates that adapting vision transformers (SwinV2) to audio spectrograms, combined with careful curriculum learning for semantic alignment, can outperform larger, waveform-based models in both fidelity and utility for downstream generative tasks.
Methodology
The paper proposes SAGE, a 105M parameter Variational Autoencoder (VAE) for audio that operates on the complex Short-Time Fourier Transform (STFT) rather than raw waveforms or magnitude spectrograms. The core architectural innovation is the adaptation of the SwinV2 vision transformer backbone to audio, utilizing rectangular patches and attention windows to handle the time-frequency structure of spectrograms. A key methodological contribution is the "semantic distillation" strategy, where the latent space is aligned with embeddings from a frozen CLAP (Contrastive Language-Audio Pretraining) model. This alignment is introduced via a delayed schedule (detached warm-up) to ensure reconstruction quality is established before semantic constraints are applied, preventing the semantic loss from degrading fidelity. The training process is divided into two phases: pretraining with a full adversarial objective (using a WavTokenizer discriminator) and a fine-tuning phase where the encoder is frozen and the decoder is refined. The use of a sum-and-difference loss for stereo imaging is also a notable technical detail, addressing the common issue of stereo collapse in autoencoders.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors compare SAGE against five strong baselines (Stable Audio Open, SAME-L, SAME-S, CoDiCodec, Music2Latent) across five different held-out datasets, including in-domain (FMA) and out-of-domain (MoisesDB, MusicCaps, Song Describer) sets. Metrics cover perceptual (CLAP similarity), distributional (Fréchet Audio Distance in MERT, PANN, and CLAP spaces), and sample-exact (SI-SDR, STFT distance) categories. Crucially, the paper includes a MUSHRA listening test with 21 raters, showing SAGE is statistically indistinguishable from the much larger SAME-L model. The semantic evaluation is extensive, using 19 probing tasks (genre, artist, instrument, etc.) in the MAEB format, where SAGE achieves state-of-the-art results on all tasks. The ablation studies on the semantic distillation schedule and adversarial components provide strong evidence for the design choices.
Reproducibility
The paper offers high reproducibility. The authors provide a link to the GitHub repository containing code, weights, and the evaluation harness. The training data consists of publicly available corpora (FMA, MTG-Jamendo, M4Singer), and the paper details the specific splits and preprocessing steps. The hyperparameters for both training phases are listed in the appendix. The use of standard libraries and the release of the evaluation harness allows other researchers to easily verify the results and compare their own models.
Limitations
The primary limitation is the reliance on the CLAP model for semantic alignment; if the CLAP embeddings are biased or limited in their musical understanding, SAGE's latent space will inherit these limitations. The model is trained on music, so its performance on non-music audio (speech, sound effects) is not evaluated and likely poor. The inference cost, while competitive, is still higher than simple convolutional codecs, which may be a barrier for real-time mobile applications. The paper focuses on music, so generalization to other audio domains is an open question.
Broader impact
SAGE provides a lightweight, high-fidelity, and semantically rich audio encoder that can serve as a drop-in replacement for existing autoencoders in latent diffusion pipelines for music generation. Its semantic structure could enable better control and conditioning in generative models. The efficient inference cost makes it suitable for interactive applications. The comprehensive benchmarking harness contributes to the standardization of audio autoencoder evaluation. SAGE introduces a compact, spectrogram-native VAE with semantic distillation that achieves state-of-the-art reconstruction and latent semantics at low inference cost. The paper demonstrates that adapting vision transformers (SwinV2) to audio spectrograms, combined with careful curriculum learning for semantic alignment, can outperform larger, waveform-based models in both fidelity and utility for downstream generative tasks.
Author of this paper? Send a correction.