AudioCodex: Streaming Autoencoders for Live Audio
Abstract
We introduce AudioCodex, a fully causal autoencoder for live generative audio applications that compresses stereo audio into compact continuous latent sequences at ~10 Hz. It combines a Mean Flow magnitude autoencoder with latent hypersphere projection, bounded-noise augmentation, representation alignment, and adversarial phase recovery to learn representations that support both efficient downstream generation and high-quality reconstruction. Through extensive ablations, we show that AudioCodex outperforms the state-of-the-art in terms of generation-reconstruction trade-off, whilst remaining strictly causal in both encoding and decoding. This work paves the way towards truly real-time interactive and streaming generative audio applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AudioCodex introduces a fully causal, continuous audio autoencoder leveraging Mean Flow and adversarial phase recovery to achieve a superior reconstruction-generation trade-off for real-time applications. The paper demonstrates a rigorous methodology for decoupling magnitude and phase modeling, providing a robust framework for streaming generative audio that outperforms existing non-causal baselines in the critical balance of fidelity and generative usability.
Methodology
The paper proposes AudioCodex, a fully causal autoencoder for live audio generation. The core methodological contribution is the decoupling of magnitude and phase modeling. The magnitude autoencoder utilizes a Mean Flow (specifically Improved Mean Flow, iMF) framework to learn a continuous latent representation from log-magnitude spectrograms. This is a novel application of flow-based generative modeling to audio autoencoding, aiming for single-step decoding stability. The architecture employs a causal convolutional patchifier and frequency-wise transformers. A key innovation is the "bounded-noise augmentation" strategy, which synchronizes latent noise with data-space noise levels to regularize the latent space without degrading reconstruction fidelity. Additionally, Representation Alignment (REPA) is used to align intermediate features with a frozen self-supervised encoder (USAD-Small), enhancing the semantic structure of the latents for downstream generation. The phase is recovered separately using an adversarial estimator (R3GAN-based), which allows the magnitude model to focus on spectral structure without the instability introduced by phase prediction.
Results & evaluation
The experimental section is rigorous, featuring extensive ablations on both the magnitude autoencoder and the phase estimator. The authors evaluate the reconstruction-generation trade-off using Fréchet Audio Distance (FAD) for both reconstruction (rFAD) and downstream generation (gFAD) with a lightweight proxy model. They compare against strong baselines including M2L, CDC, SAO, and DAC. The results demonstrate that AudioCodex achieves a superior balance between reconstruction quality and generative usability compared to non-causal baselines, while maintaining strict causality. The inference speed analysis confirms real-time capability on a single A100 GPU, which is critical for the stated application domain. The use of multiple audio domains (speech, music, general audio) strengthens the generalizability claims.
Reproducibility
The paper provides detailed implementation specifics, including hyperparameters for training (learning rates, batch sizes, optimizer settings), architectural details (channel counts, layer types, downsampling strides), and data preprocessing steps. The release of evaluation metadata and the companion website with audio samples further supports reproducibility. The code and model weights are implied to be available via the project page, though the text does not explicitly state a GitHub link, the companion site is provided.
Limitations
The primary limitation acknowledged is that non-causal decoding still yields better quality than causal decoding, indicating a performance ceiling imposed by the causality constraint. The downstream generation evaluation uses short 1.5-second samples, which may not fully capture long-range musical or speech structure. The reliance on a separate phase estimator adds architectural complexity and potential points of failure in the reconstruction pipeline.
Broader impact
This work is significant for the development of real-time interactive audio systems, such as live music accompaniment, conversational AI with low-latency voice, and interactive sound design tools. By providing a high-quality, streamable continuous latent space, it enables new classes of generative applications that were previously hindered by the latency of non-causal models or the discrete nature of token-based codecs. It bridges the gap between high-fidelity offline audio generation and real-time streaming constraints. AudioCodex introduces a fully causal, continuous audio autoencoder leveraging Mean Flow and adversarial phase recovery to achieve a superior reconstruction-generation trade-off for real-time applications. The paper demonstrates a rigorous methodology for decoupling magnitude and phase modeling, providing a robust framework for streaming generative audio that outperforms existing non-causal baselines in the critical balance of fidelity and generative usability.
Author of this paper? Send a correction.