GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding
Abstract
Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization is applied only post-training to the fitted parameters. Rather than relying on discrete codebook stages for bitrate control, our representation exposes a fine-grained rate-quality tradeoff: a single trained checkpoint supports post-training bitrate control by varying the number of primitives and the per-parameter bit depth, with no retraining required. GS-Codec matches or exceeds well-established open-source codecs such as EnCodec and DAC on speaker similarity (SIM), intelligibility (STOI), and perceptual quality (UTMOS) at comparable bitrates, while achieving comparable semantic performance (WER). Code and audio samples are available at https://ronaluf.github.io/gs-codec/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces GS-Codec, a neural audio codec that replaces quantization bottlenecks with a parametric 1D Gaussian-splatting decomposition, achieving competitive quality with fine-grained post-training bitrate control. By adapting Gaussian splatting from 3D graphics to 1D audio latents and amortizing the fitting process with a predictor network, the authors provide a robust alternative to VQ-VAE templates, demonstrating strong perceptual and objective performance across multiple datasets and bitrates.
Methodology
The paper proposes a novel architectural shift in neural audio codecs, replacing the standard Vector Quantization (VQ) or Finite Scalar Quantization (FSQ) bottleneck with a parametric 1D Gaussian Splatting (GS) decomposition. The core idea is to fit the encoder's latent representation as a weighted sum of Gaussian primitives via an inner optimization loop during training. To address the computational cost of this iterative fitting at inference, the authors introduce a "GS Predictor Net," a lightweight transformer-based module that amortizes the optimization into a single forward pass. This approach allows for fine-grained post-training bitrate control by varying the number of primitives and bit depth without retraining, a significant advantage over discrete codebook-based methods. The methodology is sound, leveraging differentiable rendering concepts from 3D graphics (Gaussian Splatting) and applying them to 1D signal processing. The use of a straight-through estimator and commitment loss to stabilize the end-to-end training through the inner loop is a clever engineering solution to the non-differentiability of the optimization steps.
Results & evaluation
The experimental evaluation is comprehensive, comparing GS-Codec against strong baselines like EnCodec, DAC, and WavTokenizer on standard datasets (LibriTTS, LJSpeech, LibriSpeech). The paper reports competitive or superior results on perceptual quality (UTMOS, PESQ) and speaker similarity (SIM) at comparable bitrates (3-6 kbps). The inclusion of human listening tests (MOS and MUSHRA) adds significant weight to the objective metrics, confirming that the Gaussian decomposition yields perceptually pleasing audio. The ablation studies effectively isolate the contribution of the GS bottleneck compared to FSQ and RVQ, and the analysis of encoding time demonstrates the practical viability of the GS Predictor Net, which reduces latency to levels competitive with standard codecs.
Reproducibility
The paper provides high reproducibility. It includes detailed hyperparameters for the SEANet backbone, the GS bottleneck (number of primitives, inner loop steps, learning rates), and the GS Predictor Net. The authors explicitly state that code and audio samples are available at the provided URL. The training setup (single GPU, specific dataset hours) is clearly defined, allowing other researchers to replicate the results with moderate computational resources.
Limitations
The primary limitation is the encoding latency. While the GS Predictor Net mitigates this, the iterative version is significantly slower (34x slower than DAC), and even the predictor version is slightly slower than feed-forward baselines. The paper also notes that performance at very low bitrates (<3 kbps) is not as strong as specialized low-bitrate codecs like WavTokenizer. Additionally, the current implementation is non-causal, limiting its immediate application in real-time streaming scenarios without further architectural modifications.
Broader impact
This work opens a new design axis for neural audio compression by demonstrating that parametric signal decompositions can serve as effective bottlenecks. It challenges the dominance of discrete codebooks and offers a path toward more flexible, continuous rate control. This could influence future designs in speech language models and real-time communication systems where adaptive bitrate is crucial. The paper introduces GS-Codec, a neural audio codec that replaces quantization bottlenecks with a parametric 1D Gaussian-splatting decomposition, achieving competitive quality with fine-grained post-training bitrate control. By adapting Gaussian splatting from 3D graphics to 1D audio latents and amortizing the fitting process with a predictor network, the authors provide a robust alternative to VQ-VAE templates, demonstrating strong perceptual and objective performance across multiple datasets and bitrates.
Author of this paper? Send a correction.