ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
Abstract
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Methodology
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
Results & evaluation
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
Reproducibility
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
Limitations
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
Broader impact
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Author of this paper? Send a correction.