Optimising Neural Speech Codecs for 300bps Communication using Reinforcement Learning
Abstract
In bandwidth-constrained communication such as satellite and underwater channels, speech must often be transmitted at ultra-low bitrates where intelligibility is the primary objective. At such extreme compression levels, codecs trained with acoustic reconstruction losses tend to allocate bits to perceptual detail, leading to substantial degradation in word error rate (WER). This paper proposes ClariCodec, a neural speech codec operating at 300 bit per second (bps) that reformulates quantisation as a stochastic policy, enabling reinforcement learning (RL)-based optimisation of intelligibility. Specifically, the encoder is fine-tuned using WER-driven rewards while the acoustic reconstruction pipeline remains frozen. Even without RL, ClariCodec achieves 4.64% WER on the LibriSpeech test-clean set at 300 bps, already competitive with codecs operating at higher bitrates. Further RL fine-tuning reduces WER to 3.55% on test-clean and 10.4% on test-other, corresponding to a 23% relative reduction while preserving perceptual quality.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of ClariCodec, a neural speech codec optimized for intelligibility at ultra-low bitrates using reinforcement learning. This work represents a significant advancement in codec design, addressing the challenges of speech intelligibility in bandwidth-constrained environments while maintaining competitive acoustic quality.
Methodology
The proposed methodology introduces ClariCodec, a neural speech codec that operates at an ultra-low bitrate of 300 bps. The two-stage training strategy is innovative, combining a reconstruction-based pre-training phase with a reinforcement learning (RL) fine-tuning phase. The reformulation of quantisation as a stochastic policy allows for direct optimisation against a non-differentiable WER metric, which is a significant advancement in codec training. The use of group relative policy optimisation (GRPO) is a novel approach in this context, enabling the encoder to learn representations that prioritize intelligibility over acoustic fidelity. The methodology is well-structured and clearly articulated, demonstrating a thoughtful integration of RL into codec training.
Results & evaluation
The experiments are robust, utilizing the LibriSpeech dataset, which is a standard benchmark in speech processing. The results show a clear improvement in WER, with ClariCodec achieving a WER of 3.55% on the test-clean set, which is competitive with higher bitrate codecs. The paper includes comprehensive comparisons with various baseline systems, providing a thorough analysis of both intelligibility and acoustic quality metrics. The ablation studies further validate the effectiveness of the proposed RL fine-tuning strategy, highlighting the balance between intelligibility and perceptual quality.
Reproducibility
The paper provides sufficient details regarding the training setup, loss functions, and evaluation metrics, which would allow other researchers to replicate the experiments. However, the absence of a publicly available code repository limits full reproducibility. The authors mention the use of generative AI for language polishing, which does not affect the technical content but is worth noting for transparency.
Limitations
One limitation is the potential trade-off between intelligibility and acoustic fidelity, as indicated by the PESQ scores during RL fine-tuning. The model's performance may vary under different acoustic conditions or with different languages, which is not explored in the current study. Additionally, the non-causal architecture may pose latency issues for real-time applications, which the authors plan to address in future work.
Broader impact
The development of ClariCodec has significant implications for communication in bandwidth-constrained environments, such as satellite and underwater channels. By prioritizing intelligibility, this work could enhance speech communication in critical applications where clarity is paramount. The approach may also inspire further research into RL applications in audio processing and codec design, potentially leading to advancements in other areas of speech technology. The main contribution of this paper is the introduction of ClariCodec, a neural speech codec optimized for intelligibility at ultra-low bitrates using reinforcement learning. This work represents a significant advancement in codec design, addressing the challenges of speech intelligibility in bandwidth-constrained environments while maintaining competitive acoustic quality.
Author of this paper? Send a correction.