CFMDCTCodec: A Low-Bitrate Neural Speech Codec with Noise-Prior-aware Conditional Flow Matching for MDCT-Spectral Enhancement
Abstract
High-quality speech coding at low bitrates is crucial for bandwidth-constrained applications, yet remains challenging due to the severe loss of quality-critical information in highly compressed representations. To overcome this challenge, we propose CFMDCTCodec, a low-bitrate neural speech codec that operates entirely in the modified discrete cosine transform (MDCT) domain. CFMDCTCodec integrates a lightweight encoder-quantizer-decoder-style MDCT-spectral codec with a noise-prior-aware, conditional-flow-matching (CFM)-based MDCT-spectral enhancer. Within this framework, the codec serves as a base module that compactly discretizes the MDCT spectrum extracted from speech and produces an initial coarse reconstruction, while the enhancer further restores fine-grained spectral details. The enhancer improves the decoded MDCT spectrum by integrating a conditional MDCT velocity-field filter with an ordinary differential equation (ODE) solver, under the guidance of an MDCT-derived magnitude-adaptive noise prior, aiming to emphasize perceptually significant high-energy regions while stabilizing low-energy and silent regions. Finally, the enhanced MDCT spectrum is reconstructed into the decoded speech using the inverse MDCT. When optimizing CFMDCTCodec, we adopt a unified non-adversarial training strategy that jointly combines reconstruction, quantization and CFM objectives. Both objective and subjective evaluations show that CFMDCTCodec outperforms competitive baselines in low-bitrate regimes, e.g., 0.65 kbps, while approaching the perceptual quality of large-scale codecs with significantly fewer parameters and computations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of CFMDCTCodec, a low-bitrate neural speech codec that effectively enhances spectral quality through a novel conditional flow matching approach, demonstrating significant improvements in speech quality while maintaining low computational complexity. This work represents a meaningful advancement in the field of speech coding, particularly for applications requiring efficient bandwidth usage without compromising audio fidelity.
Methodology
The proposed CFMDCTCodec introduces a novel architecture for low-bitrate speech coding that operates entirely in the MDCT domain, integrating a single-codebook quantization strategy with a noise-prior-aware conditional flow matching (CFM) enhancement mechanism. This approach effectively addresses the limitations of existing codecs by enhancing the spectral quality of decoded speech without increasing bitrate, utilizing a joint training strategy that simplifies the learning process. The methodology is well-structured, with clear descriptions of the encoder, decoder, and enhancer components, and the use of ordinary differential equations (ODE) for state evolution is particularly innovative.
Results & evaluation
The experimental setup is robust, utilizing two different speech corpora and multiple bitrate settings to evaluate the codec's performance. The paper provides both objective and subjective evaluation metrics, including MUSHRA tests and various objective measures (STOI, SI-SDR, etc.), which demonstrate the codec's superiority over competitive baselines. The results indicate significant improvements in speech quality at low bitrates, validating the effectiveness of the proposed enhancements.
Reproducibility
The paper includes detailed descriptions of the experimental setup, including hyperparameters, training configurations, and evaluation metrics, which facilitate reproducibility. However, the absence of a publicly available code repository limits the ease of replication for other researchers.
Limitations
One limitation is the reliance on a single-codebook quantization strategy, which may not capture the full diversity of speech signals as effectively as multi-codebook approaches. Additionally, while the results are promising, further testing across a wider range of speech datasets and real-world scenarios would strengthen the findings.
Broader impact
The CFMDCTCodec has significant potential applications in bandwidth-constrained environments such as satellite communications, teleconferencing, and mobile applications, where high-quality speech transmission is critical. Its lightweight design and efficient processing could facilitate broader adoption in various speech processing applications, contributing to advancements in telecommunications and accessibility technologies. The main contribution of this paper is the development of CFMDCTCodec, a low-bitrate neural speech codec that effectively enhances spectral quality through a novel conditional flow matching approach, demonstrating significant improvements in speech quality while maintaining low computational complexity. This work represents a meaningful advancement in the field of speech coding, particularly for applications requiring efficient bandwidth usage without compromising audio fidelity.
Author of this paper? Send a correction.