STCTS: Generative Semantic Compression for Ultra-Low Bitrate Speech via Explicit Text-Prosody-Timbre Decomposition
Abstract
Voice communication in bandwidth-constrained environments--maritime, satellite, and tactical networks--remains prohibitively expensive. Traditional codecs struggle below 1 kbps, while existing semantic approaches (STT-TTS) sacrifice prosody and speaker identity. We present STCTS, a generative semantic compression framework enabling natural voice communication at approximately 80 bps. STCTS explicitly decomposes speech into linguistic content, prosodic expression, and speaker timbre, applying tailored compression: context-aware text encoding (approximately 70 bps), sparse prosody transmission via TTS interpolation (less than 14 bps at 0.1-1 Hz), and amortized speaker embedding. Evaluations on LibriSpeech demonstrate a 75x bitrate reduction versus Opus (6 kbps) and 12x versus EnCodec (1 kbps), while maintaining perceptual quality (NISQA MOS greater than 4.26). We also discover a bimodal quality distribution with prosody sampling rate: sparse and dense updates both achieve high quality, while mid-range rates degrade due to perceptual discontinuities--guiding optimal configuration design. Beyond efficiency, our modular architecture supports privacy-preserving encryption, human-interpretable transmission, and flexible deployment on edge devices, offering a robust solution for ultra-low bandwidth scenarios.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the STCTS framework, which achieves ultra-low bitrate speech communication through a novel approach of explicitly decomposing speech into linguistic, prosodic, and timbral components, significantly enhancing the efficiency and quality of voice transmission in constrained environments. This work represents a meaningful advancement in the field of audio processing and communication technologies, with potential applications in various critical domains.
Methodology
The paper introduces STCTS, a novel framework for ultra-low bitrate speech communication, which decomposes speech into three components: linguistic content, prosody, and timbre. This explicit decomposition allows for tailored compression strategies that significantly reduce bandwidth usage while maintaining perceptual quality. The methodology is well-structured, leveraging existing technologies (STT, TTS) and introducing innovative strategies for prosody transmission and speaker embedding. The use of context-aware text encoding and sparse prosody transmission is particularly noteworthy, as it showcases a deep understanding of the temporal dynamics of speech components.
Results & evaluation
The experimental evaluation is robust, utilizing the LibriSpeech dataset to benchmark STCTS against established codecs like Opus and EnCodec. The reported results demonstrate a significant bitrate reduction while achieving high perceptual quality (NISQA MOS > 4.26). The discovery of a bimodal quality distribution concerning prosody sampling rates provides valuable insights for future configurations. However, the paper could benefit from more extensive user studies to assess real-world performance in diverse communication scenarios.
Reproducibility
The authors provide an open-source implementation of their system, which is a strong point for reproducibility. The detailed description of the system architecture, configuration options, and the availability of the source code facilitate replication of the experiments. However, the paper lacks comprehensive benchmarking infrastructure details that could aid other researchers in reproducing the results precisely.
Limitations
One limitation is the reliance on specific datasets for evaluation, which may not fully capture the variability of real-world speech communication in bandwidth-constrained environments. Additionally, the paper does not address potential challenges in adapting the system to different languages or dialects, which could affect its generalizability. The performance under extreme network conditions or with varying speaker characteristics also requires further exploration.
Broader impact
The STCTS framework has significant implications for voice communication in bandwidth-constrained environments, such as maritime, satellite, and tactical networks. By enabling natural and expressive communication at ultra-low bitrates, it addresses critical needs in various fields, including emergency response, remote work, and IoT applications. The modular architecture also supports future advancements in speech technology, making it a versatile tool for diverse applications. The main contribution of this paper is the STCTS framework, which achieves ultra-low bitrate speech communication through a novel approach of explicitly decomposing speech into linguistic, prosodic, and timbral components, significantly enhancing the efficiency and quality of voice transmission in constrained environments. This work represents a meaningful advancement in the field of audio processing and communication technologies, with potential applications in various critical domains.
Author of this paper? Send a correction.