Latency-Configurable Streaming Speech Enhancement via Asymmetric Temporal Padding
Abstract
Streaming speech enhancement requires balancing algorithmic latency against quality, yet existing approaches largely treat this as a binary causal versus non-causal choice. LaCo-SENet addresses this issue with two mechanisms parameterized by a single training-time hyperparameter. First, asymmetric temporal padding redistributes past and future context in convolutions, enabling systematic latency configuration. Second, dual-buffer streaming combines state buffers for past context with lookahead buffers that supply future context at both the input and feature levels. Selective state updates also prevent future-frame leakage into the streaming state, ensuring training-inference consistency. On VoiceBank+DEMAND, a fixed-budget (1.37M parameters) backbone yields a family of models spanning 12.5-75.0 ms, with PESQ rising from 3.35 to 3.43. At just 12.5 ms (fully causal), a PESQ of 3.35 matches or exceeds the prior causal state-of-the-art (3.27 at 46.5 ms).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LaCo-SENet introduces a latency-configurable streaming speech enhancement framework using asymmetric temporal padding and selective state updates to prevent future-frame leakage, achieving state-of-the-art causal performance at ultra-low latency. The paper presents a robust solution to the latency-quality trade-off in streaming speech enhancement. By decoupling these factors through asymmetric padding and a novel dual-buffer mechanism with selective state updates, the authors enable a single fixed-parameter model to span a wide range of algorithmic latencies. The technical contribution is significant because it addresses a fundamental implementation challenge (state corruption) that has previously hindered flexible streaming designs. The results demonstrate that careful architectural management of context buffers can yield superior causal performance compared to models specifically designed for higher latencies. This approach offers a practical path for deploying high-quality speech enhancement on resource-constrained devices where latency is a hard constraint.
Methodology
The paper proposes LaCo-SENet, a streaming speech enhancement architecture that decouples algorithmic latency from model quality by introducing asymmetric temporal padding. The core innovation lies in the "dual-buffer" streaming framework, which manages state buffers for past context and lookahead buffers for future context. Crucially, the authors identify and solve the "state corruption" problem inherent in naive asymmetric padding implementations, where future frames recorded in state buffers would leak into subsequent chunks. They resolve this via "selective state updates," ensuring that only current-chunk frames update the recurrent state, thereby maintaining training-inference consistency. The approach is technically sound, leveraging standard convolutional operations (Dense Dilated Depthwise Blocks) but applying them in a novel streaming configuration. The method is elegant in its simplicity: a single hyperparameter (padding ratio) controls the latency-quality trade-off without altering the parameter count or receptive field size.
Results & evaluation
The evaluation is conducted on the standard VoiceBank+DEMAND dataset. The results are compelling: at 12.5 ms latency (fully causal), LaCo-SENet achieves a PESQ of 3.35, outperforming the previous causal state-of-the-art (aTENNuate at 3.27 PESQ, but with 46.5 ms latency). The model scales gracefully up to 75.0 ms (PESQ 3.43) and 200.0 ms (PESQ 3.47). The ablation study effectively demonstrates the necessity of the selective state update mechanism, showing catastrophic performance drops (below noisy baseline) when it is disabled. The throughput analysis (RTF) is also included, showing real-time capability. However, the evaluation is limited to a single dataset (VoiceBank+DEMAND) and a single metric suite (PESQ, STOI, CSIG, etc.). While the results are strong for the benchmark, the lack of evaluation on more recent, larger-scale datasets (e.g., DNS Challenge, CHiME) limits the generalizability claims.
Reproducibility
The paper provides sufficient detail for reproduction. The architecture (PrimeK-Net backbone), hyperparameters (channel dimensions, kernel sizes, STFT parameters), and training loss components are explicitly listed. The mathematical formulation of the asymmetric padding and selective state updates is clear. The code is not provided, but the method relies on standard PyTorch/TensorFlow operations (convolutions, buffering, masking), making implementation feasible for researchers in the field.
Limitations
The primary limitation is the scope of evaluation. VoiceBank+DEMAND is a relatively small, older dataset with specific noise types. Performance on real-world, non-stationary noise or diverse acoustic environments is not reported. Additionally, while the latency is configurable, the model does not adapt latency dynamically based on input SNR or content; it is fixed per model instance. The PESQ gains at higher latencies (75-200ms) are marginal compared to the latency cost, suggesting diminishing returns for very high lookahead in this specific architecture.
Broader impact
This work has significant implications for real-time audio applications such as teleconferencing, hearing aids, and voice assistants, where low latency is critical but quality cannot be sacrificed. By enabling a single model to serve a range of latency requirements, it simplifies deployment pipelines for edge devices with varying computational constraints. The technique of selective state updates for preventing future-frame leakage is a generalizable contribution to streaming deep learning architectures beyond just speech enhancement. LaCo-SENet introduces a latency-configurable streaming speech enhancement framework using asymmetric temporal padding and selective state updates to prevent future-frame leakage, achieving state-of-the-art causal performance at ultra-low latency. The paper presents a robust solution to the latency-quality trade-off in streaming speech enhancement. By decoupling these factors through asymmetric padding and a novel dual-buffer mechanism with selective state updates, the authors enable a single fixed-parameter model to span a wide range of algorithmic latencies. The technical contribution is significant because it addresses a fundamental implementation challenge (state corruption) that has previously hindered flexible streaming designs. The results demonstrate that careful architectural management of context buffers can yield superior causal performance compared to models specifically designed for higher latencies. This approach offers a practical path for deploying high-quality speech enhancement on resource-constrained devices where latency is a hard constraint.
Author of this paper? Send a correction.