faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs
Abstract
This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
Methodology
The paper presents a highly specialized, dependency-free C runtime (`faster-enhancer.c`) designed to deploy the FastEnhancer-Medium speech enhancement model on commodity CPUs. The methodology focuses on extreme engineering optimization rather than algorithmic novelty. Key technical strategies include: (1) Training-free int8 quantization with per-frame activation range recomputation, eliminating the need for calibration datasets; (2) End-to-end runtime specialization with six SIMD tiers (ARM NEON, DOTPROD, I8MM, x86 AVX2, AVX-VNNI, AVX-512 VNNI) selected at initialization; (3) Use of Winograd F(2,3) for $k=3$ convolutions to reduce multiply-accumulate counts; (4) Fusion of GRU gates and dequantization epilogues to minimize memory traffic; (5) Strict byte-identical output guarantees across different compilers and architectures by addressing floating-point non-determinism in the Hann window generation and integer overflow edge cases (clamping to [-127, 127]). The approach is a masterclass in low-level systems engineering for ML deployment.
Results & evaluation
The evaluation is rigorous and addresses the specific constraints of audio processing. The authors measure Real-Time Factor (RTF) and energy efficiency (joules per second) on two distinct platforms: Apple M2 (macOS) and Galaxy S23+ (Android). They demonstrate a 3.3x speedup over fp32 ONNX Runtime on the M2 (0.069 RTF vs 0.230 RTF). Crucially, they introduce a "duty-cycled" benchmarking protocol that accounts for the 6.67 ms frame deadline, revealing that "race-to-idle" strategies are energy-inefficient compared to pacing. They also evaluate quality using PESQ, STOI, SNR, SI-SDR, and SIGMOS, showing negligible degradation (-0.006 PESQ, -0.08 dB SNR) compared to the fp32 baseline. The analysis of long-term stability (30-minute runs) highlights scheduler-induced jitter, a critical insight for real-world deployment.
Reproducibility
The paper provides a high degree of reproducibility. The code is released as a dependency-free library on GitHub. The authors provide detailed descriptions of the quantization scheme, kernel implementations, and the specific steps taken to ensure bit-identical outputs across platforms (e.g., double-precision window generation). The evaluation protocol is clearly defined, including the use of VoiceBank-DEMAND and specific timing methodologies. The inclusion of hash comparisons for output verification adds a layer of rigor often missing in systems papers.
Limitations
The primary limitation is the scope of the model: it is a port of an existing architecture (FastEnhancer-Medium) and does not propose a new model. The x86 tiers are verified for bit-equality but not timed due to lack of hardware, limiting the generality of the performance claims. The runtime is single-threaded and CPU-only, excluding GPU/TPU acceleration which is common in other deployment stacks. The quality metrics, while robust, rely on standard objective measures; subjective listening tests are not included, though the objective deltas are argued to be inaudible. The "cheapest core" placement on Android was found to miss deadlines, limiting the flexibility of core placement in heterogeneous systems.
Broader impact
This work has significant impact for on-device AI, particularly in privacy-sensitive applications like real-time communication, hearing aids, and voice assistants where latency and power consumption are critical. By providing a lightweight, dependency-free runtime, it lowers the barrier to deploying high-quality neural speech enhancement on resource-constrained devices. It challenges the industry's reliance on heavy general-purpose inference engines (like ONNX Runtime) for specific, fixed models, advocating for specialized, hand-tuned kernels. The emphasis on energy efficiency and deadline adherence provides a valuable framework for evaluating the true cost of ML deployment in audio pipelines. This paper delivers a highly optimized, dependency-free C runtime for streaming speech enhancement, achieving significant speedups and energy savings on commodity CPUs through specialized int8 quantization and SIMD kernel fusion, while maintaining bit-identical output and negligible quality loss.
Author of this paper? Send a correction.