Bridging Self-Supervised Learning and Speech Enhancement: A Wav2Vec2-Conditioned Framework
Abstract
Diffusion models show potential for speech enhancement but lack linguistic guidance. We condition a diffusion-based model on wav2vec 2.0 features from noisy input, injected at the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). Phonetic representations from wav2vec 2.0 features of degraded speech, anchor the reverse diffusion process. While a frozen wav2vec 2.0 encoder extracts features, a learned FiLM generator produces scale and shift parameters modulating the bottleneck with minimal overhead. Motivated by the optimal Bayesian causal estimator under a linear-Gaussian state-space model, FiLM coefficients are aggregated via exponential smoothing for temporal compression. Evaluation on VoiceBank-DEMAND and LibriMix shows competitive performance against the unconditioned baseline in PESQ, STOI, SI-SDR and DNSMOS. We consistently record an improvement of 0.4 on PESQ score, suggesting self-supervised representations effectively condition diffusion-based speech enhancement.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a theoretically motivated and empirically effective method for conditioning diffusion-based speech enhancement with self-supervised features, offering a compelling alternative to standard conditioning strategies despite a noted trade-off in source separation metrics.
Methodology
The paper proposes a novel conditioning mechanism for diffusion-based speech enhancement by integrating self-supervised learning (SSL) features from wav2vec 2.0. The core technical contribution lies in the injection of these phonetic representations into the U-Net bottleneck via Feature-wise Linear Modulation (FiLM). A significant methodological strength is the theoretical derivation of the temporal aggregation strategy for the FiLM coefficients. By modeling the phonetic state as a random walk and the projected coefficients as noisy observations, the authors derive that the optimal causal estimator is a Kalman filter, which simplifies to exponential moving average (EMA) at steady state. This provides a principled, theoretically grounded alternative to ad-hoc pooling methods (like mean pooling) for handling the temporal mismatch between frame-level SSL features and the global context required at the diffusion bottleneck. The choice to apply FiLM only at the bottleneck, supported by ablation studies, demonstrates a nuanced understanding of feature abstraction levels in U-Nets.
Results & evaluation
The experimental evaluation is conducted on two standard benchmarks: VoiceBank-DEMAND and LibriMix. The results show consistent improvements in perceptual metrics (PESQ, STOI, DNSMOS) compared to the unconditioned StoRM baseline, with a notable 0.4 improvement in PESQ on VB-DEMAND. However, the paper acknowledges a trade-off: a degradation in SI-SDR, attributed to aggressive noise suppression. This is a critical observation; while perceptual quality improves, the objective source separation metric suffers, suggesting the model may be over-smoothing or removing non-speech components that contribute to the separation score. The evaluation includes ablations on the smoothing coefficient and conditioning location, which strengthen the claims. However, the lack of subjective listening tests (MOS) limits the validation of the perceptual gains claimed by the DNSMOS and PESQ improvements. The comparison is primarily against StoRM and a few other diffusion baselines, which is appropriate but could be broader to include recent SSL-enhanced discriminative models.
Reproducibility
The paper provides sufficient detail regarding the model architecture (U-Net configuration, wav2vec 2.0 base model), training hyperparameters (learning rate, epochs, optimizer), and datasets. The theoretical derivation of the EMA smoothing is clearly explained. However, the code is not publicly linked in the text provided, and specific details on the noise types and mixing conditions for LibriMix are somewhat generic ("min" mixing mode). The claim of "minimal overhead" is supported by FLOPs analysis, but exact inference latency comparisons would be more useful for reproducibility in real-time applications.
Limitations
The primary limitation is the trade-off between perceptual quality and source separation fidelity (SI-SDR). The model improves PESQ but reduces SI-SDR, which may be undesirable for applications requiring strict source isolation. Additionally, the reliance on wav2vec 2.0, while effective, ties the method to a specific SSL model; the authors mention future work with WavLM/HuBERT, but the current work does not explore the sensitivity to the choice of SSL encoder. The theoretical derivation assumes a linear-Gaussian state-space model, which is a simplification of the complex, non-linear dynamics of speech and noise. Finally, the evaluation lacks subjective human listening tests, which are the gold standard for speech enhancement quality.
Broader impact
This work contributes to the field of audio processing by bridging self-supervised learning and generative modeling. It demonstrates that linguistic/phonetic information can effectively guide diffusion processes, potentially leading to more robust speech enhancement systems that preserve speech content even in low-SNR conditions. This has implications for telecommunications, hearing aids, and speech recognition preprocessing. The theoretical connection between Kalman filtering and exponential smoothing for feature aggregation is a generalizable insight that could apply to other temporal sequence modeling tasks. This paper presents a theoretically motivated and empirically effective method for conditioning diffusion-based speech enhancement with self-supervised features, offering a compelling alternative to standard conditioning strategies despite a noted trade-off in source separation metrics.
Author of this paper? Send a correction.