Self-Supervised Source-Filter Model of Speech Production
Abstract
Deep learning models of speech seldom represent the source-filter process by which speech is produced, relying instead on latent features with no physical correspondence to articulation or phonation. We propose SoFilNet, a self-supervised, differentiable Source-Filter Network that estimates the parameters of speech production directly from the raw waveform: a source, specified by fundamental frequency (F0), voicing and the aperiodicity of five frequency bands, and a filter, specified by ordered formant poles, additional poles that shape the rest of the spectral envelope, and antiresonance zeros. The model is trained only to reconstruct the input speech; no formant or pitch annotation is used. Since per-bin spectral losses favor blurred spectra wherever the model is uncertain, we train F0 and voicing against autocorrelation pseudo-labels refined with the model's own filter, aperiodicity against the periodic fraction of each band, and formant sharpness against spectral contrast. On VTR-FORMANTS, the estimated formants attain the lowest mean error among the reported methods, including supervised trackers, and on the unseen PTDB-TUG corpus the F0 frame error matches the best classical tracker and is lower than that of all learned trackers. The parameters are also consistent with speech production theory: aperiodicity rises with frequency and orders phone classes by the turbulence of their excitation, and the zeros deepen on nasal consonants, the lower one rising with place of articulation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
Methodology
The paper proposes SoFilNet, a self-supervised differentiable source-filter model that estimates speech production parameters (F0, voicing, band aperiodicity, formants, zeros) directly from raw waveforms. The methodology is technically sophisticated, addressing the "blur" problem of per-bin spectral losses by introducing specific auxiliary losses: autocorrelation-based pseudo-labels for F0/voicing (refined via inverse filtering), periodic fraction loss for aperiodicity, and spectral contrast loss for formant sharpness. The use of a multi-resolution encoder with separate gates for source and filter parameters is a well-motivated architectural choice that aligns with the different temporal scales of glottal and vocal tract dynamics. The stick-breaking recursion for formant ordering is a clever constraint to ensure physical consistency without post-hoc tracking.
Results & evaluation
The evaluation is rigorous and comprehensive. The model is tested on VTR-FORMANTS for formant tracking, achieving the lowest mean error among reported methods, including supervised baselines. Pitch tracking is evaluated on the unseen PTDB-TUG corpus, where SoFilNet matches the best classical tracker (RAPT) and outperforms learned trackers. The interpretability analysis provides strong evidence that the learned parameters align with speech production theory (e.g., aperiodicity rising with frequency, zeros deepening on nasals). The ablation studies effectively demonstrate the contribution of zeros, the multi-resolution encoder, and the specific loss functions. The reconstruction quality (PESQ/STOI) from parameters alone further validates the information content of the estimated features.
Reproducibility
The paper provides detailed descriptions of the architecture, loss functions, and training procedure. Hyperparameters such as learning rate, batch size, and number of epochs are specified. The use of standard datasets (TIMIT, VTR-FORMANTS, PTDB-TUG) and well-known baselines facilitates reproduction. However, the specific implementation details of the differentiable synthesizer and the exact code for the pseudo-label refinement process are not provided in the text, though a demo link is available.
Limitations
The model is trained on TIMIT, which is a relatively small and dated dataset, potentially limiting generalization to diverse accents or noisy environments. The evaluation of formants is limited to F1-F3 (F4 excluded due to lack of hand-correction in the benchmark), which may not capture the full vocal tract response. The pitch evaluation, while strong, is on a single corpus (PTDB-TUG), and the model's performance on highly noisy or pathological speech is not assessed. The computational cost of the multi-resolution encoder and the iterative pseudo-label refinement may be higher than simpler baselines.
Broader impact
This work bridges the gap between data-driven deep learning models and physical speech production theory. By providing interpretable parameters that correspond to articulatory and phonatory features, SoFilNet can be used for speech analysis, synthesis, and modification tasks where physical interpretability is crucial. The self-supervised approach reduces the need for expensive expert annotations, making it scalable to larger datasets. The insights into the behavior of aperiodicity and zeros could inform future speech coding and synthesis models. The paper presents a novel self-supervised differentiable source-filter model that accurately estimates speech production parameters from raw waveforms. By addressing the limitations of per-bin spectral losses with targeted auxiliary losses and a multi-resolution architecture, SoFilNet achieves state-of-the-art formant and pitch tracking accuracy while providing physically interpretable parameters consistent with speech production theory.
Author of this paper? Send a correction.