Reconstructing the Vocal Tract with Differentiable Acoustic Simulation
Abstract
The vocal tract is the region of the human body responsible for filtering one's voice to create speech. In this paper, we present a differentiable and GPU accelerated acoustic simulator for the vocal tract. The differentiable simulator synthesizes speech by propagating sound along an acoustic tube model of the vocal tract, and via its gradients, can solve the inverse problem: reconstructing the shape of the vocal tract solely from the sound it produces. Although the inverse mapping between geometry and sound is notoriously non-convex, we discover that gradient descent succeeds with three technical contributions: (1) we design a frequency domain formulation of the vocal tract's fluid dynamics that is 70x more GPU parallelizable than finite differences in time, (2) we integrate a differentiable model for turbulence to synthesize consonants, and (3) similar to prior work in implicit neural representations (INRs) and neural fields, we find that parameterizing the geometry with a neural network accelerates convergence and escapes local minima that trap discrete representations. Because the simulator is differentiable, it is readily integrated with other deep learning pipelines to enable novel linguistics and medical imaging applications. (1) We demonstrate self-supervised autoencoding of vocal tract shapes across 11 languages, and (2) we couple our simulator with a generative model of MRI (magnetic resonance imaging) images to reconstruct one's moving vocal tract from only their speech without paired data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel differentiable acoustic simulator that enables the reconstruction of vocal tract geometry and MRI videos from speech alone. By combining frequency-domain physics with neural field parameterization, the authors solve a notoriously non-convex inverse problem, offering a robust framework for physically grounded speech analysis and synthesis.
Methodology
The paper introduces a differentiable, GPU-accelerated acoustic simulator for the vocal tract based on the linearized Euler equations. The core technical contribution is the shift from time-domain finite difference methods to a frequency-domain formulation using transmission line matrices (TLM). This allows for parallelization across frequencies, resulting in a 70x speedup and, crucially, smoother gradient landscapes that facilitate stable optimization. The authors also propose a differentiable turbulence model using softplus gating and softmax-based constriction localization to handle consonants, unifying vowel and consonant synthesis. Finally, they leverage neural fields (implicit neural representations) to parameterize the vocal tract geometry, which acts as a regularizer to escape local minima in the non-convex inverse problem.
Results & evaluation
The experiments are extensive and well-designed. The authors validate the simulator's accuracy by comparing Frequency Domain Synthesis (FDS) against Time Domain Synthesis (TDS), showing significant improvements in SI-SDR, STOI, and PESQ metrics. They demonstrate the utility of the simulator in two novel applications: (1) a self-supervised autoencoder that maps audio to vocal tract area functions across 11 languages, outperforming paired-data baselines in intelligibility and speaker identity preservation; and (2) a generative model that reconstructs MRI videos of the vocal tract from speech alone, without paired data. The emergence of the IPA vowel chart from the latent space of the MRI generator provides strong evidence for the physical grounding of the model.
Reproducibility
The paper provides detailed derivations of the physics, including the linearized Euler equations, Webster's equation, and the circuit interpretation. Specific hyperparameters, such as window lengths, hop sizes, and Reynolds number thresholds, are provided. The use of standard architectures like Wav2Vec 2.0 and StyleGAN2 aids reproducibility. However, the specific implementation of the differentiable turbulence model and the exact neural field architectures (RFF vs MFN) would benefit from more code-level details or a public repository link beyond the project page.
Limitations
The model relies on a 1D acoustic tube approximation, which may not capture all 3D effects of the vocal tract, particularly for complex articulations or nasal sounds (though the nasal tract is noted as a limitation due to MRI data availability). The inverse problem is inherently ill-posed, meaning multiple geometries can produce the same sound, leading to potential ambiguities in reconstruction. The MRI reconstruction quality is limited by the resolution and coverage of the training MRI dataset.
Broader impact
This work has significant implications for linguistics, speech pathology, and medical imaging. By providing a physically grounded link between speech and articulatory geometry, it enables new tools for voice coaching, language acquisition studies, and non-invasive visualization of vocal tract movements. The self-supervised nature of the autoencoder and MRI generator makes these tools scalable and accessible without the need for expensive paired data collection. The paper presents a novel differentiable acoustic simulator that enables the reconstruction of vocal tract geometry and MRI videos from speech alone. By combining frequency-domain physics with neural field parameterization, the authors solve a notoriously non-convex inverse problem, offering a robust framework for physically grounded speech analysis and synthesis.
Author of this paper? Send a correction.