Synthesizing Ultrasonic Echo Profiles from Text for Open-Vocabulary Silent Speech Recognition
Abstract
Eyewear that senses articulation with inaudible ultrasound can now recognize silent speech over an open vocabulary, but its accuracy is limited by the cost of training data. We present SoniGAN, which synthesizes the ultrasonic echo profiles of silent articulation from text and uses them as synthetic supervision. A voice-cloned text-to-speech (TTS) model reads a text corpus, and a two-stage generator synthesizes echo profiles from representations of the spoken content of that audio. Pretraining a recognizer on 104 hours of the synthesized data improves the word error rate (WER) of a public open-vocabulary benchmark by 2.3 points. Adding the sessions the corpus discards for a faulty microphone brings the system to 22.2% without any language model. Lexicon-constrained decoding, LLM rescoring, and ensembling further bring it to 14.9%, the first minimally obtrusive silent speech interface to fall inside the error range in which users report no loss of willingness to use silent speech input.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
Methodology
The paper proposes SoniGAN, a two-stage generative framework for synthesizing ultrasonic echo profiles from text. The methodology is technically sound and well-motivated. It leverages a voice-cloned TTS model (F5-TTS) to generate audio from text, converts this to soft speech units to strip timbre, and then uses a conditional GAN architecture to map these units to ultrasonic signals. The two-stage approach (envelope first, then fringe structure) is a clever solution to the high-frequency nature of ultrasonic fringes, which are difficult for single-step generators to capture. The use of a frozen recognizer as a perceptual loss is a strong choice to ensure semantic fidelity. The inclusion of cross-modal contrastive learning (crossCon) in the recognizer training is a significant methodological contribution that effectively utilizes the paired voiced/silent data available in the benchmark.
Results & evaluation
The experiments are rigorous and comprehensive. The authors provide a clear ablation study showing the contribution of each component (synthetic pretraining, faulty session handling, decoding strategies). The controls for "extra optimizer steps" vs. "data content" are particularly well-designed, proving that the gain comes from the synthetic data's content rather than just increased training time. The results show a significant improvement in Word Error Rate (WER), moving from a baseline of ~26% to 14.9%, which is a substantial leap for this modality. The evaluation on the public SoniSpeech benchmark ensures comparability.
Reproducibility
The paper provides detailed hyperparameters, architecture descriptions, and training procedures. However, the specific implementation of the "soft speech units" extraction and the exact configuration of the F5-TTS cloning process are not fully detailed, which might pose challenges for exact reproduction. The code is not explicitly linked in the provided text, though the use of standard components (ResNet, Transformer, F5-TTS) aids in reconstruction.
Limitations
The system relies on a specific benchmark (SoniSpeech) and a specific TTS model. The generalizability to other ultrasonic sensing setups or different languages is not tested. The "faulty microphone" augmentation is specific to the hardware failure mode of the benchmark device. The 14.9% WER, while a major improvement, is still higher than typical audible speech recognition systems, indicating room for further improvement.
Broader impact
This work has significant implications for accessibility and human-computer interaction. Silent speech interfaces are crucial for individuals with vocal impairments or for use in noisy environments. By reducing the data collection burden via synthetic supervision, this approach makes it easier to scale silent speech systems to open vocabularies, potentially enabling more natural and less obtrusive communication interfaces. The paper introduces a novel synthetic data generation pipeline for ultrasonic silent speech recognition, significantly improving WER through a two-stage GAN and cross-modal contrastive learning. The rigorous ablations and controls demonstrate the specific value of the synthetic data, establishing a new state-of-the-art for minimally obtrusive silent speech interfaces.
Author of this paper? Send a correction.