Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization
Abstract
Some text-to-speech systems ship a synthesis model and preset style vectors but not the reference encoder that turns audio into such a vector. The model still accepts a style vector; a user with a voice of their own cannot produce one. We solve for that input directly, inverting the released pipeline by gradient descent: every weight stays frozen and only the style vector is optimized, against time-pooled WavLM statistics of one recording. Because the objective discards the time axis, the synthesized text may differ from the recording, so no transcript and no alignment are needed. On 154 speakers from two corpora, ECAPA-TDNN similarity rises from 0.132 to 0.413 and ResNet from 0.099 to 0.401, improving for every speaker; a verifier at its equal-error point accepts 53% of the recovered voices as the target, against 1% for the presets they start from.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
Methodology
The paper proposes a gradient-based inverse optimization method to extract style vectors from frozen Text-to-Speech (TTS) models when the reference encoder is withheld. The core innovation lies in bypassing the need for transcribed reference audio or forced alignment by using a content-independent loss function. Specifically, it optimizes the style vector to match the time-pooled mean and standard deviation of hidden states from an early layer (Layer 4) of a frozen WavLM model. This approach effectively decouples speaker identity from linguistic content, allowing the system to work with untranscribed, arbitrary audio recordings. The method is technically sound, leveraging existing differentiable wrappers for ONNX models and standard optimization techniques (Adam, learning rate scheduling). It draws strong parallels to GAN inversion and Textual Inversion in the image domain, adapting these concepts to the audio TTS space.
Results & evaluation
The evaluation is conducted on 154 speakers from two corpora (VCTK and Common Voice subset of Seed-TTS Eval). The results demonstrate a significant improvement in speaker similarity metrics (ECAPA-TDNN and ResNet) compared to using the nearest preset style vector. The improvement is universal across all tested speakers. The paper provides a thorough analysis of the stopping criterion, characterizing the "content floor" to prevent overfitting into unrealistic style regions. It also evaluates intelligibility (WER) and naturalness (UTMOS), showing that the extracted voices remain intelligible and natural-sounding. The use of independent verification metrics (ECAPA, ResNet) to validate the optimization objective (WavLM statistics) strengthens the claim of successful voice cloning.
Reproducibility
The authors release code and provide detailed hyperparameters, including the optimization setup, loss function definition, and stopping criteria. The use of public models (SupertonicTTS, WavLM, ECAPA-TDNN) and public datasets (VCTK, Common Voice) ensures that the experiments are fully reproducible. The description of the gradient flow through the ONNX modules is sufficiently detailed for implementation.
Limitations
The method is currently validated on a single TTS model (SupertonicTTS). While the authors argue it should generalize to other models with continuous style inputs, this has not been empirically demonstrated. The optimization is sensitive to the choice of the WavLM layer and the stopping threshold, which are somewhat heuristic. The method does not recover duration/prosody styles, inheriting them from the initialization preset. Furthermore, the evaluation prompts are the same ones used during optimization, so generalization to unseen text is not fully tested. The paper acknowledges that the extracted voices might still be detectable by spoofing countermeasures, even if they pass speaker verification.
Broader impact
This work has significant dual-use implications. On one hand, it highlights a security vulnerability in TTS systems that withhold reference encoders, demonstrating that partial releases do not fully protect voice identity. This can inform better security practices for TTS providers. On the other hand, it provides a powerful tool for voice cloning, which can be misused for impersonation and fraud. The authors address this by releasing code with ethical warnings and not packaging it as a turnkey cloning tool. The work contributes to the broader understanding of latent spaces in generative models and the transferability of self-supervised speech representations. This paper presents a novel gradient-based method for extracting voice styles from frozen TTS models without reference encoders, demonstrating that withheld encoders do not fully secure voice identity and offering a reproducible framework for analyzing TTS latent spaces.
Author of this paper? Send a correction.