Fréchet Distance Loss on Speech Representations for Text-to-Speech Synthesis
Abstract
Few-step diffusion and flow-matching text-to-speech (TTS) models are usually trained with local objectives, such as conditional flow matching, reconstruction, and stop prediction. These losses provide stable optimization, but they never ask whether sampled speech follows the distribution of high-quality speech. We propose Speech Representation Fr'echet Distance loss (SR-FD), a training-time distributional regularizer for tokenizer-free flow-matching autoregressive TTS. During fine-tuning, the model synthesizes speech with the same few-step sampler used at deployment, and SR-FD matches the mean and covariance of frozen Whisper and CTC features of this speech to reference statistics computed offline from three complementary content targets. The loss requires no discriminator and no inference-time computation. On Seed-TTS English, four-step SR-FD fine-tuning reduces WER from the original four-step VoxCPM2 baseline's 2.2279% to 1.4147%, a 36.5% relative reduction, and also surpasses the original ten-step baseline at 1.7366%; both gains are significant under an utterance-level paired bootstrap. Speaker similarity and objective quality proxies are preserved at the ten-step level, and an error analysis shows the gain comes from content substitutions across all prompt lengths. SR-FD is thus an intelligibility-improving distributional regularizer for few-step TTS.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces SR-FD, a distributional regularizer that aligns the statistical moments of few-step generated speech representations with reference targets, significantly improving intelligibility in low-step flow-matching TTS. The paper presents a well-motivated and technically robust solution to the problem of distributional drift in few-step continuous TTS. By formulating a Fréchet Distance loss on frozen speech representations, the authors successfully bridge the gap between local training objectives and global output distribution quality. The experimental results are compelling, showing substantial WER improvements while maintaining perceptual quality. The method is complementary to existing distillation techniques and offers a practical path to deploying real-time, high-fidelity TTS systems. The rigorous evaluation, including statistical testing and listening tests, adds credibility to the claims.
Methodology
The paper proposes Speech Representation Fréchet Distance loss (SR-FD), a training-time regularizer for few-step flow-matching Text-to-Speech (TTS) models. The core innovation lies in addressing the distributional drift that occurs when compressing continuous flow-matching samplers to few steps (e.g., 4 steps). Standard local losses (conditional flow matching) do not guarantee that the sampled output distribution matches the high-quality reference distribution. SR-FD computes the Fréchet distance between the mean and covariance of frozen speech representations (Whisper semantic features and wav2vec 2.0 CTC posteriors) of generated samples and offline reference statistics. This approach is discriminator-free and adds no inference-time overhead. The methodology is technically sound, leveraging the stability of moment matching while avoiding the instability of adversarial training. The use of a feature queue to estimate covariances efficiently during training is a practical engineering contribution.
Results & evaluation
The evaluation is conducted on the Seed-TTS English benchmark using the VoxCPM2 backbone. The results are significant: SR-FD fine-tuning reduces Word Error Rate (WER) by 36.5% relative to the 4-step baseline, even outperforming the 10-step baseline. The authors provide rigorous statistical validation using utterance-level paired bootstrapping. They also include a blinded listening test (TOST) confirming that the quality and speaker similarity are perceptually indistinguishable from the 10-step baseline. The ablation study effectively demonstrates the contribution of each of the three reference targets (Whisper anchor, CTC teacher, CTC real-speech). The error analysis correctly identifies that gains come from reduced content substitutions.
Reproducibility
The paper provides extensive implementation details, including model architecture (VoxCPM2), optimization hyperparameters (LoRA rank, learning rate schedule), and specific sampler settings (guidance, temperature). The reference statistics computation is described in detail, including the handling of rank-deficient covariance matrices. The use of standard open-source models (Whisper, wav2vec 2.0) and datasets (LibriTTS, Seed-TTS) enhances reproducibility. The code is not explicitly linked in the text provided, but the methodological description is sufficient for implementation.
Limitations
The primary limitation is that SR-FD is evaluated on a single backbone model (VoxCPM2) and a single language (English). Its generalizability to other flow-matching architectures (e.g., F5-TTS, ARCHI-TTS) or multilingual settings remains unproven. The method relies on the quality of the frozen encoders; if the encoders have biases, the regularizer might propagate them. Additionally, the raw Fréchet Distance score was found to be a poor diagnostic for WER, suggesting that the loss landscape is complex and not directly correlated with the final metric in a simple way. The method also requires offline computation of reference statistics, which adds a pre-processing step.
Broader impact
This work contributes to the field of efficient speech synthesis by enabling high-quality, low-latency TTS systems without sacrificing intelligibility. By decoupling training stability from inference-time distributional fidelity, it offers a new paradigm for regularizing generative models. The potential for improving accessibility technologies through more robust and efficient TTS systems is significant. This paper introduces SR-FD, a distributional regularizer that aligns the statistical moments of few-step generated speech representations with reference targets, significantly improving intelligibility in low-step flow-matching TTS. The paper presents a well-motivated and technically robust solution to the problem of distributional drift in few-step continuous TTS. By formulating a Fréchet Distance loss on frozen speech representations, the authors successfully bridge the gap between local training objectives and global output distribution quality. The experimental results are compelling, showing substantial WER improvements while maintaining perceptual quality. The method is complementary to existing distillation techniques and offers a practical path to deploying real-time, high-fidelity TTS systems. The rigorous evaluation, including statistical testing and listening tests, adds credibility to the claims.
Author of this paper? Send a correction.