ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis
Abstract
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
Methodology
The methodology proposes ImageTTS, a novel framework that repurposes a pretrained image diffusion transformer (FLUX.2) for speech synthesis. A key strength is the systematic disentanglement of two forms of vision-to-speech transfer: representation transfer via a frozen image VAE and generative-prior transfer via DiT initialization. The approach involves converting log-mel spectrograms into pseudo-RGB images, encoding them with the frozen FLUX.2 image VAE, and then fine-tuning only the pretrained DiT backbone using conditional latent flow matching. This pseudo-RGB conversion and subsequent VAE encoding is a creative way to bridge the modalities, though it relies on the image VAE's ability to meaningfully compress and reconstruct spectrograms. The formulation of zero-shot TTS as reference-conditioned latent infilling and instruction-guided TTS as full latent generation conditioned on text and description is well-defined. The use of classifier-free guidance for both text and speech conditions (and style for instruction-guided) is standard practice in generative models and appropriately applied here. The architecture leverages FLUX.2's double-stream and single-stream Transformer blocks for multimodal interaction, which is a sensible adaptation for handling both text and speech latents. The overall methodology is sound, innovative, and systematically designed to answer the research questions posed.
Results & evaluation
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
Author of this paper? Send a correction.