FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
Abstract
Recent continuous autoregressive TTS models operate directly on continuous speech representations, preserving rich acoustic details while leveraging the instruction-following capabilities of text LLMs. This paradigm opens new possibilities for voice cloning, instruction-controlled voice design, and speech editing, but remains susceptible to error accumulation during autoregressive generation. Existing solutions often require additional semantic modules, multi-stage tokenizer training pipelines, or complex autoregressive architectures. In this work, we propose FireRedTTS3, a simple yet effective speech generation and editing framework that mitigates error accumulation at the representation level. Specifically, we leverage a frozen Audio Encoder trained on diverse speech understanding tasks as a semantic teacher to regularize the audio feature space. This improves text-speech alignment and stabilizes autoregressive generation while keeping the overall system simple. FireRedTTS3 provides two variants: FireRedTTS3-Base for multilingual and multi-dialect zero-shot voice cloning, and FireRedTTS3-Instruct for unified voice cloning, instruction-controlled voice design, and speech editing. Experiments show that FireRedTTS3-Base achieves the best average speech intelligibility and speaker similarity among compared systems on Seed-TTS-Eval and MiniMax-MLS-Test, while FireRedTTS3-Instruct outperforms competing systems on InstructTTSEval and Ming-Freeform-Audio-Edit. These results demonstrate that semantically enriched continuous speech representations, combined with a simple architecture, enable stable, controllable, and high-fidelity speech generation and editing. Code and models are available at https://github.com/FireRedTeam/FireRedTTS3.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
Methodology
The paper proposes FireRedTTS3, a framework for continuous autoregressive text-to-speech (TTS) that addresses error accumulation through semantically enriched representations. The core innovation is the RedAE tokenizer, which uses a frozen, pretrained Audio Encoder as a semantic teacher to regularize the latent space during training. This approach aims to improve text-speech alignment and stability without adding complex architectural components or multi-stage training pipelines. The generation backbone utilizes an LLM-DiT (Diffusion Transformer) architecture, leveraging the instruction-following capabilities of large language models (Qwen3) for both zero-shot cloning (Base variant) and instruction-controlled editing/design (Instruct variant). The methodology is sound and addresses a known pain point in continuous autoregressive TTS (error accumulation), offering a relatively simple solution compared to prior works that require additional semantic modules or complex tokenizers.
Results & evaluation
The authors evaluate FireRedTTS3 on four benchmarks: Seed-TTS-Eval, MiniMax-MLS-Test, InstructTTSEval, and Ming-Freeform-Audio-Edit. The results claim state-of-the-art performance in terms of speech intelligibility, speaker similarity, and instruction-following capabilities. The evaluation covers multilingual zero-shot cloning, instruction-controlled voice design, and speech editing. The use of established benchmarks adds credibility. However, the reliance on automated metrics (WER/CER, WavLM similarity, LLM-as-a-judge) for instruction following introduces potential bias, although this is standard in the field. The claim of "best average" performance is supported by the provided tables (referenced as [REF]), suggesting strong empirical results.
Reproducibility
The paper provides a GitHub link for code and models. The methodology describes the architecture (Qwen3-style Transformers, DiT, RedAE) and training objectives (GAN, flow-matching, semantic distillation). The use of pretrained components (Qwen3, CAM++, FireRedAudio) aids reproducibility. However, the specific details of the "FireRedAudio" model and the exact composition of the 500k hours of training data are not fully detailed, which may hinder exact replication. The description of the Aggregator and DiT modules is sufficiently detailed for implementation by researchers familiar with these architectures.
Limitations
The paper acknowledges that Cantonese error rates are high due to the recognition model (Whisper-large-v3) limitations, not necessarily synthesis quality. The reliance on a frozen semantic teacher requires the existence of a high-quality pretrained audio understanding model, which may not be available for all languages or domains. The "simple" architecture still relies on large pretrained LLMs and DiTs, implying significant computational resources for training and inference. The generalization to unseen languages (Portuguese, Ukrainian) is claimed but not deeply analyzed regarding the source of this generalization (linguistic universality vs. acoustic similarity).
Broader impact
FireRedTTS3 contributes to the advancement of controllable and editable speech synthesis, which has applications in accessibility, entertainment, and content creation. The ability to perform instruction-controlled editing and voice design enhances user interaction with speech technologies. However, the ease of voice cloning and editing raises concerns about deepfakes and misuse for fraudulent activities. The open-source release of code and models necessitates careful consideration of safety guardrails and usage policies. [This paper presents FireRedTTS3, a unified speech generation and editing framework that mitigates error accumulation in continuous autoregressive TTS by using a frozen audio encoder as a semantic teacher to regularize the latent space, achieving state-of-the-art performance in zero-shot cloning and instruction-controlled editing.]
Author of this paper? Send a correction.