Fish Audio Drama 3 Technical Report
Abstract
We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
Methodology
The paper introduces Fish Audio Drama 3, a 35B-parameter Mixture-of-Experts (MoE) speech generation model. The core architectural innovation is the replacement of the previous "Fast AR" module with a "Codebook Multi-Token-Prediction (MTP) Decoder." This decoder uses dedicated layers for each residual codebook that attend causally over the entire sequence, allowing fine acoustic details from reference audio to directly influence codebook predictions, rather than being compressed into a single backbone state. The backbone utilizes a hybrid of Gated DeltaNet (linear attention) and full attention layers to handle long contexts efficiently. A significant methodological contribution is the training pipeline, which employs a single learned reward model for Reinforcement Learning (RL) instead of rule-based rewards (like WER or speaker similarity). This addresses the conflict where expressive performance (whispering, laughing) often penalizes standard intelligibility/similarity metrics. The RL uses Group Relative Policy Optimization (GRPO) with large batches and no KL penalty, optimizing a preference model trained on human judgments.
Results & evaluation
The evaluation is comprehensive, covering multilingual cloning (MiniMax Multilingual Testset), editing (Ming-Freeform-Audio-Edit), and revision (Step-Audio-Edit). Drama 3 demonstrates state-of-the-art or competitive performance, particularly in multilingual cloning where it outperforms OmniVoice and FireRedTTS3-Base in multiple languages. The paper provides a strong ablation on the reward model, showing that the preference-based reward correctly prioritizes styled speech (e.g., whispering) over neutral speech, whereas rule-based rewards would penalize it. Inference efficiency is also rigorously evaluated, showing that despite having 7x more parameters than the previous model (S2), the new architecture and FP8 quantization allow for faster decoding (271 fps vs 177 fps on H200).
Reproducibility
High. The model weights are released on Hugging Face and GitHub. The paper provides detailed descriptions of the architecture, training stages, and inference engine optimizations. While the specific data curation details are proprietary, the architectural and algorithmic details are sufficient for reproduction of the model structure. The use of open-source benchmarks and public baselines enhances reproducibility of the evaluation.
Limitations
The model is large (35B parameters), which may limit deployment on edge devices despite the MoE efficiency. The paper relies on a proprietary reward model for RL, which is not released, making it difficult to fully reproduce the RL alignment process. The evaluation of "directed performance" is somewhat subjective and relies on the specific benchmarks provided, which may not cover all nuances of human direction.
Broader impact
This work significantly advances the field of expressive speech synthesis by unifying cloning, direction, editing, and multi-speaker scenes in a single model. The shift from rule-based to preference-based RL rewards is a crucial step towards more natural and controllable audio generation. The open-source release of such a large, high-performing model lowers the barrier for research and application in interactive characters, dubbing, and audiobooks. Fish Audio Drama 3 introduces a 35B MoE speech model with a novel codebook-MTP decoder and preference-based RL, achieving state-of-the-art results in multilingual cloning and expressive performance while improving inference efficiency through architectural and engineering optimizations.
Author of this paper? Send a correction.