SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks
Abstract
Speech and audio generation is often needed in animation dubbing, audio drama, movies, advertising, games, podcasts, and short-video production. In these scenarios, creators may need to design voices without reference recordings, control speaker styles with natural language, support acoustic scenes with environments and audio effects, and later reuse the designed voices. Therefore, it is important to support multi-speaker speech and audio generation for both instruct and zero-shot tasks. The instruct task requires a caption of the environment, speaker styles, and fine-grained content, while the zero-shot task uses reference audio together with the same fine-grained content. We address these tasks from both the data and model sides. First, we propose SwanData-Caption, which cleans raw speech and audio data, adds targeted synthetic coverage, and annotates diverse and accurate multi-level captions. Then, we propose SwanTale, a multi-speaker expressive speech and audio generation model that supports both zero-shot and instruct tasks. We introduce SwanVAE to support high-quality multi-audio-modality generation. Then, we adopt reward-conditioned quality control and Engram conditioning, along with Unified MoE for multi-task and multi-audio-modality modeling. In addition, we use curriculum learning and GRPO post-training to let the model progressively learn and strengthen its capabilities. Experimental results show that SwanTale leads on multiple key zero-shot and instruct metrics, achieves the best expressiveness scores in both tasks, and supports complex instruct generation involving multi-speaker speech and audio. Demos can be found at https://swanaigc.github.io/#swantale.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Methodology
The paper proposes SwanTale, a unified model for multi-speaker speech and audio generation supporting both zero-shot (reference-based) and instruct (caption-based) tasks. The methodology is comprehensive, addressing data, model architecture, and training. Key technical contributions include: 1) SwanData-Caption, a pipeline for generating fine-grained multi-level captions using an LLM (Seed2.0 Lite) with a persona library to guide annotation. 2) SwanVAE, a custom VAE with a Gaussian bottleneck and Transformer-based decoder for 48kHz audio, designed to balance reconstruction quality and latent learnability. 3) A flow-based Transformer prior with Unified MoE (Mixture of Experts) to handle multiple tasks and audio modalities (speech, environment, effects, singing). 4) Reward-conditioned quality control and GRPO (Group Relative Policy Optimization) post-training to improve pronunciation, stability, and caption adherence. The approach to unify zero-shot and instruct generation in a single model with complex audio scene modeling is technically sound and represents a significant engineering effort. The use of Engram conditioning and curriculum learning is also noted.
Results & evaluation
The evaluation covers zero-shot TTS, instruct TTS, and acoustic quality. Benchmarks used include SwanBench-Speech, InstructTTSEval, SwanBench-Scene, and SwanBench-Caption. The paper claims leadership on multiple key metrics. The data pipeline includes rigorous filtering (DNSMOS, SQUIM) and human auditing (best-worst comparison for expressiveness). The inclusion of synthetic data for underrepresented demographics (elderly) and challenging pronunciations is a strong point for robustness. The evaluation of multi-speaker dialogue and complex instruct generation adds depth beyond standard single-speaker TTS.
Reproducibility
The paper provides detailed descriptions of the data pipeline, model architecture (SwanVAE, Unified MoE), and training procedures (curriculum, GRPO). However, the code is not publicly available (Project URL is none), and some components (e.g., Seed2.0 Lite, SwanAligner, SwanVerifier) are referenced as internal or proprietary tools. While the methodology is described well, full reproducibility might be challenging without access to the specific internal models and the full dataset (70M records).
Limitations
The reliance on internal data and proprietary models (Seed2.0, SwanAligner) limits immediate reproducibility. The paper mentions "SwanBench" which appears to be a new benchmark, potentially lacking the long-term community validation of established benchmarks like LibriSpeech or VCTK. The complexity of the model (Unified MoE, GRPO) might make inference and fine-tuning resource-intensive. The "Engram conditioning" is mentioned but not fully detailed in the provided text snippet, which might obscure its specific novelty or implementation details.
Broader impact
This work has significant potential for creative industries (animation, gaming, film) by enabling more expressive and controllable audio generation. The ability to generate complex audio scenes with speech and effects from natural language instructions is a powerful tool for content creation. However, the ease of voice cloning and impersonation raises ethical concerns regarding deepfakes and misinformation, which the authors should address in terms of safety guardrails. SwanTale presents a robust and unified framework for multi-speaker speech and audio generation, effectively bridging the gap between zero-shot cloning and instruct-based creative control through a sophisticated data pipeline and a novel VAE-MoE architecture, significantly advancing the state-of-the-art in expressive audio synthesis.
Author of this paper? Send a correction.