AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Abstract
We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Methodology
The paper proposes AuK, a unified foundational model for speech generation and editing. The architecture is a hybrid of a Multimodal Large Language Model (MLLM) for semantic conditioning, a Variational Autoencoder (VAE) for acoustic conditioning, and a Transformer backbone using a FLUX-style design (dual-stream MMDiT blocks followed by single-stream DiT blocks). The training pipeline is extensive, involving a two-stage pre-training curriculum (generation-only warm-up followed by joint generation-editing), post-training with human-feedback preference optimization (DPO) for editing and Flow-GRPO for generation, and a distillation stage using Consistency Initialization and Task-Routed Decoupled DMD to achieve 4-step inference. The methodology is sophisticated, particularly the integration of RL and preference optimization for audio tasks, which is relatively novel in this specific combination. The data construction pipeline is also detailed, covering 3.03 billion instruction-audio instances across five task families.
Results & evaluation
The experiments cover a wide range of tasks: zero-shot TTS, instruction TTS, acoustic editing, paralinguistic editing, content editing, enhancement, and separation. The paper claims leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing. It reports a 4.5x wall-clock speedup for the distilled model (AuK-Flash). However, the provided text is truncated before the detailed results tables, so specific quantitative comparisons (e.g., MOS scores, WER, speaker similarity scores) against baselines are not fully visible in the excerpt. The reliance on "leading performance" claims without visible comparative tables in the provided text limits the ability to fully verify the magnitude of improvement, though the scope of evaluation is comprehensive.
Reproducibility
The paper claims to release both source code and model weights. It provides detailed hyperparameters for training (learning rates, batch sizes, GPU counts, optimizer settings) and architectural details (layer counts, hidden dimensions, strides). The data construction pipeline is described in detail, including specific tools used for annotation and synthesis (Qwen3-Omni, F5-TTS, etc.). This level of detail supports high reproducibility, assuming the released code and weights are complete and functional.
Limitations
The model is large (1.5B parameters in the backbone alone, plus the MLLM and VAE), which may limit deployment on edge devices despite the distillation. The reliance on a complex MLLM for semantic conditioning adds computational overhead during inference compared to simpler text encoders. The paper is a technical report, so peer review rigor may be lower than a top-tier conference paper. The truncated text prevents a full assessment of the experimental results' depth.
Broader impact
The unification of speech generation and editing under a single instruction-based interface has significant potential for practical applications in content creation, accessibility, and interactive voice assistants. The open-source release of such a comprehensive model could accelerate research in unified audio models. The techniques for post-training (RL and DPO) and distillation are applicable to other generative audio tasks. The paper introduces AuK, a unified foundational model for speech generation and editing that leverages a hybrid MLLM-VAE-Transformer architecture, extensive data construction, and advanced post-training strategies including preference optimization and reinforcement learning. It represents a significant step towards general-purpose audio AI by consolidating diverse tasks into a single instruction-driven framework, offering a robust and efficient solution for complex speech manipulation tasks.
Author of this paper? Send a correction.