Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
Abstract
Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
Methodology
The paper proposes a "Listen, Critique, and Refine" framework for instruction-following speech synthesis using Large Audio Language Models (LALMs). The core methodological contribution is extending the chain-of-thought/reasoning paradigm from text to audio tokens. The model generates a draft speech ($v_1$), generates a textual critique ($c$) by "listening" to $v_1$, and then generates a refined speech ($v_2$) conditioned on both. The training utilizes Group Relative Policy Optimization (GRPO) with a novel reward function. A key technical insight is the use of a non-linear transformation for the improvement term in the reward to prevent cancellation of the constant first-pass baseline within the GRPO group-relative advantage calculation. This is a sound and well-motivated technical detail that addresses a specific failure mode in applying GRPO to iterative refinement tasks.
Results & evaluation
The experiments are conducted on the InstructTTSEval benchmark, covering in-domain (DSD) and out-of-domain (APS, RP) tasks. The evaluation uses both objective metrics (CLSP score, WER) and subjective metrics (LALM-as-a-judge, human evaluation). The results show consistent improvements over zero-shot and RL one-hop baselines. The ablation studies effectively isolate the contributions of the non-linear reward shaping, speaker retrieval, and the critique mechanism. The human evaluation with 5 senior researchers adds credibility to the subjective gains. However, the training dataset is small (1000 examples from ParaSpeechCaps), which raises questions about the generalizability and robustness of the learned refinement behavior.
Reproducibility
The authors provide a GitHub repository link, which is a positive step. The paper details the model architecture (Step-Audio-2-mini), hyperparameters, and implementation details (LoRA, vLLM, NCCL). However, the reliance on specific proprietary or less common models (Step-Audio-2-mini, CLSP) might limit reproducibility for the broader community compared to using more standard open-source components. The small training set size is a specific detail that is reproducible but may not be representative of large-scale applications.
Limitations
The primary limitation is the small training dataset (1000 samples), which may lead to overfitting or limited generalization. The method relies on the base model's ability to generate meaningful critiques, which may not always be accurate or useful. The two-hop inference process doubles the computational cost compared to single-pass generation. The evaluation is limited to English speech. The reliance on CLSP for style adherence may not capture all nuances of complex instructions.
Broader impact
This work demonstrates the potential of self-refinement and reasoning in audio generation, a promising direction for improving the controllability and quality of LALMs. The technique of using non-linear reward shaping to handle iterative refinement under GRPO could be applicable to other multi-step generation tasks in other modalities. The framework provides a blueprint for integrating understanding and generation in a unified reasoning loop. The paper introduces a novel RL-based self-refinement framework for speech synthesis that enables a LALM to reason over its own audio output. By combining a generate-critique-refine pipeline with a specifically designed GRPO reward function that accounts for iterative improvement, the method achieves significant gains in instruction-following capability, demonstrating the viability of audio-token reasoning for enhancing generative quality.
Author of this paper? Send a correction.