Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
Abstract
Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
Methodology
The paper proposes "LoopTTS," a closed-loop TTS framework comprising a Filter, Judge (AudioLLM), and Refiner. The core methodological contribution is the Refiner, a fine-tuned TTS model (based on EmoVoice) that performs guided expressive re-synthesis conditioned on the initial defective audio, target text, and structured instructions from the AudioLLM Judge. The training methodology introduces a "Position-Weighted Cross-Entropy Loss" and a "Structural Alignment Loss" using Representational Similarity Matrices (RSMs) to focus supervision on sparse prosodic spans (stress/pause). The data construction pipeline ("Refiner-DB") uses contrastive AudioLLM annotation to generate weak supervision for word-level prosody. The approach is technically sound, addressing the specific gap of local prosodic defects that utterance-level metrics miss. The use of RSMs for span-level consistency is a creative application of representation learning to TTS fine-tuning.
Results & evaluation
The experiments are rigorous and well-structured. The authors evaluate the full pipeline on a subset of flagged utterances (200 samples) and a larger scaled evaluation (3,000 samples). They include strong baselines: open-loop re-generation (CosyVoice2, EmoVoice), global-only control, and a human-instruction upper bound. The use of a different AudioLLM (Qwen3-Omni) for the Judge stage demonstrates robustness. Subjective evaluations (MOS, MOS-I) are conducted by professional evaluators with reported ICC reliability. The results show consistent improvements in recovery quality and instruction-following fidelity (stress/pause) over baselines. The ablation studies effectively isolate the contributions of the position-weighting and structural alignment losses.
Reproducibility
The paper provides high reproducibility. Code, data construction scripts, prompts, and model checkpoints are promised for release. The GitHub link is provided. Detailed hyperparameters, loss formulations, and dataset composition are included in the main text and appendices. The use of public datasets (ESD, RAVDESS, etc.) and standard models (CosyVoice2, Whisper) further enhances reproducibility.
Limitations
The primary limitation is the reliance on a proprietary AudioLLM (Gemini-3-pro) for both data annotation and inference-time judging, which may limit accessibility and introduce potential biases. The evaluation is focused on English speech, and the framework's applicability to tonal languages is not tested. The "weak supervision" from AudioLLMs is acknowledged as imperfect, with moderate F1 scores for pause detection. The system is designed for offline batch processing, not streaming, due to the multi-stage latency.
Broader impact
This work has significant implications for the deployment of high-quality TTS systems in production environments where sporadic prosodic errors are unacceptable. It demonstrates a viable path for integrating LLM-based diagnostic capabilities into generative audio pipelines, moving beyond simple generation to generation-and-correction. The methodology of using AudioLLMs for scalable weak supervision of fine-grained prosodic features could be extended to other audio tasks, such as music production or sound effect design. The paper presents a novel closed-loop TTS system that integrates AudioLLM-based diagnosis with a specialized Refiner model to correct local prosodic defects. By introducing a position-aware training objective and a contrastive data construction pipeline, the authors effectively address the challenge of fine-grained prosodic control in TTS, demonstrating significant improvements in recovery quality and instruction-following fidelity over open-loop baselines.
Author of this paper? Send a correction.