Taming Long-form Text-to-Speech
Abstract
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
Methodology
The paper proposes Localized Attention-Constrained Inference (LACI), an inference-only technique to address reliability issues in long-form autoregressive TTS. The core insight is identifying "emergent alignment heads" within the model's attention mechanism that track the mapping between input text tokens and generated audio tokens. The authors define two specific failure modes: "skips" (deletions) and "hallucinations" (insertions/substitutions). LACI monitors the attention weights of these specific heads in real-time. If the attention pattern deviates from the expected monotonic progress (e.g., skipping a bin of text tokens or stalling), the system detects the error, rolls back the generation state to the error onset, and applies a temporary hard attention mask (guardrail) to force correct alignment during regeneration. This is a clever, low-overhead approach that leverages internal model states rather than external ASR feedback for detection, which is a significant methodological improvement over standard best-of-N sampling.
Results & evaluation
The experiments are rigorous and well-targeted. The authors use the AppTek Call Center dataset, a real-world long-form TTS use case, which is more relevant than standard short-utterance benchmarks for this specific problem. They evaluate on multiple model scales (Qwen3-TTS-0.6B, 1.7B) and families (VoxCPM2), demonstrating generalizability. The metrics include standard WER and SIM, but the introduction of wSIM (sliding-window speaker similarity) is a valuable contribution that exposes failure patterns hidden by global averaging. The results show dramatic improvements in worst-of-N WER (35.2% to 3.4%) and wSIM (0.01 to 0.47), effectively closing the gap between short-form and long-form reliability. The comparison against ACI (always-on guardrails) clearly demonstrates the superiority of the detection-and-recovery approach.
Reproducibility
The paper states that code will be open-sourced upon publication, which is standard for arXiv preprints. However, the specific details of the "instability measure" and detector settings are referenced as being in the code, which is a slight barrier to immediate reproducibility without access to the repository. The use of standard open-weights models (Qwen3-TTS, VoxCPM2) and public datasets (AppTek) aids reproducibility. The manual verification of wSIM failures adds a layer of qualitative validation that is hard to automate but important for trust.
Limitations
The method relies on the existence of "emergent alignment heads," which may not be present or stable in all TTS architectures, particularly non-autoregressive or diffusion-based models. The paper focuses on autoregressive neural codec language models. The computational overhead, while described as "negligible," involves rollback and regeneration, which could be significant for very long texts if failures are frequent. The evaluation is limited to English (implied by the datasets and models used). The "worst-of-N" metric, while useful for reliability, does not reflect the average user experience if N is large, though the paper argues for the importance of reliability in professional settings.
Broader impact
This work has high practical impact for deploying TTS in real-world applications like audiobooks, customer support agents, and long-form voice cloning. The ability to reliably generate long-form speech with consistent speaker identity is a critical bottleneck for current TTS systems. The proposed wSIM metric is a valuable tool for the community to better evaluate speaker consistency in long-form generation. The inference-only nature of the solution makes it easily adoptable by users of existing open-weights models without retraining. LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
Author of this paper? Send a correction.