Harness TTS: Towards Context-Aware Expressive Speech Synthesis with Harness Layer
Abstract
Expressive speech synthesis for voice assistants requires flexible style control that adapts to explicit requests and broader interaction context. We propose Harness TTS, a lightweight control layer that wraps around a TTS engine to externalize and govern its expressive behavior. It reformulates style control as closed-set prompt-tool routing: offline, a compact registry of stylistic prompt tools is constructed with structured metadata; online, an LLM planner selects the appropriate tool based on a priority-aware observation schema, and the TTS executor synthesizes speech using the corresponding prompt audio. We evaluate Harness TTS on both routing and synthesis tasks. In routing, Qwen3-4B achieves Top-1 accuracies of 74.3%, 43.0%, and 64.6% on explicit, implicit, and conflict subsets. For synthesis, experiments on CosyVoice3 and VoxCPM2 show that Harness TTS outperforms instruction-only control, achieving higher instruction-following win rates (margins of 23.1-35.6 points on CosyVoice3 and 13.8-20.0 points on VoxCPM2) and improving UTMOSv2 scores by 0.11-0.38. Moreover, the 4B planner delivers its first tool recommendation in under 50 ms in standard mode, introducing negligible latency for real-time interaction. These results demonstrate that equipping TTS engines with a dedicated Harness layer offers a practical, auditable, and context-aware solution for voice assistant expression control.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Harness TTS presents a practical and well-structured approach to stabilizing expressive TTS in voice assistants by externalizing style selection into an LLM-driven routing layer, demonstrating significant improvements in instruction following and naturalness over direct instruction methods, though its reliance on synthetic evaluation and closed-set tools limits its generalizability and open-ended expressiveness.
Methodology
The paper proposes "Harness TTS," a control layer that wraps around existing Text-to-Speech (TTS) engines to manage expressive speech synthesis. The core innovation lies in reformulating style control as a closed-set tool routing problem. Instead of relying on continuous acoustic parameters or free-form instruction embeddings which can be unstable, the system uses a structured registry of "tools" (audio prompts) with metadata. An LLM planner observes multi-source context (user profile, scenario, explicit instructions) and selects the most appropriate tool based on a priority-aware schema. This approach separates the decision-making logic (Harness) from the generative engine (TTS), aiming for improved stability, interpretability, and context-awareness. The methodology is sound and addresses a genuine gap in making expressive TTS robust for voice assistants, particularly regarding conflict resolution between implicit context and explicit user intent.
Results & evaluation
The evaluation is comprehensive, covering both the routing accuracy of the LLM planner and the synthesis quality of the end-to-end system. The routing experiments demonstrate that Qwen3-4B can effectively select tools, with particularly strong performance on explicit instructions (74.3% Top-1) and reasonable performance on implicit (43.0%) and conflict (64.6%) scenarios. The synthesis experiments on CosyVoice3 and VoxCPM2 show significant improvements in instruction-following win rates compared to direct instruction conditioning, validating the utility of the Harness layer. The latency analysis is also a strong point, showing that First-ID latency is low enough for real-time interaction. However, the reliance on LLM-generated data for evaluation datasets and LLM-based judges (Gemini-2.5-Pro) introduces potential bias and limits the generalizability of the results to human perception.
Reproducibility
The paper provides detailed descriptions of the tool registry structure, observation schema, and priority policies. The use of standard LLMs (Qwen3 series) and TTS engines (CosyVoice3, VoxCPM2) aids reproducibility. However, the specific prompts, few-shot examples, and the exact construction of the synthetic evaluation datasets are not fully detailed, which might make exact replication difficult. The lack of code release (no project URL provided) further hinders immediate reproducibility.
Limitations
The authors acknowledge several limitations: the use of CoT increases latency, the evaluation relies on LLM teachers/judges rather than human annotations, and the test data is synthetically generated rather than from real-world logs. Additionally, the closed-set nature of the tool registry limits the expressiveness to pre-defined styles, which might not cover all edge cases or novel user requests. The dependency on high-quality pre-recorded prompt audio for the registry also poses a scalability and curation challenge.
Broader impact
This work has significant potential for improving the user experience of voice assistants by making them more natural and contextually appropriate. By providing a stable and auditable interface for style control, it could lower the barrier for developers integrating expressive TTS into applications. However, the reliance on LLMs for decision-making raises concerns about cost, latency, and potential biases in style selection. The closed-set approach also limits the open-ended creativity of the TTS system. Harness TTS presents a practical and well-structured approach to stabilizing expressive TTS in voice assistants by externalizing style selection into an LLM-driven routing layer, demonstrating significant improvements in instruction following and naturalness over direct instruction methods, though its reliance on synthetic evaluation and closed-set tools limits its generalizability and open-ended expressiveness.
Author of this paper? Send a correction.