DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Abstract
Zero-shot text-to-speech (TTS) can reproduce an unseen speaker from a short reference recording, but typically entangles speaker identity and accent within the same reference. We introduce DEFINE, an end-to-end framework that decouples these factors by conditioning speaker identity and target accent on separate audio exemplars. A single inference-time guidance weight continuously controls accent strength without retraining. Built on F5-TTS with parameter-efficient LoRA adaptation, DEFINE maps short accent exemplars into a conditioning space using an exemplar encoder supervised through learned accent prototypes, requiring neither accent labels at inference time nor post-synthesis waveform conversion. On seen accents, increasing accent guidance improves accent-probe accuracy from 6.5% to 19.6%. More importantly, a single DEFINE model generalizes accent control beyond its training accent set: on seen and out-of-domain accents, though not on held-out accents, it matches the accent transfer performance of a two-model TTS-voice-conversion cascade while achieving higher speaker similarity and comparable predicted speech quality. These results demonstrate that speaker identity and accent can be independently controlled from audio exemplars within a single zero-shot TTS model, including for accents unseen during training.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces DEFINE, a framework that decouples speaker identity and accent in zero-shot TTS by using separate audio exemplars and a prototype-anchored encoder, achieving comparable accent transfer to a two-model cascade while better preserving speaker identity. The technical contribution is significant, with a novel supervision mechanism for accent embeddings and a continuous inference-time control for accent strength. The experiments are thorough, including objective metrics and a small-scale listening test, demonstrating the effectiveness of the approach. The work is well-positioned to influence future research on disentangled speech synthesis and fine-grained control in TTS systems.
Methodology
The paper proposes DEFINE, a framework for disentangling speaker identity and accent in zero-shot TTS. The core technical contribution is the "prototype anchoring" mechanism, which uses a learned table of accent prototypes to supervise an exemplar encoder (based on frozen XLS-R features). This addresses the weak supervision signal for accent representations in flow-matching objectives. The method injects the accent embedding into the F5-TTS backbone via additive shifts to timestep and text embeddings, scaled by RMS to ensure proper magnitude. A key feature is the inference-time guidance weight $w$ that allows continuous control over accent strength without retraining. The use of LoRA for parameter-efficient adaptation is standard but effective. The methodology is sound, though the reliance on a specific backbone (F5-TTS) and the specific injection point (adaLN modulation via embedding shifts) limits generalizability to other architectures.
Results & evaluation
The experiments are comprehensive, evaluating seen, held-out, and out-of-domain accents. The use of a 15-way logistic regression probe on XLS-R features for accent accuracy is a reasonable objective metric, though it is a proxy for perceptual accent. The comparison against a Seed-VC cascade is strong, showing that DEFINE matches accent transfer performance while improving speaker similarity. The listening test with 11 listeners is a positive addition, confirming that the accent changes are perceptually real and that the guidance weight controls the perceived accent strength. However, the sample size for the listening test (11 listeners, 10 trials) is relatively small, and the statistical significance, while present, is based on a limited number of votes. The WER and UTMOS metrics are standard and reported, showing that intelligibility and quality are maintained.
Reproducibility
The paper provides a GitHub link, which is a strong indicator of reproducibility. The training data sources (Common Voice, in-house studio recordings) are described, but the in-house data is not publicly available, which limits full reproducibility. The hyperparameters for LoRA, the prototype table size, and the guidance weight are specified. The use of frozen XLS-R and specific layer selection (layer 15) is detailed. Overall, reproducibility is good, assuming access to the code and the ability to recreate the in-house speaker prompts or use a similar public dataset.
Limitations
The main limitation is the reliance on a specific TTS backbone (F5-TTS), which may not generalize to other architectures without significant modification. The accent control is limited to the accents seen during training or those that can be represented by the exemplar encoder; truly novel accents not in the training distribution may not be well-captured. The listening test is small-scale, and the objective accent probe may not fully capture perceptual accent nuances. The method requires separate exemplar clips for accent, which adds a data requirement at inference time.
Broader impact
The work has significant implications for personalized TTS systems, allowing users to control not just the voice but also the accent of the synthesized speech. This could be useful for language learning, accessibility, and creative applications. The decoupling of identity and accent is a step towards more fine-grained control over speech synthesis, which is a key goal in the field. The method's ability to generalize to out-of-domain accents is particularly promising for real-world applications where the target accent may not be in the training set. The paper introduces DEFINE, a framework that decouples speaker identity and accent in zero-shot TTS by using separate audio exemplars and a prototype-anchored encoder, achieving comparable accent transfer to a two-model cascade while better preserving speaker identity. The technical contribution is significant, with a novel supervision mechanism for accent embeddings and a continuous inference-time control for accent strength. The experiments are thorough, including objective metrics and a small-scale listening test, demonstrating the effectiveness of the approach. The work is well-positioned to influence future research on disentangled speech synthesis and fine-grained control in TTS systems.
Author of this paper? Send a correction.