TokAN: Accent Normalization Using Self-Supervised Speech Tokens
Abstract
Accent normalization (AN) seeks to convert non-native (L2) accented speech into standard (L1) speech while preserving speaker identity. The current techniques either require naturally recorded parallel L1-L2 speech for training, or suffer from quality degradation when supervised by synthesized targets. In this paper, we present TokAN, a token-based accent normalization framework that operates on self-supervised discrete speech tokens extracted from a L1-L2 jointly trained vector-quantization (VQ) tokenizer, without the need of synthetic supervisory speech. An autoregressive encoder-decoder model performs token-to-token conversion, translating L2-accented token sequences into the tokens of standard voice. We also introduce reinforcement learning (RL) post-training based on Group Relative Policy Optimization (GRPO), using word error rate and accent classifier confidence as complementary rewards. A non-autoregressive flow-matching synthesizer recovers the Mel-spectrogram from the converted tokens, conditioned on the source speaker embedding. We also develop a flow-matching duration predictor that supports total-duration-aware synthesis, making TokAN applicable to duration-critical tasks such as voice dubbing and live casting. Experiments on seven English accents demonstrate that TokAN reduced the word error rate from 12.40% to 9.89% after supervised fine-tuning, and further to 9.23% after RL post-training, consistently outperforming frame-to-frame, direct flow-matching, and prompt-based token-conversion baselines in terms of accent reduction and intelligibility.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
Methodology
The paper proposes TokAN, a token-based accent normalization framework that addresses the data scarcity and quality degradation issues in existing methods. The core methodological contributions are threefold: (1) A jointly trained Vector Quantization (VQ) tokenizer that co-optimizes phonetic content representation with speech synthesis and ASR supervision, replacing static K-Means clustering. This ensures the discrete tokens are more discriminative for accent-related phonetic features. (2) An autoregressive encoder-decoder conversion model using Rotary Positional Encodings (RoPE) and a self-attention-only decoder, trained with BART-style pre-training and supervised fine-tuning on semi-synthetic pairs. (3) A novel Reinforcement Learning (RL) post-training stage using Group Relative Policy Optimization (GRPO). This is a significant technical innovation in the speech domain, as it directly optimizes task-level objectives (Word Error Rate and Accent Classifier Confidence) rather than relying solely on token-level cross-entropy loss. The use of GRPO eliminates the need for a value network, making it computationally efficient for sequence generation. The system also includes a flow-matching synthesizer and a duration predictor for total-duration-aware synthesis, enhancing applicability to dubbing.
Results & evaluation
The evaluation is comprehensive, covering seven English accents from the L2-ARCTIC dataset. The authors compare TokAN against strong baselines: FramAN (frame-to-frame), CosyAccent (direct flow-matching), and VEVO (token-based). Results show TokAN significantly reduces Word Error Rate (WER) from 12.40% (SFT only) to 9.23% (after RL), outperforming all baselines in intelligibility and accent reduction (L1-Prob). Subjective evaluations (MUSHRA) confirm higher naturalness and accentedness reduction. The ablation studies effectively isolate the contributions of the joint tokenizer training, BART pre-training, SFT, and RL post-training. The phonemic distribution analysis provides deep insight into how the model corrects specific accent-related errors (e.g., dental-alveolar fricative confusion for Chinese speakers). The inclusion of both duration-free and source-length modes adds practical value.
Reproducibility
The paper provides extensive implementation details, including hyperparameters for all training stages, dataset descriptions (LibriTTS-R, Emilia-EN, L2-ARCTIC, GLOBE), and model architectures. The code and demo samples are publicly available on GitHub and a dedicated webpage, which significantly enhances reproducibility. The use of standard components (WavLM, HiFT, Whisper) and open-source RL libraries (TRL) further supports reproducibility.
Limitations
The reliance on semi-synthetic data for SFT means the model's performance is still somewhat tied to the quality of the TTS system used to generate native targets. While RL mitigates this by optimizing on real data, the initial mapping is learned from synthetic pairs. Additionally, the speaker similarity is slightly lower than some baselines (like VEVO), indicating a trade-off between accent normalization and voice preservation that might be critical for some applications. The method assumes access to a pre-trained accent classifier and ASR model for rewards, which introduces potential bias if these models are not robust to the specific accents being normalized.
Broader impact
TokAN has significant potential for improving accessibility and communication for non-native speakers, aiding language learning, and enhancing the quality of automated dubbing and live translation services. By reducing the reliance on parallel L1-L2 data, it lowers the barrier to creating accent normalization systems for low-resource languages or less common accents. The use of RL for speech generation also opens new avenues for optimizing speech models against complex, non-differentiable metrics. TokAN introduces a novel token-based accent normalization framework that leverages a jointly trained VQ tokenizer and RL post-training with GRPO to achieve superior accent reduction and intelligibility without requiring naturally recorded parallel speech data.
Author of this paper? Send a correction.