Data Augmentation for L2 English Speaking Assessment using TTS
Abstract
Automated assessment of second language (L2) speaking proficiency relies on large-scale annotated speech data, which remains scarce compared to widely available written learner corpora. A promising direction for addressing this imbalance is to use text-to-speech (TTS) and voice cloning to convert written L2 production into synthetic speech. However, written and spoken L2 differ fundamentally: spontaneous speech includes disfluencies and discourse markers, while writing is more planned and complex. This raises the question of what is required to generate synthetic L2 speech suitable for assessment. We address this through a systematic analysis of speaker-text relationships using COREFL, a publicly available corpus containing paired spoken and written responses from the same L2 learners to the same questions across modalities. In our proposed framework, we first address the structural differences between written and spoken language by transforming written responses into spoken-style transcripts ("speechification") using a large language model. These transcripts are then converted into speech using a TTS/voice-cloning model. To assign a voice to each synthetic response, we investigate different speaker-text pairing strategies based on shared learner attributes (proficiency level, first language, both, or neither). We evaluate our data augmentation techniques on the language assessment task, with improvements shown in both wav2vec2 (audio-based) and ModernBERT (text-based) scoring systems. Results show that matching speakers and texts by proficiency level yields the most robust synthetic speech. Moreover, raw written text leads to a strong mismatch with spoken language, while speechification substantially reduces this gap and improves grading performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
Methodology
The paper proposes a data augmentation framework for L2 speaking assessment by converting written learner essays into synthetic speech. The core methodological contribution is the "speechification" step, where an LLM (GPT-4.1) rewrites written text into spoken-style transcripts incorporating disfluencies, hesitations, and learner errors, conditioned on CEFR proficiency levels. This is followed by zero-shot voice cloning using OmniVoice. The study systematically evaluates speaker-text pairing strategies (matching by CEFR, L1, both, or neither) to determine the optimal conditions for generating synthetic data that mimics real L2 speech characteristics. The approach is well-motivated by the scarcity of annotated L2 speech data and the structural differences between written and spoken language.
Results & evaluation
The authors utilize the COREFL corpus, which provides paired written and spoken responses, allowing for a controlled analysis of modality gaps. They evaluate the synthetic data using two downstream grading models: wav2vec2 (audio-based) and ModernBERT (text-based). The experiments demonstrate that speechification significantly improves performance, particularly for the text-based grader, by reducing the linguistic mismatch. Matching speakers and texts by proficiency level was found to be the most robust strategy. The results show consistent improvements in PCC, SRC, and RMSE metrics when using augmented data. However, the evaluation relies heavily on automatic metrics (WER, PCC, SRC) and speaker verification cosine similarity, lacking subjective human evaluation (e.g., MOS) for speech naturalness or assessment accuracy, which is a notable gap for TTS research.
Reproducibility
The paper provides detailed descriptions of the pipeline, including the LLM prompt for speechification, the TTS model (OmniVoice), and the grading architectures. The COREFL dataset is publicly available. The code for the grading systems is not explicitly linked, but the model architectures (wav2vec2, ModernBERT) are standard. The specific hyperparameters are mentioned to be in a table (referenced but not fully detailed in the snippet provided, though standard practices are implied). The cost of LLM usage is noted, adding transparency. Overall, the methodology is clear enough for reproduction, assuming access to the proprietary OmniVoice model or a suitable open-source alternative.
Limitations
The reliance on GPT-4.1 for speechification introduces a dependency on a specific, potentially costly, and non-deterministic component. The synthetic speech, while linguistically improved via speechification, may still lack the acoustic variability and prosodic nuances of real spontaneous speech, as evidenced by the performance gap between real and synthetic data. The study focuses on English L2 learners, limiting generalizability to other languages without further validation. The lack of human evaluation for the synthetic speech quality and the grading performance limits the confidence in the "realism" of the data from a pedagogical perspective. The use of zero-shot TTS means the synthetic voices are clones but may not capture the full idiosyncratic speech patterns of the original speakers.
Broader impact
This work has significant potential to address the data scarcity problem in automated language assessment, particularly for L2 speaking. By enabling the generation of large-scale, annotated synthetic speech data from abundant written corpora, it could accelerate the development of more robust and fair assessment tools. However, ethical considerations regarding the use of synthetic data in high-stakes testing environments must be addressed, including issues of bias, transparency, and the potential for over-reliance on synthetic data that may not fully represent human speech variability. The paper presents a systematic framework for augmenting L2 speaking assessment data by speechifying written texts and synthesizing speech, demonstrating that matching proficiency levels and modeling spoken linguistic structures significantly improves downstream grading performance.
Author of this paper? Send a correction.