Advancing Speaker-Based Vocal Effort Classification with WavLM and Data Augmentation in Naturalistic Non-Calibrated Speech Recordings
Abstract
The variations in vocal effort range (e.g. whisper, soft, neutral, loud, shout) alter production and speech acoustics, reducing intelligibility and limiting the robustness of any subsequent speech technology. Classification is challenging since effort lies on a continuum, adjacent categories are easily confused, and labeled data remain scarce. Prior SSL approaches with wav2vec2, HuBERT, and AST improve performance on the AVID corpus but still suffer from boundary errors. In this study, we introduce WavLM for the first time in vocal effort classification and benchmark it against wav2vec2 and HuBERT. To address data scarcity, we conduct a systematic study of augmentation strategies, covering RIR convolution, additive noise, time masking, speed perturbation, band-limiting, MixUp, and CutMix. Augmentation consistently improves WavLM, with gains ranging from +0.6% to +1.8% absolute. We further propose Gaussian-neighbor soft labels, which further reduce near-boundary confusions by modeling the vocal effort continuum. Our best system, WavLM-BASE with gradual unfreezing, augmentation, and Gaussian-neighbor soft labels, achieves 78.2% mean accuracy, establishing a new state-of-the-art on AVID.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a rigorous benchmarking of SSL models for vocal effort classification, introducing WavLM and Gaussian-neighbor soft labels to mitigate boundary errors, thereby establishing a new state-of-the-art on the AVID corpus with incremental but meaningful improvements in robustness and accuracy.
Methodology
The paper proposes a systematic fine-tuning of Self-Supervised Learning (SSL) models, specifically introducing WavLM-Base to the vocal effort classification (VE-ID) task. The core methodological contributions lie in three areas: (1) Benchmarking WavLM against wav2vec2 and HuBERT, finding WavLM superior; (2) A comprehensive study of waveform-level and mix-based data augmentations; and (3) The proposal of "Gaussian-neighbor soft labels," which replaces standard label smoothing with a distribution that accounts for the ordinal proximity of vocal effort classes (e.g., 'soft' is closer to 'normal' than to 'very loud'). The methodology is sound and logically structured, addressing the specific challenge of boundary confusion in a continuous-like classification task. However, the novelty is moderate as SSL fine-tuning is now standard practice, and the soft-labeling technique, while well-motivated, is a variation of existing ordinal regression or label smoothing techniques.
Results & evaluation
The experiments are conducted on the AVID corpus, a standard dataset for this task, using 10-fold group cross-validation. The results show a clear improvement over previous baselines, achieving 78.2% mean accuracy. The ablation studies effectively demonstrate the individual contributions of WavLM, specific augmentations (MixUp being most effective), and the Gaussian soft labels. The statistical reporting includes standard deviations, adding credibility. However, the gains, while consistent, are incremental (e.g., +0.6% to +1.8% from augmentation). The comparison is limited to Base-sized models, ignoring Large variants which might offer different trade-offs, though the authors justify this based on data scarcity. The confusion matrix analysis supports the claim of reduced boundary errors.
Reproducibility
The paper provides sufficient detail regarding the dataset (AVID non-calibrated), model architectures (Base variants), training hyperparameters (learning rates, batch size, epochs), and augmentation techniques. The use of standard libraries (implied by the model names) and standard evaluation metrics (accuracy, group K-fold) enhances reproducibility. The specific implementation of the Gaussian-neighbor soft labels is described mathematically and conceptually, allowing for replication.
Limitations
The study is confined to the AVID corpus, which consists of read speech in a controlled laboratory setting (close-talking microphone), despite the title's claim of "naturalistic" recordings (the non-calibrated aspect adds some realism, but it is not truly naturalistic/conversational). The results may not generalize to spontaneous speech or noisy environments not covered by the augmentation strategies. The focus on Base models limits the exploration of scaling laws. The performance gain, while statistically significant, is modest in absolute terms.
Broader impact
This work contributes to the robustness of speech technologies, particularly in applications where vocal effort is a critical feature, such as hearing aid adaptation, speaker state monitoring, and robust ASR front-ends. By demonstrating the efficacy of WavLM and tailored regularization techniques, it provides a blueprint for handling ordinal classification problems in speech processing. The focus on data scarcity and augmentation strategies is broadly applicable to low-resource speech tasks. This paper presents a rigorous benchmarking of SSL models for vocal effort classification, introducing WavLM and Gaussian-neighbor soft labels to mitigate boundary errors, thereby establishing a new state-of-the-art on the AVID corpus with incremental but meaningful improvements in robustness and accuracy.
Author of this paper? Send a correction.