QuaSR: Quality-Aware Sample Reweighting for Pacific Indigenous Speech Recognition
Abstract
Training automatic speech recognition (ASR) models for low-resource languages is challenging due to limited data and highly variable supervision quality. In particular, Pacific Indigenous speech corpora often exhibit heterogeneous acoustic conditions, transcript inconsistencies, and varying degrees of acoustic-text alignment reliability, making standard fine-tuning approaches sensitive to noisy or misleading supervision signals. In this work, we propose QuaSR, a simple yet effective weighting framework that combines data-side reliability with model-side learnability to improve ASR adaptation. Specifically, we estimate data reliability from acoustic, transcription, and alignment, while measuring learnability using training loss from the model. These two complementary signals are integrated into a unified sample utility score to produce training weights for the samples. We also evaluated across four Pacific Indigenous languages, which shows that the proposed utility scores reliably correlate with adaptation performance. Furthermore, QuaSR consistently improves ASR adaptation over standard fine-tuning and alternative data selection strategies, highlighting a new way to leverage difficulty scores for low-resource speech learning.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
Methodology
The paper proposes QuaSR, a sample reweighting framework for Automatic Speech Recognition (ASR) adaptation in low-resource settings. The core methodology involves calculating a "sample utility" score by multiplying a data-side reliability term (derived from acoustic quality, transcript stability, and alignment reliability) with a model-side learnability term (derived from teacher-forced loss). This approach attempts to disentangle "hard but learnable" samples from "noisy/unreliable" samples, which is a critical distinction in low-resource domains where data quality is heterogeneous. The methodology is well-motivated and addresses a genuine gap in current PEFT (Parameter-Efficient Fine-Tuning) practices for ASR, which often treat all samples equally or rely solely on loss-based filtering. However, the novelty is moderate as sample reweighting based on quality heuristics is not entirely new in the broader ML community, though its specific application and combination for Pacific Indigenous languages is a novel contribution. The technical implementation details, particularly the alignment reliability metric, are somewhat opaque in the text (referencing undefined variables like 'm' and 'pho' in the raw text), suggesting a need for clearer formalization.
Results & evaluation
The evaluation is conducted on four Pacific Indigenous languages (Bislama, Nafsan, Lelepa, Nguna) using Whisper-small fine-tuned with LoRA. The results demonstrate that QuaSR consistently improves WER and CER compared to standard unweighted fine-tuning and hard filtering baselines. The gains are most significant for languages with lower data volume and higher quality heterogeneity (Lelepa, Nguna), which aligns with the paper's hypothesis. The analysis of why certain reliability signals (e.g., transcript-alignment vs. acoustic-alignment) work better for specific languages adds valuable empirical insight. However, the comparison is limited to Whisper-small and LoRA; it does not explore other foundation models or adaptation techniques. The ablation studies on the reliability components are helpful but could be more extensive. The claim of "largest gains" is supported by the data, but the absolute performance metrics remain relatively high, indicating the inherent difficulty of the task.
Reproducibility
The paper provides sufficient detail on the experimental setup, including model architecture (Whisper-small, LoRA ranks, learning rates), dataset sources (PARADISEC), and the specific metrics used for reliability estimation (SQUIM/STOI, MMS forced alignment). The use of standard tools (SQUIM, MMS) enhances reproducibility. However, the raw text contains placeholders and undefined variables in the equations section (e.g., "what is pho here?", "m is just a symbol"), which suggests that the final published version must address these clarity issues. The code is not explicitly linked, which hinders immediate reproducibility.
Limitations
The primary limitation is the dependency on external tools for reliability estimation (SQUIM, MMS), which may not be available or accurate for all low-resource languages. The method assumes that the "unweighted baseline" run is a good proxy for learnability, which might not hold if the baseline fails to converge. The paper does not address the computational overhead of computing these reliability scores, which could be significant for very large datasets. Furthermore, the "transcript stability" metric relies on heuristic counting of irregularities, which may not capture all forms of transcription error. The evaluation is restricted to four languages, limiting the generalizability of the findings to other low-resource language families.
Broader impact
This work has significant positive impact for the preservation and technological empowerment of Indigenous communities in the Pacific region. By improving ASR performance on under-resourced languages, it facilitates better access to technology and documentation for these communities. The framework also offers a generalizable approach for handling noisy, low-quality data in other domains beyond speech, such as text classification or computer vision, where data quality is a major bottleneck. The emphasis on data quality over quantity shifts the narrative in low-resource ML towards more sustainable and ethical data practices. QuaSR presents a robust, multi-dimensional quality-aware reweighting framework that effectively improves ASR adaptation for low-resource Pacific Indigenous languages by distinguishing between data reliability and model learnability, offering a scalable solution for noisy supervision in under-resourced domains.
Author of this paper? Send a correction.