Is One Score Enough? Assessing Singing Quality of Songs with Temporal Score Curves
Abstract
Singing Quality Assessment (SQA) has become increasingly important for practical multimedia applications and Music AI systems, yet existing studies predominantly focus on short singing clips and remain insufficient for full-length songs. Unlike clip-level assessment, full-length song SQA requires modeling how singing quality varies across different audio segments and how these local variations influence the overall evaluation of vocal performance. Moreover, the scarcity of segment-level annotations makes effective supervision challenging, as directly assigning a single overall score label to every segment tends to treat different segment qualities as equivalent. To address these challenges, we propose SongSQA, a two-stage framework for full-length song SQA. In the first stage, a Segment Score Predictor is trained with pseudo labels generated by a pre-trained teacher model, enabling segment-level singing quality prediction without requiring manual segment annotations. In the second stage, a Song Quality Aggregator integrates segment features and predicted segment scores into unified segment embeddings, and employs a learnable song embedding together with self-attention to capture the connection between segment-level vocal performance and overall song quality. In this way, SongSQA dynamically aggregates critical quality cues across the song to produce a holistic quality prediction, while also generating a temporal segment-level quality curve. Experimental results demonstrate the effectiveness of SongSQA for full-length song SQA, achieving up to a 13.95% relative improvement in KTAU over the strongest baseline, while consistently improving other evaluation metrics across all datasets.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
Methodology
The paper proposes SongSQA, a two-stage framework for full-length song singing quality assessment (SQA). The core methodological innovation lies in addressing the lack of segment-level annotations by using a pre-trained teacher model to generate pseudo-labels for a Segment Score Predictor. This predictor extracts segment-level features and scores, which are then fused and aggregated by a Song Quality Aggregator using a Transformer with a learnable [CLS]-like token. The approach effectively bridges the gap between clip-level SQA and full-song assessment. The use of pseudo-labels from a specialized teacher (SongEval) rather than simple label sharing is a pragmatic and technically sound solution to the data scarcity problem. The aggregation mechanism using self-attention to model the non-linear influence of segments on overall quality is well-motivated by human perception studies.
Results & evaluation
The experimental evaluation is robust, utilizing two datasets: the public Lyra-SA dataset and a proprietary Internal Dataset annotated by experts. The paper compares SongSQA against strong baselines including clip-level SOTA models (UTMOSv2, PS-SQA, RAMP+) and SSL-based full-song adaptations. SongSQA achieves significant relative improvements (up to 13.95% in KTAU) over the strongest baseline. The ablation studies are thorough, validating the contributions of the teacher model choice, feature fusion strategy, and aggregation design. The case study and subjective evaluation further support the perceptual validity of the temporal score curves generated by the model. The results on the Internal Dataset show even higher performance, suggesting the model benefits from high-quality expert annotations.
Reproducibility
The paper provides detailed implementation details, including dataset splits, segment duration (10s with 5s overlap), model architecture parameters (number of layers, loss weights), and optimizer settings. The use of publicly available datasets (Lyra-SA) and pre-trained models (MuQ, SongEval) enhances reproducibility. However, the Internal Dataset is proprietary, which limits the ability to fully replicate the results on that specific benchmark. The code is not explicitly linked in the text provided, though a demo page is available.
Limitations
The primary limitation is the reliance on a pre-trained teacher model for pseudo-label generation. The quality of the final model is thus bounded by the teacher's ability to discern segment-level singing quality. If the teacher model has biases or errors, these will propagate to the student model. Additionally, the performance on the proprietary Internal Dataset, while impressive, cannot be directly compared to other works due to the lack of public access. The model's performance on non-Mandarin/Cantonese singing (only a small proportion in the internal dataset) is not thoroughly evaluated, limiting generalizability claims.
Broader impact
This work has significant implications for Music AI, particularly in the evaluation of Singing Voice Synthesis (SVS) and Singing Voice Conversion (SVC) systems. By providing a tool for fine-grained, full-song assessment, it enables more precise feedback for model improvement and content moderation on multimedia platforms. The generated temporal score curves can also be used for music education and analysis. The methodology of using pseudo-labels for segment-level supervision in audio quality assessment could be adapted to other domains lacking granular annotations. SongSQA introduces a two-stage framework for full-length song singing quality assessment that utilizes pseudo-labels from a teacher model for segment-level supervision and a Transformer-based aggregator to capture non-linear segment interactions. The paper makes a solid technical contribution to the field of audio quality assessment by shifting the paradigm from clip-level to full-song evaluation. The proposed methodology is innovative in its handling of the annotation scarcity problem through a teacher-student pseudo-labeling strategy, which is a clever adaptation of semi-supervised learning principles to a specific perceptual task. The integration of segment-level predictions into a global context via self-attention allows the model to mimic human perceptual weighting of vocal highlights and flaws. The experimental results are strong and well-supported by ablation studies and subjective evaluations. While the novelty is incremental in terms of architectural components (using Transformers and SSL features is common), the specific application to full-song SQA with temporal curves and the pseudo-labeling strategy for segment supervision is a valuable addition to the literature. The work is technically sound and addresses a clear gap in current SQA systems.
Author of this paper? Send a correction.