Robust Audio Tagging under Class-wise Supervision Unreliability
Abstract
Weakly labeled datasets such as AudioSet have driven recent progress in audio tagging. However, annotation quality varies across sound classes. Labels may be incomplete, ambiguous, or unreliable, which introduces class-dependent supervision bias during optimisation. The issue becomes harder as real and generated audio are increasingly mixed in training, and generated samples do not always match their intended semantic labels. Prior work mainly addressed unreliable supervision from missing-positive labels, while this paper targets three other sources of unreliable supervision: spurious additions, misassignments between similar classes, and weakened label evidence. These effects introduce class-dependent optimisation bias that is not explicitly modeled by most existing methods. To bridge this gap, the paper proposes a Class-wise Supervision Unreliability (CSU) framework that controls supervision strength at the class level during training. CSU learns a separate unreliability parameter for each class and down-weights less reliable supervision without changing the model architecture or inference process. To support evaluations, this paper also introduces ESC-FreeGen50, a manually verified benchmark of 50 sound classes that combines real and generated audio. Experiments on controlled benchmarks and AudioSet show that CSU improves robustness across different architectures and different sources of supervision unreliability. The results indicate that explicit class-wise modeling of supervision unreliability is an effective and practical strategy for robust audio tagging under large-scale weakly labeled training. Code and data are available at: https://github.com/Yuanbo2020/CSU
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the Class-wise Supervision Unreliability (CSU) framework, which effectively addresses the challenges posed by unreliable supervision in audio tagging tasks. The comprehensive evaluation and the introduction of a new benchmark dataset significantly advance the state of the art in robust audio tagging methodologies.
Methodology
The paper introduces the Class-wise Supervision Unreliability (CSU) framework, which innovatively addresses the problem of unreliable supervision in audio tagging by learning separate unreliability parameters for each class. This approach allows for dynamic down-weighting of less reliable supervision without altering the model architecture or inference process. The methodology is well-structured, addressing three specific types of supervision unreliability (spurious additions, misassignments, and weakened label evidence) and providing a clear rationale for the need for class-wise control mechanisms. The incorporation of a new benchmark dataset, ESC-FreeGen50, further enhances the methodology by allowing controlled evaluations of the proposed framework.
Results & evaluation
The experiments are comprehensive, utilizing both the newly introduced ESC-FreeGen50 dataset and the well-established AudioSet for validation. The results demonstrate that CSU significantly improves robustness across various architectures and types of supervision unreliability. The evaluation metrics used, including mean Average Precision (mAP) and F1-score, are appropriate for the audio tagging task. The paper effectively shows the performance gains of CSU over baseline models and other robust learning methods, providing strong empirical support for the proposed framework.
Reproducibility
The paper provides sufficient details regarding the implementation of the CSU framework and the experimental setup, including model architectures, training procedures, and evaluation metrics. However, the lack of a demo URL or direct access to the experimental results may hinder full reproducibility for external researchers. Nonetheless, the availability of the code and dataset on GitHub is a positive aspect for reproducibility.
Limitations
While the paper presents a robust framework, it does not thoroughly explore the potential limitations of the CSU approach, such as the impact of varying the number of classes or the generalizability of the learned unreliability parameters across different datasets. Additionally, the reliance on manually verified labels for the ESC-FreeGen50 dataset may limit its scalability and applicability to larger, less curated datasets.
Broader impact
The proposed CSU framework has significant implications for the field of audio tagging and weakly supervised learning, particularly in real-world applications where annotation quality is often inconsistent. By improving robustness against unreliable supervision, this work can enhance the performance of audio tagging systems in various domains, including environmental sound recognition and multimedia content analysis. The introduction of the ESC-FreeGen50 dataset also provides a valuable resource for future research in this area. The main contribution of this paper is the introduction of the Class-wise Supervision Unreliability (CSU) framework, which effectively addresses the challenges posed by unreliable supervision in audio tagging tasks. The comprehensive evaluation and the introduction of a new benchmark dataset significantly advance the state of the art in robust audio tagging methodologies.
Author of this paper? Send a correction.