VoxWatermark: A Large-Scale Benchmark for Audio Watermark Detection under Perturbations
Abstract
With the rapid deployment of speech generation systems in open environments, providing verifiable source attribution and copyright accountability for audio content has become critical. A gap in current research is the lack of a unified benchmark that systematically compares different watermark injection methods under realistic distribution shifts. To address this, we build VoxWatermark by applying 10 watermarking methods (4 neural and 6 traditional) with unified injection and annotation on multilingual, multi-source corpora, and introducing no-box, black-box, and white-box perturbations to simulate real recording and transmission conditions. Based on this benchmark, we propose AudioWMD as a robust baseline detector for large-scale, multi-method, cross-distribution settings. Results show that injection-method diversity and distribution shifts affect detection stability, while validating the effectiveness and scalability of AudioWMD. Dataset and code are publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents VoxWatermark, a large-scale benchmark for audio watermark detection, and proposes the AudioWMD framework, significantly advancing the state of research in audio watermarking and detection methodologies. The comprehensive evaluation of various watermarking methods under realistic perturbations provides valuable insights into the robustness of detection systems, paving the way for future advancements in the field.
Methodology
The paper introduces a comprehensive methodology for audio watermark detection by constructing the VoxWatermark benchmark, which systematically evaluates various watermarking methods under different perturbation scenarios. The proposed AudioWMD framework employs a two-stage detection process that incorporates query-response stability analysis, enhancing robustness against adversarial attacks. The methodology is well-structured, addressing a significant gap in the existing literature regarding the evaluation of watermark detection systems.
Results & evaluation
The experiments are rigorously designed, utilizing a large-scale dataset of over 126,000 hours of audio across multiple languages and perturbation types. The authors provide a detailed comparison of their AudioWMD detector against a baseline model (WMD), demonstrating superior performance across various out-of-domain (OOD) test sets. The results highlight the effectiveness of the proposed approach in maintaining detection stability under realistic conditions, although performance does degrade under certain adversarial attacks.
Reproducibility
The authors have made their dataset and code publicly available, which is a significant step towards ensuring reproducibility. The detailed description of the experimental setup, including data partitioning and evaluation protocols, further supports the reproducibility of the results. However, the reliance on specific hyperparameters and configurations may still pose challenges for complete replication.
Limitations
One limitation noted is the vulnerability of the AudioWMD detector to certain black-box attacks, indicating that while the model improves robustness, it is not entirely immune to sophisticated adversarial strategies. Additionally, the performance under no-box perturbations approaches chance levels, suggesting that further work is needed to enhance resilience against common audio processing distortions.
Broader impact
This research has significant implications for the fields of audio security and copyright protection, particularly as synthetic audio generation becomes more prevalent. The development of a robust watermark detection system is crucial for ensuring the integrity and authenticity of audio content in various applications, including media production and digital forensics. The paper presents VoxWatermark, a large-scale benchmark for audio watermark detection, and proposes the AudioWMD framework, significantly advancing the state of research in audio watermarking and detection methodologies. The comprehensive evaluation of various watermarking methods under realistic perturbations provides valuable insights into the robustness of detection systems, paving the way for future advancements in the field.
Author of this paper? Send a correction.