The Watermark Shortcut: How Provenance Marking Sabotages Audio Deepfake Detection
Abstract
Provenance watermarking is increasingly treated as a safeguard for synthetic speech, whether built directly into speech-generation models such as Chatterbox, provided through dedicated techniques such as AudioSeal, or deployed by commercial platforms such as ElevenLabs. We identify a previously uncharacterized liability: when synthetic speech is watermarked and human speech is not, detectors trained alongside latch onto the watermark as a spurious "watermark => fake" shortcut. This single feature yields three coupled failures: generalization degradation (model performance deteriorates on unseen data), strip-to-evade (a watermarked fake escapes once unwatermarked), and mark-to-frame (watermarking a real voice flags it as fake). In a controlled white-box experiment, a watermark-trained detector shows all three (for example, mark-to-frame lifts Equal Error Rate from 16% to 75%). In a black-box test of a commercial API, we show that adding a watermark to real speech disguises it as fake. However, this shortcut is fixable: retraining with the watermark on both classes decorrelates it and restores clean behavior. We release experiment data as a paired clean-versus-watermarked corpus (WASP).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies a critical spurious correlation in audio deepfake detectors where provenance watermarks on synthetic speech lead to severe generalization failures and evasion/framing attacks, demonstrating that training detectors on watermarked spoofs causes them to latch onto the watermark as a shortcut rather than genuine synthesis artifacts.
Methodology
The paper employs a rigorous causal analysis methodology to identify a spurious correlation (shortcut) in audio deepfake detection systems. By comparing two detectors trained on identical architectures (AASIST) but differing only in the presence of provenance watermarks on the spoof class during training, the authors isolate the "watermark => fake" shortcut. The methodology is clean and effective, utilizing both white-box experiments on standard benchmarks (ASVspoof19) and black-box queries of a commercial API. The mitigation strategy—watermark augmentation (applying watermarks to both classes)—is a simple but theoretically sound application of decorrelation techniques to remove spurious features.
Results & evaluation
The experimental design is robust. The white-box experiments clearly demonstrate the three failure modes: generalization degradation, strip-to-evade, and mark-to-frame. The results are statistically significant, with EERs shifting dramatically (e.g., from 0.7% to 74.6% in mark-to-frame scenarios). The black-box experiment adds significant real-world validity, showing that this is not just a lab artifact but a live issue in commercial systems (specifically identifying PerTh watermark exposure in the vendor's training data). The release of the WASP corpus provides a valuable resource for future research, allowing for paired clean-versus-watermarked evaluation.
Reproducibility
High. The authors provide a clear description of the experimental setup, including the specific detectors (AASIST), datasets (ASVspoof19, ASVspoof21-LA, In-the-Wild), and watermarking schemes (PerTh, WavMark, AudioSeal, etc.). Crucially, they release the WASP dataset, which enables exact replication of the black-box findings and further analysis of watermark-detector interactions. The code for the augmentation strategy is implied to be straightforward, and the dataset release ensures transparency.
Limitations
The study focuses primarily on text-to-speech (TTS) systems and specific watermarking schemes (PerTh, AudioSeal, etc.). It does not extensively cover other forms of audio deepfakes (e.g., voice conversion, speech editing) which may have different artifact profiles. Additionally, while the black-box experiment is compelling, it relies on a single commercial API, limiting the generalizability of the specific vendor findings, though the white-box results support the broader claim. The paper also notes that strip-to-evade did not transfer in the black-box test, suggesting that while the shortcut is strong, it may not be the *only* signal the detector uses, which is a nuance worth exploring further.
Broader impact
This paper has significant implications for the deployment of provenance watermarking and deepfake detection systems. It highlights a critical conflict between the industry trend of default watermarking and the need for robust detection. The findings suggest that current evaluation protocols for detectors are flawed if they do account for the training data's watermarking status. The proposed mitigation (augmentation) offers a practical path forward for developers. The work encourages a more holistic approach to security, where provenance and detection are co-designed rather than developed in isolation. The paper identifies a critical spurious correlation in audio deepfake detectors where provenance watermarks on synthetic speech lead to severe generalization failures and evasion/framing attacks, demonstrating that training detectors on watermarked spoofs causes them to latch onto the watermark as a shortcut rather than genuine synthesis artifacts.
Author of this paper? Send a correction.