Tokens Change, Structure Endures: Spectral Watermarking for Generated Speech
Abstract
Watermarking is a promising tool for establishing the provenance of AI-generated speech. While many neural audio watermarking methods rely on a separately trained watermark generator, token-level watermarking is a training-free alternative that operates directly during generation. Its main weakness is retokenization: decoding generated speech to a waveform and encoding it again can change token identities and erode the watermark. To make the watermark robust to these changes, we propose Redwing, REtokenization-Durable Watermarking IN Generation. It builds a graph from the token substitutions observed under retokenization, whose Laplacian yields a basis that assigns similar values to tokens likely to substitute for one another. Over this basis, embedding and detection functions are jointly optimized to preserve watermark signal through retokenization while limiting embedding distortion and detector variability on unwatermarked speech. On the Moshi full-duplex system, after eight consecutive passes of Mimi resynthesis, Redwing achieves 80.7% TPR at a calibrated 1% FPR, compared with 8.3% for KGW and at most 7.3% for WMAR. It also has the highest TPR after eight passes through three other neural codecs (77.5-93.0%), and the gains generalize to TTS models at a speech-quality cost close to that of KGW. These results show that retokenization is not merely a source of noise: its transition structure can be exploited as a design principle for robust token-level watermarking.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Redwing, a spectral graph-based token watermarking method that exploits the structure of codec retokenization to achieve high robustness against repeated resynthesis attacks, significantly outperforming existing token-level and post-hoc watermarking techniques in speech generation systems.
Methodology
The paper proposes "Redwing," a token-level watermarking scheme for speech generation that leverages the structural properties of neural codec retokenization. Unlike standard token watermarks (like KGW) that rely on specific token identities, which are often altered during the decode-encode cycle, Redwing constructs a graph where nodes are tokens and edges represent substitution frequencies observed during retokenization. By using the eigenvectors of the graph Laplacian (specifically those with small eigenvalues) as a basis, the method ensures that tokens likely to substitute for one another have similar watermark values. The embedding and detection functions are jointly optimized over this basis to maximize robustness against resynthesis while minimizing distortion to the generated speech. This is a clever application of spectral graph theory to a practical audio security problem, moving beyond treating retokenization as noise to exploiting its deterministic structure.
Results & evaluation
The experiments are rigorous, testing the watermark on the Moshi full-duplex dialogue system and two TTS models (CosyVoice3, MOSS-TTS). The primary attack simulated is repeated resynthesis through the Mimi codec (8 passes). Redwing achieves an 80.7% True Positive Rate (TPR) at 1% False Positive Rate (FPR), significantly outperforming baselines like KGW (8.3%) and WMAR (7.3%). The paper also demonstrates generalizability to other neural codecs (77.5-93.0% TPR) and maintains speech quality comparable to the KGW baseline. The inclusion of multiple models and codecs strengthens the claim of general applicability.
Reproducibility
The method is described as "training-free" in the sense that it does not require retraining the speech model, but it does require building the substitution graph from a corpus and solving for embedding/detection functions. The paper states it applies to released models unchanged, which is a strong reproducibility feature. However, specific details on the optimization process for the embedding/detection functions and the exact corpus used for graph construction are not fully detailed in the truncated text, though the methodology is clearly defined.
Limitations
The method relies on the stability of the retokenization substitution patterns. If a codec is updated or if the attack involves a different type of processing (e.g., heavy noise addition, pitch shifting) that disrupts the token substitution graph, the watermark's robustness may degrade. Additionally, the construction of the graph requires access to the codec's encoder/decoder to perform the substitution counting, which might not be feasible for all black-box systems.
Broader impact
This work has significant implications for the provenance of AI-generated speech. As speech models become more capable and widespread, robust watermarking is essential for distinguishing synthetic from human speech. By addressing the specific weakness of token-level watermarks in audio (retokenization), this paper provides a viable path for deploying watermarks in real-world speech generation systems without modifying the underlying model architecture. The paper introduces Redwing, a spectral graph-based token watermarking method that exploits the structure of codec retokenization to achieve high robustness against repeated resynthesis attacks, significantly outperforming existing token-level and post-hoc watermarking techniques in speech generation systems.
Author of this paper? Send a correction.