DiffAnon: Diffusion-based Prosody Control for Voice Anonymization
Abstract
To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DiffAnon, a diffusion-based voice anonymization framework that enables explicit and continuous control over prosody preservation, significantly advancing the field of privacy-preserving speech technologies. This work represents a meaningful step forward in balancing the utility-privacy trade-off in voice applications, showcasing the potential for structured prosody control in enhancing both privacy and expressiveness in anonymized speech.
Methodology
The proposed methodology, DiffAnon, leverages a novel diffusion-based framework with classifier-free guidance to provide continuous control over prosody preservation in voice anonymization. This approach is innovative as it allows for the modulation of the utility-privacy trade-off in a structured manner, which is a significant advancement over existing methods that operate at fixed points. The integration of semantic embeddings from an RVQ codec with a diffusion model is particularly noteworthy, as it combines strengths from both domains to enhance the quality of anonymized speech.
Results & evaluation
The experiments are robust, utilizing the VoicePrivacy Challenge 2024 protocol, which provides a standardized framework for evaluating privacy and utility. The results demonstrate that DiffAnon achieves competitive performance across various metrics, including EER for privacy and WER for content preservation, while also showing a clear trade-off between privacy and prosodic fidelity. The systematic evaluation across different prosody guidance weights adds depth to the findings.
Reproducibility
The authors have made their code and pretrained models publicly available, which is a strong point for reproducibility. The detailed training and inference setup, including hyperparameters and datasets used, further supports replicability of the results.
Limitations
While the paper presents a significant advancement, it does not explore the potential impact of varying speaker characteristics on the performance of the model. Additionally, the reliance on specific datasets may limit the generalizability of the findings to other languages or dialects. The paper also does not address the computational costs associated with training and deploying the model in real-world applications.
Broader impact
The ability to anonymize voice while preserving prosody has significant implications for privacy in various applications, including telecommunication, virtual assistants, and voice-based interactions. This work could enhance user trust in voice technologies by providing a means to protect identity while maintaining communicative effectiveness. The structured control over prosody could also lead to advancements in emotional speech synthesis and human-computer interaction. The main contribution of this paper is the introduction of DiffAnon, a diffusion-based voice anonymization framework that enables explicit and continuous control over prosody preservation, significantly advancing the field of privacy-preserving speech technologies. This work represents a meaningful step forward in balancing the utility-privacy trade-off in voice applications, showcasing the potential for structured prosody control in enhancing both privacy and expressiveness in anonymized speech.
Author of this paper? Send a correction.