Stream-Voice-Anon: Enhancing Utility of Real-Time Speaker Anonymization via Neural Audio Codec and Language Models
Abstract
Protecting speaker identity is crucial for online voice applications, yet streaming speaker anonymization (SA) remains underexplored. Recent research has demonstrated that neural audio codec (NAC) provides superior speaker feature disentanglement and linguistic fidelity. NAC can also be used with causal language models (LM) to enhance linguistic fidelity and prompt control for streaming tasks. However, existing NAC-based online LM systems are designed for voice conversion (VC) rather than anonymization, lacking the techniques required for privacy protection. Building on these advances, we present Stream-Voice-Anon, which adapts modern causal LM-based NAC architectures specifically for streaming SA by integrating anonymization techniques. Our anonymization approach incorporates pseudo-speaker representation sampling, a speaker embedding mixing and diverse prompt selection strategies for LM conditioning that leverage the disentanglement properties of quantized content codes to prevent speaker information leakage. Additionally, we compare dynamic and fixed delay configurations to explore latency-privacy trade-offs in real-time scenarios. Under the VoicePrivacy 2024 Challenge protocol, Stream-Voice-Anon achieves substantial improvements in intelligibility (up to 46% relative WER reduction) and emotion preservation (up to 28% UAR relative) compared to the previous state-of-the-art streaming method DarkStream while maintaining comparable latency (180ms vs 200ms) and privacy protection against lazy-informed attackers, though showing 15% relative degradation against semi-informed attackers.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of Stream-Voice-Anon, a novel real-time speaker anonymization system that effectively combines neural audio codecs and language models to enhance privacy while preserving speech intelligibility and emotional content. This work represents a significant advancement in the field of audio processing and privacy-preserving technologies, addressing a critical need in online voice applications.
Methodology
The proposed methodology of Stream-Voice-Anon is innovative in its integration of neural audio codecs with causal language models for real-time speaker anonymization. The use of pseudo-speaker representation sampling and diverse prompt selection strategies demonstrates a thoughtful approach to enhancing privacy while maintaining intelligibility and emotional fidelity. The dual-decoder architecture and the dynamic delay mechanism for latency management are particularly noteworthy, showcasing a sophisticated understanding of the trade-offs involved in real-time processing.
Results & evaluation
The experimental setup is robust, utilizing the VoicePrivacy 2024 Challenge protocol for evaluation, which lends credibility to the results. The paper reports significant improvements over the state-of-the-art method DarkStream, particularly in intelligibility and emotion preservation, while maintaining comparable latency. The comprehensive evaluation against both lazy-informed and semi-informed attackers provides a nuanced understanding of the system's privacy capabilities.
Reproducibility
The paper provides detailed implementation specifics, including model configurations, training details, and datasets used, which enhances reproducibility. However, the lack of a publicly available code repository limits the ease with which other researchers can replicate the work.
Limitations
One identified limitation is the relative degradation in performance against semi-informed attackers, indicating that while the system is effective, there is room for improvement in robustness against more sophisticated adversaries. Additionally, the reliance on specific datasets for training may limit the generalizability of the model to other domains or languages.
Broader impact
The implications of this work are significant, particularly in applications requiring speaker privacy, such as call centers, legal recordings, and medical conversations. The advancements in real-time speaker anonymization could lead to broader adoption of voice technologies in sensitive contexts, enhancing user trust and compliance with privacy regulations. The main contribution of this paper is the development of Stream-Voice-Anon, a novel real-time speaker anonymization system that effectively combines neural audio codecs and language models to enhance privacy while preserving speech intelligibility and emotional content. This work represents a significant advancement in the field of audio processing and privacy-preserving technologies, addressing a critical need in online voice applications.
Author of this paper? Send a correction.