StreamVoiceAnon+: Emotion-Preserving Streaming Speaker Anonymization via Frame-Level Acoustic Distillation
Abstract
We address the challenge of preserving emotional content in streaming speaker anonymization (SA). Neural audio codec language models trained for audio continuation tend to degrade source emotion: content tokens discard emotional information, and the model defaults to dominant acoustic patterns rather than preserving paralinguistic attributes. We propose supervised finetuning with neutral-emotion utterance pairs from the same speaker, combined with frame-level emotion distillation on acoustic token hidden states. All modifications are confined to finetuning, which takes less than 2 hours on 4 GPUs and adds zero inference latency overhead, while maintaining a competitive 180ms streaming latency. On the VoicePrivacy 2024 protocol, our approach achieves a 49.2% UAR (emotion preservation) with 5.77% WER (intelligibility), a +24% relative UAR improvement over the baseline (39.7%->49.2%) and +10% over the emotion-prompt variant (44.6% UAR), while maintaining strong privacy (EER 49.0%). Demo and code are available: https://anonymous3842031239.github.io/
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a novel supervised finetuning approach combined with frame-level emotion distillation for emotion-preserving streaming speaker anonymization, which significantly improves emotion retention while maintaining privacy and intelligibility. The technical contributions and rigorous methodology present a meaningful advancement in the field of audio processing and speaker anonymization.
Methodology
The methodology presented in this paper is innovative, focusing on supervised finetuning with neutral-emotion utterance pairs and frame-level emotion distillation. This dual approach effectively addresses the limitations of existing neural audio codec models in preserving emotional content during speaker anonymization. The use of neutral-emotion pairs ensures that the model learns to generate emotional outputs without relying on emotional prompts, which can be difficult to obtain. The design choice to apply emotion distillation to the acoustic branch rather than the semantic branch is a significant improvement that allows for cleaner gradient flow and better emotion preservation.
Results & evaluation
The experiments are well-structured, adhering to the VoicePrivacy 2024 protocol, which allows for direct comparison with prior works. The results show a substantial improvement in emotion preservation (UAR) and privacy (EER) while maintaining competitive intelligibility (WER). The ablation studies provide clear evidence of the contributions of each component of the proposed method, reinforcing the claims made in the paper. The dataset used for training and evaluation is appropriate, although the reliance on acted speech corpora may limit generalizability.
Reproducibility
The paper provides sufficient details regarding the implementation, including the training setup, data preprocessing, and evaluation metrics. However, the absence of a public code repository limits reproducibility. The authors mention that the demo is available, which is a positive aspect, but a comprehensive project URL would enhance reproducibility further.
Limitations
The paper acknowledges several limitations, including the reliance on a single SER evaluator, the lack of subjective listening tests, and the evaluation being restricted to acted speech corpora. These factors could affect the generalizability and real-world applicability of the findings. Additionally, the gap in performance compared to offline methods suggests that further improvements are needed for practical deployment.
Broader impact
The proposed method has significant implications for privacy-preserving applications in various domains, including teleconferencing, call centers, and online mental health counseling. By effectively anonymizing speaker identity while preserving emotional content, this research addresses a critical need for maintaining communication effectiveness in sensitive contexts. The approach could pave the way for more sophisticated anonymization techniques that balance privacy and emotional expressiveness. The main contribution of this paper is the introduction of a novel supervised finetuning approach combined with frame-level emotion distillation for emotion-preserving streaming speaker anonymization, which significantly improves emotion retention while maintaining privacy and intelligibility. The technical contributions and rigorous methodology present a meaningful advancement in the field of audio processing and speaker anonymization.
Author of this paper? Send a correction.