Goodbye Equal Error Rate, Hello Local Information Disclosure: Evaluating Voice Anonymisation against 1-to-N Linkage Threats
Abstract
Voice anonymisation aims to protect speaker identity. Currently, its empirical privacy evaluation heavily relies on the Equal Error Rate (EER). Originally designed for biometric verification, EER aggregates scores globally, implicitly assuming an attacker is only trying to verify if two specific voice samples match (a 1-to-1 comparison). This introduces a threat model mismatch with real-world database linkage attacks, where an attacker searches across a fixed set of N enrolled identities (a 1-to-N closed-set search), allowing global averages to obscure localised privacy failures. While recent 1-to-N metrics address this aggregation issue, they abstract away the magnitude of the biometric evidence. In this paper, we propose a modular, information-theoretic evaluation framework explicitly designed for the 1-to-N linkage threat model. Within this framework, our core metric, Local Information Disclosure (LID), quantifies the exact privacy loss of a single trial utterance in bits by calibrating its raw similarity scores into the attacker's posterior confidence for each enrolled identity. Evaluating top-performing systems from the VoicePrivacy 2024 Challenge reveals that systems exhibiting near-perfect EERs (48 %) can still suffer from localised vulnerabilities with worst-case disclosures reaching 1 bit per trial utterance (effectively doubling the attacker's success rate over a random guess). We demonstrate that adopting localised privacy metrics is essential for capturing worst-case risks and aligning with strict privacy regulations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
Methodology
The paper proposes a rigorous, information-theoretic evaluation framework for voice anonymisation, specifically targeting the mismatch between standard 1-to-1 verification metrics (like EER) and real-world 1-to-N linkage threats. The core contribution is the Local Information Disclosure (LID) metric, which quantifies privacy loss in bits by calibrating raw similarity scores into posterior probabilities using logistic regression on row-normalized scores. The methodology is mathematically sound, leveraging Pointwise Mutual Information (PMI) to measure the reduction in attacker uncertainty. The framework is modular, allowing for various global aggregations (ALID, PDR, NDR, LID_max) that provide a more granular view of system vulnerabilities than global averages. The approach correctly identifies that global metrics can obscure local failures where specific utterances are highly linkable despite good average performance.
Results & evaluation
The authors evaluate their framework against systems from the VoicePrivacy 2024 Challenge (VPC 2024), including baselines and top submissions. They use the standard LibriSpeech dev/eval splits. The experiments demonstrate a critical finding: systems with near-perfect EERs (e.g., 0.48, indicating high privacy in 1-to-1 terms) can still exhibit worst-case local disclosures of ~1 bit, effectively doubling the attacker's success rate over random chance for specific trials. The results are presented clearly through CCDFs and tables comparing EER, C_llr, and the new LID-based metrics. The analysis of the calibration weight $w$ provides additional insight into the discriminative power of the anonymised embeddings. The empirical evidence strongly supports the claim that 1-to-1 metrics are insufficient for assessing 1-to-N privacy risks.
Reproducibility
The paper provides detailed mathematical formulations for the LID metric, including the row-wise z-normalization and logistic calibration steps. It specifies the datasets (LibriSpeech) and the attacker model (semi-informed ECAPA-TDNN). However, the specific implementation details of the calibration (e.g., exact hyperparameters for the logistic regression, handling of edge cases) are somewhat abstract. The code is not explicitly linked in the provided text, though the VPC 2024 data is public. Reproducibility is moderate; a researcher could implement the method, but might need to infer some implementation details.
Limitations
The primary limitation is the reliance on logistic regression for score calibration, which assumes a specific distribution of scores (normality after z-normalization). The authors acknowledge this and suggest it may not hold for all anonymisation methods. Additionally, the framework is evaluated only on the semi-informed threat model; while this is the most realistic, the framework's behavior under ignorant or lazy-informed models is not explored. The evaluation is limited to voice data; while the framework is generalizable, its applicability to other modalities (e.g., face, fingerprint) is not demonstrated. The "random" baseline comparison is useful but simplistic; a more nuanced baseline might involve a dummy classifier with similar score distributions but no identity information.
Broader impact
This work has significant implications for the deployment of voice anonymisation systems in privacy-sensitive applications (e.g., healthcare, legal, customer service). By providing a metric that aligns with regulatory concerns about "singling out" and "linkability" (e.g., GDPR), it offers a more robust tool for auditors and developers. It shifts the focus from average performance to worst-case risks, which is crucial for compliance. The information-theoretic framing also contributes to the broader field of privacy-preserving machine learning by offering a concrete way to quantify information leakage in biometric systems. The paper introduces a novel information-theoretic metric, Local Information Disclosure (LID), to evaluate voice anonymisation systems under 1-to-N linkage threats, revealing that systems with good global privacy metrics (EER) can still suffer from severe local vulnerabilities, thereby providing a more accurate and regulation-aligned assessment of privacy risks.
Author of this paper? Send a correction.