Eroding Trust in Real Speech: A Large-Scale Study of Human Audio Deepfake Perception
Abstract
Audio deepfakes have improved rapidly recently, yet their effect on human trust in real speech remains unstudied. We present the largest listening study on audio deepfake perception to date, collecting 35,532 judgments from 1,768 participants across 138 text-to-speech and voice conversion systems. Our central finding is a skepticism shift: compared to a 2021 baseline, human accuracy on fake samples barely changed (72.9% to 71.2%), but accuracy on real samples dropped from 72.7% to 64.1%. Participants are not worse at detecting synthesis artifacts; rather, they increasingly distrust authentic speech. Samples generated by commercial and autoregressive language model systems proved hardest to detect (61.3 - 65.9%), while those from traditional seq2seq and flow-matching models remain easier to spot (75.4 - 76.8%). An ML detector that served as a reference point maintained over 94.5% accuracy across all conditions. Our results suggest that the primary threat posed by modern deepfakes may not be mere deception, but the erosion of trust in genuine audio.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents the largest listening study on audio deepfake perception, revealing a critical shift in human trust towards authentic audio. The comprehensive methodology and significant findings contribute meaningfully to the understanding of deepfake technology's societal implications, marking a notable advancement in the field.
Methodology
The methodology is robust, leveraging a large-scale listening study with 35,532 judgments from 1,768 participants. The use of a web-based platform for data collection is innovative, allowing for a diverse participant pool without the need for registration. The active-learning sampling scheme enhances the study's design by focusing on difficult-to-detect deepfakes, although it introduces some bias in sample counts. The demographic analysis adds depth to the findings, though the self-reported nature of the data could introduce variability.
Results & evaluation
The experiments are well-structured, comparing human detection capabilities against a high-performing ML detector. The results indicate a significant societal concern regarding the erosion of trust in real audio, which is a novel finding. The comparison with a 2021 baseline provides a clear context for the results, and the detailed breakdown of accuracy by TTS architecture family is insightful.
Reproducibility
The authors have made efforts to ensure reproducibility by releasing the anonymized dataset and analysis code. The web platform is publicly accessible, allowing others to replicate the study design. However, the reliance on self-reported demographics may limit the reproducibility of participant characteristics.
Limitations
The study's limitations include a potential bias in participant demographics due to self-selection, as younger users may be overrepresented. The audio quality may vary based on participants' playback equipment, and the study is limited to English audio samples. Additionally, the inability to control for participants who may have taken part in both the 2021 and 2026 studies could affect the results.
Broader impact
The findings have significant implications for media literacy and public trust in audio communications. The erosion of trust in genuine audio could impact journalism, legal proceedings, and personal interactions. The study highlights the need for awareness campaigns that address both deepfake detection and the psychological effects of audio skepticism. This paper presents the largest listening study on audio deepfake perception, revealing a critical shift in human trust towards authentic audio. The comprehensive methodology and significant findings contribute meaningfully to the understanding of deepfake technology's societal implications, marking a notable advancement in the field.
Author of this paper? Send a correction.