Robust CAPTCHA Using Audio Illusions in the Era of Large Language Models: from Evaluation to Advances
Abstract
CAPTCHAs are widely used by websites to block bots and spam by presenting challenges that are easy for humans but difficult for automated programs to solve. To improve accessibility, audio CAPTCHAs are designed to complement visual ones. However, the robustness of audio CAPTCHAs against advanced Large Audio Language Models (LALMs) and Automatic Speech Recognition (ASR) models remains unclear. In this paper, we introduce AI-CAPTCHA, a unified framework that offers (i) an evaluation framework, ACEval, which includes advanced LALM- and ASR-based solvers, and (ii) a novel audio CAPTCHA approach, IllusionAudio, leveraging audio illusions. Through extensive evaluations of seven widely deployed audio CAPTCHAs, we show that most existing methods can be solved with high success rates by advanced LALMs and ASR models, exposing critical security weaknesses. To address these vulnerabilities, we design a new audio CAPTCHA approach, IllusionAudio, which exploits perceptual illusion cues rooted in human auditory mechanisms. Extensive experiments demonstrate that our method defeats all tested LALM- and ASR-based attacks while achieving a 100% human pass rate, significantly outperforming existing audio CAPTCHA methods.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a novel audio CAPTCHA system that exploits auditory illusions to enhance security against advanced AI models. The research provides a timely and relevant exploration of CAPTCHA vulnerabilities and presents a promising solution, though it could benefit from improved reproducibility and a more thorough examination of its limitations.
Methodology
The proposed methodology introduces a novel CAPTCHA system, IllusionAudio, which leverages auditory illusions to enhance security against LALMs and ASR models. The authors provide a comprehensive evaluation framework, ACEval, which is a significant contribution as it systematically assesses the vulnerabilities of existing audio CAPTCHAs. The methodology is well-structured, focusing on human auditory perception, which is a unique angle in CAPTCHA design. However, the paper lacks detailed descriptions of the specific audio illusions used and their implementation, which could enhance reproducibility and understanding of the approach.
Results & evaluation
The experimental evaluation is robust, involving extensive tests against seven widely deployed audio CAPTCHAs. The results show that existing methods are vulnerable to LALMs and ASR models, while the proposed IllusionAudio method successfully defends against these attacks with a 100% human pass rate. The experiments are well-documented, but the paper could benefit from a more detailed explanation of the experimental setup, including participant demographics and the exact metrics used for evaluation.
Reproducibility
The paper does not provide sufficient implementation details or code repositories, which raises concerns about reproducibility. While the methodology is innovative, the lack of a demo URL or project repository limits the ability of other researchers to replicate the findings. Clearer documentation of the experimental setup and access to the code would greatly enhance reproducibility.
Limitations
The paper acknowledges the limitations of the existing audio CAPTCHA systems but does not sufficiently address potential weaknesses in the proposed IllusionAudio method. For instance, the long-term effectiveness against evolving LALMs and ASR technologies is not discussed, nor are potential biases in participant responses or accessibility issues for users with hearing impairments.
Broader impact
The implications of this research are significant, as it addresses a critical security concern in online systems that rely on CAPTCHAs. The proposed method could enhance the robustness of audio CAPTCHAs, making them more secure against automated attacks while remaining accessible to users. This advancement could lead to broader applications in web security and user authentication processes. The main contribution of this paper is the introduction of a novel audio CAPTCHA system that exploits auditory illusions to enhance security against advanced AI models. The research provides a timely and relevant exploration of CAPTCHA vulnerabilities and presents a promising solution, though it could benefit from improved reproducibility and a more thorough examination of its limitations.
Author of this paper? Send a correction.