IsoSignVid2Aud: Sign Language Video to Audio Conversion without Text Intermediaries
Abstract
Sign language to spoken language audio translation is important to connect the hearing- and speech-challenged humans with others. We consider sign language videos with isolated sign sequences rather than continuous grammatical signing. Such videos are useful in educational applications and sign prompt interfaces. Towards this, we propose IsoSignVid2Aud, a novel end-to-end framework that translates sign language videos with a sequence of possibly non-grammatic continuous signs to speech without requiring intermediate text representation, providing immediate communication benefits while avoiding the latency and cascading errors inherent in multi-stage translation systems. Our approach combines an I3D-based feature extraction module with a specialized feature transformation network and an audio generation pipeline, utilizing a novel Non-Maximal Suppression (NMS) algorithm for the temporal detection of signs in non-grammatic continuous sequences. Experimental results demonstrate competitive performance on ASL-Citizen-1500 and WLASL-100 datasets with Top-1 accuracies of 72.01\% and 78.67\%, respectively, and audio quality metrics (PESQ: 2.67, STOI: 0.73) indicating intelligible speech output. Code is available at: https://github.com/BheeshmSharma/IsoSignVid2Aud_AIMLsystems-2025.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of IsoSignVid2Aud, a novel framework for direct sign language video-to-audio conversion that eliminates the need for intermediate text representation, thereby enhancing real-time communication capabilities. This work represents a meaningful advancement in the field of machine learning and sign language processing, with significant implications for accessibility and communication technologies.
Methodology
The proposed IsoSignVid2Aud framework employs a novel end-to-end architecture that directly translates sign language videos to audio without intermediate text, which is a significant departure from traditional multi-stage systems. The integration of an I3D-based feature extraction module, a specialized feature transformation network, and an audio generation pipeline showcases a comprehensive approach to addressing the challenges of sign language translation. The introduction of a Non-Maximal Suppression algorithm for temporal detection further enhances the model's ability to isolate signs effectively, which is critical for real-time applications.
Results & evaluation
The experimental results demonstrate competitive performance on two well-established datasets (ASL-Citizen-1500 and WLASL-100), achieving Top-1 accuracies of 72.01% and 78.67%, respectively. The audio quality metrics (PESQ: 2.68, STOI: 0.73) indicate intelligible speech output, which is a crucial aspect of the model's utility. However, while the results are promising, they also highlight areas for improvement, particularly in recognizing complex signs.
Reproducibility
The paper provides detailed implementation specifics, including architecture choices, training protocols, and evaluation metrics, which facilitate reproducibility. The availability of code on GitHub further supports this aspect, allowing other researchers to replicate and build upon the work.
Limitations
The model's accuracy on the WLASL dataset remains below some state-of-the-art levels, indicating potential room for improvement in recognizing complex signs. Additionally, the audio quality metrics suggest that while intelligibility is achieved, there are still artifacts and limited frequency expressiveness that need to be addressed in future iterations.
Broader impact
The IsoSignVid2Aud framework has the potential to significantly improve communication accessibility for deaf and hard-of-hearing individuals by providing a direct means of translating sign language into spoken language. Its modular design allows for adaptation to various sign languages and applications, making it a versatile tool for enhancing interaction in diverse contexts. The main contribution of this paper is the introduction of IsoSignVid2Aud, a novel framework for direct sign language video-to-audio conversion that eliminates the need for intermediate text representation, thereby enhancing real-time communication capabilities. This work represents a meaningful advancement in the field of machine learning and sign language processing, with significant implications for accessibility and communication technologies.
Author of this paper? Send a correction.