SraVaani 1.0: Scaling Inclusive Speech Recognition for Indic Languages
Abstract
India's linguistic landscape spans over 700 languages and thousands of dialects, yet the vast majority of automatic speech recognition (ASR) systems support only a small fraction of this diversity. We present SraVaani-1.0, a multilingual ASR model covering 65 Indian languages and dialects, many of which currently have no publicly available or competing ASR system. SraVaani-1.0 is built on a FastConformer architecture and trained from scratch through a three-stage pipeline.In the first stage, we perform self-supervised pretraining on 31,255 hours of unlabelled speech from the VAANI corpus using a contrastive learning objective. In the second stage, we introduce an audio-image representation alignment stage that leverages the paired images and speech available in the VAANI corpus. This multimodal alignment encourages the speech encoder to learn semantically richer representations by exploiting the relationship between visual context and spoken content, thereby improving downstream recognition, particularly for low-resource languages.In the final stage, the aligned encoder is fine-tuned end-to-end using a Hybrid Token-and-Duration Transducer (TDT)-CTC decoder on 31,263 hours of labelled multilingual Indian speech compiled from 24 public datasets spanning 65 languages and dialects. We evaluate SraVaani-1.0 against three state-of-the-art multilingual ASR systems across eight benchmarks. SraVaani-1.0 achieves the lowest word error rate (WER) on a large number of language-dataset pairs while remaining competitive with the best-performing systems on high-resource languages.Most importantly, it is the only open-source evaluated model that provides transcription capability for multiple low-resource and tribal Indian languages, which are assessed exclusively on the VAANI benchmark.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SraVaani-1.0 is a significant contribution to the field of multilingual speech recognition, particularly for low-resource Indian languages, by demonstrating the effectiveness of a three-stage pipeline combining self-supervised learning, audio-image alignment, and supervised fine-tuning, and by providing the first open-source ASR system with broad coverage of India's linguistic diversity.
Methodology
The paper proposes a three-stage training pipeline for a multilingual ASR system. Stage 1 involves self-supervised contrastive pretraining on a large corpus of unlabelled Indian speech. Stage 2 introduces a novel audio-image representation alignment stage, leveraging paired audio and images from the VAANI corpus to inject semantic context into the speech encoder using a SigLIP-style contrastive loss. Stage 3 is supervised fine-tuning using a Hybrid Token-and-Duration Transducer (TDT)-CTC decoder. The methodology is technically sound and follows established practices in self-supervised learning (wav2vec 2.0 style) and multimodal alignment (SigLIP). The specific novelty lies in the application of audio-image alignment to improve low-resource ASR performance, which is a creative use of available multimodal data. However, the core components (FastConformer, TDT, SigLIP) are not new; the contribution is primarily in the system design and data curation for a specific linguistic domain.
Results & evaluation
The evaluation is comprehensive, covering 65 languages and comparing against strong baselines (Gemini 3 Flash, Sarvam Saaras v3, IndicConformer). The use of multiple benchmarks (CommonVoice, FLEURS, etc.) provides a robust assessment of generalization. The results demonstrate that SraVaani-1.0 achieves state-of-the-art performance on many languages, particularly those with limited prior support. The inclusion of 44 languages with no competing ASR systems is a significant empirical contribution. The analysis of the coverage-accuracy trade-off is insightful. However, the WERs on some low-resource languages are quite high (e.g., Nyishi 93.9%), indicating that while the model is the best available, it is not yet production-ready for all target languages. The evaluation is limited to in-domain data (VAANI), which the authors acknowledge as a limitation.
Reproducibility
The paper provides detailed implementation details, including model architecture, hyperparameters, training schedules, and data preprocessing steps. The use of standard frameworks (NVIDIA NeMo, PyTorch Lightning) and public datasets enhances reproducibility. The model weights are made available on Hugging Face, which is a strong positive for reproducibility and community impact. The codebase is not explicitly mentioned as open-source, but the model weights are.
Limitations
The authors openly discuss several limitations: lack of support for Urdu and Kashmiri in fine-tuning, issues with code-switched speech transcription, lack of text normalization, unreliable language identification, and significant variance in data availability across languages. The evaluation is also limited to in-domain data, which may not reflect real-world performance on out-of-domain speech. The high WERs on very low-resource languages highlight the current limits of the approach.
Broader impact
This work has significant potential impact on digital equity in India by providing ASR capabilities for a vast number of underrepresented languages and dialects. This can facilitate access to government services, education, and healthcare for rural and tribal communities. The open-source release of the model promotes further research and development in low-resource NLP and speech technologies. The approach of leveraging multimodal data for ASR improvement could be applicable to other domains and languages. SraVaani-1.0 is a significant contribution to the field of multilingual speech recognition, particularly for low-resource Indian languages, by demonstrating the effectiveness of a three-stage pipeline combining self-supervised learning, audio-image alignment, and supervised fine-tuning, and by providing the first open-source ASR system with broad coverage of India's linguistic diversity.
Author of this paper? Send a correction.