Canary-1B-v2 & Parakeet-TDT-0.6B-v3: Efficient and High-Performance Models for Multilingual ASR and AST

Monica Sekoyan, Nithin Rao Koluguri, Nune Tadevosyan ...Full author list in the original paper.

These authors contributed equally. Listed in alphabetical order based on First Name · arXiv

Affiliations: These authors contributed equally. Listed in alphabetical order based on First Name

Abstract

This report introduces Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST). Built with a FastConformer encoder and Transformer decoder, it supports 25 languages primarily European. The model was trained on 1.7M hours of total data samples, including Granary and NeMo ASR Set 3.0, with non-speech audio added to reduce hallucinations for ASR and AST. We describe its two-stage pre-training and fine-tuning process with dynamic data balancing, as well as experiments with an nGPT encoder. Results show nGPT scales well with massive data, while FastConformer excels after fine-tuning. For timestamps, Canary-1B-v2 uses the NeMo Forced Aligner (NFA) with an auxiliary CTC model, providing reliable segment-level timestamps for ASR and AST. Evaluations show Canary-1B-v2 outperforms Whisper-large-v3 on English ASR while being 10x faster, and delivers competitive multilingual ASR and AST performance against larger models like Seamless-M4T-v2-large and LLM-based systems. We also release Parakeet-TDT-0.6B-v3, a successor to v2, offering multilingual ASR across the same 25 languages with just 600M parameters.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The primary contribution of this paper is the introduction of two efficient multilingual models for ASR and AST, showcasing innovative methodologies and competitive performance against larger models. The comprehensive exploration of architecture and training strategies positions this work as a significant advancement in the field of speech processing.

Methodology

The paper presents a well-structured methodology for developing the Canary-1B-v2 and Parakeet-TDT-0.6B-v3 models, including a two-stage pre-training and fine-tuning process with dynamic data balancing. The use of a FastConformer encoder combined with a Transformer decoder is a thoughtful architectural choice that enhances performance in multilingual ASR and AST tasks. The incorporation of non-speech audio to mitigate hallucinations is a novel approach that reflects a deep understanding of the challenges in ASR. The comparative study of encoder architectures, particularly the application of nGPT, adds a unique dimension to the methodology, showcasing the trade-offs between different model architectures.

Results & evaluation

The experiments conducted are robust, leveraging a substantial dataset of 1.7M hours, which is commendable. The results indicate that Canary-1B-v2 outperforms existing models like Whisper-large-v3 in English ASR while maintaining a significant speed advantage. The evaluation against larger models like Seamless-M4T-v2-large demonstrates the competitive nature of the proposed models. However, more detailed metrics and comparisons across all 25 languages would provide a clearer picture of performance consistency.

Reproducibility

The paper provides links to the model repositories on Hugging Face, which is a positive step towards reproducibility. However, it lacks detailed implementation specifics, such as hyperparameters and training configurations, which are crucial for others to replicate the findings accurately. Including these details would enhance the reproducibility of the results.

Limitations

One limitation noted is the focus on primarily European languages, which may restrict the model's applicability in global contexts. Additionally, while the paper discusses the trade-offs between the FastConformer and nGPT architectures, it does not delve deeply into scenarios where one might significantly outperform the other beyond the provided context, which could be beneficial for practitioners.

Broader impact

The development of efficient multilingual ASR and AST models has significant implications for accessibility and communication across language barriers. The release of models like Parakeet-TDT-0.6B-v3 for resource-limited settings could democratize access to advanced speech technologies, fostering inclusivity in technology use. This work contributes to the ongoing efforts to enhance machine learning applications in real-world scenarios, particularly in multilingual environments. The primary contribution of this paper is the introduction of two efficient multilingual models for ASR and AST, showcasing innovative methodologies and competitive performance against larger models. The comprehensive exploration of architecture and training strategies positions this work as a significant advancement in the field of speech processing.

Author of this paper? Send a correction.