Simultaneous Speech-to-Speech Translation Without Aligned Data
Abstract
Simultaneous speech translation requires translating source speech into a target language in real-time while handling non-monotonic word dependencies. Traditional approaches rely on supervised training with word-level aligned data, which is difficult to collect at scale and thus depends on synthetic alignments using language-specific heuristics that are suboptimal. We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely. This fundamentally simplifies the training pipeline and enables seamless scaling to diverse languages with varying grammatical structures, removing the bottleneck of designing language-specific alignment heuristics. We first train on sentence-level aligned data to learn speech translation at high latency, then apply a novel reinforcement learning strategy using GRPO to optimize latency while preserving translation quality. Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks. Moreover, we demonstrate that our model can be adapted to support a new input language with less than 1000h of speech. We provide examples, model weights, inference code and we release a benchmark containing 45h of multilingual data for speech translation evaluation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Hibiki-Zero, a novel approach to simultaneous speech-to-speech translation that eliminates the need for word-level alignments, thereby simplifying the training process and enhancing scalability across diverse languages. This work represents a meaningful advancement in the field of machine translation, particularly in addressing the challenges of real-time translation and the limitations of traditional methods.
Methodology
The proposed methodology, Hibiki-Zero, innovatively eliminates the need for word-level alignments in simultaneous speech-to-speech translation, which is a significant advancement in the field. The authors utilize a two-step training process, initially training on sentence-level aligned data and subsequently applying a reinforcement learning strategy (GRPO) to optimize latency without sacrificing translation quality. This approach is particularly noteworthy as it addresses the challenges posed by non-monotonic word dependencies and the difficulties in collecting aligned data at scale.
Results & evaluation
The authors report state-of-the-art performance across multiple metrics (translation accuracy, latency, voice transfer, and naturalness) on five X-to-English tasks, which demonstrates the effectiveness of their approach. However, the paper would benefit from a more detailed description of the datasets used and the specific experimental setups to allow for better understanding and reproducibility of results.
Reproducibility
The authors provide model weights and inference code, which is a positive aspect for reproducibility. However, the paper lacks detailed information on the training process, hyperparameters, and the specific configurations used in experiments, which could hinder full reproducibility by other researchers.
Limitations
One limitation is the reliance on sentence-level aligned data for initial training, which may still pose challenges in terms of data availability for certain languages. Additionally, while the model adapts to new languages with less than 1000 hours of speech, the practical implications of this adaptation in real-world scenarios are not thoroughly discussed.
Broader impact
The ability to translate speech in real-time without the need for aligned data has significant implications for multilingual communication, accessibility, and the development of more inclusive technologies. The release of a benchmark containing multilingual data also contributes positively to the field by providing resources for further research and development. The main contribution of this paper is the introduction of Hibiki-Zero, a novel approach to simultaneous speech-to-speech translation that eliminates the need for word-level alignments, thereby simplifying the training process and enhancing scalability across diverse languages. This work represents a meaningful advancement in the field of machine translation, particularly in addressing the challenges of real-time translation and the limitations of traditional methods.
Author of this paper? Send a correction.