VOX-KRIKRI: Unifying Speech and Language through Continuous Fusion
Abstract
We present a multimodal fusion framework that bridges pre-trained decoder-based large language models (LLM) and acoustic encoder-decoder architectures such as Whisper, with the aim of building speech-enabled LLMs. Instead of directly using audio embeddings, we explore an intermediate audio-conditioned text space as a more effective mechanism for alignment. Our method operates fully in continuous text representation spaces, fusing Whisper's hidden decoder states with those of an LLM through cross-modal attention, and supports both offline and streaming modes. We introduce \textit{VoxKrikri}, the first Greek speech LLM, and show through analysis that our approach effectively aligns representations across modalities. These results highlight continuous space fusion as a promising path for multilingual and low-resource speech LLMs, while achieving state-of-the-art results for Automatic Speech Recognition in Greek, providing an average $\sim20\%$ relative improvement across benchmarks.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the development of a novel multimodal fusion framework that bridges the gap between speech and language models, achieving state-of-the-art results in Greek ASR and providing a promising avenue for future research in multilingual speech technologies. The comprehensive analysis of the technical contributions, methodology, and significance to the field underscores the paper's relevance and potential impact on advancing speech-enabled LLMs.
Methodology
The proposed methodology introduces a novel multimodal fusion framework that effectively integrates continuous audio and text representations using cross-modal attention. The choice to operate in a continuous latent space rather than relying on discrete audio embeddings is innovative, as it allows for a more linguistically grounded representation. The framework's design supports both offline and streaming applications, which is a significant advancement in the field of speech-enabled LLMs. The use of causal masking for real-time applications is particularly noteworthy, as it addresses practical limitations in ASR systems.
Results & evaluation
The experiments are well-structured, utilizing multiple datasets to evaluate the performance of the VoxKrikri models against a strong baseline (Whisper-large-v3). The results demonstrate consistent improvements in Word Error Rate (WER) across various benchmarks, highlighting the effectiveness of the proposed fusion strategies. The analysis of cross-modal alignment through correlation metrics provides additional insight into the model's performance and the effectiveness of the proposed methodology.
Reproducibility
The paper provides sufficient details on the experimental setup, including the choice of datasets, model configurations, and hyperparameters. However, the absence of a publicly accessible demo or code repository limits the reproducibility of the results. Future work should consider making the models and training scripts available to the research community.
Limitations
One limitation of the study is the focus on Greek ASR, which may restrict the generalizability of the findings to other languages or dialects. Additionally, while the paper discusses future work, it does not address potential challenges in scaling the approach to more complex tasks or languages with less available data.
Broader impact
The introduction of VoxKrikri as the first Greek speech LLM has the potential to significantly impact the accessibility of speech technology for Greek speakers, particularly in low-resource settings. The framework's emphasis on continuous space fusion may also inspire further research in multilingual and cross-lingual applications, potentially broadening the reach of speech-enabled LLMs across diverse languages. The main contribution of this paper is the development of a novel multimodal fusion framework that bridges the gap between speech and language models, achieving state-of-the-art results in Greek ASR and providing a promising avenue for future research in multilingual speech technologies. The comprehensive analysis of the technical contributions, methodology, and significance to the field underscores the paper's relevance and potential impact on advancing speech-enabled LLMs.
Author of this paper? Send a correction.