Scaling Spoken Language Models with Syllabic Speech Tokenization
Abstract
Spoken language models (SLMs) typically discretize speech into high-frame-rate tokens extracted from SSL speech models. As the most successful LMs are based on the Transformer architecture, processing these long token streams with self-attention is expensive, as attention scales quadratically with sequence length. A recent SSL work introduces acoustic tokenization of speech at the syllable level, which is more interpretable and potentially more scalable with significant compression in token lengths (4-5 Hz). Yet, their value for spoken language modeling is not yet fully explored. We present the first systematic study of syllabic tokenization for spoken language modeling, evaluating models on a suite of SLU benchmarks while varying training data scale. Syllabic tokens can match or surpass the previous high-frame rate tokens while significantly cutting training and inference costs, achieving more than a 2x reduction in training time and a 5x reduction in FLOPs. Our findings highlight syllable-level language modeling as a promising path to efficient long-context spoken language models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction and evaluation of syllabic tokenization for spoken language models, demonstrating its potential to improve efficiency and performance in spoken language understanding tasks. The technical contributions are significant, as they address a critical bottleneck in the scalability of spoken language models while providing a clear path forward for future research in this area.
Methodology
The paper presents a systematic study of syllabic tokenization for spoken language modeling, which is a novel approach compared to traditional high-frame-rate tokenization methods. The methodology includes a clear experimental setup with a comparison between Sylber and Hubert tokenization methods, utilizing various datasets and model architectures. The use of k-means clustering for syllabic tokenization is a straightforward yet effective choice, although the authors acknowledge potential limitations in this approach.
Results & evaluation
The experiments are well-structured, comparing models trained with different tokenization methods across several benchmarks. The results demonstrate that syllabic tokenization can achieve comparable or superior performance to frame-based methods while significantly reducing training time and computational resources. The use of multiple datasets for training and evaluation strengthens the findings, although the paper could benefit from more extensive ablation studies to further validate the robustness of the results.
Reproducibility
The paper provides sufficient detail regarding the experimental setup, model architectures, and evaluation metrics, which supports reproducibility. However, the absence of publicly available code or datasets limits the ability for other researchers to replicate the findings fully. Including links to the datasets and code would enhance reproducibility.
Limitations
The paper acknowledges that the Sylber tokens have a coarser representation than the Hubert tokens, which may affect performance in certain contexts. Additionally, the reliance on k-means clustering for syllabic tokenization may not be optimal, and further exploration of alternative methods could yield better results. The study also does not explore the impact of different training epochs or hyperparameter tuning extensively.
Broader impact
The findings suggest that syllabic tokenization could lead to more efficient spoken language models, which have significant implications for applications in speech recognition, natural language processing, and conversational AI systems. By reducing computational costs and improving model efficiency, this research could facilitate the deployment of advanced spoken language models in real-world applications. The main contribution of this paper is the introduction and evaluation of syllabic tokenization for spoken language models, demonstrating its potential to improve efficiency and performance in spoken language understanding tasks. The technical contributions are significant, as they address a critical bottleneck in the scalability of spoken language models while providing a clear path forward for future research in this area.
Author of this paper? Send a correction.