FlexiSLM: A Dynamic and Controllable Frame Rate Spoken Language Model
Abstract
Spoken language models (SLMs) extend LLMs to speech input and output. Existing SLMs represent speech at fixed frame rates (e.g., 25 or 12.5 Hz), ignoring the time-varying information density of speech and offering no flexibility to trade off quality for speed at inference time. Recent audio tokenizer research has proposed dynamic frame rate speech coding, which exploits this non-uniformity and enables two new capabilities: very low average frame rates and frame rate controllability. However, this technique has not yet been applied to SLMs. We introduce Flexible Spoken Language Model (FlexiSLM), the first SLM that supports dynamic and controllable frame rates on both speech input and output. Using dynamic frame rate representations, FlexiSLM outperforms fixed-frame-rate 7B models including Qwen2.5-Omni and Kimi-Audio at its high-quality operating points. We further verify that FlexiSLM can be accurately steered down to 4.0 Hz; at 6.25 Hz, it roughly halves inference time relative to 12.5 Hz while retaining strong speech-to-speech quality. Audio samples are available at https://flexislm.github.io .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
FlexiSLM introduces the first spoken language model with dynamic and controllable frame rates, achieving superior efficiency and competitive performance compared to fixed-rate 7B baselines by integrating dynamic speech tokenization with direct inference-time frame rate conditioning.
Methodology
The paper proposes FlexiSLM, a spoken language model (SLM) that integrates dynamic frame rate tokenization into both the input and output streams. The core methodological contribution is the adaptation of FlexiCodec, a dynamic-rate speech tokenizer, into a full SLM pipeline. Key technical innovations include: 1) A "Thinker-Talker" architecture where the Talker module predicts dynamic-frame-rate speech tokens and their associated frame lengths. 2) Direct frame rate conditioning using sinusoidal positional encoding, allowing users to specify the target average frame rate at inference time, which is a significant improvement over the indirect threshold-based control used in prior tokenizer work. 3) A bidirectional connection between the Talker and Thinker, enabled in the final fine-tuning stage, to provide the language model with explicit knowledge of generated speech tokens. The approach effectively bridges the gap between efficient audio tokenization and end-to-end language modeling, addressing the inefficiency of fixed-rate representations in SLMs.
Results & evaluation
The authors conduct comprehensive evaluations against strong 7B baselines (Qwen2.5-Omni, Kimi-Audio, Mimo-Audio) and larger models (Gemini 2.5-Pro). Results on Kimi-Audio-Evalkit show that FlexiSLM-7B outperforms fixed-rate 7B models at 12.5 Hz and maintains strong performance at 6.25 Hz, achieving roughly half the inference time (RTF) with minimal quality degradation. The paper includes ablation studies validating the necessity of dynamic merging, the effectiveness of direct frame rate control, and the benefits of the Talker-to-Thinker connection. Additional evaluations on audio understanding (LLaSO-Eval) and ASR (LibriSpeech) further demonstrate the model's robustness. The experimental design is rigorous, covering multiple operating points and comparing against state-of-the-art systems.
Reproducibility
The paper provides detailed descriptions of the architecture, training stages (pre-training, LoRA fine-tuning, full fine-tuning), and hyperparameters. The authors commit to releasing code and data. The use of open-source components (Qwen2.5-7B, FlexiCodec, Vocos) enhances reproducibility. However, the construction of the proprietary "FlexiSLM-Data" involves distillation from a 30B model and specific filtering pipelines which may be complex to replicate exactly without access to the intermediate models or detailed filtering scripts.
Limitations
The authors acknowledge several limitations: 1) The model is not yet streaming-capable, limiting its use in real-time interactive scenarios. 2) Post-training alignment techniques like RLHF or DPO have not been explored, which could improve response quality. 3) The training data lacks reasoning-intensive tasks and multi-turn dialogues, potentially limiting generalization. 4) Performance degrades significantly at very low frame rates (4.0-5.0 Hz), indicating a need for better robustness in extreme compression regimes.
Broader impact
FlexiSLM demonstrates a practical path toward more efficient and flexible spoken language models. By enabling dynamic frame rates, it allows for better trade-offs between quality and compute resources, which is crucial for deploying SLMs on edge devices or in bandwidth-constrained environments. This work contributes to the broader goal of making multimodal AI more accessible and efficient. FlexiSLM introduces the first spoken language model with dynamic and controllable frame rates, achieving superior efficiency and competitive performance compared to fixed-rate 7B baselines by integrating dynamic speech tokenization with direct inference-time frame rate conditioning.
Author of this paper? Send a correction.