SLM-TTA: A Framework for Test-Time Adaptation of Generative Spoken Language Models
Abstract
Spoken Language Models (SLMs) are increasingly central to modern speech-driven applications, but performance degrades under acoustic shift - real-world noise, reverberation, and microphone variation. Prior solutions rely on offline domain adaptation, which is post-hoc, data-intensive, and slow. We introduce the first test-time adaptation (TTA) framework for generative SLMs that process interleaved audio-text prompts. Our method updates a small, targeted subset of parameters during inference using only the incoming utterance, requiring no source data or labels. This stabilizes token distributions and improves robustness to acoustic variability without degrading core task accuracy. Evaluated on automatic speech recognition, speech translation, and 19 audio understanding tasks from AIR-Bench, our approach yields consistent gains under diverse corruptions. Because adaptation touches only a small fraction of weights, it is both compute- and memory-efficient, supporting deployment on resource-constrained platforms. This work enhances the robustness and adaptability of generative SLMs for real-world speech-driven applications.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the SLM-TTA framework, which enables real-time adaptation of generative spoken language models to varying acoustic conditions without the need for additional data or labels. This work significantly enhances the robustness and adaptability of SLMs, making them more applicable to real-world scenarios where acoustic variability is prevalent.
Methodology
The proposed SLM-TTA framework introduces a novel approach to test-time adaptation for generative spoken language models by updating a small subset of parameters during inference. This methodology addresses the critical issue of performance degradation in SLMs under various acoustic conditions without requiring additional data or labels. The interleaved audio-text prompts are a clever way to leverage incoming utterances for adaptation, which enhances the model's robustness while maintaining core task accuracy. The focus on efficiency in terms of compute and memory usage is particularly relevant for deployment in resource-constrained environments, making the approach practical for real-world applications.
Results & evaluation
The experimental setup is robust, evaluating the framework across multiple tasks, including automatic speech recognition, speech translation, and 19 audio understanding tasks from AIR-Bench. The consistent performance gains reported under diverse corruptions provide strong evidence of the method's effectiveness. However, the paper could benefit from more detailed comparisons with existing adaptation techniques to better contextualize its contributions.
Reproducibility
The paper lacks specific implementation details that would facilitate reproducibility, such as hyperparameter settings, model architectures, and training procedures. Providing a clear description of the experimental setup and access to code or datasets would significantly enhance reproducibility.
Limitations
One limitation is the reliance on a targeted subset of parameters for adaptation, which may not capture all necessary variations in acoustic conditions. Additionally, the framework's performance in highly variable or extreme conditions is not fully explored, which could be a potential area for further research.
Broader impact
The framework has significant implications for the deployment of generative spoken language models in real-world applications, particularly in environments with varying acoustic conditions. Its focus on efficiency and adaptability could lead to improvements in speech-driven applications across various domains, including virtual assistants, automated transcription services, and accessibility tools. The main contribution of this paper is the introduction of the SLM-TTA framework, which enables real-time adaptation of generative spoken language models to varying acoustic conditions without the need for additional data or labels. This work significantly enhances the robustness and adaptability of SLMs, making them more applicable to real-world scenarios where acoustic variability is prevalent.
Author of this paper? Send a correction.