SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
Abstract
Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents SALSA, a lightweight adaptation method for speech-aware large language models that effectively improves performance on out-of-domain automatic speech recognition tasks. The innovative approach of learning steering vectors without requiring paired contrastive examples addresses a critical challenge in speech processing, demonstrating significant technical contributions and potential for broader applications in the field.
Methodology
The proposed SALSA method introduces a novel approach to adapt speech-aware large language models (SALLMs) by learning layer-wise steering vectors without requiring paired contrastive examples. This method is significant as it directly optimizes steering vectors using a supervised objective, addressing a gap in existing steering techniques that depend on contrastive activation differences. The methodology is well-structured, allowing for effective adaptation to out-of-domain speech tasks, and demonstrates a clear understanding of the challenges posed by speech variability.
Results & evaluation
The experiments are comprehensive, covering a range of datasets including children's speech, multilingual speech, and Mandarin-English code-switching. The results show substantial improvements over zero-shot and in-context learning baselines, with detailed analysis of performance across different training sizes and steering locations. The use of Word Error Rate (WER) and Mixed Error Rate (MER) as evaluation metrics is appropriate for the tasks at hand.
Reproducibility
The paper provides sufficient details regarding the experimental setup, including the optimization process, training configurations, and evaluation metrics. However, the lack of a publicly available code repository or demo limits reproducibility and practical application of the findings.
Limitations
The study is limited to automatic speech recognition tasks and does not explore the generalization of the SALSA method across other speech tasks or languages. Additionally, the reliance on a single shared set of steering vectors may not be optimal for highly heterogeneous speech distributions, suggesting a need for more input-dependent adaptations.
Broader impact
The findings have significant implications for improving speech recognition systems, particularly in low-resource and out-of-domain settings. By enhancing the adaptability of SALLMs, this work could contribute to more equitable access to speech technology across diverse linguistic populations. However, the potential for reinforcing existing biases in multilingual speech resources must be carefully considered. The paper presents SALSA, a lightweight adaptation method for speech-aware large language models that effectively improves performance on out-of-domain automatic speech recognition tasks. The innovative approach of learning steering vectors without requiring paired contrastive examples addresses a critical challenge in speech processing, demonstrating significant technical contributions and potential for broader applications in the field.
Author of this paper? Send a correction.