VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
Abstract
Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a promising paradigm by connecting pre-trained audio-visual encoders to an LLM, achieving strong results in clean conditions. However, these models are predominantly optimized for clean acoustic conditions, with limited attention to making the LLM backbone robust to noise. No explicit mechanism is employed to produce stable representations under corrupted audio, leading to performance degradation in noisy environments. To address this, we propose VIB-AVSR, which integrates Variational Information Bottleneck layers at targeted positions within the LLM backbone to regularize representations. VIB-AVSR reduces degradation under noisy conditions across multiple SNR levels and noise types, without requiring architectural modifications or additional training data.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
VIB-AVSR introduces Variational Information Bottleneck layers into the LLM backbone of AVSR models to regularize audio representations, demonstrating that variational compression can improve noise robustness and generalization without additional training data or architectural redesign.
Methodology
The paper proposes VIB-AVSR, a method to enhance the noise robustness of LLM-based Audio-Visual Speech Recognition (AVSR) models. The core innovation is the integration of Variational Information Bottleneck (VIB) layers into the intermediate layers of the LLM backbone (Llama-3.2-1B). Specifically, the method applies a variational compression objective to the audio hidden states ($H_a$) while leaving visual ($H_v$) and text ($H_t$) representations uncompressed. This is motivated by the observation that pre-trained LLMs, fine-tuned via LoRA, lack intrinsic mechanisms to filter out acoustic noise, relying solely on encoders which may not fully disentangle noise from speech features. The VIB module parameterizes the posterior distribution of the compressed representation as a diagonal Gaussian and uses a learnable prior, optimizing a lower bound on the IB objective. The approach is theoretically sound, applying a well-established information-theoretic principle to a modern multimodal architecture. However, the novelty is somewhat limited by the fact that VIB has been applied in various contexts before; the specific application to the *internal* representations of an LLM backbone for AVSR is the key contribution, but it is an incremental architectural modification rather than a new algorithmic breakthrough.
Results & evaluation
The experimental evaluation is conducted on the LRS2 dataset using Whisper-medium and AV-HuBERT encoders. The authors evaluate under two training paradigms: "Noisy" (noise augmentation during training) and "Clean" (no noise augmentation). Results are reported across multiple SNR levels (-10 to 5 dB) and noise types (Babble, Speech). The results show consistent Word Error Rate (WER) reductions for VIB-AVSR compared to the Llama-AVSR baseline, particularly in low-SNR regimes. A significant finding is that VIB-AVSR trained on *clean* data still outperforms the baseline on noisy test data, suggesting that the variational compression acts as a regularizer that improves generalization to unseen noise distributions. The ablation studies on layer placement, regularization strength, and interpolation coefficients provide good empirical grounding. However, the improvements, while consistent, are modest (e.g., Avg WER reduction from 18.85 to 17.39 in one setting). The paper lacks comparison with other robustness techniques (e.g., adversarial training, specific noise-robust encoders like Wav2Vec 2.0 with masking) which would better contextualize the gain.
Reproducibility
The paper provides sufficient implementation details, including the architecture of the VIB module (2-layer MLP), the use of LoRA, and the specific layers for bottleneck insertion. The code is available on GitHub. The use of standard datasets (LRS2, MUSAN) and models (Whisper, Llama-3.2) enhances reproducibility. The description of the training paradigms and hyperparameters is clear.
Limitations
The primary limitation is the modest magnitude of improvement. While statistically significant, the WER reductions are not transformative. The method adds computational overhead during training (sampling from the posterior) and slight complexity, though inference is unaffected. The approach assumes that noise is the primary source of variance to be discarded, which might risk discarding subtle acoustic features if the compression is too aggressive (though the interpolation term mitigates this). The evaluation is limited to LRS2; performance on more challenging, real-world datasets with diverse speaking styles and backgrounds is not reported. Furthermore, the "Clean" training paradigm's success relies on the assumption that noise robustness can be learned via representation compression alone, which might not hold for all noise types or severe distortions.
Broader impact
This work contributes to the broader goal of making multimodal AI systems more robust and reliable in real-world, uncontrolled environments. By improving the noise robustness of LLM-based AVSR, it paves the way for more accessible speech recognition systems for users with hearing impairments or in noisy environments. It also highlights the importance of representation regularization in large foundation models when adapting them to noisy sensory inputs. VIB-AVSR introduces Variational Information Bottleneck layers into the LLM backbone of AVSR models to regularize audio representations, demonstrating that variational compression can improve noise robustness and generalization without additional training data or architectural redesign.
Author of this paper? Send a correction.