UBG-Net: An Uncertainty-aware Bayesian Gating Network for Robust Audio-Visual Speech Recognition
Abstract
Audio-Visual speech recognition systems often degrade in real-world scenarios due to signal corruption and distribution shifts. To address this, we propose a unified uncertainty-modeling framework, namely the uncertainty-aware Bayesian gating network (UBG-Net). UBG-Net features a Modality Uncertainty-aware Bayesian Fusion (MUBF) mechanism that injects signal-level aleatoric uncertainty into a Bayesian network to model epistemic uncertainty, thereby ensuring robust fusion of pre-trained backbone features. For inference, we introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV) to select transcripts from Monte Carlo samples, prioritizing frequency and using inference scores in case of a tie. Experiments on the AVCocktail and LRS2 datasets demonstrate the overall superiority of UBG-Net compared to SOTA baselines. Ablation studies confirm that MUBF and DUHV effectively filter noise, enhancing fusion and decoding robustness.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
Methodology
The paper proposes UBG-Net, an Audio-Visual Speech Recognition (AVSR) framework that integrates uncertainty modeling into the fusion process. The core methodological contribution is the Modality Uncertainty-aware Bayesian Fusion (MUBF), which attempts to bridge aleatoric (data) and epistemic (model) uncertainty. Specifically, it extracts signal-level aleatoric uncertainty (mean and variance) from audio and visual streams and concatenates them as a context vector to guide a Bayesian Gating Network (BGN). The BGN uses variational inference to model weight uncertainty, aiming to produce robust fusion weights. For inference, the authors introduce Distribution Uncertainty-aware Hierarchical Voting (DUHV), a post-processing step that selects the final transcript from Monte Carlo samples by prioritizing frequency (majority vote) and using inference scores for tie-breaking. While the integration of these components is coherent, the novelty is moderate. Bayesian fusion in multimodal learning is not new; the specific twist of using aleatoric parameters as context for epistemic gating is a logical and sensible design choice, but it does not represent a fundamental shift in uncertainty modeling theory. The approach relies on standard variational inference techniques (ELBO, reparameterization) applied to a specific architectural bottleneck.
Results & evaluation
The experimental evaluation is conducted on two benchmarks: LRS2 (with simulated noise at various SNRs) and AVCocktail (real-world multi-party conversations). The results demonstrate consistent improvements over a strong baseline (AV-HuBERT based) and other SOTA methods, particularly in low-SNR conditions (-5 dB) and in the challenging AVCocktail fixed-chunk setting. The ablation studies effectively isolate the contributions of MUBF and DUHV, showing that combining both uncertainty types and the hierarchical voting strategy yields the best performance. The inclusion of general-purpose models like Whisper and Qwen3-Omni provides good context for the difficulty of the task. However, the evaluation is limited to Word Error Rate (WER). There is no analysis of the calibration of the uncertainty estimates themselves (e.g., Expected Calibration Error or reliability diagrams), which is crucial for a paper claiming to model uncertainty. Additionally, the claim of "7.9% improvement" on AVCocktail seems high and warrants scrutiny regarding the baseline comparison, though the table data (implied) supports significant gains. The lack of LRS3 evaluation is noted but acceptable given access issues.
Reproducibility
The paper provides detailed implementation details, including network architectures (2-layer MLPs, Bayesian linear layers), initialization schemes (Xavier-uniform, bias -5.0), hyperparameters (learning rate, batch size, weight decay), and training strategies (two-stage, DoRA). The use of a publicly available baseline repository (AVSRCocktail) aids reproducibility. The Monte Carlo sampling count (K=5) is specified. However, the code is not explicitly linked in the provided text (though a baseline repo is mentioned), and the specific random seeds for the 10 independent runs are not detailed, which is important for stochastic Bayesian methods. Overall, the description is sufficient for a competent researcher to reproduce the results.
Limitations
The paper acknowledges a slight performance decline at 10 dB and clean conditions, attributing it to "over-enhancement." This suggests the model may be overly aggressive in suppressing features even when they are reliable, indicating a potential lack of robustness in high-quality regimes. The reliance on Monte Carlo sampling for inference increases latency compared to deterministic baselines, although the authors argue K=5 is a good trade-off. The method assumes that the pre-trained backbone features are sufficiently expressive, which might not hold for out-of-domain data not seen during the fine-tuning stage. Furthermore, the complexity of the Bayesian layers adds computational overhead during training and inference.
Broader impact
This work contributes to the robustness of multimodal AI systems, which is critical for real-world deployment in noisy environments (e.g., autonomous vehicles, smart homes, hearing aids). By providing a framework that quantifies and mitigates uncertainty, it offers a pathway to more reliable AVSR systems. The techniques could be extended to other multimodal tasks where sensor reliability varies dynamically. The paper presents a well-motivated and technically sound framework for robust audio-visual speech recognition by integrating aleatoric and epistemic uncertainty modeling through a novel Bayesian gating mechanism and hierarchical voting strategy, demonstrating significant improvements on challenging real-world and simulated benchmarks.
Author of this paper? Send a correction.