Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition
Abstract
Large language model (LLM)-based audio-visual speech recognition (LLM-AVSR) has recently demonstrated strong robustness in adverse acoustic environments by leveraging complementary audio and visual information. Existing approaches typically employ independently pretrained acoustic and visual encoders, whose outputs are projected and fused as soft prompts to condition an LLM for speech recognition. However, most methods perform multimodal fusion without explicitly addressing the representational discrepancy between audio, visual and text modalities, potentially limiting the effectiveness of cross-modal integration. In this paper, we propose an optimal transport (OT)-based semantic alignment framework for LLM-AVSR. The proposed method explicitly bridges the modality gap by aligning the acoustic and visual representations with reference to the linguistic embedding space of the LLM before multimodal fusion. Specifically, OT is used to estimate probabilistic coupling matrices that characterize structured correspondences between modality-specific features and linguistic embeddings. The resulting OT couplings are further utilized as soft pseudo-labels to supervise contrastive learning, encouraging the extraction of semantically coherent and cross-modal consistent audio-visual representations. By anchoring multimodal features to the linguistic space of the LLM, the proposed framework facilitates more effective multimodal fusion and decoding. We implement the proposed framework using a Whisper-based acoustic encoder, an AV-HuBERT-based visual encoder, and a LLaMA3.2-3B decoder. Experiments conducted on the LRS3-TED benchmark demonstrate consistent improvements over strong baselines and achieve state-of-the-art performance under both clean and noisy evaluation conditions across a wide range of signal-to-noise ratios (SNRs).
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
Methodology
The paper proposes an Optimal Transport (OT)-based semantic alignment framework for LLM-based Audio-Visual Speech Recognition (LLM-AVSR). The core methodological contribution is the use of entropy-regularized OT to compute probabilistic coupling matrices between acoustic/visual features and linguistic token embeddings. These couplings are then used as soft pseudo-labels to supervise a contrastive learning objective, aiming to bridge the representational gap between modality-specific encoders (Whisper, AV-HuBERT) and the LLM decoder (LLaMA3.2-3B). The approach also introduces "virtual buckets" in the OT formulation to handle non-informative frames (noise/silence). While the application of OT to multimodal alignment is not entirely new, its specific integration into the pre-fusion stage of LLM-AVSR to align with the linguistic embedding space is a distinct methodological choice. The use of OT couplings as supervision for contrastive learning is a clever mechanism to enforce semantic consistency without requiring hard alignments. However, the novelty is somewhat tempered by the fact that OT-based alignment has been explored in other multimodal contexts, and the specific adaptation here is incremental rather than foundational.
Results & evaluation
The experimental evaluation is conducted on the LRS3-TED benchmark, which is a standard dataset for AVSR. The authors evaluate performance under clean and noisy conditions (various SNRs). They compare against strong baselines and ablate different fusion strategies (concatenation, addition, cross-attention, Q-former) and hyperparameters. The results show consistent improvements over baselines, achieving state-of-the-art performance on LRS3-TED. The inclusion of noisy evaluation is crucial for AVSR and demonstrates the robustness of the method. However, the paper lacks comparison with some very recent LLM-AVSR baselines that might use different architectural designs (e.g., direct projection vs. complex fusion). The ablation studies are present but could be more extensive regarding the sensitivity of the OT parameters. The results are solid but not overwhelmingly superior to the baselines, suggesting that while effective, the gains are marginal in some settings.
Reproducibility
The paper provides detailed descriptions of the model architecture, including the encoders (Whisper, AV-HuBERT), the LLM (LLaMA3.2-3B), and the OT alignment module. The loss functions and optimization strategies (Adam, LoRA) are specified. The dataset splits and preprocessing steps (lip cropping, audio downsampling) are described. However, the code is not publicly available (no project URL provided), and some hyperparameter tuning details are empirical. The "virtual bucket" mechanism is described but the exact implementation details (e.g., how the similarity margin $s$ is dynamically adjusted or fixed) could be clearer. Overall, reproducibility is moderate to high given the detailed mathematical formulation, but the lack of code release hinders immediate verification.
Limitations
The method introduces additional computational overhead due to the OT calculation (Sinkhorn iterations) during training. The paper acknowledges that hyperparameter tuning for the OT and contrastive learning components is non-trivial and sensitive. The "virtual bucket" mechanism, while effective, adds complexity to the OT formulation. The evaluation is limited to the LRS3-TED dataset; performance on other AVSR benchmarks (e.g., LRS2, VoxCeleb) is not reported, limiting the generalizability claims. The method relies on pretrained encoders, so improvements are constrained by the quality of these base models.
Broader impact
This work contributes to the field of multimodal learning, specifically in robust speech recognition. By improving the alignment between audio, visual, and linguistic modalities, it enhances the reliability of AVSR systems in adverse environments, which has practical applications in assistive technologies, secure communications, and human-computer interaction. The use of OT for semantic alignment could inspire similar approaches in other multimodal tasks where cross-modal correspondence is challenging. The paper presents a method for aligning audio-visual features with linguistic embeddings in LLM-based ASR using Optimal Transport, demonstrating improved robustness and state-of-the-art results on LRS3-TED.
Author of this paper? Send a correction.