Multimodal Confidence Modeling in Audio-Visual Quality Assessment
Abstract
Audio-visual quality assessment (AVQA) is essential for streaming, teleconferencing, and immersive media. In realistic streaming scenarios, distortions are often asymmetric, where one modality may be severely degraded while the other remains clean. Still, most contemporary AVQA metrics treat audio and video as equally reliable, causing confidence-unaware fusion to emphasize unreliable signals. This paper proposes MCM-AVQA, a multimodal confidence-aware AVQA framework that explicitly estimates modality-specific confidence and injects it into a dedicated audio-visual mixer for cross-modal attention. The Audio-Visual Mixer utilizes frame-level, confidence-guided channel attention to gate fusion, modulating feature interaction between modalities so that high-confidence streams dominate while unreliable inputs are suppressed, preserving temporal degradation patterns. A multi-head visual confidence estimator turns frame-level artifact probabilities into temporally smoothed, clip-level visual confidence scores, while an audio confidence module derives confidence from speech-quality cues without requiring a clean reference. Experiments on multiple AVQA benchmarks show that MCM-AVQA, and specifically its confidence-guided Audio-Visual Mixer, improve correlation with human mean opinion scores and yield more interpretable behavior under real-world asymmetric audio-visual distortions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents MCM-AVQA, a confidence-aware audio-visual quality assessment framework that improves the robustness of quality evaluation under asymmetric distortions. This work significantly advances the state of the art in AVQA by integrating modality-specific confidence into the fusion process, leading to more accurate and interpretable quality assessments.
Methodology
The proposed MCM-AVQA framework introduces a novel approach to audio-visual quality assessment by explicitly modeling modality-specific confidence and integrating it into a dedicated Audio-Visual Mixer. This methodology allows for dynamic feature gating based on confidence levels, which is a significant advancement over traditional methods that treat audio and video as equally reliable. The use of a multi-head visual confidence estimator and an audio confidence module enhances the robustness of the model under asymmetric distortions, which is a common scenario in real-world applications. The architecture is well-structured, leveraging state-of-the-art transformer models and attention mechanisms, making it a strong contribution to the field.
Results & evaluation
The experiments conducted across multiple AVQA benchmarks (LIVE-SJTU, UnB-AV, UnB-AVQ) demonstrate the effectiveness of MCM-AVQA in improving correlation with human mean opinion scores. The results indicate that the model outperforms existing state-of-the-art methods, particularly in scenarios with asymmetric distortions. The ablation studies provide valuable insights into the contributions of each component of the model, reinforcing the importance of confidence-aware fusion. The use of statistical tests to validate performance improvements adds rigor to the evaluation.
Reproducibility
The paper provides sufficient details regarding the architecture, training procedures, and evaluation metrics, which supports reproducibility. However, the absence of publicly available code or datasets limits the ability for other researchers to replicate the results directly. Including a project URL or demo would significantly enhance reproducibility.
Limitations
One limitation of the study is the lack of a comprehensive comparison with more recent AVQA methods that may not have been included in the evaluation. Additionally, while the model shows robustness under asymmetric distortions, its performance in extreme distortion scenarios or with novel types of distortions remains untested. The reliance on subjective mean opinion scores for evaluation, while standard, could also introduce variability based on human judgment.
Broader impact
The MCM-AVQA framework has significant implications for real-world applications in streaming, teleconferencing, and immersive media, where audio-visual quality is critical. By improving the accuracy of quality assessments in asymmetric distortion scenarios, this work can enhance user experiences in various multimedia applications. The approach could also be extended to other multimodal quality assessment tasks, potentially influencing future research directions in the field. The paper presents MCM-AVQA, a confidence-aware audio-visual quality assessment framework that improves the robustness of quality evaluation under asymmetric distortions. This work significantly advances the state of the art in AVQA by integrating modality-specific confidence into the fusion process, leading to more accurate and interpretable quality assessments.
Author of this paper? Send a correction.