Bounds on Agreement between Subjective and Objective Measurements
Abstract
Objective estimators of multimedia quality are often judged by comparing estimates with subjective "truth data," most often via Pearson correlation coefficient (PCC) or mean-squared error (MSE). But subjective test results contain noise, so striving for a PCC of 1.0 or an MSE of 0.0 is neither realistic nor repeatable. Numerous efforts have been made to acknowledge and appropriately accommodate subjective test noise in objective-subjective comparisons, typically resulting in new analysis frameworks and figures-of-merit. We take a different approach. By making only basic assumptions, we derive bounds on PCC and MSE that can be expected for a subjective test. Consistent with intuition, these bounds are functions of subjective vote variance. When a subjective test includes vote variance information, the calculation of the bounds is easy, and in this case we say the resulting bounds are "fully data-driven." We provide two options for calculating bounds in cases where vote variance information is not available. One option is to use vote variance information from other subjective tests that do provide such information, and the second option is to use a model for subjective votes. Thus we introduce a binomial-based model for subjective votes (BinoVotes) that naturally leads to a mean opinion score (MOS) model, named BinoMOS, with multiple unique desirable properties. BinoMOS reproduces the discrete nature of MOS values and its dependence on the number of votes per file. This modeling provides vote variance information required by the PCC and MSE bounds and we compare this modeling with data from 18 subjective tests. The modeling yields PCC and MSE bounds that agree very well with those found from the data directly. These results allow one to set expectations for the PCC and MSE that might be achieved for any subjective test, even those where vote variance information is not available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper makes a meaningful contribution to the field of multimedia quality assessment by deriving actionable bounds for the performance of objective estimators based on subjective test results. The introduction of the BinoVotes model presents a novel perspective that enhances the understanding of the relationship between subjective and objective measurements, offering valuable insights for future research and applications in the area.
Methodology
The paper presents a novel approach to deriving bounds on the Pearson correlation coefficient (PCC) and mean-squared error (MSE) for subjective tests in multimedia quality assessment. The authors introduce the BinoVotes model, which is based on the binomial distribution, to effectively capture the discrete nature of subjective ratings. This model allows for the derivation of bounds that are grounded in the intrinsic properties of the voting process, making the methodology both intuitive and mathematically sound. The approach is distinct from previous work that often relies on more complex models or assumptions about the distribution of votes.
Results & evaluation
The authors validate their theoretical bounds using data from 18 subjective tests, demonstrating that their BinoVotes and BinoMOS models yield results that align well with empirical data. The experiments are comprehensive, covering a range of multimedia types and subjective testing conditions, which strengthens the credibility of their findings. However, the paper could benefit from additional experiments that explore the performance of objective estimators across diverse datasets beyond the 18 tests analyzed.
Reproducibility
The paper provides a GitHub repository for the implementation of the BinoVotes model and the associated bounds, which enhances reproducibility. However, detailed descriptions of the datasets used in the experiments, including their characteristics and how they were processed, are somewhat limited. More thorough documentation would facilitate better reproducibility of the results.
Limitations
One limitation of the study is the reliance on the BinoVotes model for cases where vote variance information is not available, which may lead to overestimation of vote variance in some scenarios. Additionally, while the bounds derived are informative, they do not account for the potential biases introduced by the subjective nature of the voting process. Future work could explore incorporating individual subject biases into the model for a more nuanced understanding of vote variance.
Broader impact
The findings of this paper have significant implications for the development of objective quality estimators in multimedia applications. By providing realistic performance bounds, researchers can better understand the limitations of their models and set achievable goals for improvement. The methodology could be applied to various domains beyond multimedia, including any field that relies on subjective assessments, thus broadening its impact. This paper makes a meaningful contribution to the field of multimedia quality assessment by deriving actionable bounds for the performance of objective estimators based on subjective test results. The introduction of the BinoVotes model presents a novel perspective that enhances the understanding of the relationship between subjective and objective measurements, offering valuable insights for future research and applications in the area.
Author of this paper? Send a correction.