Source-linked AI summary
A Mixed-Behavior Vote Model for Multimedia Subjective Quality Votes, Means, and Variances
Jaden Pieper, Stephen D. Voran
TL;DR
The paper addresses the mismatch between realistic subjective vote behavior and variance models that can violate admissible limits. It defines the UVR, develops UVR-respecting variance fitting, and introduces a mixed-behavior vote model; across 16 datasets, results lie inside the UVR, with weighted least squares needed for three.
Problem
Existing parabolic variance models can fit broad trends but often fall below the minimum admissible variance, violating a fundamental constraint on subjective votes.
Method
The paper defines the UVR from maximum-variance unimodal and adjacent two-choice vote models, then combines UVR-respecting variance fitting with a mixed-behavior vote model.
Results
Results lie inside the UVR for 13 of 16 datasets directly, while weighted least squares is needed for the remaining three; one dataset has 28% of values on the minimum-variance curve.
Takeaways & Limitations
The UVR and mixed-behavior model quantify voting behaviors that can produce observed subjective-test variance and provide interpretable insights into experiments.
Takeaways & Limitations
BinoVotes cannot directly model voting behavior or specific vote variances from individual subjective experiments, motivating the more flexible mixed model.
Abstract
from arXiv · showhide
The relationship between subjective test vote variance and vote mean (or MOS) is well-studied, and the mathematically admissible vote variance region has been previously defined. We propose a reduced admissible variance region called the Unimodal Variance Region (UVR) that better describes real subjective rating behavior of multimedia. Further, subjective vote variance is often modeled as parabolic. We explain that, in practice, the parabolic model often violates the admissible region in the variance vs. MOS plane and we propose alternatives that respect the admissible region. We also present a parametrized random process to model votes that mixes voting processes and produces a realistic range of vote variances within the UVR at any desired MOS. This process was inspired by and comports with voting behavior that is observed in many subjective tests. By modeling vote variance from a subjective experiment, this vote model offers additional interpretable insights into voting behavior observed in a given experiment. We present example results from 16 datasets spanning speech, image, and video subjective quality experiments.
1. INTRODUCTION
Subjective tests produce MOS as their principal summary, but vote variance also matters because it measures agreement and is constrained by the rating scale. Existing parabolic variance fits can match broad trends yet violate admissible variance limits, motivating UVR and new modeling approaches.
- MOS summarizes votes for each stimulus, while vote variance measures the level of agreement.
- 0 variance occurs at MOS values 1.0 and 5.0, because all votes must agree at the scale endpoints.
- The admissible variance bounds depend on MOS and the number of votes; maximum variance uses endpoint votes, whereas minimum variance uses adjacent votes.
- 0.038 to 0.249 and 0.066 to 0.251 are reported scale-factor ranges for fitting parabolic variance models across prior datasets.The ranges come from 14 datasets in and six datasets in.
- Scale factors below 0.25 can produce fitted variances below the minimum admissible curve, so the parabolic model cannot fully describe real voting behavior.
- The paper proposes UVR, variance-shape alternatives, and a mixed vote model to represent realistic multimedia voting behavior and interpret observed variance.
2. UNIMODAL VARIANCE REGION
The UVR restricts mathematically possible variance to behavior consistent with predominantly unimodal subjective votes. It is bounded below by adjacent two-choice voting and above by maximum-variance unimodal voting.
- Subjective ratings are expected to have a unimodal PMF centered on a consensus with stimulus-dependent variability.
- Multimodal voting represents distinct perception groupings, while consistent extreme bimodal voting would raise concerns about experimental design or voter behavior.
- The MVU vote PMF maximizes variance while preserving unimodality, arbitrary means, and E(R|y)=y.
- The ATC PMF minimizes variance using only two adjacent vote options while preserving unimodality and E(R|y)=y.
- The UVR is the shaded region between the MVU maximum-variance curve and the ATC minimum-variance curve.
3. MODELING VOTE VARIANCE
The proposed variance function is designed to fit observed data while respecting the UVR and preserving endpoint behavior, symmetry, continuity, and smoothness.
- The variance function must be zero at both scale extremes, symmetric about the midpoint, continuous, smooth, UVR-compliant, and data-fitting.
- A fourth-degree polynomial models vote variance using a second parabola to attenuate the maximum-variance parabola rather than a single scale constant.
- The attenuated-parabola construction produces variance functions that are flatter near the middle of the quality scale.
- Least squares estimates w0 and w1 from MOS values and corresponding vote variances.
- Weighted least squares gives extra weight to MOS values near 1 and 5 when the fitted curve slightly violates the minimum variance bound.
4. MIXED-BEHAVIOR VOTE MODEL
The mixed-BinoVotes (MBV) model extends BinoVotes by mixing it with either ATC or MVU voting behavior, enabling interpretable modeling of vote variances throughout the UVR.
- BinoVotes: BinoVotes produces unimodal, discrete five-choice vote distributions with non-zero probability for every vote option.Its single quality parameter produces the desired MOS, but it cannot directly model experiment-specific vote variances.
- BinoVotes: BinoVotes cannot directly represent voting behavior or associated variances from specific subjective experiments.Its variance relationship corresponds to the smallest allowable parabolic relationship and lies roughly in the middle of the UVR.
- Model construction: The MVU and ATC PMFs bound the UVR and, like BinoVotes, are unimodal, single-parameter, and highly interpretable.These properties provide alternative voting behaviors for constructing a flexible model.
- Model construction: The MBV PMF mixes BinoVotes with either ATC or MVU using a mixing parameter α to model votes throughout the UVR.The mixing values can be interpreted as the degree to which voters followed different voting behaviors.
- Variance targeting: Because all component PMFs satisfy E(R|y) = y, the MBV PMF also preserves the desired mean while targeting variances inside the UVR.The target variance vr(y) is combined with the component variances through a mixing function, with separate expressions depending on whether ATC or MVU is used.
5. APPLICATION, ANALYSIS, AND DISCUSSION
The fourth-degree variance fit stays within the UVR for all 16 datasets, with weighted least squares required for three. Dataset examples show that fitted curves and mixture parameters distinguish voting behaviors across experiments.
- Variance fitting: 13 of 16 datasets fit inside the UVR with unweighted least squares, while the remaining three require weighted least squares.
- Dataset comparisons: ITS2013 and NISQA P501 MOS represent the largest and smallest mean variance curves among the 16 datasets, respectively, and both fitted curves closely track their data within the UVR.
- Dataset comparisons: 28% of NISQA P501 MOS variance values—67 of 240 files—lie on the minimum variance curve, with each file receiving 18 to 32 votes that were adjacent values only.
- Mixture behavior: ITS2013 closely matches the BinoVotes variance curve, consistent with mixture parameter values near 1.0.
- Mixture behavior: Evaluating α(y) across file MOS values estimates the relative contributions of BinoVotes, ATC, and MVU to each dataset’s observed vote variances.
- Parabolic model analysis: Parabola scale factors below 0.05 warrant data-integrity examination, whereas values above 0.54 suggest significant multimodal PMF contributions and possible subject-behavior or experimental-design issues.
6. CONCLUSION
The conclusion presents the UVR, UVR-respecting variance fitting, and a mixed-behavior vote model as complementary tools for interpreting subjective vote variance and its behavioral sources.
- The UVR replaces the mathematically possible but unrealistic maximum variance curve with the MVU model’s variance curve.
- The proposed fitting methodology respects the UVR, while the mixed-behavior model quantifies three voting behaviors in subjective test results.
- The fitting and vote-modeling steps can be used independently or together to gain insights into vote variance and behaviors producing it.