Source-linked AI summary
A practical guide and software for analysing pairwise comparison experiments
Maria Perez-Ortiz, Rafal K. Mantiuk
TL;DR
Pairwise comparisons are easy to collect, but analyzing them into an interpretable scale is more complex than testing for statistical significance alone. The paper presents scaling methods and analysis software, with design guidance and improvements for low-observer settings. Its conclusions describe competitive performance for appropriately chosen incomplete designs, while noting limitations of the assumed observer model.
Problem
Pairwise-comparison analysis is difficult, and statistical significance alone may not show the practical magnitude of differences between conditions.
Source: §1 · p. 1Method
The paper explains pairwise-comparison scaling and provides analysis methods, a low-observer prior, experimental-design guidance, and a Matlab toolbox.
Source: §1 · p. 2Results
Incomplete designs can achieve competitive performance when comparisons are appropriately chosen, while ties generally underestimate the scaling solution.
Source: §11 · p. 18Takeaways & Limitations
Pairwise scaling can rank similar-quality items, support incomplete designs, express differences in probability-related units, and address measurement noise in a principled way.
Source: §11 · p. 18The method assumes a simple Thurstone Case V observer model, without dependence between repetitions, observers, or conditions, and represents quality with one scalar value.
Source: §11 · p. 18
Abstract
from arXiv · showhide
Most popular strategies to capture subjective judgments from humans involve the construction of a unidimensional relative measurement scale, representing order preferences or judgments about a set of objects or conditions. This information is generally captured by means of direct scoring, either in the form of a Likert or cardinal scale, or by comparative judgments in pairs or sets. In this sense, the use of pairwise comparisons is becoming increasingly popular because of the simplicity of this experimental procedure. However, this strategy requires non-trivial data analysis to aggregate the comparison ranks into a quality scale and analyse the results, in order to take full advantage of the collected data. This paper explains the process of translating pairwise comparison data into a measurement scale, discusses the benefits and limitations of such scaling methods and introduces a publicly available software in Matlab. We improve on existing scaling methods by introducing outlier analysis, providing methods for computing confidence intervals and statistical testing and introducing a prior, which reduces estimation error when the number of observers is low. Most of our examples focus on image quality assessment.
1 Introduction
Pairwise comparisons are simple to run, but turning their responses into a useful quality scale requires more than statistical testing or vote counts. The paper explains scaling and introduces analysis methods and a Matlab toolbox, including a prior for low-observer settings.
1 Introduction
- The paper contributes outlier analysis, confidence intervals, statistical testing, a prior that improves accuracy with few observers, experimental-design guidance, and a Matlab toolbox.These contributions accompany a practical explanation of scaling methods.Source: §1 · p. 2
1.1 Direct rating vs. pairwise comparisons
- Direct ratings require training and can produce scales that vary across participants or sessions, while pairwise comparisons offer a simpler task and avoid calibration issues.Pairwise comparisons are also reported to provide higher sensitivity and lower measurement error than direct rating.Source: §1.1 · pp. 2–3
1.2 Vote counts vs. scaling
- Vote counts usually rank conditions correctly but do not capture difference magnitude, whereas scaling places them on a continuous interval scale and can handle incomplete or unbalanced designs.Scaled data was more strongly related to direct-rating scores than vote counts in a cited comparison.Source: §1.2 · p. 3
2 Related work
Prior work offers many approaches to aggregate pairwise judgments, including Thurstone and Bradley–Terry models and extensions for ties, covariates, or multiple dimensions. This paper emphasizes practical scaling guidance and software for analysis, experimental design, and low-sample scenarios.
2 Related work
- Thurstone and Bradley–Terry are common models; extensions add ties, preference strength, observer or condition covariates, and temporal effects, often requiring more data.The paper uses the Thurstone model; the choice between the two common models is described as a matter of preference.Source: §2 · pp. 3–4
- Pairwise comparisons can also discover explanatory variables or represent intransitive preferences, but multidimensional solutions are harder to interpret.The cited approaches include regression, multidimensional scaling, and ranking machine learning methods.Source: §2 · p. 4
- For larger quality ranges, the paper says its pairwise-comparison scaling is suitable only when compared conditions differ little enough for observers' answers to vary.Difference scaling is offered as a potentially more appropriate alternative for larger ranges.Source: §2 · p. 4
- The authors focus on practical scaling issues, including experimental design, statistical analysis, and low-sample scenarios, and provide software for scaling and analysis.They note that listed software packages do not offer a complete set of outlier analysis, confidence-interval, and statistical-testing methods.Source: §2 · p. 4
3 Example of pairwise comparison data analysis
The example walks through preparing comparison data, investigating potential outliers, and then scaling responses with confidence intervals and statistical tests. It cautions that a distribution plot is not a substitute for scaling, especially with incomplete designs.
3 Example of pairwise comparison data analysis
- The general-quality distribution plot resembles the scaling here because the example uses a full design, but may differ substantially for incomplete designs.The figure caption warns not to use the distribution plot as a replacement for scaling.Source: Figure 2 · p. 5
- The workflow converts tabulated responses into one comparison matrix per observer, with rows and columns for conditions and entries recording selection counts.A reference condition, if present, is placed first and assigned a fixed quality value of 0.Source: §3 · p. 5
- Potential outliers are flagged for investigation, not automatic removal; the example's observer with an L_dist score of 2.72 was not excluded after detailed comparison.The authors advise examining observers near or above the customary 1.5 threshold rather than using an arbitrary exclusion rule.Source: §3 · p. 5
- Scaling and 95% confidence intervals are computed after outlier review, while a separate statistical test identifies significant condition differences.The confidence intervals should not be used to infer statistical significance.Source: §3 · p. 5, Figure 3 · p. 6, Figure 4 · p. 6
4 Designing pairwise comparison experiments
Pairwise experiment design balances the comparisons needed for useful estimates against participant effort. Selecting informative, neighboring pairs can reduce unnecessary trials, but unanimous answers limit how reliably distances can be estimated.
4 Designing pairwise comparison experiments
- A complete design asks each observer to compare every pair, requiring 1/2·n·(n −1)·k·t comparisons, a count that grows quickly with n.Here, n is the number of conditions, k the number of content items, and t the number of repetitions.Source: §4 · p. 6
- Incomplete designs compare only selected pairs; obvious comparisons, such as the highest versus lowest distortion, can be omitted.If observers receive different designs, the experiment is imbalanced, which is generally inadvisable without accounting for observer covariates.Source: §4 · p. 6
- Comparing neighboring quality levels helps avoid unanimous answers, which do not permit reliable estimation of the scaled difference.The passage recommends comparisons between relatively similar images that observers may judge differently.Source: §4 · p. 7
- Offering ties usually makes modeling more difficult, so the authors recommend two-alternative forced-choice experiments without ties.The paper discusses ties separately as a modeling problem.Source: §4 · p. 7
- The report excludes discussion of viewing-condition control, learning effects, participant training, and experimental fatigue.The authors point readers to other references for these experimental-design factors.Source: §4 · p. 7
5 Problem formulation
The paper formulates pairwise scaling as estimating relative latent quality scores from observers’ choices. It uses a Gaussian observer model and distinguishes quality differences from visual detectability.
5 Problem formulation
- The goal is to estimate scores approximating unknown condition qualities from comparisons, with the first score fixed at zero because comparisons are relative.Comparisons may come from m observers, t trials, and, where relevant, k pieces of content.Source: §5 · p. 7
5.1 Comparison matrix
- The comparison-count matrix C records how often each condition is chosen as better than each other condition.For example, c_ij counts choices of O_i over O_j.Source: §5.1 · p. 7
- The empirical probability that one condition beats another is estimated from the corresponding opposing counts; the example gives 27/(27+3) = 0.9 for O2 over O1.The count matrix therefore supplies pairwise choice probabilities for scaling.Source: §5.1 · p. 7
5.3 JNDs and JODs
- Figure 6 contrasts equal-JOD blur and noise distortions that are nevertheless several JNDs apart, illustrating that the scales measure different things.The mapping between JODs and JNDs may be complex; the illustrated relation is only illustrative.Source: Figure 6 · p. 8, §5.3 · p. 8
5.2 Observer model
- Thurstone Case V models each condition’s perceived quality as a Normal random variable whose mean is its true quality score.The model attributes variation to observers’ differing judgments and changes across repeated trials.Source: §5.2 · pp. 7–8
- Thurstone Case V assumes the same noise variance for every condition and seeks expected quality scores, or their relative distances.The shared noise parameter represents uncertainty in comparisons.Source: §5.2 · p. 8
5.3 JNDs and JODs
- JODs describe quality differences relative to a perfect-quality reference, rather than visual differences between distorted images.The measure is described as closer to visual equivalence or quality expressed as a difference-mean-opinion-score than to JNDs.Source: §5.3 · p. 8
6 Scaling methods
The scaling methods convert pairwise choice probabilities into relative quality distances, then estimate scores that fit those distances or maximize the likelihood of the observed choices. The paper favors maximum likelihood estimation for comparison-count weighting and difficult response patterns.
6.1 From probabilities to distances
- The inverse cumulative Normal function maps choice probabilities to signed JOD distances; a 0.75 preference for the first condition corresponds to one JOD.A random-guess probability of 0.5 maps to zero distance, and negative distances indicate more frequent selection of the second condition.Source: Figure 7 · p. 9, §6.1 · p. 9
6.3 Maximum likelihood estimation
- MLE accounts for the number of comparisons: larger samples yield narrower ranges of likely score differences.This is useful for unbalanced designs.Source: Figure 8 · p. 10, §6.3 · p. 11
6.2 Least-square distance solution
- Because pairwise distances cannot determine absolute score values, least-squares scaling fixes one score, commonly at zero.The objective minimizes squared mismatch between estimated score differences and the signed distances.Source: §6.2 · p. 10
- Least-squares scaling fits score differences to the transformed distance matrix, but unanimous choices yield infinite distances and make this solution infeasible.Ignoring unanimous responses discards valid data, while restricting distances introduces bias.Source: §6.2 · pp. 9–10
6.3 Maximum likelihood estimation
- Maximum likelihood estimation (MLE) selects quality-score differences that maximize the probability of the observed comparison counts across compared pairs.The pairwise choice probability is linked to score differences through the cumulative Normal distribution.Source: §6.3 · p. 10
- With unanimous choices, MLE retains a likelihood over possible distances rather than forcing an infinite estimate; smaller distances remain more plausible when observer counts are low.For the illustrated unanimous-choice cases, the most likely distance exceeds 5, but smaller distances also retain likelihood.Source: Figure 9 · p. 11, §6.3 · p. 11
- MLE can scale incomplete designs in which only a subset of condition pairs has been compared.The method maximizes the likelihood over pairs with at least one comparison.Source: §6.3 · pp. 10–11
7 Statistical analysis
The paper estimates uncertainty in scaled scores by bootstrapping comparison data and tests score differences using their covariance. Simulations indicate that bootstrap intervals track true intervals on average, though estimates are less dependable with few observers.
7.1 Confidence intervals
- Bootstrapping resamples participants with replacement, scales each pseudo-sample using MLE, and takes the 2.5th and 97.5th percentiles of JOD scores as 95% intervals.The paper says a large number of pseudo-samples, usually more than 500, are generated.Source: §7.1 · p. 11
- On average, bootstrap confidence intervals closely follow the true intervals in a 10,000-experiment Monte Carlo comparison, but may be under- or overestimated, especially with few observers.The figure compares bootstrap estimates with the simulated true intervals and shows variation across 50 bootstrap runs.Source: Figure 11 · p. 12, §7.1 · p. 12
- Confidence intervals widen both farther from the reference score and as the distance between conditions grows.Errors propagate through linked score estimates; at large JOD distances, small probability errors can produce large distance errors.Source: §7.1 · p. 12, Figure 10 · p. 12
- Adding the distance prior reduces bias in simulated JOD estimates, whereas removing unanimous comparisons shifts estimates toward underestimating distances.Figure 13 compares the original MLE, the prior-augmented estimate, and estimates with unanimous comparisons removed.Source: Figure 13 · p. 13
7.2 Statistical difference between two conditions
- Because condition estimates are correlated, testing a JOD difference requires its variance from the covariance matrix, followed by a two-tailed test.The covariance captures how changing one condition’s value affects directly or indirectly linked conditions.Source: §7.2 · p. 13
8 Finite distance prior
The section addresses bias from unanimous answers by introducing a distance prior for scaling. It also evaluates the prior’s accuracy and robustness, particularly with small samples.
8 Finite distance prior
- Unanimous answers leave distances between conditions unbounded, biasing estimates; this problem is most noticeable with few observers.Ignoring unanimous answers instead leads to underestimated JOD values.Source: §8 · pp. 13–14
- The prior estimates distance probabilities by summing pairwise likelihoods, replacing unanimous responses with the closest non-unanimous response so the distribution can be normalized.The prior changes iteratively with the estimated distances.Source: §8 · p. 14, Figure 15 · p. 14
- The proposed prior modulates rather than constrains distances; adding an offset of γ = 0.1 allows other distances to be selected and reduces bias.With the prior, the likelihood has a single maximum rather than a plateau, improving optimization stability.Source: Figure 14 · p. 14, §8 · p. 15
- The proposed distance prior significantly improves estimation accuracy and robustness, especially for small samples, where RMSE and confidence intervals can be very large with fewer than 20 observers.The prior is evaluated through simulations across increasing numbers of observers.Source: §8 · p. 15
- The normal prior of Tsukida and Gupta (2011) reduces confidence intervals but introduces large estimation error, measured by RMSE.The comparison is shown for the previously presented metrics.Source: §8 · p. 15, Figure 16 · p. 15
9 Outlier detection
The section presents an outlier-detection method that compares each observer’s answers with the rest of the sample. The method flags potential outliers for experimenter review rather than making the removal decision automatically.
9 Outlier detection
- The method excludes each observer in turn, scales the remaining data with MLE, and computes the likelihood of that observer’s answers under the resulting comparison probabilities.A low likelihood indicates that the observer may not be consistent with the rest of the sample.Source: §9 · p. 16
- Outlier scores use Tukey’s quartile rules on log-likelihoods, and flag only the low-likelihood side by distance from the first quartile.Scores express distance from the central range in multiples of the interquartile range.Source: §9 · p. 16
- Outliers are harder to detect with few repetitions, so the authors recommend at least three repetitions and experimenter review of flagged observers.Automatic criteria were not always accurate on actual datasets, given the subjectivity of the experiments.Source: §9 · p. 16
10 Practical issues
The section examines design completeness, score separation, and ties as practical choices in pairwise comparison experiments. Its simulations show trade-offs in accuracy, stability, and estimation error across these choices.
10.1 Complete or incomplete design
- For the same experimental effort, incomplete designs produce more stable and similarly accurate estimates, while requiring fewer comparisons than full designs.With 10 conditions, the example requires 45 pairs for full design and 9 for incomplete design.Source: §10.1 · pp. 16–17, Figure 17 · p. 17
10.2 Distance between quality scores
- Scaling becomes especially unreliable when quality-score distances exceed 2 JODs; for larger ranges, the authors suggest difference scaling as more appropriate.In the full-design simulation, RMSE and confidence intervals increase rapidly as score distances grow.Source: §10.2 · p. 17, Figure 18 · p. 17
10.3 Experiments with ties
- Offering a “no-preference” option reduces confidence-interval size and improves effect size, but increases error and produces underestimates.The resulting negative bias can offset gains from reduced confidence intervals; eliminating it requires modeling tie selection and collecting responses for identical conditions.Source: §10.3 · p. 18
11 Conclusions and limitations
Pairwise scaling offers practical advantages over simpler analysis, and the experiments identify design choices that affect its performance. The method assumes a simple Thurstone Case V observer model and a single scalar quality value.
11 Conclusions and limitations
- Pairwise scaling can rank similar-quality items, support incomplete designs, provide interpretable difference measures, and address measurement noise in a principled way.Incomplete designs can decrease the amount of data collected while still allowing accurate predictions.Source: §11 · p. 18
- Incomplete designs can perform competitively when comparisons are appropriately chosen, while ties tend to underestimate scaling and compared conditions should differ relatively little.The experiments also show the importance of the finite distance prior and screening outlier observers.Source: §11 · p. 18
- The work assumes a simple Thurstone Case V observer model, which omits dependence among repetitions, observers, and conditions and represents quality with a single scalar.Source: §11 · p. 18
- Future work would extend the analysis and software with adaptive sampling to reduce required comparisons and machine learning to link explanatory variables to scaling.Source: §11 · p. 18