Source-linked AI summary
An Aligned Rank Transform Procedure for Multifactor Contrast Tests
Lisa A. Elkin, Matthew Kay, James J. Higgins, Jacob O. Wobbrock
TL;DR
Multifactor HCI experiments often produce nonconforming data, and the popular ART procedure can yield incorrect multifactor contrast results. The paper introduces ART-C, an aligning-and-ranking procedure validated on 72,000 synthetic data sets, and finds that it controls Type I error without inflation while offering higher power than several comparison tests and ART.
Problem
ART can detect main effects and interactions in nonconforming factorial data but lacks a suitable method for multifactor contrasts, which compare levels across multiple factors.
Method
The authors developed ART-C, a new aligning-and-ranking procedure for multifactor contrasts, and validated it through simulation studies using 72,000 synthetic data sets.
Results
ART-C does not inflate Type I error and has higher statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART.
Takeaways & Limitations
ART-C provides a nonparametric technique for multifactor contrasts within the ART paradigm, supported by updated ARTool tools for R and Windows.
Takeaways & Limitations
Validation omitted cases with condition-specific location differences, mixed factorial designs, and random slopes, so these settings remain outside the tested scope.
Abstract
from arXiv · showhide
Data from multifactor HCI experiments often violates the normality assumption of parametric tests (i.e., nonconforming data). The Aligned Rank Transform (ART) is a popular nonparametric analysis technique that can find main and interaction effects in nonconforming data, but leads to incorrect results when used to conduct contrast tests. We created a new algorithm called ART-C for conducting contrasts within the ART paradigm and validated it on 72,000 data sets. Our results indicate that ART-C does not inflate Type I error rates, unlike contrasts based on ART, and that ART-C has more statistical power than a t-test, Mann-Whitney U test, Wilcoxon signed-rank test, and ART. We also extended a tool called ARTool with our ART-C algorithm for both Windows and R. Our validation had some limitations (e.g., only six distribution types, no mixed factorial designs, no random slopes), and data drawn from Cauchy distributions should not be analyzed with ART-C.
1 INTRODUCTION
ART is widely used for nonconforming multifactor HCI data, but its original procedure is unsuitable for multifactor contrasts. This work introduces ART-C, validates its error control and power, and integrates it into ARTool.
- Background: ART supports nonparametric detection of main and interaction effects when multifactor HCI data violate normality assumptions.It aligns and ranks responses before applying an omnibus test, typically ANOVA.
- Problem: The original ART can produce incorrect results when used for multifactor contrast tests.The paper distinguishes ART’s success for omnibus effects from its lack of a suitable contrast method.
- Contribution: ART-C is a new aligning-and-ranking algorithm for conducting multifactor contrasts within the ART paradigm.The authors validated it through simulation studies and compared it with established contrast tests and ART.
- Results: ART-C’s Type I error rates are generally near expected values, while original ART contrast tests often have far-from-expected rates and low power.The comparisons included a t-test, Wilcoxon signed-rank test, Mann-Whitney U test, and original ART.
- Limitations: The validation excluded mixed factorial designs and random slopes, and ART-C should not be used for data with Cauchy-distributed residuals.The validation also covered only six population distributions and selected sample-size and factor-design ranges.
- Practical support: ART-C was added to the ARTool R package and Windows application, enabling multifactor contrasts without leaving the ART workflow.The updated tools are intended to make correct analyses easier for HCI researchers and other users.
2 RELATED WORK
Prior work established ART for nonparametric factorial analysis but did not provide a clearly suitable, validated method for multifactor contrasts. ART-C addresses this gap while extending usable statistical tooling for HCI.
- Prior methods: ART was developed to avoid the inflated interaction error rates associated with the earlier rank-transform procedure.ART enables ANOVA-like nonparametric analysis of factorial experiments.
- Gap: ART works well for detecting main effects and interactions but lacks a suitable method for multifactor contrast tests.Contrasts identify differences between factor levels after or alongside omnibus effects.
- Gap: Existing ART contrast studies demonstrated only single-factor examples or simulations, leaving applicability to multifactor contrasts unspecified.This limitation persisted even where studies considered interactions or two-factor designs.
- Novelty: The new method changes the alignment process itself rather than merely substituting a different alignment statistic.This distinguishes ART-C from Peterson et al.’s comparison of six alignment statistics.
- Usability: ART-C and updated ARTool versions lower the barrier to conducting correct multifactor contrast analyses on nonconforming data.The tools target both the HCI community and users beyond HCI.
3 THE PROBLEM: MULTIFACTOR CONTRASTS IN ART
ART is effective for detecting multifactor main and interaction effects, but using it for multifactor contrasts can produce incorrect conclusions. The running example illustrates both false positives and false negatives, motivating a different contrast procedure.
- The problem: ART detects main effects and interactions, but lacks a suitable method for multifactor contrast tests.Contrasts compare levels within factors or combinations of levels from multiple factors.
- The problem: Multifactor contrasts on ART-aligned-and-ranked data can produce incorrect results.Single-factor contrasts can be conducted safely with a t-test on ART-processed data, whereas multifactor contrasts cannot.
- Running example: In the running example, the data were generated with no true difference between (A1, B1) and (A1, B2), but a true difference between (A1, B1) and (A2, B2).The example uses a within-subjects experiment with three two-level factors and log-normal data.
- Running example: LMM contrasts on log-transformed data matched the ground truth: p = .1792 for the nonexistent difference and p < .0001 for the true difference.The contrasts averaged over factor C, which was not directly involved in the comparisons.
- Running example: ART contrasts produced both errors: p < .0001 falsely detected the nonexistent difference, while p = .9144 missed the true difference.These are respectively a Type I error and a Type II error.
4 THE SOLUTION: AN ALGORITHM FOR ART CONTRASTS
ART-C adapts alignment and ranking specifically for multifactor contrasts, rather than applying ART’s original alignment directly. It then conducts contrasts on the resulting aligned-and-ranked responses while preserving the relevant factorial structure.
- ART-C procedure: ART-C aligns data for the desired contrast tests, not for generic main effects or interactions.The procedure is designed to prevent the incorrect multifactor contrast results associated with using ART directly.
- Validation results: ART-C’s Type I error rates are closer to α = .05 than ART’s across all distributions, while its power is higher except for Cauchy distributions.The figure comparisons use proximity to .05 as the correctness criterion and larger values as the power criterion.
- ART-C procedure: Each desired contrast requires separately aligning and ranking the response for the factors whose levels will be compared.For example, contrasts involving A and B require separate processing from contrasts involving A and C.
- ART-C procedure: ART-C prepares data by concatenating the contrast factors, removing their original copies, and retaining uninvolved factors unchanged.For contrasts on A and B, these become a concatenated AB factor while C remains as-is.
- ART-C procedure: After alignment, ART-C assigns ascending midranks to all aligned responses and conducts contrasts on the resulting ranked responses.Tied values receive the mean of the ranks they would otherwise occupy.
- ART-C procedure: ART-C uses a full factorial model containing the retained factors, but interprets only comparisons among levels of the concatenated contrast factor.Comparisons involving non-concatenated factors are considered meaningless, and omnibus results for this model are ignored before the intended contrasts.
5 VALIDATING OUR APPROACH
The validation used 72,000 synthetic data sets spanning experimental layouts, distributions, sample sizes, and between- or within-subjects designs to assess ART-C's Type I error rates and power. ART-C generally maintained appropriate error rates and exceeded comparator methods in power, with Cauchy-distribution data as an important exception.
- Type I error: For Cauchy-distribution designs with contrast size one and 3 × 3 or 2 × 2 × 2 layouts, ART-C's observed Type I error rate was inflated to M= .373 and SD= .076.The corresponding t-test rate was M= .025 and SD= .004, so these designs were treated as outliers and excluded from the remaining Type I error analysis.
- Type I error: ART-C observed Type I error rates clustered around .05 on the remaining designs, with M= .050 and SD= .009, comparable to the t-test's M= .048 and SD= .012.Cauchy designs were excluded from this analysis because ART-C produced egregiously inflated error rates there.
- Statistical power: ART-C had the highest average observed power at M= .598 and SD= .143, compared with M= .521 and SD= .149 for Mann-Whitney U/Wilcoxon signed-rank tests and M= .461 and SD= .149 for the t-test.ART-C exceeded the t-test for every non-normal distribution, matched it for the normal distribution, and exceeded the Mann-Whitney U and Wilcoxon tests across all distributions.
- Comparison to ART: Compared with original ART contrasts, ART-C had lower observed Type I error, M= .067 versus M= .121, and higher observed power, M= .598 versus M= .511.ART-C's power was higher than ART's for every population distribution except Cauchy, while ART's power decreased as contrast size increased.
- Comparison to ART: In the running example, ART-C matched the linear mixed model and ground truth by detecting one true difference and not detecting one nonexistent difference.The procedure correctly compared levels of factors A and B while averaging over factor C.
6 ARTOOL.EXE AND R PACKAGE “ARTOOL”
The authors extended ARTool.exe and the ARTool R package to support ART-C multifactor contrasts, including user interfaces for selecting contrast factors and producing aligned-and-ranked output.
- ART-C was added to both the open-source ARTool.exe Windows application and the ARTool R package.
- ARTool.exe now lets users request contrasts with a checkbox and select the factors whose levels define the contrast.
- The Windows tool applies ART-C to produce aligned-and-ranked output suitable for analysis.
- In R, ARTool aligns and ranks data for each fixed effect and uses anova to run separate ANOVAs for those effects.
- ART-C correctly replaces ART for multifactor contrasts in the running example, which previously produced incorrect contrast results.
7 DISCUSSION
ART-C avoids inflated Type I error rates and generally improves power for multifactor contrasts, while Cauchy-distributed data remain problematic. The paper also illustrates software workflows for specifying and conducting ART-C contrasts.
- Type I error rates for ART-C were typically clustered around α = .05, supporting its correctness.
- ART-C had more power than a t-test for all non-normal distributions and more power than Mann-Whitney U and Wilcoxon tests for all distributions.
- The largest ART-C power increases occurred for data drawn from lognormal and exponential distributions.
- Single-factor ART-C contrasts on Cauchy-distributed data had high observed Type I error rates because Cauchy means and variances are not well defined.
- Quantile-quantile plots and worm plots can help diagnose fat-tailed residual distributions relevant to ART assumptions.
- ART-C contrasts can be specified in software for multifactor level comparisons and post hoc pairwise tests following significant interactions.
8 LIMITATIONS AND FUTURE WORK
The validation covered a deliberately limited set of distributions, population-location patterns, and model structures. Future work includes broader validation, alternative contrast tests, and additional software support.
- The validation sampled only a finite, non-exhaustive set of layouts, population distributions, and condition sample sizes.
- Validation included either common or differing population locations across all conditions, but not mixed location differences within a data set.
- Conditions within each data set were always drawn from the same distribution, excluding combinations where only some conditions differ in distribution.
- The models included random intercepts but not random slopes, which can better fit subject responses that vary across another variable such as time.
- ART-C was evaluated with t-tests as contrast tests, leaving its performance with other contrast tests for future study.
- Future tools could provide platform-agnostic, no-programming ART-C workflows and packages for software such as SAS and SPSS.
9 CONCLUSION
The paper introduces and validates ART-C as a higher-power alternative for multifactor contrasts within the ART paradigm. Its open-source implementations aim to make granular analysis of nonconforming multifactor HCI data more accessible.
- ART was shown to inflate Type I error rates and have low statistical power for multifactor contrasts.
- ART-C was validated on 72,000 synthetic data sets representing properties commonly found in HCI and statistics.
- ART-C did not inflate Type I error and had higher statistical power than t-tests, Mann-Whitney U tests, Wilcoxon signed-rank tests, and ART.
- ART-C was added to the ARTool R package and ARTool.exe Windows application.
- The method is intended to let researchers investigate nonconforming multifactor data at finer granularity without leaving the ART paradigm or requiring specialized statistical expertise.