Source-linked AI summary
A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics
M. Baak, R. Koopman, H. Snoek, S. Klous
TL;DR
The paper addresses the lack of a correlation coefficient with Pearson-like interpretation across categorical, ordinal, and interval variables. It proposes φK from refinements of Pearson’s χ2 independence test, combining mixed-type and nonlinear correlation measurement with significance and contingency-table interpretation. The authors report practical use cases and a publicly available Python library, while noting limitations in independent-frequency estimation and small-sample behavior of existing measures.
Problem
Existing coefficients do not provide Pearson-like characteristics and sound statistical interpretation consistently across interval, ordinal, and categorical variables.
Method
φK is derived from Pearson’s χ2 contingency test and includes significance evaluation plus outlier-frequency analysis for interpreting contingency-table dependencies.
Results
The paper presents φK as stable against interval-variable binning, corrected for statistical noise, and useful for mixed-type correlation matrices and dependency interpretation.
Takeaways & Limitations
The methods support correlation analysis across mixed variable types, nonlinear relationships, and practical applications through a publicly available Python library.
Takeaways & Limitations
The independent-frequency expectation uses fewer statistics and is therefore slightly less accurate than the alternative, while not being a true product of factorized marginal distributions.
Abstract
from arXiv · showhide
A prescription is presented for a new and practical correlation coefficient, $φ_K$, based on several refinements to Pearson's hypothesis test of independence of two variables. The combined features of $φ_K$ form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation between variables with mixed types. Particular emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples and significant dependencies. Three practical applications are discussed. The presented algorithms are easy to use and available through a public Python library.
1 Introduction
The paper introduces φK as a practical correlation coefficient intended to provide Pearson-like behavior across categorical, ordinal, and interval variables. It combines correlation measurement, significance evaluation, and contingency-table interpretation, including nonlinear dependencies and sparse samples.
- Motivation: The paper identifies a gap for a Pearson-like coefficient with sound statistical interpretation across interval, ordinal, and categorical variables.Pearson’s coefficient is described as applying only to interval variables.
- φK properties: φK uniformly treats interval, ordinal, and categorical variables, with values from 0 for no association to +1 for complete association.It is equivalent to Pearson’s coefficient for bivariate normal inputs and also captures nonlinear relations.
- Method: The coefficient is derived from Pearson’s χ2 contingency test of independence between variables in a contingency table.Rows and columns represent variable categories, while cells count records occurring in paired categories.
- Statistical significance: The paper provides a robust prescription for evaluating statistical significance, addressing incorrect asymptotic p-values in sparse and low- to medium-statistics samples.The prescription adjusts the χ2 distribution when using the G-test statistic.
- Interpretation and applications: Significant excesses and deficits in contingency-table cells are evaluated as outlier frequencies to help interpret dependencies between variables.The paper also discusses applications including insurance claims, survey responses, and clustering compatibility.
2 Measures of variable association
Existing association measures differ in the variable types they support, relationships they capture, and how readily their values can be interpreted. The paper introduces φK to address these issues across interval, ordinal, and categorical variables.
- Interval variables: Pearson’s coefficient is standard for interval variables but measures linear relationships, misses non-linear dependencies, and cannot directly evaluate categorical variables.Ordinal variables require ranking before Pearson’s coefficient can be used.
- Ordinal variables: Rank correlations support ordered variables but implicitly treat category ranks as equidistant, an assumption that can be difficult to justify.Adding categories can change the implied distances between existing levels.
- Categorical variables: Categorical association tests compare the joint distribution p(x, y) with the factorized product p(x)p(y), but their statistics depend on table size and record count.Their values are not generally bounded between 0 and 1 when applied to interval or ordinal data.
- Existing alternatives: Cramér’s φ ranges from 0 to +1 but varies with the contingency table’s binning and can become large from statistical fluctuations in low-statistics tables.For binned bivariate normal data, it is low for most Pearson ρ values and rises toward one only near ρ = 1.
- φK: The paper defines φK from Pearson’s χ2 contingency test and derives its statistical significance using the G-test.This prescription is intended to address the limitations of existing coefficients across variable types.
3 Test of variable independence
The contingency test evaluates whether variables are independent by comparing observed and expected cell frequencies. The paper uses dependent estimates for φK and independent estimates for interpreting individual contingency-table excesses and deficits.
- Contingency test: The contingency test assesses whether the joint distribution p(x, y) is described by the factorized marginals p(x)p(y), including for ordinal and binned interval variables.Each table cell records the frequency of a pair of categories.
- Test statistics: Pearson’s χ2 test defines φK, while the G-test evaluates the statistical significance of the contingency test.Both tests compare observed and expected frequencies across non-empty cells.
- Dependent estimates: Expected frequencies from the default method include the cell’s observed frequency, making them statistically dependent on that observation.The paper uses this estimate for φK because it matches observed frequencies most closely.
- Independent estimates: The ABCD estimate excludes the target cell’s observation, enabling an objective observed-versus-expected comparison for each bin.Its denominator can become zero for low-statistics samples, and it is slightly less accurate than the dependent estimate.
- Outlier interpretation: The paper uses independent frequency estimates to detect significant excesses or deficits of records in contingency tables.These outliers are evaluated with a statistically independent expected frequency.
4 Definition of φK
φK is constructed by mapping a contingency-table χ2 statistic onto the correlation scale of a fitted bi-variate normal model, with a sample-specific correction for statistical noise. The procedure supports mixed variable types, captures nonlinear dependence, and reproduces Pearson’s correlation for bi-variate normal inputs.
- Definition of φK: φK is obtained by inverting the χ2 contingency-test statistic while interpreting it through a bi-variate normal model.The procedure is described for two variables and can be extended to more variables.
- Statistical noise correction: A sample-specific noise pedestal accounts for statistical fluctuations using the effective number of degrees of freedom and the number of expected empty cells.The pedestal is configurable through c, with default c = 0; values below the resulting noise threshold yield φK = 0.
- Definition of φK: The calculation bins unbinned interval variables, fills a contingency table, evaluates Pearson’s χ2 test, and maps the result to a correlation parameter.The default setting uses 10 uniform bins per interval variable, although binning is generally use-case specific.
- Properties: φK treats categorical, ordinal, and interval variables uniformly, captures nonlinear relations, and is equivalent to Pearson’s correlation for bi-variate normal inputs.Unlike Cramér’s φ, its value is reported as stable against the number of bins chosen per interval variable.
- Example correlation matrix: A φK correlation matrix summarizes all pairwise dependencies in mixed-type data, while global correlations gk estimate how well each variable can be modeled from the others.The car-insurance illustration uses 2000 records and five correlated variables of mixed types.
- Empirical behavior: In synthetic benchmarks, φK captures nonlinear correlations for which Pearson’s ρ is close to zero, while its noise correction affects small samples most strongly.For 1000 samples of 500 records with zero true correlation, the threshold assigns about 50% of samples φK = 0; for larger meaningful correlations, the pedestal’s impact is limited.
5 Statistical significance
The paper argues that correlation magnitude and statistical significance must be evaluated together, especially when contingency-table approximations fail for sparse or low-statistics data. It therefore uses effective degrees of freedom and a modified χ2 distribution to obtain more reliable p-values across sample sizes.
- The usual χ2 approximation breaks down for sparse or low-statistics contingency tables, where effective degrees of freedom can be below the theoretical value.Cochran’s rule recommends that at least 80% of expected cells have at least 5 counts and none have fewer than 1.
- A hybrid approach fits an analytical empirical description of the χ2 distribution using a limited number of Monte Carlo simulations.The method introduces corrections for effective degrees of freedom and low-statistics behavior.
- For the smiley data set, the advocated ndof = 361 is reached only at very large sample sizes (≥10^6), while effective degrees of freedom drop at smaller N.The decrease is slightly steeper for the G-test statistic.
- At low statistics, Pearson χ2 distributions are wider and G-test distributions narrower than the nominal χ2 distribution, distorting p-values when theoretical ndof is used.Using the effective number of degrees of freedom avoids the uneven p-value distribution observed when ndof is used.
- The recommended p-value calculation uses the modified χ2 distribution h(x|f) with nedof, because naive calculations can underestimate significance by multiple Z units.In one example, Z rises from 1.9 with asymptotic χ2 and nedof to 2.5 with h(x|f), while the three approaches differ by more than 8 units overall.
- Correlation magnitude and statistical significance can lead to opposite impressions, so both should be studied together.
6 Interpretation of relation between two variables
The paper interprets dependencies by testing cell-level excesses and deficits against statistically independent expected frequencies. These outlier significances help reveal which contingency-table cells drive relationships, including in low-statistics samples.
- Cell-level outlier analysis compares observed and expected frequencies to identify significant excesses or deficits in a contingency table.The expected frequency and its uncertainty are evaluated independently of the observed cell count.
- Each cell’s significance is evaluated with a background-only versus signal-plus-background hypothesis test for a Poisson process.The calculation uses the observed count, expected count, and uncertainty on the expectation.
- Four corrections handle zero counts, zero uncertainty, zero expected frequencies, and the continuous statistic produced by combining counts with uncertain expectations.The final correction applies Lancaster’s mid-P value.
- For low-statistics samples, Lancaster’s mid-P correction centers background-only Z distributions near zero, while fewer than 1.0 records per bin produces more distorted distributions.With an average of 2.0 records per cell in 200-record samples, the Z distribution remains fairly close to normal.
- Cells with |Z| greater than a chosen threshold, such as 5 standard deviations, are filtered as significant excesses or deficits.For dependent variables, excesses and deficits appear together because the frequency estimates smooth the input distribution.
- In the synthetic insurance examples, black cars are overrepresented in suburbs and underrepresented downtown, while XXL cars have significantly higher mileage.Large absolute Z-values in contingency-table cells show the variable dependency and provide practical interpretation, especially for categorical variables.
7 Three practical applications
The paper applies φK-based correlation analysis to mixed-type modeling, survey interpretation, and clustering compatibility. These applications use correlations, significance, and contingency-table patterns to identify dependencies and guide follow-up analysis.
- Mixed-type variable pairs can be screened for unexpected correlations, evaluated for statistical significance, and interpreted through their dependencies.These three capabilities are presented as a general workflow for analyzing sets of mixed-type variables.
- 7.1 Modeling the frequency of insurance claims: Insurance-claim frequency modeling combines categorical and interval variables, including car type, residential area, mileage, and driver age.The application focuses on modeling expected claim counts across policy-holder classes with a generalized linear model.
- 7.1 Modeling the frequency of insurance claims: Initial claim-frequency models assume independent inputs and assign one-hot-encoded categories or interval bins separate regression parameters.Low-statistics categories are commonly merged until they contain sufficient records.
- 7.1 Modeling the frequency of insurance claims: Cross-terms can refine factorized claim estimates, but the exponentially large candidate set motivates selecting terms through φK values, significances, and correlation-plot outliers.The proposed workflow selects strongly correlated variable pairs, then identifies relevant model cross-terms and can constrain their number with shared parameters.
- 7.2 Interpreting survey responses: For survey responses, question pairs are selected using interesting φK correlations and significance, while high absolute Z values identify answer pairs deviating from independence.The resulting list contains all answer-pairs significantly departing from no correlation, including the unexpected subset.
- 7.3 Studying clustering compatibility: For clustering compatibility, cluster IDs from multiple algorithms support a correlation matrix and analysis of where two algorithms’ cluster assignments overlap or deviate.This compares the outputs of unsupervised clustering algorithms at the record level.
8 Public implementation
The φK correlation analyzer is publicly available as a Python library. Documentation provides source-code descriptions, setup guidance, tutorials, and working examples.
- The φK correlation analyzer can be installed as a Python library through PyPi and accessed from GitHub.The paper provides the project repository and an installation command.
- The project documentation describes the source code and includes setup tutorials and working usage examples.The documentation is hosted at phik.readthedocs.io.
9 Conclusion
The paper presents φK as a χ2-test-based correlation coefficient for mixed variable types, with Pearson-like behavior and noise correction. It also proposes a hybrid significance procedure for sparse data and evaluates outlier frequencies for interpreting dependencies.
- φK applies to interval, ordinal, and categorical variables alike, has Pearson-like behavior, remains stable against interval-variable binning, and includes noise correction.The coefficient is based on the χ2 contingency test and is intended to be practical for mixed-type analysis.
- The asymptotic significance approximation breaks down for sparse and low-statistics data, motivating a hybrid approach using Monte Carlo simulations and an empirical χ2 distribution description.The procedure uses the G-test statistic to estimate effective degrees of freedom before fitting the analytical description.
- Statistical significance of outlier frequencies relative to the factorization assumption helps interpret dependencies, including dependencies between categorical variables.The paper evaluates outlier frequencies against the expected independent structure.
- Three use-cases cover insurance claims, survey responses, and clustering compatibility, supported by a publicly available Python analysis library.The paper presents these applications as practical examples of the methods.