Source-linked AI summary
Cross-conformal predictors
Vladimir Vovk
TL;DR
The note addresses how to improve the predictive efficiency of inductive conformal prediction while retaining its validity properties. It introduces cross-conformal prediction, combining predictors built across folds, and evaluates calibration and efficiency empirically. On Spambase, the cross-conformal predictors appear well-calibrated and provide more stable, higher confidence than inductive conformal predictors, although their validity lacks a theoretical result.
Problem
Inductive conformal prediction is computationally efficient but typically less predictively efficient, motivating a hybrid that uses more training data for calibration and prediction-rule development.
Method
Cross-conformal prediction builds separate inductive conformal predictors across training-set folds and combines their p-values, using averaging in the main method.
Results
On Spambase, cross-conformal calibration plots are visually close to the bisector, and its confidence values have much lower standard deviation and are higher than those of inductive conformal prediction.
Takeaways & Limitations
The empirical studies support cross-conformal prediction as a more stable and efficient alternative to inductive conformal prediction in the examined setting.
Takeaways & Limitations
There are no theoretical results about the validity of cross-conformal predictors, making establishment of such results an open problem.
Abstract
from arXiv · showhide
This note introduces the method of cross-conformal prediction, which is a hybrid of the methods of inductive conformal prediction and cross-validation, and studies its validity and predictive efficiency empirically.
1 Introduction
Cross-conformal prediction is introduced as a hybrid of inductive conformal prediction and cross-validation, intended to retain validity while improving predictive efficiency. The method combines p-values from predictors built across training-set folds, and its empirical calibration and efficiency are studied on Spambase.
- Conformal prediction provides set predictions with unconditional coverage at least equal to a preset confidence level.
- Inductive conformal prediction is more computationally efficient but typically produces larger prediction sets than conformal prediction.
- Cross-conformal prediction splits the training set into K folds and constructs an inductive conformal predictor using each fold for calibration and the remaining data for training.
- The resulting p-values are merged into combined p-values, which form the procedure’s output.
- Averaging p-values is used in the main method because Fisher’s method assumes independence and produces badly miscalibrated results here.
2 Inductive conformal predictors
Inductive conformal prediction splits training data into proper-training and calibration sets, scores conformity, and outputs labels whose p-values exceed the chosen significance level. Under i.i.d. examples, its error probability is bounded by that level, while the proper-training/calibration split creates bias–variance trade-offs.
- Inductive conformal prediction partitions the training set into a proper training set and a calibration set.
- A conformity measure scores how well each calibration or test example conforms to the proper training set.
- The predictor outputs the set of labels whose p-values exceed the chosen significance level ϵ.
- Under i.i.d. random examples, the probability of an error does not exceed ϵ for any inductive conformal predictor.
- A 2:1 proper-training/calibration split is standard, but too little calibration data raises confidence variance while too little training data biases confidence downward.
3 Cross-conformal predictors
Cross-conformal prediction combines fold-specific inductive conformal predictors, using the full training set for calibration and substantial training subsets for prediction rules. In Spambase experiments, its calibration was visually good and its confidence values were more stable than ICP’s.
- Method: Cross-conformal prediction splits the training set into K folds and constructs one inductive conformal predictor per fold, using the remaining folds for training.Calibration combines the test example’s ranks across folds.
- Method: The combined p-value is essentially the arithmetic mean of the separate fold-specific p-values when folds have equal size.The note gives this relationship explicitly for equal-sized folds.
- Experiments: The experiments use 5-fold and 10-fold CCP on the 4,601-example Spambase dataset, with MART producing the conformity measure.Spambase labels spam as 1 and email as 0.
- Results: Figure 1’s calibration plots for CCP and random train/test and fold splits are visually close to the first-quadrant bisector.The plotted functions map significance levels to percentages of erroneous set predictions.
- Results: CCP’s main efficiency advantage is substantially lower standard deviation of mean confidence than ICP, while CCP also gives higher confidence.For the first 100 seeds, ICP’s average mean confidence is reported as 99.16%, with standard deviation 0.149%.
4 Conclusion
Conformal and inductive conformal predictors have known tolerance-region validity results, but cross-conformal predictors lack theoretical validity results in this note. The note therefore leaves establishing CCP validity as an open problem.
- Conclusion: Conformal and inductive conformal predictors can be expressed as 1−ϵ expectation tolerance regions.Inductive conformal predictors are also described as 1−δ tolerance regions for a proportion 1−ϵ under suitable parameters.
- Conclusion: Cross-conformal predictors have no theoretical validity results in this note, making establishment of such results an open problem.The empirical efficiency results are not presented for naive CCP because efficiency without validity is considered meaningless.
A An approach based on Fisher’s method
The Fisher-based approach combines fold-specific p-values using Fisher’s method, but its independence assumption conflicts with the heavy dependence among folds, producing badly miscalibrated predictions.
- Fisher’s method combines K p-values into a statistic assumed to follow a chi-squared distribution with 2K degrees of freedom.The resulting p-value is denoted F(p1, . . . , pK).
- Naive cross-conformal predictors use the same construction as cross-conformal predictors, replacing their fold-combination function with Fisher’s method.
- The naive predictors’ set predictions are badly miscalibrated because p-values computed from different folds are heavily dependent.
- The authors omit efficiency results for naive cross-conformal prediction because efficiency without validity is meaningless.