Seer: Maximum Likelihood Regression for Learning-Speed Curves

1. Introduction

• Regression can indicate whether a required accuracy is achievable and estimate how many classified cases are needed, addressing costly case collection.The introduction notes that collecting and encoding a case in equipment and medical diagnosis can take days and considerable cost.Source: §1 · p. 6
• Seer combines more useful learning-performance models and efficient maximum-likelihood fitting with C4.5 validation on soybean disease, heart disease, and audiological problems.The thesis also presents Seer as a practical regression tool for empirical machine learning.Source: §1 · pp. 6–7

1.1. Inductive Classification Learning

• Inductive classification learning, as modeled by Seer, uses classified training examples to produce a rule whose accuracy is assessed on labeled test examples.The task also includes an example space, class space, target function, and an unknown fixed distribution from which examples are drawn.Source: §1.1 · pp. 7–8

1.2. Regression on Inductive Learning

• Seer uses classified examples and a learner to generate learning-performance data, then fits a model that answers questions about required sample size and achievable accuracy.The model can also characterize performance in terms of VC dimension and the effects of class frequencies.Source: §1.2 · pp. 8–11
• 218 examples were predicted to yield 78.8% accuracy, the model predicted a maximum accuracy of 86.2%, and it assigned the task a VC dimension d=0.559.The observed accuracy with 218 examples was reported as very close to the prediction; the model also had start=0.149 and skew=1.191 parameters.Source: §1.2 · pp. 10–11

1.3. Overview of Thesis

• The thesis previews prior work on learning performance, learning-curve forms, and skew and multiple classes, then introduces models and argues for maximum likelihood over popular alternatives.It says ordinary linear regression is inappropriate for learning-performance models.Source: §1.3 · p. 11
• Chapter 4 frames candidate-model selection as a design problem, exploring deterministic and nondeterministic choices and heuristics for noise, skewed classes, and multiple classes.It also reduces model selection to nonlinear regression solvable by an efficient iterative algorithm.Source: §1.3 · p. 11
• Chapter 5 tests the models in almost 100 experiments on real data from soybean disease, heart disease, and audiological problems, and analyzes the experiments.Source: §1.3 · pp. 11–12
• Chapter 6 concludes the thesis with a summary and suggestions for future work.Source: §1.3 · p. 12

2.1. Theoretical Approaches: Computational Learning Theory

• Computational learning theory offers worst-case learning bounds, but its practical use is limited by simple representations, worst-case assumptions, inability to quantify noise, and focus on consistent algorithms.These limitations make its results poorly suited to predicting average-case learning performance in real-world applications.Source: §2.1 · p. 14

2.2. Empirical Approaches

• Empirical approaches begin with observed learning performance, using observations either to select a model or directly as analysis.Unlike theoretical approaches, which start from a task model to predict or bound performance, empirical approaches may fit candidate models to observations.Source: §2.2 · p. 15

2.2.1. Learning Performance Analysis Without Model Selection

• INDUCT created satisfactory rules with 18 critical cases; it required 326 cases even when correct expert cases contained 25% errors, and 1,970 cases with 10% error and one irrelevant attribute on average.Source: §2.2.1 · p. 15

2.2.2. Learning Performance Analysis With Model Selection

• Thorstone’s typewriter-learning analysis favored a hyperbolic curve, and Mazur and Hastie found the hyperbolic curve fit a word-recall task better than the exponential curve.Newell and Rosenbloom tested the Power Law across a dozen skill-acquisition domains and concluded that practice time approximately follows a power law.Source: §2.2.2 · pp. 16, 18
• Seer models classification accuracy within [0,1], unlike models that can predict invalid accuracies, and models of the same order fit learning-performance data about equally well.The thesis argues that model constraints and parameters, rather than general curve shape, distinguish models.Source: §2.2.2 · pp. 18–19
• Seer models variance as discrete and distinguishes training-example from testing-example variation, while its start and skew parameters represent effects of multiple and skewed classes.Its learning-difficulty parameter d can be calibrated to the Vapnik-Chervonenkis dimension.Source: §2.2.2 · p. 19
• Seer works directly on classification-learning data to find maximum-likelihood models, reducing the fitting task to nonlinear regression solvable by a quickly converging iterative algorithm.The thesis contrasts this with transformation-based regression and least-squares fitting, which can fail on zero observed error and do not generally find the maximum-likelihood model for classification data.Source: §2.2.2 · p. 19

2.3. Effect of Skew and Multiple Classes on Learning Performance

• Seer provides predictive empirical models of how skewed and multiple classes affect learning performance, with class-frequency information constraining the learning-curve form.The thesis describes the work as an application to noisy data from a practical learner such as C4.5 and to natural learning examples, supporting predictions rather than characterization alone.Source: §2.3 · pp. 20–21

2.4. Summary

• The thesis identifies Seer’s contributions as classification-specific models and parameters, efficient maximum-likelihood fitting, and practical demonstration on real-world data.It notes that theory cannot yet predict average-case performance for practical noisy problems using the best machine-learning algorithms.Source: §2.4 · p. 21

3. Overview of Learning-Performance Models

• The chapter’s goal is to build statistical models from learning-performance observations to predict future performance, addressing model-fit evaluation and data generation from few classified examples.Source: §3 · p. 22

3.1. Good-Fitting Models of Learning-Performance

• Section 3.1 asks what makes a learning-performance model fit the observed data well.Source: §3.1 · p. 23

3.1.1. Models of Learning Performance

• A learning-performance model maps training-set size to a probability density for the number of correctly classified test cases, combining a deterministic curve with a discrete density.Given training size m and test size k, it predicts the probability of each possible correct-count y.Source: §3.1.1 · p. 23

3.1.2. Regression Methods

• Learning-performance data are binary-response observations, so continuous-response regression should not be used, especially for small samples or accuracies near 1.Learning curves are typically nonlinear and bounded by 0% and 100%, while error variance shrinks as accuracy approaches 100%; the binomial distribution may not be well approximated by a normal distribution in these conditions.Source: §3.1.2 · pp. 24–25

3.1.3. Maximizing Likelihood

• Maximum-likelihood selection chooses the candidate model that assigns the greatest probability to the observed learning-performance data, equivalently minimizing log likelihood.The equivalence is used to simplify some calculations.Source: §3.1.3 · pp. 25–26

3.2. Generalized Cross-Validation

• With a fixed case library used for both training and testing, generalized cross-validation is introduced as a heuristic for collecting learning-performance data; standard v-fold cross-validation holds each subset out once.The resulting observations record training cases, correctly classified test cases, and total test cases.Source: §3.2 · pp. 26–27
• Generalized cross-validation performs cross-validation for each training-set size of interest, allowing Seer to gather data across many distinct m values.For each m, every case is used for testing once; training subsets are sampled to the desired size.Source: §3.2 · pp. 28–29
• Because test examples are reused across different m values, the learning-performance data are not mutually independent, limiting statistical conclusions such as confidence intervals.The thesis notes that this limitation is inevitable when working with a small amount of data.Source: §3.2 · p. 29

3.3. Conclusion

• The chapter concludes that continuous-response regression methods do not find maximum-likelihood models of learning-performance data.When the total number of classified examples is small, generalized cross-validation can gather the data, at a cost of some statistical conclusions.Source: §3.3 · p. 29

Design and Selection Method

• The chapter frames model selection as two questions: which candidate-model sets to consider and how to algorithmically find the best model from a set.The latter question is addressed through optimization techniques in Section 4.4.Source: Design and Selection Method · p. 31
• Candidate models are designed in three stages: deterministic models for noise-free, equiprobable two-class learning; extensions for multiple, skewed, or noisy classes; and a nondeterministic component.Figure 4.1 maps the design choices and the five tested paths.Source: Design and Selection Method · p. 31, Figure 4.1 · p. 31
• This chapter draws no conclusions about which designs are best; their performance is tested on real learning-performance data in Chapter 5.The candidate paths in Figure 4.1 are the designs put to that test.Source: Design and Selection Method · p. 31, Figure 4.1 · p. 31

4.1. Candidate Deterministic Models

• Deterministic models predict expected accuracy from the number of training examples; the initial candidate setting assumes two equally probable classes and noise-free classification.The section distinguishes this expected-accuracy prediction from the nondeterministic component’s prediction of accuracy error.Source: §4.1 · p. 32

4.1.1. Popular Binary-Response Models

• Logit and probit are common binary-response links, and their corresponding regression models map values from the real line to probabilities between 0 and 1.The logistic model uses the logistic distribution’s cumulative distribution function; the probit model uses the standard normal CDF.Source: §4.1.1 · p. 33

4.1.2. Models Inspired by Computational Learning Theory

• EDit and Burr1 are links designed for learning-performance curves; EDit draws on computational learning theory, while Burr1 is a simplification of EDit.The section introduces Burr1 as a candidate for binary-response regression, which Nelson said, to his knowledge, had not previously been used for that purpose.Source: §4.1.2 · p. 33, §4.1.2.2 · p. 37

4.1.2.1. The Effective-Dimension Model

• The original VC relation is unsuitable as an average-case regression model: it is defined only for integer dimensions greater than 2 and yields a slower-rising curve than typically observed.The adapted EDit curve is close to the original VC curve even at d=2, where they differ most.Source: §4.1.2.1 · pp. 35–36
• EDit adapts the VC-bound curve to be defined for nonnegative effective dimension and expresses it in accuracy rather than error; its dimension parameter expands or compresses the curve horizontally.Effective dimension is the value that defines the curve best fitting the observed learning-performance data, enabling quantitative predictions from empirical data.Source: §4.1.2.1 · pp. 35–36

4.1.2.2. The Burr1 Model

• Burr1 has a shape similar to EDmodel but is simpler; it is presented as a candidate binary-response regression model.The passage notes that, to Nelson’s knowledge, Burr1 had not previously been used for binary-response regression.Source: §4.1.2.2 · p. 37

4.1.3. A Comparison of the Deterministic Models

• EDmodel and BurrModel, based on computational learning theory, have different shapes than the better-known logistic and probit (normal) binary-response models; the four regression models intersect at accuracies 0.70 and 0.75.Source: §4.1.3 · p. 38, Figure 4.4 · p. 38
• The slopes of EDmodel and BurrModel are similar to each other and differ from logistic and probit; as accuracy approaches 1, EDit becomes infinitely larger than Burr1.The comparison also notes that EDit and Burr1 are not identical at high accuracies.Source: §4.1.3 · p. 38

Noise

• Class skew, class multiplicity, and noise have predictable effects: class frequencies determine skew and multiplicity effects, while factoring these effects out leaves a single learning-performance parameter.For skewed classes, error decreases exponentially rather than linearly with the reciprocal of the number of examples; the model is intended as an empirical heuristic, not a statement of computational learning theory.Source: Noise · p. 40

4.2.1. Learning from Only One Example

• For the guess-what-you-see classification rule on independent testing data, expected accuracy is 0.5 with two equally probable classes; with one class shared by training and testing examples, it is 1.As the number of classes with infinitesimal probability goes to infinity, expected accuracy goes to 0.Source: §4.2.1 · p. 42

4.2.2. Learning from m Examples

• For learning tasks with two classes each of frequency 0.5, expected accuracy p can be well estimated by model5[z], where z=(m-1)/d+1, m counts training examples and d relates to learning rate.With only one class of probability 1, expected accuracy is 1 for all m. As the number of classes with infinitesimal probability goes to infinity, expected accuracy goes to 0 for all finite m.Source: §4.2.2 · pp. 43–44

4.2.3. Heuristic for modelcla[z,f] in Terms of model5[z]

• The class-specific heuristic adjusts the standard model5[z] curve according to class frequency: lower-frequency classes fall below it and higher-frequency classes rise above it.For frequencies between 0 and 0.5, modelcla[z,f] lies between 0 and model5[z]; for frequencies between 0.5 and 1, it lies between model5[z] and 1.Source: §4.2.3 · pp. 45–46

4.2.4. Learning with Multiple, Equiprobable Classes

• For multiple equiprobable classes, the heuristic predicts overall accuracy from the number of classes.The supplied passage gives no resulting formula or numerical values.Source: §4.2.4 · p. 47

4.2.5. Learning with Skewed Classes

• For classes with varied probabilities, the heuristic predicts overall accuracy from the class-frequency vector.The supplied passage does not give the predicted expression or numerical results.Source: §4.2.5 · p. 47
• For skewed classes, a generalized modelskew curve approximates the weighted-sum modelall heuristic by independently setting its start and skew parameters.The parameters are chosen to match the curves as z approaches infinity, at z=1, and at a third point; skew can be solved numerically.Source: §4.2.5 · pp. 47–48
• For soybean disease, the approximation uses start=0.0868867 and skew=1.07556, producing a much simpler curve that is almost indistinguishable from modelall.Figure 4.13 compares the two models.Source: §4.2.5 · pp. 50–51, Figure 4.13 · p. 51

4.2.6. Learning with Noise

• Noise is the gap between 100% accuracy and the asymptotic accuracy max, modeled by scaling the curve without changing its class-frequency-based start.The class frequencies already reflect noise in start, so the noise model should preserve it.Source: §4.2.6 · p. 52

4.2.7. Working with modelgen[z,start,skew,max]

• Fitting modelgen requires measuring class frequencies, computing start and skew, substituting z as a function of training examples, then fitting d and max to observed accuracy.The text illustrates the curve with soybean frequencies and model5[z]=z/(z+1].Source: §4.2.7 · pp. 52–53

4.2.8. Simplified Model

• The simplified modelnoexp removes the exponents from the noise model; for two equiprobable classes, it reduces to the prior model, and its importance is tested in the next chapter.The proposed test concerns other situations.Source: §4.2.8 · p. 54

4.3. Nondeterministic Design Choices

• Independent test examples make the number y classified correctly out of k follow a binomial distribution.The binomial component accounts for randomness in the testing examples.Source: §4.3 · p. 54
• The binomial model omits training-set randomness, which can make learning curves inherently rough even with many training examples.A learner may produce a more accurate rule with 49 examples than with 50, for example, if it makes a different decision-tree split.Source: §4.3 · p. 55
• A reasonable way to model this roughness is with the beta distribution, parameterized for this purpose by p, its mean, and pvar, a parameter related to variance.Source: §4.3 · p. 55
• Combining beta and binomial distributions yields a beta-binomial model that reduces to the binomial when pvar is zero.The author notes that this distribution had previously modeled student test scores, but, to the author’s knowledge, not a regression model’s nondeterministic component.Source: §4.3 · p. 56
• Seer estimates pvar by maximizing likelihood rather than minimizing least-squares error.The observed y and k are supplied by the learning-performance data, while p comes from the deterministic model.Source: §4.3 · p. 56

4.4. Fitting Models to Data Efficiently

• Seer transforms constrained parameters such as d and pvar into functions of unconstrained parameters, turning nonlinear programming into unconstrained nonlinear optimization.The max parameter is also reparameterized to remain between start and 1.Source: §4.4 · p. 57
• Seer uses Levenberg-Marquardt optimization, estimating the Hessian from first derivatives rather than calculating second derivatives.The method varies between inverse-Hessian and steepest-descent approaches.Source: §4.4 · p. 57
• When learning-performance data are dependent under generalized cross-validation, Seer cannot estimate parameter covariance.For independent data, it approximates the asymptotic covariance matrix using the inverse negative Hessian.Source: §4.4 · p. 58

4.5. Summary

• EDmodel and BurrModel, inspired by computational learning theory, have shapes substantially different from popular binary-response regression models.Source: §4.5 · p. 58
• The modelgen heuristic represents effects of multiple classes, skewed classes, and noise with remarkably few parameters.Its design was developed by considering progressively more complex cases.Source: §4.5 · p. 58
• The binomial model addresses testing-data variation, while the Beta-Binomial is suggested when training-data variation also matters.The chapter then specifies nonlinear optimization for efficiently finding the best model in a possibly infinite set.Source: §4.5 · p. 58

• The experiments assembled five learning-performance model sets and tested them in three real-world domains.Source: §5 · p. 59
• Each model set was tested in 3 characterization trials and 30 prediction trials.Source: §5 · p. 59
• The experimental variables included domain, task, and learning-performance model set.Source: §5 · p. 59, Figure 5.1 · p. 59
• Many model sets characterized learning performance well and predicted as well as the data permitted, with choice trading computational ease against measuring interesting aspects of the learning problem.Source: §5 · p. 59

5.1.1. Experimental Variable: Domain

• The evaluation covered soybean disease diagnosis (Soy; 307 examples, 35 attributes, 19 classes), heart disease diagnosis (Heart; 303 examples, 13 attributes, 2 classes), and audiological problems (Audio; 226 examples, 69 attributes).Source: §5.1.1 · p. 60

5.1.2. Experimental Variable: Task

• Characterization used all available examples, while prediction fit models from 100 randomly selected examples and evaluated their predictions using the full dataset.The experiments ran three trials per domain/model set for characterization and 30 for prediction.Source: §5.1.2 · pp. 60–61

5.1.3. Inductive Learning Program

• All experiments used C4.5, which builds decision trees and then turns them into rules to improve generalization and noise handling, with default settings.Source: §5.1.3 · p. 61

5.1.4. Measuring Results

• Results were measured primarily with loglikelihood, which depends on the amount of learning-performance data, motivating a normalized R2-style measure.Doubling the data tends to double loglikelihood.Source: §5.1.4 · pp. 61–62

5.1.4.1. Input to the R2

• The R2 measure takes a learning-performance model, a likelihood criterion, and data from a larger set of classified examples as inputs.Each datum records training-example count, correctly classified test examples, and total test examples.Source: §5.1.4.1 · pp. 62–63

5.1.4.2. Output of the Measure

• Seer’s new R2 measure avoids assuming that a fitted model beats the constant-accuracy baseline on unseen learning data, comparing model, baseline, and saturated-model likelihoods.The baseline predicts the same accuracy regardless of training-set size; the saturated model matches observed frequencies.Source: §5.1.4.2 · pp. 63–64

5.1.4.3. Composite R2 Seer Results

• Seer generalizes its measure to related models by summing their loglikelihoods and those of their saturated models.Source: §5.1.4.3 · p. 65

5.1.5. Comparing Results from Two Model Sets

• Seer compares model-set composite R2 values using paired-difference analysis to assess statistical significance.Source: §5.1.5 · p. 65
• For each trial, methods are applied to the same learning-performance data, their models’ loglikelihoods are differenced, and a two-tailed Student t-test tests whether the mean difference departs from zero.Table 5.2 illustrates the paired-difference procedure.Source: §5.1.5 · pp. 65–66, Table 5.2 · p. 66

5.2.1. Modelgen Experiments

• The Modelgen set combined modelgen and Burr1 as its deterministic component with a binomial distribution, and tested characterization, prediction, and benchmarking.The model set is described in Table 5.3.Source: §5.2.1 · p. 66, Table 5.3 · p. 67

5.2.1.1. Characterizing Data with Modelgen

• Modelgen characterized learning-performance data well across three domains, with goodness-of-fit quantified over three trials per domain.The characterization task generated accuracy points and fit a curve to them; a perfect R2 Seer value of 1.0 would require the model to pass through every accuracy point, which is generally not the case.Source: §5.2.1.1 · p. 67, Table 5.4 · p. 67

5.2.1.2. Predicting in the Audiological Domain

• For Modelgen, composite R2 Seer was 0.981 fitting learning-performance data from 100 classified examples and 0.925 predicting p218 on the full data set, suggesting those examples may be unrepresentative.Source: §5.2.1.2 · p. 69, Table 5.6 · p. 72

5.2.1.3. Predicting in All Three Domains

• Across the three domains, predictions varied with the starting samples; Soy and Heart predictions appeared unbiased, while Audio prediction was relatively low.The table reports composite results from 30 trials, but the supplied prose does not give all of its values.Source: §5.2.1.3 · p. 72, Table 5.7 · p. 72
• Even the worst-predicting models fit their observed data well; in Heart, 100 examples showed accuracy above 0.80, versus p290 of 0.752 on all 303 examples.The passages attribute poor predictions mainly to a random subset of classified examples being unrepresentative of a larger set, and say Seer seems to do as well as possible with the examples available.Source: §5.2.1.3 · p. 74, Table 5.8 · p. 74

5.2.1.4. Estimating Parameters in All Three Domains

• For the four-parameter Modelgen set, Seer estimated start and skew from class frequency, and d and max from learning-performance data.The experiments used 30 trials per domain to estimate standard deviations after the fact.Source: §5.2.1.4 · p. 74
• Soy and Heart parameter estimates clustered around the true values, while Audio’s d estimate was off, apparently because its max estimates were too low.Estimates based on 100 classified examples also clustered around the characterization points based on full sets of classified examples.Source: §5.2.1.4 · pp. 74, 76

5.2.1.5. Summary of Modelgen experiments

• Modelgen characterized learning-performance data well and generally predicted performance well, though predictions were worse on the worst trials, apparently because samples were unrepresentative.Even in the worst trials, Modelgen’s predictions seemed to be doing about as well as possible.Source: §5.2.1.5 · p. 76

5.2.2. Logistic Experiments

• Logistic characterization fit was poor: its curves had the wrong shape, and its composite R2 Seer value was less than 0.9 in two domains.It performed best in Heart, where the data was most linear and easiest to model.Source: §5.2.2 · p. 78
• Although characterization of the 100 examples had R2 greater than 0.90, the best prediction R2 Seer value was 0.501, and all 90 predictions were too high.The predictions were biased toward optimism across all 30 trials and all three domains.Source: §5.2.2 · p. 78, Table 5.12 · p. 78, Figure 5.10 · p. 80
• Across all six task-domain combinations, Modelgen produced statistically significantly better loglikelihoods than Logistic.The paired-difference analysis covered both characterization and prediction tasks in all three domains.Source: Table 5.13 · p. 81, §5.2.2 · p. 81

5.2.3. ED Experiments

• ED’s d parameter can be interpreted as an empirical analog of VC dimension, but its EDit component is computationally harder to work with than Modelgen’s Burr1.For the Soy characterization task, mean d was 47.692, approximating a worst-case learning model with VC dimension 48.Source: §5.2.3 · pp. 83–85
• In the ED comparison, prediction showed no reliable performance difference from Modelgen; Modelgen characterized better in two of three domains.Across the prediction task, Modelgen did better in one domain, ED in another, and neither significantly better in the third.Source: §5.2.3 · p. 84, Table 5.18 · p. 85

5.2.4. NoExp Experiments

• NoExp characterization R2 Seer values exceeded 0.96, but its lowest Heart prediction value was 0.866, with maximum-accuracy estimates far off in a minority of trials.Modelgen predicted better in Heart; the other two domains showed no significant difference in predictive power.Source: §5.2.4 · pp. 85, 86, 88, Table 5.23 · p. 88
• NoExp was simpler, but its d conflates skew and learning difficulty; Modelgen’s d more closely measures inherent learning hardness independent of class frequencies.In most investigated scenarios, Modelgen’s exponent was not important because d could compensate for its absence.Source: §5.2.4 · p. 88

5.2.5. Pvar Model Set

• Pvar appeared significant for characterization, but its values were close to zero for prediction, suggesting beta-binomial was not useful for those prediction tasks.The prediction-task mean pvar was within two standard deviations of zero and, in Soy, within one.Source: §5.2.5 · p. 89
• Under the binomial criterion, Modelgen significantly outperformed Pvar in two of six scenarios and neither won in the other four; under beta-binomial, Pvar won four, Modelgen one, and one was tied.Neither set consistently beat the other by the other’s criterion, and the binomial distribution seemed sufficient for these scenarios.Source: §5.2.5 · p. 89, Table 5.26 · p. 90

5.3. Summary and Conclusion

• Logistic was the only model set that performed poorly on both characterization and prediction.The study compared five model sets over the 99 learning-performance datasets.Source: §5.3 · p. 91
• All model sets except Logistic characterized well and seemed to predict as well as possible; poor predictions in some trials were mostly attributed to unrepresentative example subsets.Seer generated 99 datasets from three domains and compared maximum-likelihood fits using loglikelihood and R2 goodness-of-fit.Source: §5.3 · p. 91
• The ED model set uniquely yields a d comparable to the parameter from Vapnik-Chervonenkis-dimension analysis.Most model sets fit max and d; start and skew can be determined from class frequencies, while Pvar adds a fitted parameter.Source: §5.3 · pp. 91–92
• Among the well-performing sets, model choice trades simplicity against useful, meaningful parameters, including asymptotic accuracy and learning difficulty.NoExp is computationally simplest, followed by Modelgen; ED and Pvar are most complex.Source: §5.3 · pp. 91–92

6.1. Summary

• Seer’s learning-performance models incorporate constraints suited to classification, useful parameters, and algorithms for efficiently finding maximum-likelihood models.Its model sets address multiple classes, skew, and noise with a heuristic modelgen approach using few parameters.Source: §6.1 · pp. 93–94
• Seer made maximum-likelihood fitting practicable by exploiting problem constraints and transformations to reduce nonlinear programming to nonlinear regression solved by an efficient iterative algorithm.Least-squares regression was unsuitable because its assumptions, including constant variance, do not fit learning-performance data.Source: §6.1 · p. 94
• All model sets except Logistic characterized learning well and predicted as well as the data allowed; poor prediction in some trials appeared mainly due to unrepresentative example subsets.The thesis reports experiments across soybean disease, heart disease, and audiological problems.Source: §6.1 · p. 95
• The max parameter estimates asymptotic accuracy, while start and skew characterize effects of multiple and skewed classes; d measures learning difficulty, and ED’s d is comparable to the VC-dimension estimate.The pvar parameter represents variation in training examples, but it did not compellingly improve prediction accuracy in these experiments.Source: §6.1 · p. 95, §6.2 · p. 96

6.2. Future Work

• Future work includes multivariate models because training-example count is not the only dependent variable of interest when predicting learning performance.The conclusion also identifies expanding available performance data and exploring pvar and dED as research directions.Source: §6.2 · pp. 95–96
• Future work could use an inexhaustible synthetic example supply to create statistically independent performance data and confidence intervals, though this would replace real-world data with a synthetic generator.The thesis also calls for determining when pvar can be estimated reliably and further exploring dED as a possible measure of inductive bias.Source: §6.2 · p. 96
• Future work proposes modeling how learning algorithm, its configuration, irrelevant attributes, and target complexity affect learning performance.Algorithm choice and configuration are typically known; target complexity, such as the number of disjuncts, may be known only for synthetic problems.Source: §6.2 · pp. 96–97
• Multivariate regression could map these task attributes to expected accuracy across training-set sizes, summarizing their general effects with few numbers.The proposed example model uses accuracy, number of training examples, and number of target disjuncts as its axes.Source: §6.2 · p. 97, Figure 6.15 · p. 97

6.3. Contributions

• Given classified examples and a learning system, regression predicts average-case learning performance and asymptotic accuracy, including with noisy data and multiple, skewed classes.The thesis presents this as a practical and efficient way to analyze inductive classification learning.Source: §6.3 · p. 98
• The thesis developed several model sets with different levels of simplicity and parameter usefulness; EDmodel’s d parameter is comparable to the Vapnik-Chervonenkis dimension.It also developed a heuristic to predict the effects of skewed and multiple classes without learning-performance data.Source: §6.3 · p. 98
• The thesis introduced pvar to measure variation across training-example sets, and argued that maximum likelihood best selects models within a family and can be found efficiently.Most similar statistical methods measure variation caused by different testing-example sets instead.Source: §6.3 · p. 98
• Generalized cross-validation creates learning-performance data from finite classified examples when dependencies are allowed, although subsets of those examples can produce unrepresentative data..Source: §6.3 · p. 98
• The thesis identifies likelihood as the preferred measure of model fit, defines likelihood-based R2 to normalize it, and uses pair-difference analysis to compare competing models.Source: §6.3 · p. 98

Vita

• Kadie was born in Denver in 1962 and earned a computer science degree at the University of Illinois at Urbana-Champaign.The vita states that he graduated with highest honors in January 1995.Source: Vita · p. 103
• He completed a master’s degree in 1989 and a Ph.D. focused on maximum-likelihood regression for predicting learning-speed curves in 1995.His graduate studies began in 1985.Source: Vita · p. 103
• His listed work includes publications and presentations on inductive learning, robot operators, and academic freedom.The vita also records his software engineering work at Microsoft on Internet features for MSN.Source: Vita · pp. 103–104