PaperlayerView exact revision on arXiv

Source-linked AI summary

A systematic comparison of supervised classifiers

D. R. Amancio, C. H. Comin, D. Casanova, G. Travieso, O. M. Bruno, F. A. Rodrigues, L. da F. Costa

arXiv:1311.0202v1cs.LG

TL;DR

Choosing classifiers and parameters remains difficult for practical pattern-recognition tasks, particularly for users without detailed knowledge of algorithm mechanisms. The paper evaluates nine Weka classifiers under default, individual-parameter, and randomly sampled configurations. Results show strong dimensionality effects, with kNN strongest under defaults in high dimensions and random parameter settings improving SVM performance by more than 20%.

  • Problem

    Choosing suitable classifiers and parameters remains difficult for practitioners who use machine-learning tools without in-depth knowledge of algorithm mechanisms.

  • Method

    The study compares nine Weka classifiers using default parameters, single-parameter variations, and randomly sampled parameter configurations on synthetic datasets.

  • Results

    In high-dimensional datasets, kNN outperformed other classifiers with default parameters, while random parameter sampling increased SVM accuracy by more than 20%.

  • Takeaways & Limitations

    Weka defaults are generally close to optimal for low-dimensional tasks, but default SVM parameters are not recommended for high-dimensional problems.

  • Takeaways & Limitations

    The study’s conclusions are based on synthetic datasets and leave factors such as class count, instances per class, class overlap, and scarce training data for further investigation.

Abstract

from arXiv · show

Pattern recognition techniques have been employed in a myriad of industrial, medical, commercial and academic applications. To tackle such a diversity of data, many techniques have been devised. However, despite the long tradition of pattern recognition research, there is no technique that yields the best classification in all scenarios. Therefore, the consideration of as many as possible techniques presents itself as an fundamental practice in applications aiming at high accuracy. Typical works comparing methods either emphasize the performance of a given algorithm in validation tests or systematically compare various algorithms, assuming that the practical use of these methods is done by experts. In many occasions, however, researchers have to deal with their practical classification tasks without an in-depth knowledge about the underlying mechanisms behind parameters. Actually, the adequate choice of classifiers and parameters alike in such practical circumstances constitutes a long-standing problem and is the subject of the current paper. We carried out a study on the performance of nine well-known classifiers implemented by the Weka framework and compared the dependence of the accuracy with their configuration parameter configurations. The analysis of performance with default parameters revealed that the k-nearest neighbors method exceeds by a large margin the other methods when high dimensional datasets are considered. When other configuration of parameters were allowed, we found that it is possible to improve the quality of SVM in more than 20% even if parameters are set randomly. Taken together, the investigation conducted in this paper suggests that, apart from the SVM implementation, Weka's default configuration of parameters provides an performance close the one achieved with the optimal configuration.

Introduction

The paper addresses the difficulty of choosing classifiers and parameters for practical pattern-recognition tasks by systematically evaluating Weka implementations and parameter sensitivity. It compares default, single-parameter, and randomly sampled configurations while considering synthetic datasets and classifier robustness.

  • Motivation: Pattern recognition automates the discovery of concealed patterns in datasets used across scientific, medical, industrial, and commercial applications.Manual identification is infeasible for large data collections, but pattern recognition also supports smaller datasets.
  • Research problem: Systematic quantitative evidence comparing many classifiers remains limited, and results from real datasets may not generalize beyond the datasets studied.Real-world data can also be expensive to obtain when evaluating additional classifier characteristics.
  • Study scope: The study evaluates nine commonly used classifiers available in the open-source Weka framework.The subset was selected according to prior work and enables researchers to inspect the implemented procedures.
  • Evaluation principle: Classifier quality is assessed not only by high accuracy but also by whether good accuracy persists across a broad range of parameter values.Sensitivity to parameter changes matters because highly specific settings may perform worse on different datasets.
  • Evaluation design: The analyses compare Weka defaults, vary individual parameters, and randomly sample parameter sets to estimate attainable accuracy and parameter sensitivity.The default configuration reflects a common practical use of the software.
  • Paper organization: The paper reviews prior classifier comparisons, describes synthetic dataset generation, quantifies performance, and presents a comparative analysis followed by conclusions.These components define the paper’s overall organization.

Related works

Prior classifier-comparison studies fall into validation-focused comparisons of a few methods and systematic comparisons of representative classifiers. Systematic studies may be qualitative or quantitative.

  • Related works: Classifier-comparison studies include validation-oriented comparisons of a few methods and systematic comparisons among many representative classifiers.The first group commonly validates or justifies a new approach.
  • Related works: Qualitative systematic studies describe classifiers’ advantages and drawbacks without conducting quantitative performance tests.The supplied passage contrasts these studies with quantitative analyses.

Materials and Methods

The paper uses a generic methodology for constructing artificial datasets with varied characteristics and defines measurements for evaluating classifier quality.

  • Materials and Methods: The methodology constructs artificial datasets that model different characteristics of real datasets.The approach is intended to provide varied data for classifier comparison.
  • Materials and Methods: Classifier quality is evaluated using specified performance measurements.These measurements support the later quantitative comparison.

Artificial Data

The study generates artificial datasets to vary important data characteristics systematically while retaining realistic distributional assumptions. Its method constructs valid class-specific covariance structures and controls dimensionality, separation, and related properties.

  • Artificial Data: The study uses multivariate Gaussian artificial data to vary correlations, variances, dimensionality, class separation, and other dataset properties systematically.The Gaussian choice is justified as representing a broad ensemble of possible real-world data occurrences.
  • Artificial Data: The proposed generator creates C classes with F features and prescribed class sizes while enforcing distributions for variances, correlations, and class separation.C, F, and the class-size vector are treated as strong parameters.
  • Artificial Data: Unlike fixed-spectrum approaches, the method allows different classes to exhibit varied correlation structures drawn from a chosen distribution.The authors describe this as more general than requiring similar feature relationships across classes.
  • Artificial Data: Valid covariance matrices are generated through a root matrix G, using the property that G G^T is positive and semi-definite.Diagonal and off-diagonal elements are associated with the variance and correlation distributions, respectively.
  • Artificial Data: The generated datasets use ten classes and equal numbers of instances per class, while the number of classes is not varied throughout the study.The authors report that results did not appreciably change for different class counts, including C = 2.
  • Artificial Data: The experiments vary feature count F from 2 to 10 and vary separation α, which quantifies class separation relative to feature standard deviation.Larger α produces more localized classes with less overlap.
  • Artificial Data: Feature standard deviations are drawn per class from fσ, and feature correlations are drawn per class from fc, with fc chosen as uniform in the study.This permits class-specific variation in feature relationships.

Evaluating the performance of the classifiers

The study evaluates classifier quality primarily through accuracy rate, while also examining accuracy variation across datasets and parameter settings. Artificial datasets vary in class separation and feature structure.

  • Accuracy rate is defined as true positives plus true negatives divided by the total number of instances.
  • The study computes average accuracy and variation across datasets to assess performance and confidence when datasets change.
  • Figure 1 varies class separation in a two-feature, ten-class artificial dataset using α = 1, α = 5, and α = 7.
  • Accuracy standard deviation across parameter settings measures classifier sensitivity to a given parameter.

Results and Discussion

Classifier performance is evaluated through default settings, individual parameter changes, and a multivariate parameter strategy across datasets with different feature counts.

  • The study evaluates classifiers using default parameters, single-parameter variation, and a multivariate strategy.
  • DB2F through DB10F denote datasets whose instances have 2 through 10 features, respectively.
  • The evaluated classifier list and abbreviated names are presented in Table 1.

Comparison of classifiers using their default parameters

Default-parameter performance varies with dataset dimensionality. Naive Bayes performs best in DB2F, whereas kNN becomes strongest by a large margin in high-dimensional datasets.

  • Researchers commonly compare classifiers using default parameters because tuning requires knowledge of algorithm mechanisms and can be cumbersome.
  • In DB2F, Naive Bayes significantly outperforms C4.5, Simple Cart, Bayes Net, Random Forest, kNN, and SVM under Weka’s default parameters.
  • The DB2F accuracy difference between best and worst cases reaches 30% for some classifiers, including C4.5, despite identical dataset-generation parameters.
  • In DB10F, kNN provides the highest average accuracy by a large margin, with the usual neighbor setting K = 1.
  • As features increase from two to ten, kNN accuracy increases, while C4.5, Simple Cart, and Bayes Net show more prominent decreases.
  • The difference between kNN and Bayes Net in DB10F is almost 40%.

Varying parameters: one-dimensional analysis

The one-dimensional analysis varies individual classifier parameters to assess how accuracy changes relative to Weka’s defaults. Default settings are usually near-optimal, but SVM can gain substantially from suitable tuning.

  • Analysis design: The analysis varies one classifier parameter at a time while keeping the remaining parameters at their default values.It compares the resulting accuracy with the default configuration using the difference S(p) = Γmax −Γdef.
  • Analysis design: S(p) measures how much classification accuracy changes when parameter p differs from its default value.Γmax is the maximum observed accuracy while p varies, whereas Γdef is the accuracy with all parameters at their defaults.
  • Results: For almost all parameters, ⟨S⟩≤0, indicating that default parameters usually provide performance close to the optimum.The exceptions are the kNN seed number K and the SVM type S.
  • Results: Maximum accuracy gains usually do not exceed 6.25% in any particular dataset, excluding kNN and SVM.This summarizes the comparison between default and best parameter values in DB2F.
  • Results: In DB10F, kNN’s default accuracy is Γdef ≃94%, while its average one-dimensional improvement is only ∆S(K) = 0.01%.The high default accuracy leaves little room for improvement through varying K alone.
  • Results: Six SVM parameters enabled increases of about 20%, producing accuracy rates higher than 94% in many cases.The result suggests that suitable SVM parameter tuning can significantly improve discriminability.

Multidimensional analysis

The multidimensional analysis randomly samples joint parameter configurations to capture interactions among parameters. Default settings are generally robust, whereas SVM benefits strongly from parameter changes and SVM and kNN rank best among sampled configurations.

  • Analysis design: The multidimensional analysis samples 1,000 random parameter configurations for each classifier to account for parameter interdependencies.This complements one-dimensional variation, which considers parameters separately.
  • Results: Random parameter choices usually reduce accuracy for Simple Cart, Perceptron, C4.5, and Logistic relative to their default configurations.For these classifiers, the reported p-value is below 50.0%.
  • Results: 96.89% of random SVM configurations improve accuracy over the default configuration in DB2F.For kNN, improvement occurs in 76.15% of random configurations; Random Forest shows a small improvement in 52%.
  • Scope: Naive Bayes and Bayesian Net are excluded from the multidimensional analysis because they have only binary parameters.
  • Results: In the multidimensional analysis, performance can be improved by 16%, 13%, and 10% for SVM, kNN, and Random Forest, respectively.These improvements make SVM and kNN the best classifiers when selecting the best realization among 1,000 random configurations for each dataset.
  • Results: In DB10F, the average SVM improvement is 20.35%, while significant improvement is much less likely for the other classifiers.The 20.35% average equals the improvement obtained by varying SVM’s complexity parameter C alone.

Conclusions

The study evaluates Weka classifiers under default, single-parameter, and random configurations across low- and high-dimensional datasets. Results show strong effects of feature count, substantial potential SVM gains from parameter changes, and generally near-optimal Weka defaults except for SVM in high dimensions.

  • Default configurations: Naive Bayes achieved the best performance among nine classifiers on two-dimensional problems with default parameters, while SVM performed worst.For DB2F, the reported accuracy distributions were Gaussian and all classifiers had high average accuracy rates.
  • Comparison design: The comparisons considered default and random parameter configurations on DB2F and DB10F, reporting the best- and worst-performing classifiers.Tables 8–10 summarize multidimensional analyses, rankings, and classifier comparisons across these configurations and datasets.
  • Parameter effects: 6% average accuracy improvement was enabled by appropriate choices of kNN parameter K and Puk-kernel SVM parameter S.These were the exceptions to generally small accuracy changes when only one parameter varied from its default.
  • Parameter effects: More than 20% average accuracy improvement was obtained for SVM on ten-dimensional datasets when parameters were varied randomly.The random-configuration strategy was especially useful for optimizing SVM performance in high-dimensional problems.
  • Dimensionality: kNN and SVM were the most accurate techniques in high-dimensional tasks under both default and alternative parameter configurations.Performance differences between classifiers were stronger when datasets involved several features.
  • Conclusions: Weka’s default parameters produced accuracy rates close to optimal values for low-dimensional tasks, with a few exceptions.The largest discrepancies occurred for SVM in high-dimensional problems, where default parameters were not recommended when SVM had to be used.
Loading 1311.0202v1…