PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Review On Twin Support Vector Machines

M. Tanveer, T. Rajani, R. Rastogi, Y. H. Shao, M. A. Ganaie

arXiv:2105.00336v3cs.LGcs.AI

TL;DR

TWSVM and TSVR address classification and regression, but comparison of TSVR variants remains limited. This review synthesizes recent methods, improvements, applications, advantages, limitations, and future prospects across both techniques. It reports faster or more accurate variants in several reviewed settings, including RELS-TSVM's leading performance in a 187-classifier, 90-dataset evaluation.

  • Problem

    Literature comparing different TSVR variants is limited, despite rapid development and application of twin support vector classification and regression algorithms.

  • Method

    The paper reviews SVM, TWSVM, and TSVR theory, then synthesizes TWSVM and TSVR improvements, applications, limitations, advantages, and future research.

  • Results

    The review reports promising performance across variants, including RELS-TSVM emerging as the best classifier among 187 classifiers evaluated on 90 UCI datasets.

  • Takeaways & Limitations

    TWSVM and TSVR offer active research directions for classification and regression, while their variants and applications remain areas for continued development.

  • Takeaways & Limitations

    TWSVM remains at a primitive development stage, with low generalization ability, limited sparsity, and limited research on large-scale applications and kernel selection.

Abstract

from arXiv · show

Twin support vector machine (TWSVM) and twin support vector regression (TSVR) are newly emerging efficient machine learning techniques which offer promising solutions for classification and regression challenges respectively. TWSVM is based upon the idea to identify two nonparallel hyperplanes which classify the data points to their respective classes. It requires to solve two small sized quadratic programming problems (QPPs) in lieu of solving single large size QPP in support vector machine (SVM) while TSVR is formulated on the lines of TWSVM and requires to solve two SVM kind problems. Although there has been good research progress on these techniques; there is limited literature on the comparison of different variants of TSVR. Thus, this review presents a rigorous analysis of recent research in TWSVM and TSVR simultaneously mentioning their limitations and advantages. To begin with we first introduce the basic theory of support vector machine, TWSVM and then focus on the various improvements and applications of TWSVM, and then we introduce TSVR and its various enhancements. Finally, we suggest future research and development prospects.

1 Introduction

The introduction positions SVM and SVR as established margin-based learning methods, then motivates a review of rapidly developing twin support vector classification and regression techniques.

  • SVM: SVM determines a binary decision boundary using support hyperplanes whose separation is maximized through a quadratic programming problem.Kernel methods extend SVM to non-linearly separable data, while structural risk minimization supports generalization.
  • SVR: SVR fits a hyperplane while allowing errors within an ϵ-tolerance margin and minimizing deviations beyond that margin.Its learning speed is low because it solves a large-sized quadratic programming problem.
  • Twin methods: Twin support vector classification and regression algorithms have developed rapidly and have been applied to real-life challenges.
  • Review scope: Because literature on twin support vector regression remains limited, this paper compiles recent developments, advantages, limitations, and future prospects for TWSVM and TSVR.The review covers foundations, TWSVM improvements and applications, TSVR enhancements and applications, and future research directions.

2 Related Work

The paper introduces SVM optimization and then describes nonlinear TWSVM, which uses two kernel-generated hyperplanes and smaller class-specific constraint sets.

  • Data representation: A binary dataset is represented by positive-class matrix A and negative-class matrix B, with labels in {−1, 1}.The matrices contain l1 and l2 samples respectively, while n denotes feature-space dimension.
  • Support Vector Machine: SVM minimizes a regularized objective subject to margin constraints involving every training sample and nonnegative slack variables.The regularization parameter is c, the nonlinear mapping is φ(x), and ξ denotes slack variables.
  • Support Vector Machine: Because all samples appear in SVM constraints, solving one large quadratic programming problem gives complexity O(l3).
  • Twin Support Vector Machines: Nonlinear TWSVM maps data through a kernel function and constructs two kernel-generated surfaces, K(x^T, D_T)u+ + b+ = 0 and K(x^T, D_T)u− + b− = 0.Here D = [A; B] and K is the kernel function.
  • Twin Support Vector Machines: A new sample is assigned according to its proximity to the kernel-generated surfaces, with the signum function determining the class.
  • Twin Support Vector Machines: TWSVM solves two quadratic programming problems in which each problem uses constraints from only one class while generating the hyperplane for the other class.The resulting proximal hyperplanes use regularization through δI, with δ > 0.

3 Research Progress on Twin Support Vector Machines

Research on TWSVM has expanded through least-squares, robust, weighted, feature-selective, knowledge-based, multicategory, and multitask variants, targeting lower computation, stronger generalization, noise resistance, and specialized data settings.

  • Least-squares variants: LS-TWSVM replaces TWSVM quadratic programming problems with linear equations, reducing computational complexity while retaining comparable accuracy but weakening generalization.Equal penalties can cause over-fitting and ignore differing sample locations.
  • Least-squares variants: Weighted LS-TWSVM assigns location-dependent sample weights and achieves better testing accuracy than TWSVM and LS-TWSVM, at higher computational cost.The weighting addresses the limitation of treating positive and negative samples identically.
  • Least-squares variants: Knowledge-based LS-TWSVM incorporates polyhedral expert-knowledge sets, while manifold-based LS-TWSVM models each class’s geometric structure using linear equations.These variants extend LS-TWSVM beyond uniform sample treatment by adding prior knowledge or data geometry.
  • Least-squares variants: NLS-TWSVM uses l1-norm regularization for relevant-feature selection, ILS-TWSVM applies structural risk minimization, and capped L2,p models reduce outlier and noise effects.The variants target high dimensionality, over-fitting, and the sensitivity of least-squares models to outliers.
  • Least-squares variants: Density-weighted LS classification improves accuracy over TWSVM and LS-TWSVM by weighting samples according to density and exploiting correlations among data points.The weighting is also intended to reduce the effect of noise.
  • Robust and specialized variants: Robust ELS-TSVM and related RLS-TWSVM methods incorporate robustness mechanisms for noise and outliers while reporting promising generalization, speed, or benchmark performance.RELS-TSVM emerged as the best classifier in an evaluation of 187 classifiers on 90 datasets.

3.2 Projection Twin Support Vector Machine

Projection-based TWSVM variants replace or augment twin hyperplanes with projection axes or directions, improving accuracy or generalization on complex distributions while introducing computational trade-offs.

  • Projection TWSVM: P-TWSVM finds projection axes instead of hyperplanes and can better classify complex distributions such as XOR problems.It minimizes within-class variance and can recursively consider multiple projection axes for each class.
  • Projection TWSVM: P-TWSVM achieves better accuracy than GEPSVM, TWSVM, LS-TWSVM, and MVSVM, but is computationally more complex than TWSVM.
  • Regularized variants: RP-TWSVM replaces empirical risk with regularized risk, improving accuracy and generalization while addressing possible singularity problems.The method also uses successive over-relaxation and genetic algorithms for parameter selection.
  • Least-squares variants: Kernelized LSP-TWSVM applies projection-based least squares to nonlinear problems while retaining linear-equation optimization.Its nonlinear recursive algorithm produced better classification accuracy than LSP-TWSVM and P-TWSVM.
  • Least-squares variants: Feature-selection LSP-TWSVM obtains comparable prediction accuracy to TWSVM, LS-TWSVM, and LSP-TWSVM, with similar generalization to TWSVM and LS-TWSVM.
  • Least-squares variants: LIWLSP-TWSVM incorporates local information through weighted means, improving generalization but increasing computational cost for multiclass problems.
  • Projection variants: NP-TWSVM determines projection axes with nonnegative margin constraints and reports better generalization and accuracy than baseline algorithms.The passage describes it as faster than P-TWSVM because it does not avoid inverse-matrix calculations.

3.3 Twin Parametric Margin Support Vector Machine

Twin parametric-margin methods introduce flexible, structurally informed margins and related robust or sparse variants, while several enhancements target training speed, noise handling, and sparsity.

  • Parametric-margin foundations: TPMSVM determines two parametric-margin hyperplanes defining positive and negative margins, automatically adapting margins to heteroscedastic noise.Its generalization is comparable to SVM, Par-ν-SVM, and TWSVM.
  • Parametric-margin foundations: LSTPMSVM solves two primal problems instead of dual problems, reducing complexity and training time while maintaining comparable or better classification accuracy.Particle swarm optimization is used to optimize its four parameters.
  • Structural and sparse variants: STPMSVM incorporates class structure based on cluster granularity, achieving good generalization, fast learning, and better performance than TPMSVM.
  • Structural and sparse variants: CTPSVM addresses TPMSVM’s loss of sparsity by optimizing centroid projection values to produce a sparse optimal hyperplane.
  • Robust variants: RP-TWSVM and ATPSVM adjust margins or maximize angles to handle heteroscedastic noise and incorporate structural information.
  • Robust variants: AULSTWSVM combines linear loss with universum information and solves linear equations, leading to less computation time.
  • Robust variants: Primal-space TWSVM adds regularization to obtain sparse hyperplanes with comparable generalization and rapid learning, though computational cost remains high.
  • Robust variants: NLP-TWSVM uses l1-norm regularization and linear equations instead of QPPs, yielding a fast, robust, sparse, and simple algorithm.Experiments report better generalization and lower computational time than GEPSVM, SVM, and TWSVM.

3.6 Twin Support Vector Machine for Universum Data and Imbalanced Datasets

Universum, imbalanced-data, and fuzzy TWSVM variants incorporate prior information, class imbalance handling, or sample memberships to improve classification, while complexity and overfitting remain concerns.

  • Universum data: U-TWSVM uses universum samples and assigns them to classes according to proximity to hyperplanes, with better accuracy than TWSVM.
  • Imbalanced datasets: Weighted Lagrangian TWSVM combines quadratic loss with OSS-SMOTE resampling to address imbalanced classification.SMOTE can produce overfitting because it relies on K-nearest neighbors.
  • Universum data: Uν-TWSVM flexibly exploits universum prior knowledge and reports better accuracy with lower running time than baseline algorithms.
  • Imbalanced datasets: MMTSSVM determines two homocentric spheres, with the smaller capturing positive samples and the larger repelling negative samples while increasing the inter-sphere margin.
  • Universum data: Improved universum TWSVM regularizes the optimization problem to make matrices nonsingular and improve generalization and training time over USVM and UTWSVM.
  • Fuzzy variants: Fuzzy TWSVM variants assign memberships using fuzzy models, entropy, or intuitionistic information to represent differing sample roles.

3.8 Some other improvements of Twin Support Vector Machines

Other TWSVM improvements target smoothing, specialized data structures, large-scale learning, robustness, parameter selection, and multilabel or noise-sensitive classification.

  • Optimization and efficiency: Smooth TWSVM transforms primal problems into smooth unconstrained minimization and uses Newton–Armijo optimization, achieving comparable generalization with significantly faster training.
  • Specialized learning settings: MI-TWSVM applies bilevel programming to multiple-instance classification, where bags are labeled by the classes of their constituent instances.
  • Sparse and weighted variants: ν-TWSVM uses ν to control the trade-off between support vectors and marginal error, while adaptive p reduces over-restricted constraints and support vectors.
  • Sparse and weighted variants: Rough ν-TWSVM and KNN-weighted rough ν-TWSVM assign location-dependent penalties, improving generalization, accuracy, or computational cost over earlier ν-TWSVM variants.
  • Optimization and efficiency: WL-TWSVM uses weighted linear loss and only linear equations to accelerate training on large-scale datasets while improving generalization over TWSVM.
  • Optimization and efficiency: TBSVM adds regularization to embody structural risk minimization and uses SOR, producing faster training and better generalization than TWSVM.
  • Data structure and locality: WL-TWSVM incorporates local correlation through sample weights, but finding K-nearest neighbors for every sample limits scalability.
  • Data structure and locality: Structural TWSVM incorporates cluster structure and improves training time and accuracy, whereas KNN-based structural TWSVM can overfit under empirical risk minimization.

3.9 Twin Support Vector Machine for Multi-class Classification

TWSVM has been extended from binary classification to multicategory problems through diverse class-structure strategies, including one-versus-rest, one-versus-one, binary-tree, hypersphere, and projection-based formulations.

  • Multi-class structures: Twin-KSVC extends TWSVM to multicategory classification using a “1-versus-1-rest” structure and ternary outputs.It requires two smaller-sized QPPs, improves accuracy over 1-versus-rest TWSVM, but loses sparsity.
  • Multi-class structures: OVA-TWSVM solves k-category problems by constructing k TWSVM models under a one-versus-all strategy.Decision-tree TWSVM instead builds binary TWSVM models at nodes of a separating binary tree.
  • Multi-class structures: THKSVM uses a “rest-versus-1” structure, constructing k hypersphere classifiers with class-specific centers and radiuses.It is reported to compute faster than Twin-KSVC.
  • Imbalanced data: Weighted and regularized least-squares TWSVM variants address multi-class imbalanced data by assigning loss-function weights to control classifier sensitivity.Experimental results report superiority and feasibility for multi-class imbalanced problems.
  • Projection-based methods: Projection-based variants determine class-specific axes through linear equations or multiple QPPs, with recursive regularization improving generalization but increasing complexity.Multiple recursive projection TWSVM becomes complex when more orthogonal projection axes are generated.
  • Comparative review: A review of multi-class TWSVMs compares one-versus-rest, one-versus-one, binary-tree, all-versus-one, and related structures, reporting generally higher performance for one-versus-one methods.The surveyed structures each have advantages and disadvantages.

3.10 Twin Support Vector Machine for Semi-Supervised Learning

Semi-supervised TWSVM methods use labeled and unlabeled data to address insufficient labels, incorporating geometric, graph, tree, clustering, and fuzzy information into classification or clustering.

  • Motivation: Semi-supervised learning combines labeled and unlabeled data because labeled examples are not easily available in many real-world challenges.Insufficient supervised information can deteriorate supervised-learning performance.
  • Laplacian methods: Laplacian-TWSVM uses the geometric information of unlabeled samples under a low-dimensional data assumption while solving two QPPs.It is reported to improve flexibility, prediction accuracy, and generalization over conventional TWSVM.
  • Laplacian methods: LTPMSVM constructs a graph of training points using graph Laplacian information and solves two SVM-type QPPs.Experiments report higher classification accuracy than SVM, TPMSVM, Lap-TWSVM, and TWSVM.
  • Tree and active-learning methods: Tree-based and active-learning extensions combine Laplacian or decision-tree structures to support semi-supervised multiclass classification and identify informative samples.A tree-based classifier was evaluated on color images, while FLap−TWSVM AL targets informative sample selection.
  • Clustering: Twin support vector clustering partitions samples into k clusters around centers using between-cluster and within-cluster information, with NNG initialization improving stability and efficiency.The cluster-center planes are determined by solving a series of QPPs.
  • Clustering: Fuzzy least-squares TSVC assigns fuzzy memberships to samples, solves primal rather than dual problems, and achieves comparable clustering accuracy with less training time.Its reported accuracy is comparable to TSVC.
  • Robust clustering: Weighted-linear-loss and fuzzy semi-supervised TSVC variants use regularization or continuous loss functions to improve accuracy, computational time, and noise robustness.The fuzzy extension is described as less sensitive to noise and better than TSVC and WLL-TSVC in clustering accuracy and computational time.
  • Robust clustering: Robust TSVC replaces squared L2-norm distance with L1-norm distance and uses linear equations instead of repeated QPPs to reduce noise sensitivity and computational complexity.Fast variants are also proposed for nonlinear cases.

4 Applications of Twin Support Vector Classification

TWSVM has been applied across medical diagnosis, speech and text recognition, intrusion detection, activity and image recognition, EEG analysis, denoising, and software or industrial defect detection, with many variants reporting promising or improved performance.

  • Medical applications: TWSVM applications span medical diagnosis, including breast cancer, hepatitis, diabetes, pathological brain detection, cardiac disease, and Alzheimer’s disease.AULSTWSVM is reported at 95% accuracy for Alzheimer’s disease detection.
  • Speech and text: TWSVM variants have been used for speaker recognition and text recognition or categorization, with several studies reporting better or highest accuracy than comparison methods.Applications include GMM-based speaker recognition, manifold-regularized text recognition, and pinball-based text categorization.
  • Security: TWSVM has been applied to network intrusion detection, where reported results show better accuracy than other intrusion-detection algorithms.The passage frames intrusion detection as a critical network-security component.
  • Activity recognition: Least-squares, energy-based, robust, and Harris-detector-assisted TWSVM variants have been applied to human activity and action recognition, including settings with outliers or noise.The Harris-detector-assisted method is reported to achieve the highest accuracy among state-of-the-art methods.
  • Image processing: TWSVM-based methods support image denoising by preserving informative edges and textures while improving image quality under subjective and objective assessments.The edge/texture-preserving method is reported as better than other available image-denoising methods.
  • EEG applications: EEG applications use universum selection, discriminative frequency bands, entropy-based features, wavelet transforms, and multiple TWSVM variants for mental-activity and seizure classification.Universum TWSVM is described as insensitive to outliers because it selects universum points from EEG datasets.
  • Other applications: TWSVM has also been used for software-bug prediction, steel-surface defect detection, image or face recognition, and facial-expression recognition.Cuckoo-search optimization is combined with TWSVM for software-defect prediction, with reported accuracy improvements.

5 Basic theory of Twin Support Vector Regression

TSVR adapts the twin formulation to regression by learning two nonparallel bound functions through two QPPs and averaging them into a final regressor. Subsequent work questions whether the original formulation fully matches TWSVM’s spirit and proposes alternative formulations and enhancements.

  • TSVR formulation: TSVR was proposed as a TWSVM-inspired regression algorithm that obtains a regressor by solving two quadratic programming problems.The original formulation is presented as an efficient twin-support-vector regression approach.
  • TSVR formulation: The final TSVR prediction is the mean of ε-insensitive up- and down-bound functions.For nonlinear regression, kernel surfaces replace linear hyperplanes.
  • Reported properties: TSVR is reported to require less computational time than standard SVR and to have better generalization ability.These are reported properties of TSVR in the review’s basic-theory discussion.
  • Linear case: The linear TSVR formulation defines two hyperplanes and two QPPs with positive regularization parameters, ε-insensitivity parameters, and nonnegative slack variables.The constraints separately bound the target from above and below.
  • Formulation debate: A later critique argues that the original TSVR is not in the true spirit of TWSVM and motivates an alternative formulation based on the twin-support-vector idea.The alternative approach is explicitly presented as a response to that formulation concern.

6 Research progress on Twin Support Vector Regression

Research on TSVR has expanded through weighting, robust losses, regularization, primal formulations, and specialized optimization strategies. These variants target over-fitting, noise sensitivity, sparsity, computational time, and generalization.

  • Weighting and robustness: Weighted TSVR assigns different sample influences, while KNNWTSVR bases penalties on local K-nearest-neighbor information.The weighted formulation was reported to avoid over-fitting and provide good generalization ability.
  • Weighting and robustness: TPSVR incorporates prior structural information by seeking projection axes with small projected-point variance around the bound functions.It was proposed because empirical-risk-only TSVR variants can be noise-sensitive.
  • Weighting and robustness: Wavelet-weighted projection TWSVM for regression uses wavelet-based weights to reduce the effect of outliers.The approach addresses the assumption that all samples are equally important.
  • Robustness and optimization: 1-norm TSVR reduces training time and improves generalization, while smooth TSVR makes its QPP positive definite for a unique global solution.These developments address TSVR’s reduced robustness from the squared 2-norm and its optimization properties.
  • Robustness and optimization: Regularized and sparse TSVR variants use l1-norm loss or regularization to improve robustness, sparsity, convexity, or computational time.Sparse least-square TSVR converts primal problems into linear programming problems and reports significantly less computational time.
  • Computational efficiency: Primal, linear-equation, and Lagrangian TSVR formulations reduce learning time while retaining comparable or better generalization and accuracy than TSVR.Reported strategies include P-TSVR, TLSSVR, LTSVR, and Lagrangian dual formulations.
  • Specialized formulations: Further variants address simultaneous learning, interval data, structural distributions, automatic parameter adjustment, and asymmetric penalties for points above and below regression bounds.Asymmetric ν-TSVR uses pinball loss and is reported to be insensitive to noise with better generalization ability.

7 Applications of Twin Support Vector Regression

TSVR variants have been applied to forecasting and positioning problems, including inflation forecasting, stock-price prediction, and indoor positioning. The section also organizes non-linear TSVR methods by performance and algorithmic properties.

  • Forecasting: L1 −ϵ-TSVR ranked important features for China’s inflation forecasting and achieved better accuracy than ordinary least squares models.An ϵ-wavelet TSVR was also implemented for inflation forecasting.
  • Comparative summaries: Table 6 presents the performance of various non-linear TSVR-based algorithms.
  • Forecasting: Polynomial smooth TSVR achieved better regression performance than SVR and TSVR in stock-price prediction experiments.
  • Positioning: GA-based TSVR improved indoor-positioning precision over k-nearest neighbor and neural-network methods, while remaining comparable to SVR with significantly less computational time.
  • Comparative summaries: Table 3 compares major TSVR methods using structural risk minimization, sparsity, matrix inversion, and noise insensitivity.
  • Comparative summaries: Table 7 presents the performance of various non-linear TSVR-based algorithms.

8 Future research and development prospects

The review identifies unresolved limitations in TWSVM and TSVR and proposes research on scalability, kernels, multiclass learning, data handling, applications, sparsity, and parameter selection.

  • TWSVM prospects: TWSVM remains at a primitive development stage, lacks practical application background, and has low generalization ability and limited sparsity.
  • TWSVM prospects: Future TWSVM work includes large-scale classification, kernel and parameter selection, multiclass imbalance, sparsity, LDA comparisons, and broader application development.
  • TSVR prospects: TSVR is faster than conventional SVR with better generalization ability but lacks model-complexity control, loses sparsity, and is sensitive to outliers.
  • TSVR prospects: Future TSVR research should develop efficient sparse algorithms, improve data cleaning and preprocessing, broaden applications beyond limited continuous-variable problems, and improve hyperparameter selection.
Loading 2105.00336v3…