Source-linked AI summary
Optimize TSK Fuzzy Systems for Regression Problems: Mini-Batch Gradient Descent with Regularization, DropRule and AdaBound (MBGD-RDA)
Dongrui Wu, Ye Yuan, Yihua Tan
TL;DR
Large-N regression poses challenges for existing TSK fuzzy-system training strategies. The paper adapts neural-network optimization methods and adds TSK-specific techniques in MBGD-RDA, which outperforms RR on all 10 datasets and is more stable than MBGD-A.
Problem
Existing TSK optimization approaches face challenges with large datasets, and the authors identify a lack of efficient training methods for that setting.
Source: §I · p. 1Method
MBGD-RDA extends mini-batch gradient descent, regularization, and AdaBound to TSK systems and adds DropRule, DropMF, and DropMembership.
Source: §I · p. 2, §IV · p. 11Results
MBGD-RDA achieved smaller best test RMSEs than RR on all 10 datasets and performed more consistently than MBGD-A.
Source: C. Experimental Results · p. 7Takeaways & Limitations
The authors recommend MBGD-RDA over MBGD-A in practice because its performance was more stable.
Source: C. Experimental Results · p. 7
Abstract
from arXiv · showhide
Takagi-Sugeno-Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance, and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., mini-batch gradient descent, regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) specifically for training TSK fuzzy systems. Our final algorithm, mini-batch gradient descent with regularization, DropRule and AdaBound (MBGD-RDA), can achieve fast convergence in training TSK fuzzy systems, and also superior generalization performance in testing. It can be used for training TSK fuzzy systems on datasets of any size; however, it is particularly useful for big datasets, on which currently no other efficient training algorithms exist.
I. INTRODUCTION
The paper targets efficient, effective training of TSK fuzzy systems for regression, especially when datasets are large. It adapts neural-network optimization methods and proposes TSK-specific techniques, combining them in MBGD-RDA.
I. INTRODUCTION
- Large training sets challenge existing TSK optimization strategies; the paper focuses on large N after assuming input dimensionality can be reduced effectively.Evolutionary methods require evaluating each population member on the full dataset, while ANFIS may overfit regression problems.Source: §I · p. 1
- Mini-batch gradient descent offers a compromise when batch GD is too slow or impossible and stochastic GD is unstable.It computes updates from randomly selected small batches of training examples.Source: §I · p. 2
- The authors identify gaps in applying AdaBound, regularization, and dropout-like techniques to TSK fuzzy-system training.Prior TSK regularization approaches used LSE for consequents, and the paper notes that DropOut- and DropConnect-like methods were absent for TSK systems.Source: §I · p. 2
- MBGD-RDA extends MBGD, regularization, and AdaBound to TSK systems and introduces DropRule, DropMF, and DropMembership.The authors report superior performance on 10 real-world datasets spanning different domains and sizes.Source: §I · p. 2
II. THE MBGD-RDA ALGORITHM
MBGD-RDA trains TSK fuzzy systems by updating their antecedent and consequent parameters through a mini-batch procedure. Its setup uses Gaussian membership functions and combines DropRule, regularization, and AdaBound in the training algorithm.
A. The TSK Fuzzy System
- The TSK system uses rules with Gaussian membership functions, whose centers, standard deviations, and consequent parameters are tuned.The paper chooses Gaussian functions for easier derivative computation but says the algorithm also applies to other membership-function shapes with computable derivatives.Source: A. The TSK Fuzzy System · p. 2, // DropRule · p. 4
II. THE MBGD-RDA ALGORITHM
- The algorithm takes labeled examples, a loss, model parameters, batch size, DropRule rate, learning-rate and regularization settings, and AdaBound parameters as inputs.It initializes membership-function centers and standard deviations from the input ranges and sets rule consequents to zero.Source: A. The TSK Fuzzy System · p. 3
A. The TSK Fuzzy System
- Each iteration samples a mini-batch, computes the system outputs and gradients, and applies DropRule by setting a rule's firing level to zero when selected.The sampled rule is selected using a uniform random draw and the DropRule rate.Source: A. The TSK Fuzzy System · p. 3, // DropRule · p. 3
- The algorithm adds regularization to consequent-coefficient gradients and then uses AdaBound moment estimates in its parameter update.After training, all rules contribute to the output for a new input.Source: // DropRule · p. 3
B. Regularization
The section specifies an ℓ2-regularized loss for mini-batch training and excludes rule intercept terms from the penalty. The authors follow a neural-network practice of leaving biases unregularized.
B. Regularization
- The loss adds an ℓ2 regularization term to the mini-batch objective, with λ ≥ 0 controlling its strength.The batch size Nbs is between 1 and N.Source: B. Regularization · p. 4
- Rule intercepts br,0 are excluded from the regularization penalty.The paper treats these terms like neural-network biases.Source: B. Regularization · p. 4
- The authors leave biases unregularized because they state biases typically require less data to fit accurately than weights.They describe this as a typical neural-network approach.Source: B. Regularization · p. 4
C. Mini-Batch Gradient Descent (MBGD)
MBGD computes gradients from randomly sampled training examples and uses them to update TSK antecedent and consequent parameters. The batch size determines whether it behaves as stochastic or batch gradient descent.
C. Mini-Batch Gradient Descent (MBGD)
- The loss gradients include an indicator function that ensures rule intercepts br,0 are not regularized.Source: C. Mini-Batch Gradient Descent (MBGD) · p. 4
- Each update uses gradients computed from a randomly sampled batch and moves the parameter vector according to the learning rate α.Source: C. Mini-Batch Gradient Descent (MBGD) · p. 4
- With Nbs = 1, MBGD is stochastic GD; with Nbs = N, it is batch GD.Source: C. Mini-Batch Gradient Descent (MBGD) · p. 4
D. DropRule
DropRule randomly removes rules during training and restores all rules at test time. The authors motivate it as a way to make each rule model with less reliance on other rules, while noting it may work best with AdaBound.
D. DropRule
- DropRule randomly sets selected rules’ firing levels to zero during training, updates only the rules not dropped, and uses all rules when testing.For each training example, a rule keeps its true firing level with probability P and is set to zero with probability 1 − P.Source: D. DropRule · pp. 4–5
- The figure marks the first rule for dropping and shows the equivalent fuzzy system after that rule is removed.Source: Figure 1 · p. 5
- DropRule aims to make each rule work with randomly chosen subsets of rules rather than rely heavily on other rules, which may help generalization.The authors report that DropRule alone may not always offer advantages, but works well when integrated with AdaBound.Source: D. DropRule · p. 5
E. Adam and AdaBound
Adam adapts learning rates separately for model parameters using gradient-moment estimates; AdaBound bounds those rates and tightens the bounds over iterations. The paper adopts AdaBound for TSK fuzzy-system training.
E. Adam and AdaBound
- Adam computes an individualized adaptive learning rate for each model parameter from first- and second-moment gradient estimates.The paper describes Adam as combining AdaGrad’s performance with sparse gradients and RMSProp’s performance in online and non-stationary settings.Source: E. Adam and AdaBound · p. 5
- AdaBound bounds each parameter’s adaptive learning rate above and below, with tighter bounds over iterations that make the rate approach a constant.The paper says the bounds start at [0, +∞) and approach [0.01, 0.01] as k approaches +∞.Source: E. Adam and AdaBound · pp. 5–6
- AdaBound was adopted because prior work reported faster convergence and better generalization than Adam.The paper applies its bounded individualized learning rates to the TSK system’s parameter vector.Source: E. Adam and AdaBound · p. 6
III. EXPERIMENTS
The experiments use ten regression datasets ranging from small to large. The dataset setup standardizes numerical features and limits input dimensionality to five, using PCA when needed.
III. EXPERIMENTS
- The experiment evaluates the proposed MBGD-RDA on ten regression datasets from the CMU StatLib archive and UCI repository, spanning small to large sizes.Source: A. Datasets · p. 6, §III · p. 6, Table II · p. 6
A. Datasets
- Abalone’s categorical sex feature was ignored, because converting it to one-hot features produced worse RMSEs than ignoring it.Source: A. Datasets · p. 6
- Numerical features were z-normalized and the output mean was subtracted; datasets with more than five features were reduced to five inputs with PCA.Source: A. Datasets · p. 6
- Each input domain used two Gaussian membership functions, and the resulting TSK system’s parameter count depends on the number of inputs and rules.Source: A. Datasets · p. 6
B. Algorithms
The experiments compare seven TSK regression algorithms and report that MBGD-RDA and MBGD-A achieve the strongest test performance and faster convergence. MBGD-RDA is more stable than MBGD-A and is the fastest of the MBGD-based methods.
B. Algorithms
- The seven algorithms comprise ridge regression and six MBGD variants, combining regularization, DropRule, and AdaBound in different configurations.MBGD-RDA combines all three additions; the variants use mini-batches of 64 and the reported settings include λ = 0.05 for regularization and P = 0.5 for DropRule.Source: B. Algorithms · p. 6
- Each dataset was split into 70% training and 30% test examples; the TSK systems used two Gaussian membership functions per input and ran up to 500 iterations, with results averaged over 10 repetitions.The repeated runs account for randomness in the data split and mini-batch selection.Source: B. Algorithms · pp. 6–7
- Support vector regression was tried but omitted because it was too time-consuming to train on big datasets.Source: B. Algorithms · p. 6
C. Experimental Results
- MBGD-RDA’s best test RMSE was smaller than RR’s on all 10 datasets; MBGD-A achieved this on nine, and both converged much faster than the other MBGD variants.The exception for MBGD-A was Wine-Red.Source: C. Experimental Results · p. 7
- MBGD-RDA was the only enhancement that consistently outperformed MBGD and was more stable than MBGD-A, although it did not always outperform AdaBound alone.MBGD-RDA outperformed MBGD-A on three datasets and had comparable performance on many others.Source: C. Experimental Results · p. 7
- MBGD-RDA was the fastest of the six MBGD-based algorithms over 500 training iterations; DropRule reduced the parameters adjusted per iteration.The reported timing covers 10 regression datasets.Source: C. Experimental Results · p. 7, Table IV · p. 7
D. Parameter Sensitivity
MBGD-RDA was stable across four tested hyper-parameters on PM10 and, on six smaller datasets, converged faster than ANFIS-GD while achieving smaller best test RMSEs. ANFIS-GD-LSE had the smallest training RMSE but almost always the largest test RMSE, with large oscillations.
D. Parameter Sensitivity
- On PM10, MBGD-RDA was stable across a wide range of batch sizes, DropRule rates, initial learning rates, and ℓ2 regularization coefficients.Each hyper-parameter experiment was repeated 100 times and averaged; other parameters were held at the captioned settings.Source: D. Parameter Sensitivity · p. 9, Figure 4 · p. 9
E. Comparison with ANFIS
- MBGD-RDA always converged much faster than ANFIS-GD and always achieved a smaller best test RMSE on the six smaller datasets.The comparison averaged training and test RMSEs over 10 runs; the figure uses batch gradient descent.Source: E. Comparison with ANFIS · p. 9, Figure 6 · p. 10
- ANFIS-GD-LSE had the smallest training RMSE but almost always the largest test RMSE, with large oscillations, suggesting significant overfitting.Reducing that overfitting, for example through regularization, is identified as future research.Source: E. Comparison with ANFIS · p. 9
- MBGD-RDA uses different learning rates for different parameters, while ANFIS-GD and ANFIS-GD-LSE share one rate across all parameters.Its learning rate starts high and rapidly decreases; the authors say the high initial rate helped rapid convergence.Source: E. Comparison with ANFIS · pp. 9–10
F. Comparison with DropMF and DropMembership
DropMF and DropMembership remove different amounts of antecedent structure than DropRule. In the reported comparison, DropRule generally performed best and DropMembership worst.
F. Comparison with DropMF and DropMembership
- DropMF drops each input membership function with probability 1 − P, setting its firing level to 1; unlike DropRule, it does not reduce the number of rules.Instead, it reduces antecedents in multiple rules by one.Source: F. Comparison with DropMF and… · p. 10, Figure 7 · p. 10
- DropMembership drops an input membership with probability 1 − P by setting it to 1, affecting fewer rule firing levels than DropMF.The example contrasts one affected rule for DropMembership with two for DropMF.Source: F. Comparison with DropMF and… · p. 10, Figure 8 · p. 11
- DropRule generally performed best and DropMembership worst; the authors suggest dropout that changes more of the TSK structure may be more beneficial.The performance comparison averages training and test RMSEs over 10 runs.Source: F. Comparison with DropMF and… · pp. 10–11
G. Comparison with Adam
MBGD-RDA converged at least as fast as the Adam variant and usually had smaller or comparable best test RMSEs. The paper also identifies unmodeled real-world data uncertainties as a limitation.
G. Comparison with Adam
- MBGD-RDA converged faster than or equally fast with MBGD-RD-Adam and had smaller or comparable best test RMSEs on most datasets.The algorithms were identical except that AdaBound was replaced by Adam in MBGD-RD-Adam.Source: G. Comparison with Adam · p. 11
IV. Conclusion and Future Research
- The authors position MBGD-RDA as usable on datasets of any size and particularly useful for big datasets lacking other efficient training algorithms.They describe it as combining MBGD, regularization, AdaBound, and DropRule for fast convergence and superior test generalization.Source: §IV · p. 11
- The study does not consider data uncertainties such as missing or wrong values, noise, and outliers, which frequently occur in real-world applications.The authors list rough sets or extensions to interval or general type-2 fuzzy systems as future directions for handling uncertainties.Source: §IV · p. 11