PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Imbalanced Classification

Enlu Lin, Qiong Chen, Xiaoming Qi

arXiv:1901.01379v1cs.LGcs.AIstat.ML

TL;DR

Highly imbalanced data challenge conventional classifiers because they favor majority instances and can miss minority cases. This paper models classification as sequential decision-making and trains a DQN agent with minority-sensitive rewards. Experiments report better performance than other imbalanced-classification methods, including a 7.7% higher G-mean score than CSM on IMDB in the reported setting.

  • Problem

    Highly skewed class distributions make conventional classifiers favor majority instances and distort minority-instance recognition.

  • Method

    The paper formulates imbalanced classification as sequential decision-making and uses a DQN agent with a reward function assigning larger feedback to minority samples.

  • Results

    The model outperforms other imbalanced-classification methods; on IMDB, its G-mean score is normally 7.7% higher than CSM and 12.3% higher at an imbalance ratio of 2%.

  • Takeaways & Limitations

    The proposed reward-guided DQN policy identifies more minority samples and maintains better classification performance as imbalance increases.

Abstract

from arXiv · show

Data in real-world application often exhibit skewed class distribution which poses an intense challenge for machine learning. Conventional classification algorithms are not effective in the case of imbalanced data distribution, and may fail when the data distribution is highly imbalanced. To address this issue, we propose a general imbalanced classification model based on deep reinforcement learning. We formulate the classification problem as a sequential decision-making process and solve it by deep Q-learning network. The agent performs a classification action on one sample at each time step, and the environment evaluates the classification action and returns a reward to the agent. The reward from minority class sample is larger so the agent is more sensitive to the minority class. The agent finally finds an optimal classification policy in imbalanced data under the guidance of specific reward function and beneficial learning environment. Experiments show that our proposed model outperforms the other imbalanced classification algorithms, and it can identify more minority samples and has great classification performance.

I. INTRODUCTION

Imbalanced classification is difficult because real-world data can be highly skewed, causing conventional learners to favor majority instances and miss minority cases. The paper proposes a deep reinforcement learning framework that treats classification as sequential decision-making with minority-sensitive rewards.

  • Motivation: Highly skewed real-world data make minority instances difficult to detect, while conventional models often recognize majority instances better.The cited examples include abnormal detection, disease diagnosis, and risk behavior recognition.
  • Related work: Existing approaches mainly rebalance data through resampling or modify learners by assigning higher misclassification costs to minority classes.These are described as data-level and algorithmic-level methods, respectively.
  • Research gap: Deep reinforcement learning has been applied to classification for noise elimination and feature learning, but few studies have addressed imbalanced data learning.Prior work also constructed classification as sequential decision-making, although multi-agent simulation introduced high time complexity.
  • Proposed approach: The proposed DQN-based model formulates imbalanced classification as a sequential decision-making process in which an agent classifies one sample at each step and receives rewards.Correct actions receive positive rewards and incorrect actions receive negative rewards, with minority-class rewards weighted more heavily.
  • Evaluation: The paper evaluates its model experimentally against other imbalanced classification methods and studies its classification performance.The stated contributions include constructing the environment, defining interaction rules, and designing a specific reward function.

III. METHODOLOGY

The methodology represents imbalanced classification as the ICMDP, a sequential process where an agent receives training samples as states, predicts labels as actions, and obtains environment rewards. The agent seeks a policy that maximizes cumulative rewards while episodes proceed through the shuffled training samples.

  • Imbalanced Classification Markov Decision Process: ICMDP decomposes imbalanced classification into a sequential decision-making problem in which a classifier evolves as an agent.The training set is represented as labeled samples D = {(x_1, l_1), ..., (x_n, l_n)}.
  • State: Each environment state corresponds to the current training sample, beginning with x_1; a new episode shuffles the sample order.The state at time t corresponds to sample x_t.
  • Action: Each action predicts a class label, with binary actions 0 and 1 representing the minority and majority classes.The action is associated with the training-set label space.
  • Reward: The environment assigns larger rewards or punishments to correct or incorrect classifications of minority samples.This reward design makes the agent more sensitive to minority-class samples.
  • Transition and episode: Transitions are deterministic and follow the training-data sample order, while γ ∈[0, 1] balances immediate and future rewards.An episode follows a trajectory until all samples are classified or a terminal condition is reached.
  • Objective: The policy π_θ maps states to actions and is treated as the classifier; the target is an optimal policy π* maximizing cumulative rewards.The agent’s objective is to correctly recognize as many training samples as possible.

B. Reward function for imbalanced data classification

The reward function assigns stronger feedback to minority-class predictions and uses λ to control the importance of majority-class samples. Deep Q-learning then optimizes the policy through Q-function estimates and cumulative rewards.

  • Reward definition: The reward function gives minority samples rewards of +1 or −1 and majority samples rewards of λ or −λ, depending on prediction correctness.Here λ ∈[0, 1], and minority and majority sample sets are denoted D_P and D_N.
  • Reward interpretation: For imbalanced data with λ < 1, minority-class prediction costs exceed majority-class costs, whereas balanced data use λ = 1 for equal costs.The reward value is interpreted as the agent’s prediction cost.
  • Reward interpretation: λ acts as a trade-off parameter that adjusts the importance of majority samples in the classification objective.Changing λ changes the influence of majority samples on classification performance.
  • Reward exploration: In experiments, performance is best when the majority-class reward λ equals the imbalance ratio ρ.The paper reports this relationship across datasets with distinct imbalance degrees and across datasets sharing an imbalance ratio.
  • Deep Q-learning: The classification policy receives a sample and returns label probabilities, while Q-learning selects actions using Q-function estimates that represent state-action quality.The greedy policy under the optimal Q* function maximizes cumulative rewards.
  • Deep Q-learning: Deep Q-learning stores interaction transitions in replay memory, samples mini-batches, and updates the Q network by minimizing its loss.The target estimate uses the immediate reward for terminal transitions and a discounted next-state estimate otherwise.

2) Influence of reward function:

The reward function counteracts majority-class bias by weighting minority-class feedback differently. When λ matches the imbalance ratio, minority- and majority-class contributions to the Q-network loss become equal.

  • Different class rewards can make minority and majority samples have the same impact on the Q network.
  • The reward design separates minority- and majority-class contributions to the Q-network loss.
  • When λ = 1 and N > P, majority-class loss contribution exceeds minority-class contribution because majority samples are more numerous.
  • When λ < 1, reducing negative-sample rewards weakens their impact on the Q-network loss function.
  • λ equal to the imbalance ratio ρ makes the minority- and majority-class loss contributions equal.

3) Training details:

The simulation environment follows the ICMDP definition, and a deep Q-learning network is trained through repeated agent–environment interactions. The converged Q network, combined with softmax, serves as an imbalanced-data classifier.

  • The Q network takes training samples as input and produces one output per sample category.Its architecture depends on the complexity and amount of training data.
  • At each episode step, the agent selects an action with an ϵ-greedy policy and receives a reward through the environment's STEP function.
  • After about 120000 parameter-update iterations, the converged Q network is saved and paired with a softmax layer to form a classifier.

A. Comparison Methods and Evaluation Metrics

DQNimb is evaluated against sampling, cost-sensitive, loss-based, threshold-adjustment, and cross-entropy baseline methods using G-mean and F-measure. Higher scores on both metrics indicate better performance.

  • Comparison Methods: DQNimb is compared with five imbalanced-learning methods and a deep neural network trained with cross-entropy loss.The alternatives include sampling techniques, cost-sensitive learning, and decision-threshold adjustment.
  • Comparison Methods: ROS balances data by randomly replicating minority-class samples, whereas RUS removes majority-class samples randomly.
  • Comparison Methods: MFE uses mean false error loss, CSM assigns greater minority-class misclassification costs, and DTA adjusts the test-time decision threshold using class prior probability.
  • Evaluation Metrics: G-mean is the geometric mean of sensitivity and precision, while F-measure is the harmonic mean between recall and precision.
  • Evaluation Metrics: Higher G-mean and F-measure scores indicate better algorithmic performance on imbalanced datasets.

B. Dataset

Experiments study binary imbalanced classification on IMDB, Cifar-10, Mnist, and Fashion-Mnist using deliberately imbalanced simulated datasets. Imbalance is controlled by reducing positive samples to ρ×N relative to the negative-class count.

  • Dataset: Experiments evaluate binary imbalanced classification on IMDB, Cifar-10, Mnist, and Fashion-Mnist with deliberately imbalanced splits.
  • Dataset: IMDB contains 50000 movie reviews labeled positive or negative, with reviews preprocessed as sequences of word indexes.Positive reviews are treated as the positive class.
  • Dataset: Mnist provides 28 × 28 grayscale digit images across 10 classes; images labeled 2 are designated positive and the remaining images negative.
  • Dataset: Fashion-Mnist contains 70000 28 × 28 grayscale fashion-product images across 10 categories, with two differently sized simulated datasets extracted for evaluation.
  • Dataset: Cifar-10 contains 32x32 color images across 10 natural-object classes, and two simulated datasets use selected labels as positive and negative classes.
  • Dataset: The positive-class training count is reduced to ρ×N, where N is the total negative-class count and ρ is the dataset imbalance ratio.The experiment dataset details are reported in Table I.

C. Network Architecture

The compared models use deep neural networks for feature representation across high-dimensional datasets, while DQNimb removes the final softmax layer from its Q network. Experimental tables report results on balanced datasets and G-mean scores.

  • Network Architecture: Compared algorithms use deep neural networks to learn feature representations from imbalanced, high-dimensional datasets.The architectures include embedding and fully connected layers for text, and convolutional and fully connected layers for image classification.
  • Network Architecture: Table IV reports experiment results on balanced datasets.
  • Network Architecture: Table V reports G-mean scores for the experimental results.

D. Parameter Setting

DQNimb is evaluated with specified DQN training settings and compared against supervised and imbalanced-learning baselines. It performs slightly better than DNN on balanced data and shows strong G-mean and F-measure performance under imbalance.

  • Parameter Setting: DQNimb uses an ϵ-greedy policy, 50,000 replay-memory capacity, approximately 120,000 interactions, γ = 0.1, and Adam optimization.The exploration probability decreases linearly from 1.0 to 0.01, with a learning rate of 0.00025.
  • Experiment Result: On balanced datasets, DQNimb and DNN both demonstrate good performance, with DQNimb slightly higher in G-mean and F-measure.
  • Experiment Result: 7.7% higher G-mean than CSM is achieved by DQNimb on IMDB normally, increasing to 12.3% when the imbalance ratio is 2%.
  • Experiment Result: As imbalance increases, every algorithm’s F-measure declines, but DQNimb has the smallest decrease while DNN suffers the most serious decline.
  • Experiment Result: Experience replay and higher rewards or penalties for positive samples address skew at the data and algorithmic levels.

F. Exploration On Reward Function

The reward-function study varies the majority-class reward relative to the imbalance ratio. Performance is best when the majority-class reward equals the imbalance ratio, while deviations disrupt the reward balance.

  • F. Exploration On Reward Function: The study tests majority-class rewards λ at 0.05ρ, 0.1ρ, 0.5ρ, ρ, 5ρ, 10ρ, and 20ρ.The minority-class reward is 1, and the preceding experiments set λ = ρ.
  • F. Exploration On Reward Function: Performance is best when the majority-class reward λ equals the imbalance ratio ρ on the same dataset with different imbalance levels.
  • F. Exploration On Reward Function: When datasets share an imbalance ratio, performance improves as λ approaches ρ.
  • F. Exploration On Reward Function: Changing λ from ρ disrupts the balance of terms in the Q-network loss function and leads to poorer classification performance.
  • V. CONCLUSION: DQNimb combines a sequential decision process, reward differentiation between classes, and deep Q learning to find a classification policy.
Loading 1901.01379v1…