PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Imbalanced Classification

Enlu Lin, Qiong Chen, Xiaoming Qi

arXiv:1901.01379v1cs.LGcs.AIstat.ML

TL;DR

Highly imbalanced data challenge conventional classifiers because they favor majority instances and can miss minority cases. This paper models classification as sequential decision-making and trains a DQN agent with minority-sensitive rewards. Experiments report better performance than other imbalanced-classification methods, including a 7.7% higher G-mean score than CSM on IMDB in the reported setting.

Abstract

from arXiv · show

Data in real-world application often exhibit skewed class distribution which poses an intense challenge for machine learning. Conventional classification algorithms are not effective in the case of imbalanced data distribution, and may fail when the data distribution is highly imbalanced. To address this issue, we propose a general imbalanced classification model based on deep reinforcement learning. We formulate the classification problem as a sequential decision-making process and solve it by deep Q-learning network. The agent performs a classification action on one sample at each time step, and the environment evaluates the classification action and returns a reward to the agent. The reward from minority class sample is larger so the agent is more sensitive to the minority class. The agent finally finds an optimal classification policy in imbalanced data under the guidance of specific reward function and beneficial learning environment. Experiments show that our proposed model outperforms the other imbalanced classification algorithms, and it can identify more minority samples and has great classification performance.

I. INTRODUCTION

Imbalanced classification is difficult because real-world data can be highly skewed, causing conventional learners to favor majority instances and miss minority cases. The paper proposes a deep reinforcement learning framework that treats classification as sequential decision-making with minority-sensitive rewards.

Motivation

  • Highly skewed real-world data make minority instances difficult to detect, while conventional models often recognize majority instances better.The cited examples include abnormal detection, disease diagnosis, and risk behavior recognition.Source: §I · p. 1

Related work

  • Existing approaches mainly rebalance data through resampling or modify learners by assigning higher misclassification costs to minority classes.These are described as data-level and algorithmic-level methods, respectively.Source: §I · p. 1, A. Imbalanced data classification · p. 2

Research gap

  • Deep reinforcement learning has been applied to classification for noise elimination and feature learning, but few studies have addressed imbalanced data learning.Prior work also constructed classification as sequential decision-making, although multi-agent simulation introduced high time complexity.Source: §I · p. 1, B. Reinforcement learning for… · p. 2

Proposed approach

  • The proposed DQN-based model formulates imbalanced classification as a sequential decision-making process in which an agent classifies one sample at each step and receives rewards.Correct actions receive positive rewards and incorrect actions receive negative rewards, with minority-class rewards weighted more heavily.Source: §I · p. 1

Evaluation

  • The paper evaluates its model experimentally against other imbalanced classification methods and studies its classification performance.The stated contributions include constructing the environment, defining interaction rules, and designing a specific reward function.Source: §I · p. 1

III. METHODOLOGY

The methodology represents imbalanced classification as the ICMDP, a sequential process where an agent receives training samples as states, predicts labels as actions, and obtains environment rewards. The agent seeks a policy that maximizes cumulative rewards while episodes proceed through the shuffled training samples.

Imbalanced Classification Markov Decision Process

  • ICMDP decomposes imbalanced classification into a sequential decision-making problem in which a classifier evolves as an agent.The training set is represented as labeled samples D = {(x_1, l_1), ..., (x_n, l_n)}.Source: A. Imbalanced Classification Markov… · pp. 2–3

State

Action

Reward

  • The environment assigns larger rewards or punishments to correct or incorrect classifications of minority samples.This reward design makes the agent more sensitive to minority-class samples.Source: A. Imbalanced Classification Markov… · p. 3

Transition and episode

  • Transitions are deterministic and follow the training-data sample order, while γ ∈[0, 1] balances immediate and future rewards.An episode follows a trajectory until all samples are classified or a terminal condition is reached.Source: A. Imbalanced Classification Markov… · p. 3

Objective

  • The policy π_θ maps states to actions and is treated as the classifier; the target is an optimal policy π* maximizing cumulative rewards.The agent’s objective is to correctly recognize as many training samples as possible.Source: A. Imbalanced Classification Markov… · p. 3

B. Reward function for imbalanced data classification

The reward function assigns stronger feedback to minority-class predictions and uses λ to control the importance of majority-class samples. Deep Q-learning then optimizes the policy through Q-function estimates and cumulative rewards.

Reward definition

  • The reward function gives minority samples rewards of +1 or −1 and majority samples rewards of λ or −λ, depending on prediction correctness.Here λ ∈[0, 1], and minority and majority sample sets are denoted D_P and D_N.Source: B. Reward function for imbalanced data… · p. 3

Reward interpretation

  • For imbalanced data with λ < 1, minority-class prediction costs exceed majority-class costs, whereas balanced data use λ = 1 for equal costs.The reward value is interpreted as the agent’s prediction cost.Source: B. Reward function for imbalanced data… · p. 3
  • λ acts as a trade-off parameter that adjusts the importance of majority samples in the classification objective.Changing λ changes the influence of majority samples on classification performance.Source: B. Reward function for imbalanced data… · p. 3

Reward exploration

  • In experiments, performance is best when the majority-class reward λ equals the imbalance ratio ρ.The paper reports this relationship across datasets with distinct imbalance degrees and across datasets sharing an imbalance ratio.Source: B. Reward function for imbalanced data… · p. 3

Deep Q-learning

  • The classification policy receives a sample and returns label probabilities, while Q-learning selects actions using Q-function estimates that represent state-action quality.The greedy policy under the optimal Q* function maximizes cumulative rewards.Source: 1) Deep Q-learning for ICMDP · pp. 3–4
  • Deep Q-learning stores interaction transitions in replay memory, samples mini-batches, and updates the Q network by minimizing its loss.The target estimate uses the immediate reward for terminal transitions and a discounted next-state estimate otherwise.Source: 1) Deep Q-learning for ICMDP · p. 4

2) Influence of reward function:

The reward function counteracts majority-class bias by weighting minority-class feedback differently. When λ matches the imbalance ratio, minority- and majority-class contributions to the Q-network loss become equal.

3) Training details:

The simulation environment follows the ICMDP definition, and a deep Q-learning network is trained through repeated agent–environment interactions. The converged Q network, combined with softmax, serves as an imbalanced-data classifier.

  • The Q network takes training samples as input and produces one output per sample category.Its architecture depends on the complexity and amount of training data.Source: 3) Training details · p. 5
  • At each episode step, the agent selects an action with an ϵ-greedy policy and receives a reward through the environment's STEP function.Source: 3) Training details · p. 5
  • After about 120000 parameter-update iterations, the converged Q network is saved and paired with a softmax layer to form a classifier.Source: 3) Training details · p. 5

A. Comparison Methods and Evaluation Metrics

DQNimb is evaluated against sampling, cost-sensitive, loss-based, threshold-adjustment, and cross-entropy baseline methods using G-mean and F-measure. Higher scores on both metrics indicate better performance.

Comparison Methods

Evaluation Metrics

B. Dataset

Experiments study binary imbalanced classification on IMDB, Cifar-10, Mnist, and Fashion-Mnist using deliberately imbalanced simulated datasets. Imbalance is controlled by reducing positive samples to ρ×N relative to the negative-class count.

Dataset

  • Experiments evaluate binary imbalanced classification on IMDB, Cifar-10, Mnist, and Fashion-Mnist with deliberately imbalanced splits.Source: B. Dataset · p. 5
  • IMDB contains 50000 movie reviews labeled positive or negative, with reviews preprocessed as sequences of word indexes.Positive reviews are treated as the positive class.Source: B. Dataset · pp. 5–6
  • Mnist provides 28 × 28 grayscale digit images across 10 classes; images labeled 2 are designated positive and the remaining images negative.Source: B. Dataset · p. 6
  • Fashion-Mnist contains 70000 28 × 28 grayscale fashion-product images across 10 categories, with two differently sized simulated datasets extracted for evaluation.Source: B. Dataset · p. 6
  • Cifar-10 contains 32x32 color images across 10 natural-object classes, and two simulated datasets use selected labels as positive and negative classes.Source: B. Dataset · p. 6
  • The positive-class training count is reduced to ρ×N, where N is the total negative-class count and ρ is the dataset imbalance ratio.The experiment dataset details are reported in Table I.Source: B. Dataset · p. 6, Table I · p. 6

C. Network Architecture

The compared models use deep neural networks for feature representation across high-dimensional datasets, while DQNimb removes the final softmax layer from its Q network. Experimental tables report results on balanced datasets and G-mean scores.

Network Architecture

  • Compared algorithms use deep neural networks to learn feature representations from imbalanced, high-dimensional datasets.The architectures include embedding and fully connected layers for text, and convolutional and fully connected layers for image classification.Source: C. Network Architecture · p. 6
  • Table IV reports experiment results on balanced datasets.Source: Table IV · p. 6
  • Table V reports G-mean scores for the experimental results.Source: Table V · p. 7

D. Parameter Setting

DQNimb is evaluated with specified DQN training settings and compared against supervised and imbalanced-learning baselines. It performs slightly better than DNN on balanced data and shows strong G-mean and F-measure performance under imbalance.

Parameter Setting

  • DQNimb uses an ϵ-greedy policy, 50,000 replay-memory capacity, approximately 120,000 interactions, γ = 0.1, and Adam optimization.The exploration probability decreases linearly from 1.0 to 0.01, with a learning rate of 0.00025.Source: D. Parameter Setting · p. 7

Experiment Result

  • On balanced datasets, DQNimb and DNN both demonstrate good performance, with DQNimb slightly higher in G-mean and F-measure.Source: E. Experiment Result · p. 7
  • 7.7% higher G-mean than CSM is achieved by DQNimb on IMDB normally, increasing to 12.3% when the imbalance ratio is 2%.Source: E. Experiment Result · p. 7
  • As imbalance increases, every algorithm’s F-measure declines, but DQNimb has the smallest decrease while DNN suffers the most serious decline.Source: E. Experiment Result · p. 7
  • Experience replay and higher rewards or penalties for positive samples address skew at the data and algorithmic levels.Source: E. Experiment Result · pp. 7–8

F. Exploration On Reward Function

The reward-function study varies the majority-class reward relative to the imbalance ratio. Performance is best when the majority-class reward equals the imbalance ratio, while deviations disrupt the reward balance.

F. Exploration On Reward Function

V. CONCLUSION

  • DQNimb combines a sequential decision process, reward differentiation between classes, and deep Q learning to find a classification policy.Source: §V · pp. 8–9

Related papers

Loading 1901.01379v1…