PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving into Deep Imbalanced Regression

Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, Dina Katabi

arXiv:2102.09554v2cs.LGcs.AIcs.CV

TL;DR

Deep Imbalanced Regression addresses the gap between categorical imbalance methods and real-world tasks with continuous, potentially missing target values. The paper proposes label and feature distribution smoothing, benchmarks DIR across multiple domains, and reports consistent performance gains across tasks and target-frequency regions. Its feature-space strategy assumes that nearby targets should have similar feature statistics when the model and data are properly balanced.

  • Problem

    Existing imbalanced-learning techniques focus on categorical targets, leaving continuous-target tasks with nearby-value similarity, missing regions, and full-range generalization insufficiently addressed.

  • Method

    The paper proposes label distribution smoothing and feature distribution smoothing, using nearby-target similarity to calibrate label and learned feature distributions.

  • Results

    Across computer vision, natural language processing, and healthcare benchmarks, LDS and FDS provide consistent gains, especially in medium- and few-shot regions, while improving generalization across target regions.

  • Takeaways & Limitations

    DIR offers a benchmarked framework for evaluating continuous-target imbalance and methods that exploit similarity between nearby targets in label and feature spaces.

  • Takeaways & Limitations

    FDS is motivated by the assumption that nearby targets should have similar feature statistics when the model works properly and the data is balanced.

Abstract

from arXiv · show

Real-world data often exhibit imbalanced distributions, where certain target values have significantly fewer observations. Existing techniques for dealing with imbalanced data focus on targets with categorical indices, i.e., different classes. However, many tasks involve continuous targets, where hard boundaries between classes do not exist. We define Deep Imbalanced Regression (DIR) as learning from such imbalanced data with continuous targets, dealing with potential missing data for certain target values, and generalizing to the entire target range. Motivated by the intrinsic difference between categorical and continuous label space, we propose distribution smoothing for both labels and features, which explicitly acknowledges the effects of nearby targets, and calibrates both label and learned feature distributions. We curate and benchmark large-scale DIR datasets from common real-world tasks in computer vision, natural language processing, and healthcare domains. Extensive experiments verify the superior performance of our strategies. Our work fills the gap in benchmarks and techniques for practical imbalanced regression problems. Code and data are available at https://github.com/YyzHarry/imbalanced-regression.

1. Introduction

The paper defines Deep Imbalanced Regression (DIR) for continuous targets, where imbalance, missing target regions, and nearby-target similarity create challenges that classification methods do not address. It proposes smoothing label and feature distributions and benchmarks these strategies across real-world domains.

  • Deep Imbalanced Regression (DIR) learns continuous targets from naturally imbalanced data, including potentially missing target values, while generalizing across the full target range.
  • Continuous targets lack class boundaries, and nearby values carry meaningful similarity that categorical imbalance methods fail to exploit.Target neighborhoods can differ in effective imbalance even when individual target values have similarly few observations.
  • Label distribution smoothing (LDS) and feature distribution smoothing (FDS) use nearby-target similarity to calibrate distributions in label and feature spaces.Both methods use kernel-based distribution smoothing, integrate with existing deep networks, and support end-to-end optimization.
  • The paper curates large-scale DIR datasets and evaluation benchmarks spanning computer vision, natural language processing, and healthcare.Tasks include single-value prediction such as age and text similarity, plus dense-value prediction such as depth.
  • Extensive experiments report consistent and superior performance for the proposed strategies across benchmark DIR datasets.The contributions include formalizing DIR, developing LDS and FDS, and establishing strong baselines and evaluation benchmarks.

2. Related Work

Prior imbalanced-learning research largely addresses categorical classes, while imbalanced regression remains less explored and is often handled by adapting SMOTE. The paper distinguishes its approach by modeling continuous-target similarity and avoiding unsuitable high-dimensional interpolation.

  • Imbalanced Classification: Imbalanced classification methods use data-based sampling or model-based loss adjustments to address minority and majority classes.Examples include over- or under-sampling, SMOTE, re-weighting, and altered loss functions.
  • Imbalanced Regression: Imbalanced regression research is less developed, with many methods directly adapting SMOTE to predefined rare target regions.Approaches interpolate inputs and targets, add Gaussian noise, or combine preprocessing steps in bagging ensembles.
  • Imbalanced Regression: Existing regression adaptations do not account for distances between continuous targets and instead heuristically divide data into rare and frequent sets.
  • Imbalanced Regression: Linear interpolation is problematic for modern high-dimensional inputs such as images and physiological signals because it may not produce meaningful synthetic samples.

3. Methods

The methods address continuous-target imbalance by estimating effective label density and calibrating feature statistics across nearby target values. They are motivated by the smoother error structure of regression and the distortion of feature statistics in underrepresented target regions.

  • Problem setting: DIR represents continuous targets with equal-interval bins, using deep features z=f(x; θ) and a regression function for prediction.The bins define the minimum target resolution of interest.
  • Motivating example: −0.76 versus −0.47: test error strongly follows label density for CIFAR-100 but is smoother and less correlated for continuous-label IMDB-WIKI.Both datasets use the same training label distribution in the comparison.
  • Label distribution smoothing: Empirical label density is inaccurate for continuous targets because nearby labels share information, so imbalance must account for neighborhood density.A target with few samples can still lie in a highly represented neighborhood, unlike an equally sparse target in a weak neighborhood.
  • Label distribution smoothing: LDS convolves a symmetric kernel with empirical density to estimate effective density, which correlates with error at −0.83.The estimated density can support adaptations such as inverse-density loss re-weighting.
  • Feature distribution smoothing: FDS smooths feature means and covariance across nearby target bins to calibrate biased feature distributions, especially for underrepresented values.Its motivation is that properly learned features for nearby continuous targets should have similar statistics.

4. Benchmarking DIR

The paper benchmarks deep imbalanced regression across five datasets spanning vision, language, and healthcare, using balanced target-region evaluation and shot-based subsets. LDS and FDS consistently improve performance, particularly in underrepresented and missing-target regions.

  • Datasets: Five DIR benchmarks span computer vision, natural language processing, and healthcare, covering single-value and dense-value prediction tasks.The datasets include age, text similarity, health condition scores, and depth estimation.
  • Evaluation: Evaluation divides the target space into many-shot, medium-shot, few-shot, and zero-shot regions to assess performance across data-density levels.Many-shot bins contain over 100 training samples, medium-shot bins 20–100, few-shot bins under 20, and zero-shot regions have no training samples.
  • Main results: LDS and FDS consistently improve regression performance across image, text, depth, and health-score tasks, with the largest gains generally in medium-shot and few-shot regions.On age estimation, gains are maintained or slight in many-shot regions while substantially improving medium- and few-shot regions; depth estimation shows a similar trade-off.
  • Main results: On STS-B-DIR, LDS and FDS provide especially strong improvements under Pearson correlation, a commonly used metric for the NLP task.The methods improve results across different baselines, particularly in medium- and few-shot regions.
  • Extrapolation and interpolation: LDS and FDS improve both target interpolation and extrapolation, with larger gains in regions lacking training data.The methods improve performance on observed regions as well as target regions without data, and interpolation receives larger boosts than extrapolation.
  • Understanding FDS: FDS feature statistics become more accurate during training, allowing the smoothing module to be removed during inference.The average L1 distance between feature statistics decreases as training evolves, indicating progressively improved feature generation.
  • Robustness: LDS and FDS remain robust across common regression losses, including L1, MSE, and Huber loss.Similar results across these losses indicate that the methods are not tied to one specific regression objective.

Ablation: Hyper-parameter for LDS & FDS (Appendix

LDS and FDS remain effective across tested smoothing hyper-parameters and diverse skewed label distributions, while outperforming adapted imbalanced-classification schemes on DIR.

  • Hyper-parameter robustness: Overall MSE gains on STS-B-DIR range from 3.3% to 6.2% across tested Gaussian-kernel sizes and standard deviations.The tested settings use l ∈ {5,9,15} and σ ∈ {1,2,3}.
  • Robustness to label densities: LDS and FDS consistently boost performance across all regions under different imbalanced label distributions, with relative MAE gains of 8.8% to 12.4%.The robustness tests include skewed Gaussian distributions and potential missing target regions.
  • Comparison with classification methods: LDS and FDS outperform imbalanced-classification schemes by a large margin, reducing few-shot-region errors by up to 50% to 60%.The comparison also reports that classification schemes can perform worse than vanilla regression.
  • Methods: The proposed methods use kernel-based smoothing in label and feature spaces and can be embedded into existing deep networks for end-to-end optimization.LDS may be combined with loss inverse re-weighting, while FDS calibrates learned feature statistics.

B. Details of DIR Datasets

The paper curates five DIR datasets spanning vision, language, depth estimation, and healthcare, with balanced validation or test evaluation over continuous target ranges.

  • Dataset overview: Five curated DIR datasets support evaluation across computer vision, natural language processing, and healthcare tasks.The datasets cover single-value prediction such as age and text similarity, plus dense-value prediction such as depth.
  • Vision datasets: IMDB-WIKI-DIR contains 191.5K training images with ages from 0 to 186 and per-bin counts ranging from 1 to 7149.Validation and test sets are manually constructed to be balanced over supported ages.
  • Vision datasets: AgeDB-DIR contains 12.2K training images spanning ages 0 to 101, with bin densities from 1 to 353 and balanced validation and test sets of 2.1K images.The dataset is constructed similarly to IMDB-WIKI-DIR.
  • Language dataset: STS-B-DIR predicts continuous sentence-pair similarity scores from 0 to 5 using the original STS-B collection.The scores are averaged human annotations from multiple annotators.
  • Dense prediction dataset: NYUD2-DIR predicts depth maps from RGB indoor-scene images, using depth values bounded between 0.7 and 10 meters.The benchmark uses 50K training images and 654 test images.
  • Healthcare dataset: SHHS-DIR predicts General Health scores from EEG, ECG, and breathing signals for subjects in the SHHS polysomnography dataset.The training set contains 1,892 subjects, while validation and test sets contain 369 balanced samples each.

C.1. Implementation Details

Implementation uses task-specific neural architectures and standard optimization settings, with LDS and FDS generally applied using shared Gaussian-smoothing hyper-parameters.

  • Vision implementation: IMDB-WIKI-DIR and AgeDB-DIR experiments use ResNet-50, Adam, 90 training epochs, L1 loss, and batch size 256.The learning rate starts at 10^-3 and decays at epochs 60 and 80.
  • Smoothing configuration: LDS and FDS use a Gaussian kernel with l = 5 and σ = 2, while FDS uses feature variance and momentum 0.9.Feature variance replaces covariance for computational efficiency.
  • Language implementation: STS-B-DIR encodes paired sentences with 300D GloVe embeddings and a two-layer 1500D-per-direction BiLSTM with max pooling.The resulting vectors are combined as [u; v; |u − v|; uv] before regression.
  • Language implementation: STS-B-DIR uses Adam with a fixed learning rate of 10^-4, validates every 10 epochs, and uses MSE for validation and early stopping.The same LDS and FDS hyper-parameters are used as in the image experiments.
  • Depth implementation: NYUD2-DIR uses a ResNet-50-based encoder-decoder and trains for 20 epochs with Adam and an initial learning rate of 10^-4.The learning rate decays by 0.1 every five epochs, and depth training uses MSE loss.
  • Healthcare implementation: SHHS-DIR uses three signal encoders whose 256-dimensional outputs are concatenated and passed to a three-layer MLP regressor.The inputs are EEG, ECG, and breathing signals.

C.2. Evaluation Metrics

Evaluation combines standard regression metrics with region-specific reporting and a geometric-mean error metric intended to characterize fairness across predictions.

  • Regression metrics: MAE measures the averaged absolute difference between ground-truth and predicted values over samples.It is defined as the mean of |y_i − ŷ_i|.
  • Regression metrics: MSE measures the averaged squared prediction difference, while RMSE is the square root of MSE.MSE is defined as the mean of (y_i − ŷ_i)^2.
  • Fairness-oriented metric: GM uses the geometric mean of per-sample L1 errors to characterize fairness or uniformity in model predictions.Each error is e_i = |y_i − ŷ_i|.
  • Language metrics: STS-B-DIR evaluation uses Pearson correlation for linear relationships and Spearman correlation for monotonic rank-order relationships.These metrics follow common STS-B and GLUE evaluation practice.
  • Depth metrics: NYUD2-DIR evaluation includes mean log10 error and threshold accuracy among standard depth-estimation metrics.Threshold accuracy is based on the proportion of predictions satisfying specified δ_i thresholds.

D.1. Complete Results on IMDB-WIKI-DIR

Across complete evaluations, LDS and FDS improve imbalanced regression performance, particularly in underrepresented regions, across vision, language, depth, and healthcare tasks.

  • MIXUP and M-MIXUP improve many-shot performance but provide negligible gains in medium-shot and few-shot regions.
  • Combining LDS and FDS substantially improves IMDB-WIKI-DIR results, especially for underrepresented regions, across evaluation metrics.
  • LDS and FDS consistently boost AgeDB-DIR performance, with the largest gains in medium-shot and few-shot regions.
  • On STS-B-DIR, LDS and FDS substantially improve all evaluated metrics, with particularly pronounced advantages under Pearson correlation.
  • On NYUD2-DIR, combining LDS and FDS reduces vanilla-model overfitting to many-shot regions and generalizes better across all regions.
  • On SHHS-DIR, combining LDS and FDS produces the highest performance gains across all tested regions in healthcare data.

E.2. Training Loss for LDS & FDS

LDS and FDS show similar performance across L1, MSE, and Huber training losses on STS-B-DIR, with no significant performance differences reported.

  • LDS and FDS obtain similar STS-B-DIR results with L1, MSE, and Huber training losses, without significant performance differences.

E.3. Hyper-parameters for LDS & FDS

LDS and FDS are robust to the tested kernel-size and standard-deviation choices, while l = 5 and σ = 2 gives the best reported results on both evaluated datasets.

  • Both LDS and FDS remain robust across the tested kernel sizes l ∈{5,9,15} and standard deviations σ ∈{1,2,3}.
  • For IMDB-WIKI-DIR, LDS few-shot relative MAE improvements range from 11.4% to 15.7% across tested hyper-parameters.
  • LDS favors medium-shot and few-shot gains, whereas FDS boosts all regions more evenly, with smaller underrepresented-region improvements than LDS.
  • For both LDS and FDS on IMDB-WIKI-DIR, l = 5 and σ = 2 produces the best results.
  • On STS-B-DIR, overall MSE gains range from 3.3% to 6.2% versus the vanilla model across tested hyper-parameters.
  • On STS-B-DIR, l = 5 and σ = 2 produces the best results for both LDS and FDS.

E.4. Robustness to Diverse Skewed Label Distributions

LDS and FDS remain effective across diverse skewed label distributions and test-set distributions, including regions with missing training data.

  • The robustness study creates IMDB-WIKI-DIR training sets from 1, 2, 3, or 4 disjoint skewed Gaussian label distributions, including potentially missing target regions.
  • LDS and FDS consistently improve over the vanilla model under different skewed label distributions.
  • Improvements are substantial in observed regions and more prominent in zero-shot regions requiring target interpolation or extrapolation.
  • DIR evaluation uses a test set balanced over the entire continuous target range.
  • With a training-matched skewed test distribution, LDS and FDS slightly degrade many-shot performance but drastically improve medium-shot and few-shot performance.

E.6. Further Comparisons to Imbalanced Classification Methods

The study compares imbalanced classification methods with regression approaches on appropriate DIR datasets and examines how FDS calibrates feature statistics. LDS and FDS outperform classification schemes, while FDS concentrates feature-statistics similarity near neighboring target values.

  • Comparison setup: The comparison discretizes continuous labels into the same bins used by LDS and FDS across IMDB-WIKI-DIR, STS-B-DIR, and NYUD2-DIR.CB, CRT, and vanilla classification are compared with imbalanced regression approaches on these datasets.
  • Results: LDS and FDS outperform imbalanced classification schemes across all evaluated DIR datasets.Few-shot-region errors are reduced by up to 50% to 60%.
  • Results: Imbalanced classification schemes often perform worse than the vanilla regression model, supporting distinct approaches for regression imbalance.The comparison identifies a gap between simply discretizing continuous targets and addressing regression-specific structure.
  • FDS analysis: Without FDS, feature statistics show unexpectedly high similarity between anchor ages and regions with very few samples.The reported analysis uses anchor ages 0, 30, 60, and 90.
  • FDS analysis: With FDS, feature statistics are better calibrated, showing high similarity near each anchor and decreasing similarity farther from it.The similarity declines as the target value becomes smaller or larger than the anchor.
Loading 2102.09554v2…