PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity in Machine Learning

Zhiqiang Gong, Ping Zhong, Weidong Hu

arXiv:1807.01477v2cs.CV

TL;DR

Machine learning lacks a systematic analysis of how diversity can be promoted across its training data, models, and inference. This paper surveys methods and applications in those three areas, finding that diversification improves model performance and provides informative or complementary representations across many tasks.

  • Problem

    There is no systematic analysis of diversity-promoting methods across training data, learned models, and inference in machine learning.

  • Method

    The paper systematically categorizes and surveys data diversification, model diversification, and inference diversification methods and applications.

  • Results

    The surveyed diversity technologies improve machine learning performance across applications including remote sensing, segmentation, object detection, machine translation, and information retrieval.

  • Takeaways & Limitations

    The survey helps researchers judge whether diversity is needed and choose methods for specific machine learning requirements.

Abstract

from arXiv · show

Machine learning methods have achieved good performance and been widely applied in various real-world applications. They can learn the model adaptively and be better fit for special requirements of different tasks. Generally, a good machine learning system is composed of plentiful training data, a good model training process, and an accurate inference. Many factors can affect the performance of the machine learning process, among which the diversity of the machine learning process is an important one. The diversity can help each procedure to guarantee a total good machine learning: diversity of the training data ensures that the training data can provide more discriminative information for the model, diversity of the learned model (diversity in parameters of each model or diversity among different base models) makes each parameter/model capture unique or complement information and the diversity in inference can provide multiple choices each of which corresponds to a specific plausible local optimal result. Even though the diversity plays an important role in machine learning process, there is no systematical analysis of the diversification in machine learning system. In this paper, we systematically summarize the methods to make data diversification, model diversification, and inference diversification in the machine learning process, respectively. In addition, the typical applications where the diversity technology improved the machine learning performance have been surveyed, including the remote sensing imaging tasks, machine translation, camera relocalization, image segmentation, object detection, topic modeling, and others. Finally, we discuss some challenges of the diversity technology in machine learning and point out some directions in future work.

I. INTRODUCTION

The paper surveys diversity in machine learning across training data, learned models, and inference, addressing an absence of a systematic framework and reviewing applications and analysis questions.

  • Motivation: Limited labelled data, many model parameters, and over-fitting make accurate inference difficult, while diversity is identified as an important performance factor.These constraints motivate diversification throughout the machine learning process.
  • Motivation: The paper motivates its survey by noting that prior work addressed individual diversity methods or inference settings without providing a general machine learning framework.The cited discussion contrasts broader coverage with work focused on determinantal point processes or diversified search results.
  • Scope: Diversity reduces redundancy and provides informative data, representative models, or complementary inference choices across the machine learning process.The paper organizes these approaches into data diversification, model diversification, and inference diversification.
  • Research questions: The survey asks how to measure and enhance diversity in training data, learned models, and inference, and how these forms relate.It also examines applications in which diversification improves machine learning performance.
  • Organization: The paper is structured around general machine learning models, diversity methods in Sections III–V, and applications in Section VI.The diversity sections address training data, models, and inference.
  • Scope: The paper covers supervised, active, and a special unsupervised learning setting while ignoring reinforcement learning because its diversity technologies mainly operate on the model.Its general supervised-learning pipeline comprises data preprocessing, training, and inference.

B. ACTIVE LEARNING

This section presents active learning and related data-diversification settings for learning with limited labelled data, emphasizing informative, less redundant training samples.

  • Active learning: Active learning reduces labelling cost by selecting informative samples when labelled training data are limited.The paper uses Convex Transductive Experimental Design (CTED) as a representative active-learning method.
  • Active learning: CTED uses data reconstruction and sparse sample selection to construct a training set from selected unlabelled samples.Its reconstruction coefficients and sample-selection vector identify samples for labelling.
  • Active learning: CTED-selected samples can be similar and redundant, reducing training efficiency and requiring more samples for comparable performance.This motivates adding diversity to active learning.
  • Unsupervised learning: In unsupervised learning, pseudo classes are formed from unlabelled samples and iteratively used to train the model.Diversifying and repulsing the class centers can better discriminate pseudo classes and improve process effectiveness.
  • Organization: The paper groups diversification in machine learning into data, model, and inference diversification across supervised, active, and unsupervised settings.Training-batch diversification, active learning, and unsupervised learning primarily concern training data.

A. DIVERSIFICATION IN SUPERVISED LEARNING

This section surveys supervised-learning data diversification, especially methods that construct more diverse mini-batches and selected samples to reduce redundancy and improve training.

  • Mini-batch diversification: Random mini-batch generation can create imbalance and redundancy, negatively affecting supervised-learning effectiveness.The issue arises under random selection in conventional mini-batch training.
  • DPP: DPPs assign higher probabilities to mini-batches with dissimilar items and lower probabilities to batches containing redundant data.They use a similarity matrix and determinant-based diversity prior.
  • DPP: DPP sampling balances data and generates stochastic gradients with lower variance.The passage directly links these properties to the DPP treatment of redundant and diverse mini-batches.
  • PDS: PDS enforces a minimum pairwise distance between samples and provides arrangements similar to DPP with greater efficiency.Its threshold r is evaluated using a distance measurement D(x_i, x_j).
  • Effects: Under DPP or PDS diversification priors, mini-batches contain more diverse and informative samples that train models more effectively.The resulting models can extract more discriminative features from objects.
  • Active learning: Active-learning methods introduce diversity regularizers or similarity-based terms to select informative and complementary samples.The surveyed formulations include cosine similarity, angular cosine similarity, and RBF-kernel measurements.
  • Active learning: For active learning, diversity regularization can produce comparable or better performance with limited training samples than with plentiful samples.The method selects samples with more information and less redundancy for labelling.

C. DIVERSIFICATION IN UNSUPERVISED LEARNING

Unsupervised learning can promote diversity by separating pseudo-class centers, while model diversification reduces redundancy among learned parameters and improves representational capacity. Bayesian priors provide one framework for imposing diversity over model parameters.

  • Unsupervised learning: Repulsing pseudo-class centers encourages more discriminative features from samples belonging to different classes.The diversity term makes center points move away from one another during training.
  • Model diversification: Model diversification addresses redundancy caused by highly similar parameters learned from limited and imbalanced training samples.Such redundancy negatively affects the model’s representational ability.
  • D-model: D-model diversification encourages parameter factors to model unique information, improving the model’s representational ability.The whole set of factors can model a larger proportion of information.
  • D-models: D-models diversify multiple base models so that different models provide choices reflecting multi-modal belief.Without diversification, separately trained models can produce similar, redundant representations.
  • Bayesian method: Bayesian modeling incorporates a diversity-promoting prior P(W) over model parameters alongside the training-set likelihood P(X|W).The prior is intended to encourage learned parameter factors to be diversified.
  • Overview: Table 1 summarizes frequently used diversification methods for D-models and identifies papers containing example measurements.

2) Posterior Regularization Method

Posterior regularization imposes diversity through a regularizer that balances the model objective with a diversity term. The survey compares distance-, angular-, kernel-, and other measurements, noting trade-offs in scale sensitivity, computational cost, and correlation structure.

  • Posterior regularization: Posterior regularization adds side information through a diversity regularizer f(W) that balances model optimization against diversification.The tradeoff is controlled by γ, while L0 denotes the model optimization term.
  • Distance-based measurements: Distance-based measurements diversify factors by increasing their pairwise separation, but their effectiveness can be affected by scaling.This scale sensitivity can limit their suitability for models with large scale ranges.
  • Distance-based measurements: Heat-kernel diversity uses a Gaussian distance function whose penalization varies with factor distance.The survey reports better performance than general Euclidean distance.
  • Angular-based measurements: Angular measurements improve scale invariance, with cosine similarity encouraging different factors to become orthogonal and uncorrelated.Angular-based methods address orientation-related defects in the former cosine-similarity measurement.
  • Analysis: Across the surveyed methods, pairwise diversity commonly balances complexity and effectiveness, while applications should select measurements according to task requirements.Some methods additionally model correlations among three or more factors.

B. D-MODELS

D-models diversify multiple models so their representations and outputs provide complementary information rather than redundant alternatives. The paper groups these methods into sample-based, optimization-based, and ranking-based approaches.

  • B. D-MODELS: Ensemble diversification addresses sub-optimal single-model solutions and highly similar ensemble choices by producing outputs that reflect multi-modal belief.This is especially relevant when structured prediction tasks have multiple reasonable interpretations but only one groundtruth.
  • B. D-MODELS: The paper formulates multiple-model learning using parallel base models with parameters W_i and inferences P(W_i), where each model is trained on corresponding samples X_i.The formulation provides a common basis for diversifying models during ensemble training.
  • B. D-MODELS: Sample-based methods diversify models by assigning different training samples or sample weights to different base models.Randomly dividing samples can leave redundant representations, motivating more deliberate sample assignment.
  • B. D-MODELS: Optimization-based methods encourage diversity by measuring similarity between base models and incorporating a diversification objective during training.The term Γ(W_1, W_2, · · ·, W_s) measures diversification among the models.
  • B. D-MODELS: Ranking-based methods generate many models, rank them by a criterion, and select the top-L models for the final ensemble.The paper presents these three categories as its overview of D-model diversification methods.

1) Optimization-Based Methods

Optimization-based methods diversify ensembles by adding measurements or regularizers that reduce similarity among base models. The survey covers Bayesian, correlation, geometric, group-wise, and selective-ensemble approaches, while noting task-fit and computational trade-offs.

  • 1) Optimization-Based Methods: Optimization-based methods obtain diversified models by optimizing an objective function that includes a diversity term.The survey identifies defining suitable diversity measurements as a central challenge.
  • 1) Optimization-Based Methods: Bayesian methods increase differences between base-model inference distributions, encouraging models to provide more complementary information.Divergence is described as the most commonly used Bayesian measurement; cross entropy similarly promotes dissimilar pairwise distributions.
  • 1) Optimization-Based Methods: Renyi-entropy can adapt differences across models through σ, but it requires more computation and makes ensemble updates more complex than divergence measurement.The measurement uses a Gaussian kernel and depends on the model dimension d.
  • 1) Optimization-Based Methods: HFC-PGA accounts for multiple-model correlation by comparing each model with the ensemble’s average distribution, but its fixed weighting may not fit different tasks.The task mismatch can limit the performance of its diversity-promoting prior.
  • 1) Optimization-Based Methods: Negative correlation learning reduces covariance among models while avoiding increases in variance and bias, thereby decorrelating individual errors.Its penalty can be designed differently for sequential and parallel model training.
  • 1) Optimization-Based Methods: Geometric and group-wise measurements diversify models through cosine or exclusivity relations, pairwise uncorrelation, and L2,1-based group selection.These approaches target complementary model behavior through different notions of similarity.

2) Sample-Based Methods

Sample-based methods diversify ensembles by changing the training data or weights assigned to different base models. They aim to produce complementary model information while balancing diversity against reduced data per model.

  • 2) Sample-Based Methods: Randomly dividing training data can create overlapping representations between models, causing redundancy and potentially reducing ensemble performance.This motivates assigning samples more deliberately across base models.
  • 2) Sample-Based Methods: One method assigns each training sample to the model with the lowest prediction error, so base models focus on features from specific classes.Clustering is also described as a way to divide training samples.
  • 2) Sample-Based Methods: Dividing the full training set can reduce the number of samples available to each model and negatively affect performance.This is the main trade-off identified for subset-based diversification.
  • 2) Sample-Based Methods: Weighted-sample methods train different base models with different sample weights so each model can focus on complementary information.The described procedure initializes weights, revises them using final loss, and repeats training for M models.
  • 2) Sample-Based Methods: UDEED promotes ensemble diversity using unlabeled samples rather than relying on error-prone pseudo-labels to enlarge labeled data.It differs from existing semi-supervised ensemble methods in how unlabeled data are used.

3) Ranking-Based Methods

Ranking-based methods first produce multiple models, then select a diverse subset for the final ensemble. The survey describes criterion-based pruning, prediction clustering, and feature-set selection as examples.

  • 3) Ranking-Based Methods: Ranking-based methods rank all models according to a criterion and select the top-L models for the final ensemble.Forward and backward selection are presented as pruning strategies that directly compare model accuracy.
  • 3) Ranking-Based Methods: Prediction-based clustering groups similar models, prunes redundant members within each cluster, and combines the remaining models.The method uses prediction similarity to enforce diversity during selection.
  • 3) Ranking-Based Methods: Feature-selection methods choose multiple sets of features, train a model for each base feature set, and combine their inferences.The approach can use multi-scale or other feature-generation procedures before selecting the top-L feature sets.
  • 3) Ranking-Based Methods: The survey summarizes D-model diversification through optimization-based, sample-based, and ranking-based methods.Optimization imposes diversity regularization, whereas sample-based methods train models on specific training sets.

V. INFERENCE DIVERSIFICATION

Inference diversification generates multiple complementary choices rather than relying on a single potentially sub-optimal MAP inference. These choices can be selected through methods that promote dissimilarity while retaining high-quality solutions.

  • Inference Diversification: Inference diversification addresses sub-optimal MAP results by producing multiple choices that provide complement information.The approach targets limitations from model representational ability and limited training samples.
  • Inference Diversification: Traditional multiple-choice inference can return near-duplicate solutions, such as one-pixel-shifted versions of the best choice.This behavior can cause other probability-distribution peaks to be ignored.
  • Inference Diversification: D-MCL, submodular methods, M-NMS, and related approaches diversify candidate choices using dissimilarity or complement-information criteria.Figure 7 illustrates diversified choices followed by re-ranking to obtain a final solution.

A. DIVERSITY-PROMOTING MULTIPLE CHOICE LEARNING (D-MCL)

D-MCL seeks a diverse set of highly probable inference solutions by combining solution quality with dissimilarity to previously selected choices. It includes sequential greedy and joint formulations, but its selected choices may not be local extrema.

  • D-MCL: D-MCL searches for diverse, high-probability solutions by maximizing probability together with dissimilarity to previous choices.The method aims to produce a diverse set of low-energy solutions even when the MAP solution is poor.
  • D-MCL: The greedy formulation selects each next labelling as the lowest-energy state sufficiently dissimilar from previously chosen labellings.The parameter γ controls the trade-off between diversity and energy, while y1 is the MAP solution.
  • D-MCL: A joint formulation infers all M labellings simultaneously by minimizing a single energy objective with a total-diversity term.It introduces split-diversity, node-diversity, and node-split-diversity measures.
  • D-MCL: D-MCL choices may not be local extrema, leaving other solutions that better represent the objects potentially undiscovered.This limitation distinguishes diversity by dissimilarity from explicit local-mode selection.

B. SUBMODULAR FOR DIVERSIFICATION

Submodular diversification selects a high-quality, diverse subset by exploiting diminishing marginal gains. Its greedy optimization can provide complement information, but exhaustive candidate spaces can make even a scan infeasible.

  • Submodular Diversification: Submodularity models diverse subset selection through decreasing marginal gains as the selected set grows.For monotone submodular functions, greedy addition achieves the best possible approximation bound stated in the passage.
  • Submodular Diversification: When the ground set has exponentially many items, even a single linear scan for greedy augmentation can be infeasible.This computational boundary limits straightforward application to very large candidate spaces.
  • Submodular Diversification: The method combines a monotone, nondecreasing, normalized diversity function with an optimization objective over the selected choices.A greedy algorithm starts from the empty set and iteratively adds the best term.
  • Submodular Diversification: Submodular maximization finds multiple choices intended to provide the maximum amount of complement information.The selected solution is described as lying within ε of the optimal solution.

D. M-MODES

M-modes extends inference diversification by seeking multiple high-quality local extrema, addressing overlap and non-local-extremum issues in earlier methods. The broader survey situates these techniques within applications where diversity improves model representation and performance.

  • D. M-MODES: Earlier diversification methods can select separated choices that are not local extrema, whereas M-modes seeks both local extrema and diversity.This design is intended to provide multiple choices with complement information.
  • D. M-MODES: M-modes defines local maxima through a δ-neighborhood and computes the M labellings with minimal energies among the modes.As δ increases, the neighborhood grows and the mode set forms a monotonically decreasing nested sequence.
  • D. M-MODES: The M-modes problem can be reduced to finding the M best configurations in a new chain through a cost-preserving correspondence.The passage also describes a one-to-one correspondence between consistent configurations and the mode set.
  • Related Diversification Methods: DPP-based diversification favors representative, non-repetitive subsets that combine high quality with dissimilarity.Its kernel uses energy-based quality terms and similarity terms to favor uncorrelated choices.
  • VI. APPLICATIONS: Diversity technology is reported to improve model representational ability across remote sensing, segmentation, detection, translation, retrieval, and other applications.The survey particularly associates diversity priors with informative features in tasks involving limited training samples and complex structures.
  • A. REMOTE SENSING IMAGING TASKS: In remote sensing, diversity-promoting methods are reported to improve classification performance and hyperspectral-image representation.The survey discusses model, data, and multiple-model diversification across remote-sensing tasks.

B. IMAGE SEGMENTATION

Image segmentation uses diversity to generate multiple plausible solutions and improve performance, while related vision tasks address uncertainty through diversified choices or models.

  • B. IMAGE SEGMENTATION: Inference diversification encourages diverse low-energy solutions representing different local optima in image segmentation.Prior work introduced inference diversity into segmentation through multiple methods.
  • B. IMAGE SEGMENTATION: Inference diversification can significantly improve segmentation performance on comparison results reported using PASCAL VOC.The cited comparison results are summarized in Table 6.
  • Related Vision Tasks: Camera relocalization addresses nonconvex optimization with many local optima by seeking multiple camera-pose predictors.Diversified predictors are intended to produce different hypotheses for the observed RGB-D frame.
  • Related Vision Tasks: Greedy training can produce diversified camera-relocalization models whose complementary predictions are compared under different selection procedures.Training-sample weights are updated using the loss from the previously learned model.
  • Related Vision Tasks: Object detection also uses multiple diversified choices because its algorithms contain substantial uncertainty.M-NMS-based methods eliminate repeated detections and can obtain diversified detections through search strategies.

E. MACHINE TRANSLATION

The survey examines diversity across machine learning components and applications, including machine translation, topic modeling, search, ranking, and summarization. It presents diversification as a way to reduce redundancy and improve representational or output coverage, while remaining an additional rather than decisive technique.

  • E. MACHINE TRANSLATION: Machine translation systems can leave users guessing after errors, motivating diverse alternative translations rather than nearly identical outputs.M-best lists often differ only by punctuation or minor morphological variation.
  • E. MACHINE TRANSLATION: Diversity-promoting methods can provide users with multiple diversified translations by applying a dissimilarity function to candidate outputs.The cited dissimilarity function uses an Iverson bracket and subsequences of the translation.
  • 1) Natural Language Processing: Topic modeling applies model diversification because LDA discovers topics along one dimension while RBM can learn redundant hidden units and ignore long-tail topics.A determinantal point process prior can encourage topic groups to model different data dimensions.
  • Other Applications: Search-result diversification seeks distinct interpretations of queries to satisfy multiple user types, using submodular and distance-based approaches.The objective is broad feature coverage with limited redundancy.
  • Other Applications: Diversified top-K graph rankings address similarity among highly ranked nodes by incorporating diversity into ranking objectives.Prior work formulates diversified ranking as submodular set-function maximization or uses Heat-kernel weights.
  • Other Applications: Multi-document summarization requires both corpus coverage and diversity, which can conflict and therefore motivate tradeoffs through submodular optimization.Efficient algorithms can provide near-optimal solutions for diverse constraint sets when the submodular function is monotone.
Loading 1807.01477v2…