PaperlayerView exact revision on arXiv

Source-linked AI summary

Dice Loss for Data-imbalanced NLP Tasks

Xiaoya Li, Xiaofei Sun, Yuxian Meng, Guoyin Wang, Junjun Liang, Fei Wu, Jiwei Li

arXiv:1911.02855v5cs.CL

TL;DR

Data-imbalanced NLP tasks are dominated by negative and easy-negative examples, creating a mismatch between cross-entropy training and F1-oriented evaluation. The paper replaces cross-entropy with dice-based losses and dynamically adjusted example weights, achieving significant gains across imbalanced NLP tasks without changing model architectures.

  • Problem

    Data-imbalanced NLP tasks suffer from easy-negative examples dominating training and from a mismatch between equally weighted cross-entropy training and F1-oriented evaluation.

  • Method

    The paper replaces cross-entropy or maximum likelihood with Dice or Tversky losses and dynamically downweights easy-negative examples.

  • Results

    Significant performance boosts are reported across imbalanced NLP tasks, including new SOTA results for three POS-tagging datasets and four NER datasets.

  • Takeaways & Limitations

    The proposed loss narrows the gap between the training objective and F1 evaluation without changing model architectures.

Abstract

from arXiv · show

Many NLP tasks such as tagging and machine reading comprehension are faced with the severe data imbalance issue: negative examples significantly outnumber positive examples, and the huge number of background examples (or easy-negative examples) overwhelms the training. The most commonly used cross entropy (CE) criteria is actually an accuracy-oriented objective, and thus creates a discrepancy between training and test: at training time, each training instance contributes equally to the objective function, while at test time F1 score concerns more about positive examples. In this paper, we propose to use dice loss in replacement of the standard cross-entropy objective for data-imbalanced NLP tasks. Dice loss is based on the Sorensen-Dice coefficient or Tversky index, which attaches similar importance to false positives and false negatives, and is more immune to the data-imbalance issue. To further alleviate the dominating influence from easy-negative examples in training, we propose to associate training examples with dynamically adjusted weights to deemphasize easy-negative examples.Theoretical analysis shows that this strategy narrows down the gap between the F1 score in evaluation and the dice loss in training. With the proposed training objective, we observe significant performance boost on a wide range of data imbalanced NLP tasks. Notably, we are able to achieve SOTA results on CTB5, CTB6 and UD1.4 for the part of speech tagging task; SOTA results on CoNLL03, OntoNotes5.0, MSRA and OntoNotes4.0 for the named entity recognition task; along with competitive results on the tasks of machine reading comprehension and paraphrase identification.

1 Introduction

Data-imbalanced NLP tasks suffer from training-test discrepancy and domination by easy-negative examples under cross-entropy objectives. The paper replaces cross-entropy with dice-based losses and dynamically adjusts example weights, yielding broad performance improvements.

  • Motivation: NLP tasks such as tagging and machine reading comprehension commonly contain far more negative or background examples than positive examples.NER has 5:1 and 8:1 background-to-entity ratios in CoNLL03 and OntoNotes5.0, while MRC negative-positive ratios reach 50–200.
  • Motivation: Cross-entropy creates a training-test discrepancy because training weights instances equally while evaluation F1 gives equal importance to positive and negative examples.The resulting learning process tends to favor the majority class.
  • Motivation: Easy-negative examples can overwhelm training, limiting discrimination between positive and hard-negative examples.Cross-entropy or maximum likelihood handles neither this effect nor the training-test discrepancy.
  • Approach: Dice loss and the Tversky index replace cross-entropy or maximum likelihood to address imbalance-related training-test discrepancy.Dice loss gives similar importance to false positives and false negatives, while Tversky provides a precision-recall trade-off.
  • Results: Combining dice-based losses with dynamically adjusted example weights produces significant performance boosts across a wide range of data-imbalanced NLP tasks.The weighting strategy is intended to reduce the influence of easy-negative examples.

2 Related Work

Prior work addresses imbalance through example weighting, resampling, hard-example mining, and Dice-based objectives. These approaches span NLP-related classification, object detection, and image segmentation.

  • Imbalance handling: Importance sampling, boosting, hard-example mining, and oversampling modify training emphasis or class representation to address imbalance.These methods respectively reweight samples, select harder examples, downsample majority classes, or balance class sampling.
  • Related domains: Object detection research studies severe background-object imbalance using hard-negative mining, IoU-balanced sampling, and average-precision-based ranking.These methods provided inspiration for addressing data imbalance in NLP.
  • Related domains: Image segmentation research uses Generalized Dice Loss and related Dice-based objectives for unbalanced medical-image tasks.Reported extensions include multi-class organ segmentation and batch soft Dice loss.

3 Losses

The paper develops Dice- and Tversky-based losses to align training more closely with F1 under class imbalance, then adds dynamically adjusted example weights to reduce easy-negative dominance.

  • Cross Entropy Loss: Cross entropy gives every training instance equal objective weight, while F1-oriented evaluation treats positive and negative examples differently.The paper identifies this mismatch as a source of training-test discrepancy under imbalanced labels.
  • Dice Coefficient: The Sørensen–Dice coefficient is F1-oriented, and its loss treats false positives and false negatives with equal importance.For the paper’s set interpretation, A contains predicted positive examples and B contains gold positive examples.
  • Tversky Index: Tversky index generalizes Dice by controlling the tradeoff between false negatives and false positives, approximating the Fβ score.It reduces to DSC when α = β = 0.5.
  • Self-adjusting Dice Loss: Dice loss alone remains vulnerable to easy-negative dominance because it is a soft form of F1 and easy examples can be pushed toward probabilities of 0 or 1.This can leave the model struggling to distinguish hard negatives from positives.
  • Self-adjusting Dice Loss: The adaptive DSC multiplies each example’s soft probability by (1 − p_i1)^α, dynamically reducing the weight of easy examples during training.The weighting idea resembles focal loss, which down-weights well-classified examples.

4 Experiments

Experiments evaluate the proposed loss across part-of-speech tagging, named entity recognition, machine reading comprehension, and paraphrase identification. The method improves results across these tasks, with especially large gains reported for Chinese POS and NER.

  • Experimental settings: The evaluation covers POS tagging, NER, MRC, and paraphrase identification across multiple datasets.POS experiments include Chinese and English datasets; MRC uses SQuAD v1.1, SQuAD v2.0, and Quoref.
  • Part-of-speech tagging: The proposed DSC loss improves Chinese POS F1 over BERT-Tagger by +1.86 on CTB5, +1.80 on CTB6, and +2.19 on UD1.4.The authors report state-of-the-art performance on all three Chinese datasets.
  • Named entity recognition: DSC improves NER over BERT-MRC by +0.29 on CoNLL2003, +0.96 on OntoNotes5.0, +0.97 on MSRA, and +2.36 on OntoNotes4.0.The paper reports new state-of-the-art performance on all four NER datasets.
  • Machine reading comprehension: For MRC, DSC improves XLNet by +1.25 F1 and +0.84 EM on SQuAD v1.1, while reaching 87.65 EM and 89.51 F1 on SQuAD v2.0.On Quoref, DSC surpasses XLNet by +1.46 EM and +1.41 F1.
  • Paraphrase identification: Replacing the training objective with DSC improves paraphrase-identification performance by +0.58 on MRPC and +0.73 on QQP.F1 is the reported comparison metric for these experiments.

5 Ablation Studies

The ablations show that DSC consistently performs best across imbalance settings, while data composition and task alignment affect the benefits of alternative losses. DSC also improves imbalanced NLP tasks but is not suitable for accuracy-oriented sentiment classification, and Tversky hyperparameters materially influence performance.

  • Datasets imbalanced to different extents: +positive improves over the original dataset, whereas +negative underperforms because it creates a more imbalanced training distribution.
  • Datasets imbalanced to different extents: Negative downsampling produces inferior performance despite balancing the data because it reduces the number of training examples.
  • Datasets imbalanced to different extents: DSC achieves the highest F1 score across all synthetic QQP imbalance datasets.
  • Dice loss for accuracy-oriented tasks?: The proposed dice losses are therefore intended for F1-oriented imbalanced tasks rather than accuracy-oriented sentiment classification.
  • Dice loss for accuracy-oriented tasks?: 54.63 and 55.19: on SST-5, DL and DSC perform below BERT with CE, which achieves 55.57 accuracy.
  • Tversky index hyperparameters: 84.67 and 68.44: the highest F1 scores occur at α = 0.6 for Chinese OntoNotes4.0 and α = 0.4 for QuoRef, respectively.

6 Conclusion

The paper concludes that dice-based losses narrow the gap between the training objective and F1 evaluation, improving performance across imbalanced NLP tasks without changing model architectures.

  • Dice-based losses narrow the gap between the training objective and the F1 evaluation metric.
  • The proposed loss function achieves significant performance boosts without changing model architectures.

A.1 Part-of-Speech Tagging

The part-of-speech tagging experiments use BERT on Chinese and English datasets, evaluating predictions with span-level micro-averaged precision, recall, and F1.

  • CTB5 contains 507,222 words, 824,983 characters, and 18,782 sentences.
  • CTB6 extends CTB5 and contains 781,351 words, 1,285,149 characters, and 28,295 sentences.
  • UD denotes Universal Dependencies, a framework for consistent annotation of grammar across human languages.

A.2 Named Entity Recognition

The named entity recognition experiments cover Chinese and English benchmark datasets with differing entity inventories, domains, splits, and preprocessing protocols.

  • The NER experiments include Chinese OntoNotes4.0 and MSRA, plus English CoNLL2003 and OntoNotes5.0.
  • CoNLL2003 contains four entity types: Location, Organization, Person, and Miscellaneous.
  • English OntoNotes5.0 contains 18 entity types and uses the standard CoNLL2012 shared-task train/dev/test split.
  • Chinese MSRA is a news-domain benchmark with three entity types and a development split created from the training data.
  • The study uses the official test set for evaluation after splitting training data into training and development sets by 9:1.
  • Chinese OntoNotes4.0 contains news-domain texts with 18 entity types and follows Wu et al.'s data split.

A.3 Machine Reading Comprephension

The MRC experiments use SQuAD v1.1, SQuAD v2.0, and Quoref, covering standard question answering and coreferential reasoning.

  • The MRC evaluation uses SQuAD v1.1, SQuAD v2.0, and Quoref.SQuAD v1.1 and v2.0 are widely used QA benchmarks, while Quoref tests coreferential reasoning.

A.4 Paraphrase Identification

The paraphrase-identification experiments use MRPC and QQP, two sentence- or question-pair datasets with imbalanced class distributions.

  • The paraphrase-identification experiments are conducted on MRPC and QQP.MRPC contains sentence pairs annotated for semantic equivalence, while QQP contains question pairs from Quora.
  • MRPC contains 6,800 sentence pairs, with 68% positive and 32% negative examples.
  • QQP contains over 400,000 question pairs, with 37% positive and 63% negative examples.
Loading 1911.02855v5…