PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification

Varsha Suresh, Desmond C. Ong

arXiv:2109.05427v1cs.CLcs.AI

TL;DR

Fine-grained classification is challenging when many classes have subtle differences and are easily confused. The paper introduces LCL, which incorporates learned label relationships into contrastive training to weight examples differently. LCL outperforms existing contrastive losses and performs comparably with the state of the art, with reported benefits especially for larger or more confusable class sets.

  • Problem

    Fine-grained classification must distinguish subtle, semantically similar classes, which can interfere with one another.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    LCL incorporates learned inter-label relationships into contrastive training to weight positive and negative examples, particularly confusable negatives.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    LCL outperforms existing contrastive learning losses and performs comparably with the state of the art, with increased performance especially for larger or more confusable class sets.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    LCL’s reported gains are strongest in settings with more classes or more confusable classes, and it seems to encourage more confident decisions.

    Source: §6 · p. 9

    The study excludes fine-grained entity-type classification, which is generally multi-label.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Fine-grained classification involves dealing with datasets with larger number of classes with subtle differences between them. Guiding the model to focus on differentiating dimensions between these commonly confusable classes is key to improving performance on fine-grained tasks. In this work, we analyse the contrastive fine-tuning of pre-trained language models on two fine-grained text classification tasks, emotion classification and sentiment analysis. We adaptively embed class relationships into a contrastive objective function to help differently weigh the positives and negatives, and in particular, weighting closely confusable negatives more than less similar negative examples. We find that Label-aware Contrastive Loss outperforms previous contrastive methods, in the presence of larger number and/or more confusable classes, and helps models to produce output distributions that are more differentiated.

1 Introduction

Fine-grained text classification is difficult because classes can be closely confusable, while standard cross-entropy treats misclassifications nominally. The paper introduces Label-aware Contrastive Loss (LCL), which adaptively incorporates inter-class relationships, and reports that it outperforms existing contrastive losses while performing comparably with the state of the art.

1 Introduction

  • Fine-grained classification is challenging because closely confusable classes interfere with one another.Examples include distinctions such as “sad” versus “devastated” and “furious” versus “annoyed.”Source: §1 · p. 1
  • Standard cross-entropy treats misclassifications as nominal, despite differences in ordinal distance and semantic similarity between classes.For example, it does not distinguish the loss of confusing “positive” with “very positive” from confusing it with “very negative.”Source: §1 · p. 1
  • LCL adaptively learns inter-label relationships to weight positive and negative examples, emphasizing more confusable negatives.The approach uses a weighting model to help the primary model distinguish among closely related classes.Source: §1 · p. 1
  • LCL outperforms existing contrastive learning losses and performs comparably with the state of the art.The evaluation covers emotion recognition and sentiment analysis across coarse- and fine-grained classification settings.Source: §1 · pp. 1–2

2 Related Work

The related work covers fine-grained text classification, contrastive learning, and adaptive sample reweighting. This paper focuses on improving contrastive fine-tuning for fine-grained text classification.

2.1 Fine-grained classification

  • Fine-grained text classification distinguishes semantically similar labels, such as positive versus very positive, and remains challenging for models.Prior approaches include multi-task sentiment classification, sentiment-specific attention or pretraining, and lexicon-based emotion methods.Source: §2.1 · p. 2

2.2 Contrastive learning

  • Prior contrastive work includes self-supervised pretraining, supervised contrastive fine-tuning, and sentence-embedding objectives; this work targets fine-grained downstream classification.SimCLR uses augmentation for positive examples, while supervised contrastive learning also treats same-label examples as positives.Source: §2.2 · pp. 2–3

2.3 Other related work

  • Related dual-model and adaptive reweighting methods address model compression, label noise, or sample selection, and some require clean validation data.The paper contrasts its focus with methods that assume known or estimated noise rates or depend on clean validation data.Source: §2.3 · p. 3

3 Approach

Contrastive loss organizes latent representations by comparing positives and negatives; LCL extends supervised contrastive loss by incorporating learned inter-label relationships into those comparisons. A separate weighting network supplies class-related weights during training, while only the primary encoder is used at test time.

3.1 Contrastive Loss

  • Contrastive loss brings same-class representations closer by defining positive examples and separating them from negative examples.Self-supervised contrastive loss uses an augmented version of each sample as its positive and the other batch examples as negatives.Source: §3.1 · p. 3

3.2 Label-aware Contrastive Loss

  • LCL adapts supervised contrastive loss by using a class-weighting vector to incorporate inter-label relationships into pairwise similarities.Its positive set includes the augmented sample and samples from the same class.Source: §3.2 · p. 4
  • A second weighting network estimates class relationships, so confusable negative labels can receive greater weight than other negatives.The weighting network is optimized with cross-entropy, and its soft-label probabilities provide confidence estimates for classes.Source: §3.2 · pp. 4–5
  • During training, both networks process each batch, but testing uses only the primary encoder.The encoder’s representation is used for the contrastive loss and classification, while the weighting network produces the weight vector.Source: Figure 1 · p. 4, §3.2 · p. 5

4 Experiments

The experiments evaluate emotion recognition and sentiment analysis, using datasets that span different numbers and types of classes. Models are compared with standard losses and task-relevant pretrained language models.

4.1 Datasets

  • Emotion datasets range from 4 to 32 labels, while sentiment is evaluated on the 5-class and binary versions of SST.The emotion datasets are Empathetic Dialogues, GoEmotions, ISEAR, and EmoInt; the sentiment dataset is Stanford Sentiment Treebank.Source: §4.1 · p. 5

4.2 Implementation Details

  • Models use ELECTRAbase initialization, synonym replacement augmentation, Adam optimization, and early stopping based on validation performance.The study reports mean performance across five random seeds; synonym replacement substitutes 30% of input words when covered by WordNet.Source: §4.2 · p. 6

4.3 Model comparisons and evaluation

  • Evaluation uses accuracy and F1 for emotion classification and accuracy for sentiment, comparing LCL with cross-entropy, supervised contrastive loss, and task-relevant language models.Sentiment baselines include BERT, RoBERTa, SentiBERT, SentiLARE, and SKEP; ELECTRAbase is used for the loss comparisons.Source: §4.3 · p. 6

5 Results and Discussion

LCL performs better than comparison losses on many fine-grained emotion and sentiment tasks, with its advantages strongest when classes are numerous or confusable. Its prediction distributions are also less entropic on some tasks, though the confidence analysis has task-specific caveats.

5.1 Emotion Classification Performance

  • Across 32-, 27-, and 7-class emotion tasks, LCL significantly outperformed SCL and cross-entropy; on 4-class emotion, it beat SCL but not significantly cross-entropy.Results are averaged over five runs.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Sentiment Analysis Performance

  • On SST-5, LCL outperformed SCL and showed a non-significant advantage over cross-entropy; on SST-2, gains over both were non-significant and SKEP performed better.LCL's SST-5 performance was comparable to SentiLARE (p = .77); its SST-2 gains were more modest, with p = .78 versus cross-entropy and p = .32 versus SCL.Source: Table 2 · p. 7, §5.2 · p. 7

5.3 Case Study: Varying number of classes

  • LCL consistently improved over SCL and cross-entropy on 32-, 16-, and 8-class subsets; methods tied on the easy four-class set, while LCL usually led on confusable four-class sets.The LCL–SCL comparison on 4-hard-b was not significant; the authors attribute this to high standard deviations.Source: Table 3 · p. 8, §5.3 · p. 8

5.4 Quantifying model confidence

  • LCL yielded lower prediction-score entropy on Empathetic Dialogues; on GoEmotions both contrastive losses were lower than cross-entropy, while SST-5 suggested lower LCL entropy with a small k range.Lower entropy corresponds to peakier score distributions, but this post-hoc confidence measure is independent of prediction correctness.Source: §5.4 · p. 9, Figure 2 · p. 9

6 Conclusion

LCL weights negative classes by their confusability and is reported to improve classification performance especially with more numerous or confusable classes. The authors frame adaptive class weighting as one promising direction, with external label knowledge and alternative distance metrics as possible extensions.

6 Conclusion

  • LCL weights negative classes according to their confusability and is associated with increased classification performance, especially with more classes or more confusable classes.The authors say it also seems to encourage more confident decisions.Source: §6 · p. 9
  • The authors view adaptive class weighting as one approach, with external label knowledge and alternative class-distance metrics as possible future directions.They argue that not all negative classes should be treated equally.Source: §6 · p. 9

A.1 Evaluation metrics

The evaluation uses top-1 accuracy and weighted macro F1-score. The latter accounts for imbalance in the label distribution, with its weights based on class sample counts.

A.1 Evaluation metrics

  • Top-1 accuracy and weighted macro F1-score are the evaluation metrics.Source: §A.1 · p. 13
  • Weighted macro F1-score accounts for label-distribution imbalance, using each class’s sample count relative to the total number of samples.The passage defines n_c as the number of samples in class c and N as the total number of samples.Source: §A.1 · p. 13

B Experiment settings

The experiments use manual hyperparameter search and select models by validation top-1 accuracy. Settings include dataset-specific learning rates, loss-scaling factors, and temperatures, alongside data cleaning for EmoInt.

B Experiment settings

  • Models were selected by validation top-1 accuracy after manual search over learning rates {1e-05, 2e-05, 3e-05}, α values from 0.1 to 0.5, and τ values {0.1, 0.3, 0.5}.Batch size was 10, with one augmented sample per input and an effective batch size of 20.Source: B Experiment settings · p. 13
  • The selected learning rate was 2e-05 for EmpatheticDialogues, EmoInt, SST-5, SST-2, and GoEmotions, and 3e-05 for ISEAR.Source: B Experiment settings · p. 13
  • EmoInt text was cleaned by removing non-ASCII characters, letter repetitions, and extra whitespace, and replacing user mentions and links with unique identifiers.Source: B Experiment settings · p. 13

B.1 Average runtime and parameters

The LCL system combines a primary encoder and a weighting network, with ELECTRA-base used for both. Reported epoch runtimes vary across the six datasets.

B.1 Average runtime and parameters

  • 110M parameters are trainable across the primary encoder and weighting network, which both use ELECTRA-base.Source: §B.1 · p. 13
  • Average epoch runtimes were 2.9 minutes for EmoInt, 5.2 for ISEAR, 19.8 for GoEmotions, 19.7 for EmpatheticDialogues, 6.1 for SST-2, and 8.2 for SST-5.Source: §B.1 · p. 13

C Validation performance

The section reports validation results for sentiment analysis and emotion classification. Both tables average results over five runs and show standard deviations in brackets.

C Validation performance

  • The reported test results correspond to validation performance in Table 4 for sentiment analysis and Table 5 for emotion classification.Source: C Validation performance · p. 13, Table 4 · p. 13, Table 5 · p. 14
  • Table 4 reports validation results for the sentiment analysis task.Results are averaged over five runs, with standard deviations in brackets.Source: Table 4 · p. 13
  • Table 5 reports validation results for the emotion classification task.Results are averaged over five runs, with standard deviations in brackets.Source: Table 5 · p. 14

Related papers

Loading 2109.05427v1…