PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Fine-Tune BERT for Text Classification?

Chi Sun, Xipeng Qiu, Yige Xu, Xuanjing Huang

arXiv:1905.05583v3cs.CL

TL;DR

BERT’s potential for text classification remains underexplored. The paper exhaustively evaluates fine-tuning strategies and proposes a general solution combining further pre-training, optional multitask fine-tuning, and target-task fine-tuning, achieving state-of-the-art performance on eight datasets.

  • Problem

    BERT has achieved strong NLU results, but its potential for further improving target-task performance remains underexplored.

  • Method

    The paper exhaustively investigates BERT fine-tuning and proposes further pre-training, optional multitask fine-tuning, and target-task fine-tuning.

  • Results

    The approach achieves state-of-the-art performance on eight widely studied text classification datasets.

  • Takeaways & Limitations

    Appropriate layer selection, layer-wise learning rates, further pre-training, and multitask fine-tuning improve BERT text classification, including with small datasets.

  • Takeaways & Limitations

    The partition of the English datasets into topic, sentiment, and question domains is not strictly correct.

Abstract

from arXiv · show

Language model pre-training has proven to be useful in learning universal language representations. As a state-of-the-art language model pre-training model, BERT (Bidirectional Encoder Representations from Transformers) has achieved amazing results in many language understanding tasks. In this paper, we conduct exhaustive experiments to investigate different fine-tuning methods of BERT on text classification task and provide a general solution for BERT fine-tuning. Finally, the proposed solution obtains new state-of-the-art results on eight widely-studied text classification datasets.

1 Introduction

BERT’s potential for text classification remains underexplored, so the paper exhaustively investigates fine-tuning strategies and proposes a general solution. The solution achieves new state-of-the-art results across eight classification datasets.

  • BERT’s potential for improving target-task performance beyond its existing NLU results remains underexplored.
  • The paper conducts exhaustive experiments to investigate how to maximize BERT’s utilization for text classification.
  • The proposed fine-tuning solution further pre-trains BERT, optionally applies multitask fine-tuning, and then fine-tunes it for the target task.
  • The study investigates target-task fine-tuning methods including long-text preprocessing, layer selection, layer-wise learning rates, and catastrophic forgetting.
  • The evaluation covers seven widely studied English text classification datasets and one Chinese news classification dataset.

2 Related Work

Related work develops text classification through pre-trained representations, language-model fine-tuning, and multitask learning. BERT extends this line with bidirectional pre-training and downstream fine-tuning, while multitask methods introduce efficiency and objective-weighting considerations.

  • Pre-trained word, sentence, and paragraph embeddings are used as features to improve downstream NLP models.
  • Language-model pre-training followed by downstream fine-tuning has produced strong results across natural language understanding tasks.
  • BERT uses masked language modeling and next sentence prediction over a large cross-domain corpus for pre-training.
  • Multitask learning can jointly train language-model and task objectives, but often requires training from scratch and carefully weighting task-specific objectives.

3 BERT for Text Classification

For text classification, BERT represents the input using the final hidden state of [CLS] and adds a task-specific softmax classifier. All BERT parameters and the classifier matrix are fine-tuned jointly by maximizing the correct-label log-probability.

  • BERT-base has 12 Transformer blocks, 12 self-attention heads, hidden size 768, and a maximum input length of 512 tokens.
  • For classification, the final hidden state h of [CLS] represents the whole input sequence.
  • A softmax classifier maps the [CLS] representation to probabilities for the task’s labels.
  • The model jointly fine-tunes all BERT parameters and the task-specific matrix W by maximizing the correct label’s log-probability.

4 Methodology

The methodology evaluates BERT fine-tuning through task adaptation, layer-sensitive optimization, further pre-training, and multi-task learning. It combines these components into a general fine-tuning procedure while examining long-text preprocessing, layer selection, and overfitting-related choices.

  • 4 Methodology: The paper seeks proper BERT fine-tuning methods across fine-tuning strategies, further pre-training, and multi-task fine-tuning.These three directions address optimization choices, domain mismatch, and knowledge sharing across related tasks.
  • 4.1 Fine-Tuning Strategies: Fine-tuning analysis considers long-text preprocessing, layer selection, and overfitting, reflecting differences in the information captured by BERT layers.BERT's maximum sequence length is 512, and the model includes an embedding layer, a 12-layer encoder, and a pooling layer.
  • 4.1 Fine-Tuning Strategies: Layer-wise learning rates set the base rate to ηL and recursively use ηk−1 = ξ · ηk, with ξ < 1 assigning lower rates to lower layers.When ξ = 1, all layers use the same learning rate, equivalent to regular stochastic gradient descent.
  • 4 Methodology: The general solution further pre-trains BERT on within-task or in-domain data, optionally uses multitask fine-tuning, then fine-tunes it for the target task.The paper presents this as a three-step procedure, with multitask learning included when related tasks are available.
  • 4.2 Further Pre-training: Further pre-training uses masked language modeling and next sentence prediction on within-task, in-domain, or cross-domain data.The three approaches differ according to whether pretraining data comes from the target task, its domain, or both related and different domains.
  • 4.3 Multi-Task Fine-Tuning: In multi-task fine-tuning, tasks share BERT and embedding layers while retaining private final classification layers.The experimental analysis is conducted in Section 5.5.

5 Experiments

Experiments evaluate BERT fine-tuning on eight text classification datasets spanning sentiment, question, and topic classification in English and Chinese. The setup uses BERT-base models with specified GPU, optimization, sequence-length, and training configurations.

  • 5.2 Experimental Settings: The experiments use uncased BERT-base for English tasks and Chinese BERT-base for the Chinese task.The models are identified as the uncased BERT-base model and the Chinese BERT-base model.
  • 5 Experiments: The evaluation covers eight datasets across sentiment analysis, question classification, and topic classification.The collection includes seven English tasks and one Chinese news classification task.
  • 5.1 Datasets: The sentiment datasets are IMDb and binary and five-class Yelp, while question classification uses TREC and Yahoo! Answers.TREC contains six question categories, and Yahoo! Answers has 1,400k training samples.
  • 5.1 Datasets: Topic classification uses AG’s News and DBPedia, and the Chinese evaluation uses six categories from the Sogou news corpus.The Chinese categories are sports, house, business, entertainment, women, and technology.
  • 5.2 Experimental Settings: Further pre-training uses one TITAN Xp GPU, batch size 32, maximum sequence length 128, learning rate 5e-5, 100,000 training steps, and 10,000 warm-up steps.Fine-tuning uses four TITAN Xp GPUs, batch size 24, Adam, base learning rate 2e-5, warm-up proportion 0.1, and at most four epochs.

5.3 Exp-I: Investigating Different Fine-Tuning Strategies

The experiments compare BERT fine-tuning choices for long texts, layer selection, catastrophic forgetting, and layer-wise learning rates. The findings favor head+tail truncation, the final BERT layer, low learning rates, and decreasing rates for lower layers.

  • Dealing with long texts: Head+tail truncation achieves the best performance on IMDb and Sogou, so it is used for subsequent long-text experiments.The method keeps the first 128 and last 382 tokens.
  • Features from Different layers: The final BERT layer gives the best performance among the evaluated layer features and is used in subsequent experiments.
  • Catastrophic Forgetting: A lower learning rate such as 2e-5 is necessary for BERT to overcome catastrophic forgetting, whereas 4e-4 prevents the training set from converging.
  • Layer-wise Decreasing Layer Rate: Assigning lower learning rates to lower layers is effective, with an appropriate setting of ξ=0.95 and lr=2.0e-5.

5.4 Exp-II: Investigating the Further Pretraining

The experiments evaluate further pre-training on within-task, in-domain, and cross-domain data before fine-tuning BERT. Further pre-training generally improves performance, with in-domain data usually outperforming within-task data, while cross-domain gains are not generally obvious.

  • Within-Task Further Pre-Training: Further pre-training improves BERT on the target task and achieves the best performance after 100K training steps.
  • In-Domain and Cross-Domain Further Pre-Training: Almost all further-pre-trained models outperform the original BERT-base model on all seven English datasets.
  • In-Domain and Cross-Domain Further Pre-Training: In-domain pre-training generally performs better than within-task pre-training, while within-task pre-training harms performance on the small sentence-level TREC dataset.
  • In-Domain and Cross-Domain Further Pre-Training: Cross-domain pre-training does not bring an obvious general benefit, and IMDb and Yelp do not help each other within the sentiment domain.The authors associate the latter difference with substantial data-distribution differences between movie and food sentiment tasks.
  • Comparisons to Previous Models: BERT-IDPT-FiT performs best among the compared further-pre-training models, reducing average error rate by 18.57% relative to BERT-Feat.

5.5 Exp-III: Multi-task Fine-Tuning

The study evaluates multi-task fine-tuning across several text-classification datasets and finds benefits over standard BERT fine-tuning, though gains are not universal when domain-specific pre-training is used.

  • Data and setup: Multi-task fine-tuning uses IMDb, Yelp P., AG, and DBP while excluding datasets with test-training overlap or different domains.The exclusion criteria address overlap and domain considerations.
  • Data and setup: The experiments compare official uncased BERTbase weights with weights further pre-trained on seven English classification datasets.Each subtask receives additional fine-tuning with a lower learning rate after joint fine-tuning.
  • Results: Multi-task fine-tuning improves BERT-based classification results.The passage reports the overall effect without specifying individual dataset values.
  • Results: Multi-task fine-tuning does not help BERT-CDPT on Yelp P. and AG, where it may be unnecessary because cross-domain pre-training already provides domain-specific information.The paper presents multi-task fine-tuning and cross-domain pre-training as alternative approaches in these cases.

5.6 Exp-IV: Few-Shot Learning

The few-shot experiment tests BERT with different amounts of IMDb training data and reports substantial error reduction from further pre-training, including under small-data conditions.

  • Experimental setup: BERT-FiT and BERT-ITPT-FiT are evaluated on subsets of IMDb training data with different numbers of examples.The results are presented as test error rates in Figure 4.
  • Results: 17.26% to 9.23%: further pre-training reduces test error rates with only a small amount of training data.The passage attributes the improvement to further-pre-trained BERT.

5.7 Exp-V: Further Pre-Training on BERT Large

The BERTLARGE experiment examines whether the findings extend beyond BERTBASE. Task-specific further pre-training enables BERTBASE to outperform ULMFiT across tasks, while BERTLARGE achieves state-of-the-art results.

  • Experimental setup: The experiment further pre-trains BERTLARGE and fine-tunes it using long sequences on multiple GPUs.Further pre-training uses 120K steps, while target-task fine-tuning uses a maximum sequence length of 512.
  • Results: ULMFiT performs better than BERTBASE on almost all tasks but not BERTLARGE.This comparison is reported for the evaluated text-classification tasks.
  • Results: Task-specific further pre-training makes BERTBASE outperform ULMFiT on all tasks.The comparison changes after further pre-training on task-specific data.
  • Results: BERTLARGE with task-specific further pre-training achieves state-of-the-art results.The passage reports this as the principal outcome of the experiment.

6 Conclusion

The conclusion summarizes extensive experiments on BERT fine-tuning and identifies layer selection, learning-rate design, further pre-training, multi-task learning, and small-data performance as key findings. These findings support state-of-the-art performance on eight text-classification datasets.

  • Conclusion: The top BERT layer is more useful for text classification than lower layers.This is one of the paper’s reported experimental findings.
  • Conclusion: An appropriate layer-wise decreasing learning rate can overcome catastrophic forgetting.The conclusion links this learning-rate strategy to the forgetting problem.
  • Conclusion: Within-task and in-domain further pre-training significantly boost BERT’s performance.The conclusion reports both pre-training settings as effective.
  • Conclusion: Preceding multi-task fine-tuning helps single-task fine-tuning, but less than further pre-training.The conclusion explicitly compares the relative benefits of the two approaches.
  • Conclusion: BERT can improve text-classification tasks with small-size data and achieves state-of-the-art performance on eight widely studied datasets.The paper states this overall result after applying the reported findings.
Loading 1905.05583v3…