PaperlayerView exact revision on arXiv

Source-linked AI summary

Portuguese Named Entity Recognition using BERT-CRF

Fábio Souza, Rodrigo Nogueira, Roberto Lotufo

arXiv:1909.10649v2cs.CLcs.IRcs.LG

TL;DR

Portuguese NER needs effective modeling despite scarce annotated resources and abundant unlabeled text. The paper trains Portuguese BERT models and combines BERT with a CRF, comparing feature-based and fine-tuning strategies. Fine-tuning achieves new state-of-the-art results on HAREM I, improving F1-score by about 1 point in the selective scenario and 4 points in the total scenario.

  • Problem

    Portuguese NER is valuable because annotated resources are scarce while unlabeled text is abundant, motivating evaluation of pre-trained language models.

  • Method

    The paper trains Portuguese BERT models and uses a BERT-CRF architecture while comparing feature-based and fine-tuning strategies.

  • Results

    Fine-tuning obtains new state-of-the-art results on HAREM I, improving F1-score by about 1 point in the selective scenario and 4 points in the total scenario.

  • Takeaways & Limitations

    Portuguese BERT models can outperform the previous state of the art for Portuguese NER, and the authors release code and models for reproducibility.

Abstract

from arXiv · show

Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of a trained model to downstream natural language processing tasks, such as named entity recognition (NER) and question answering. It has been shown that the leverage of pre-trained language models improves the overall performance on many tasks and is highly beneficial when labeled data is scarce. In this work, we train Portuguese BERT models and employ a BERT-CRF architecture to the NER task on the Portuguese language, combining the transfer capabilities of BERT with the structured predictions of CRF. We explore feature-based and fine-tuning training strategies for the BERT model. Our fine-tuning approach obtains new state-of-the-art results on the HAREM I dataset, improving the F1-score by 1 point on the selective scenario (5 NE classes) and by 4 points on the total scenario (10 NE classes).

1 Introduction

NER identifies and classifies named entities, but contextual semantics and differing definitions complicate the task and its evaluation. This work applies Portuguese BERT models to NER, comparing feature-based and fine-tuning strategies where annotated resources are scarce.

  • NER identifies text spans mentioning named entities and assigns them predefined categories.
  • Contextual semantics, surrounding text, and differing entity definitions make NER and its evaluation challenging.
  • Pre-trained language models improve many NLP tasks and reduce the labeled data needed for supervised learning.
  • The work evaluates Portuguese BERT models for NER and compares feature-based with fine-tuning training strategies.
  • Portuguese NER is a valuable setting because annotated resources are scarce while unlabeled text is abundant.

2 Related Work

Prior Portuguese NER research used CRF, local grammars, neural architectures, and contextual embeddings. The proposed method is illustrated as a document-level BERT-to-CRF pipeline using overlapping spans and maximum-context predictions.

  • Earlier Portuguese NER systems combined CRF models with handcrafted features or local grammars.
  • Neural NER reduced feature engineering by learning representations, including character-level and word-level features for sequential classification.
  • The proposed pipeline tokenizes documents with WordPiece, splits them into overlapping maximum-length spans, and processes each span through BERT and a classifier.
  • Subtokens are removed before CRF processing, and tokens with maximum context are concatenated into the final predicted tag sequence.
  • Portuguese contextual-embedding systems combined Flair or ELMo representations with BiLSTM-CRF architectures.

3 Model

The model combines BERT token representations, a token-level classifier, and a linear-chain CRF for structured NER prediction. It supports frozen feature-based transfer and jointly trained fine-tuning, while using overlapping document spans to preserve context.

  • 3.1 BERT-CRF for NER: BERT encodes each token, a classifier projects representations into the tag space, and a linear-chain CRF scores tag transitions.
  • 3.1 BERT-CRF for NER: The CRF transition matrix includes start and end states, and sequence probabilities are trained using the correct tag sequence.
  • 3.1 BERT-CRF for NER: During evaluation, dynamic programming supports the sequence objective and Viterbi decoding selects the most likely tag sequence.
  • 3.2 Feature-based and Fine-tuning approaches: In feature-based transfer, BERT weights remain frozen while a BiLSTM classifier and CRF are trained on BERT representations.
  • 3.2 Feature-based and Fine-tuning approaches: In fine-tuning, a linear classifier is used and all weights, including BERT’s, are updated jointly.
  • 3.2 Feature-based and Fine-tuning approaches: Long documents are split into overlapping spans, and each token’s final prediction comes from the span where it has the most contextual information.

4 Experiments

The experiments pre-train cased Portuguese BERT Base and Large models using a large, diverse corpus, then evaluate NER training configurations. The setup includes corpus cleaning, whole-word masking, long training runs, and distinct model initializations.

  • The experiments define BERT pre-training and NER training setups, including datasets, training procedures, and hyperparameters.
  • Portuguese BERT models are trained in Base and Large sizes with a maximum sequence length of 512 tokens and cased inputs.
  • The cased 30k-subword vocabulary is built with SentencePiece BPE from 200k Portuguese Wikipedia articles and converted to WordPiece format.
  • brWaC supplies 2.68 billion tokens from 3.53 million documents and provides whole-document coverage, domain diversity, and content quality for pre-training.
  • Pre-processing removes mojibakes and residual HTML tags, producing 17.5GB of raw text.
  • Pre-training runs for 1,000,000 steps with whole-word masking, a 1e-4 learning rate, warmup over 10,000 steps, and linear decay.
  • BERT Base starts from Multilingual BERT Base, while BERT Large starts from English BERT Large and uses different sequence-length and batch-size schedules.

4.2 NER experiments

The experiments evaluate Portuguese NER on HAREM I using two entity-class scenarios, several BERT models, and feature-based or fine-tuning strategies. Dataset ambiguity is resolved into single labels for sequence tagging, and evaluation uses MiniHAREM with CoNLL-style exact-match metrics.

  • NER datasets: HAREM I provides 10 manually annotated entity classes, while experiments use Selective (5 classes) and Total (10 classes) scenarios.The Selective scenario includes Person, Organization, Location, Value, and Date; the Total scenario includes all ten classes.
  • NER datasets: First HAREM is used for training and MiniHAREM for testing.
  • Dataset preprocessing: Ambiguous <ALT> segments are resolved by selecting the alternative with the most named entities, breaking ties by choosing the first.Entities assigned multiple classes receive the first valid class for the scenario.
  • NER experimental setup: Three BERT models are evaluated with IOB2 tagging and a stride of D = 128 tokens for splitting inputs into spans.The models are Multilingual BERT-Base, Portuguese BERT-Base, and Portuguese BERT-Large.
  • NER experimental setup: Table 2 reports Precision, Recall, and F1-score on MiniHAREM using the CoNLL 2003 exact-match evaluation script.Reported values are averages across runs with different random seeds, and † marks feature-based results.
  • Evaluation: Invalid IOB2 transitions are removed during evaluation, trading recall for potentially higher precision.

5 Results

Fine-tuned Portuguese BERT-CRF achieves the strongest reported Portuguese NER results, outperforming prior systems across both evaluation scenarios. Results also show precision–recall trade-offs for CRF and post-processing choices, while fine-tuning consistently exceeds feature-based training.

  • Overall results: About 1 F1 point on Selective and 4 points on Total separate Portuguese BERT-CRF from the previous state of the art.The previous state of the art is BiLSTM-CRF+FlairBBP.
  • Overall results: 8.3 absolute F1 points on Total and 7.0 on Selective separate Portuguese BERT-CRF from LSTM-CRF without contextual embeddings.
  • CRF contribution: CRF models improve or match simpler variants in overall F1, usually with higher precision but lower recall.
  • Model size and training strategy: Portuguese BERT-Large models perform best in both scenarios, but feature-based use degrades relative to smaller variants while remaining above Multilingual BERT.The authors hypothesize that the limited NER dataset size contributes to this behavior.
  • Model size and training strategy: Feature-based models perform significantly worse than fine-tuned models.The performance gap exceeds reported English NER values.
  • Post-processing: Filtering invalid IOB2 transitions raises average F1 by 1.9 points for feature-based and 1.2 points for fine-tuning approaches.It reduces recall by 0.4 points while increasing precision by 3.5 points on average.

6 Conclusion

The paper establishes Portuguese BERT-CRF as a new state of the art on HAREM I by pre-training Portuguese BERT models and fine-tuning them for Portuguese NER. It emphasizes reproducibility and releases models and code, while leaving newer architectures for future work.

  • Conclusion: The work reports a new state of the art on the HAREM I corpora.
  • Conclusion: Portuguese BERT models are pre-trained on unlabeled text and then fine-tuned in a BERT-CRF model for Portuguese NER.
  • Conclusion: The proposed model outperforms BiLSTM-CRF+FlairBBP despite being pre-trained on substantially less data.
  • Conclusion: The authors release code and models to support reproducibility and benchmarking of Portuguese NLP tasks.
  • Future work: Experiments with RoBERTa and T5 are left for future work.

A.1 SentencePiece to WordPiece conversion

The conversion procedure adapts a SentencePiece vocabulary to BERT’s WordPiece tokenization rules by adding required special tokens and punctuation handling. Tokens containing punctuation are split at those characters.

  • Vocabulary conversion: The generated SentencePiece vocabulary is converted to WordPiece following BERT’s tokenization rules.
  • Vocabulary conversion: The conversion inserts [CLS], [MASK], [SEP], and [UNK], then adds punctuation characters from the Multilingual vocabulary.
  • Vocabulary conversion: SentencePiece tokens containing punctuation are split at those characters after whitespace and punctuation chunking.

A.2 HAREM Dataset preprocessing example

The HAREM preprocessing example illustrates how one ambiguous annotation can encode alternative named-entity solutions. Applying the stated selection rule chooses the alternative containing the most named entities.

  • A.2 HAREM Dataset preprocessing example: HAREM annotations can represent multiple alternative named-entity solutions within an <ALT> tag.The example uses “|” to separate alternatives.
  • A.2 HAREM Dataset preprocessing example: The example offers one Person entity spanning “Governo de Cavaco Silva.”
  • A.2 HAREM Dataset preprocessing example: The example also offers one Organization entity spanning “Governo de Cavaco Silva.”
  • A.2 HAREM Dataset preprocessing example: A third solution identifies “Governo” as Organization and “Cavaco Silva” as Person, yielding two named entities.
  • A.2 HAREM Dataset preprocessing example: The preprocessing rules select the third solution because it contains the highest number of named entities.
Loading 1909.10649v2…