PaperlayerView exact revision on arXiv

Source-linked AI summary

Portuguese Named Entity Recognition using BERT-CRF

Fábio Souza, Rodrigo Nogueira, Roberto Lotufo

arXiv:1909.10649v2cs.CLcs.IRcs.LG

TL;DR

Portuguese NER needs effective modeling despite scarce annotated resources and abundant unlabeled text. The paper trains Portuguese BERT models and combines BERT with a CRF, comparing feature-based and fine-tuning strategies. Fine-tuning achieves new state-of-the-art results on HAREM I, improving F1-score by about 1 point in the selective scenario and 4 points in the total scenario.

  • Problem

    Portuguese NER is valuable because annotated resources are scarce while unlabeled text is abundant, motivating evaluation of pre-trained language models.

    Source: §1 · p. 1

  • Method

    The paper trains Portuguese BERT models and uses a BERT-CRF architecture while comparing feature-based and fine-tuning strategies.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3

  • Results

    Fine-tuning obtains new state-of-the-art results on HAREM I, improving F1-score by about 1 point in the selective scenario and 4 points in the total scenario.

    Source: §5 · p. 5, §6 · p. 6

  • Takeaways & Limitations

    Portuguese BERT models can outperform the previous state of the art for Portuguese NER, and the authors release code and models for reproducibility.

    Source: §6 · p. 6

Abstract

from arXiv · show

Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of a trained model to downstream natural language processing tasks, such as named entity recognition (NER) and question answering. It has been shown that the leverage of pre-trained language models improves the overall performance on many tasks and is highly beneficial when labeled data is scarce. In this work, we train Portuguese BERT models and employ a BERT-CRF architecture to the NER task on the Portuguese language, combining the transfer capabilities of BERT with the structured predictions of CRF. We explore feature-based and fine-tuning training strategies for the BERT model. Our fine-tuning approach obtains new state-of-the-art results on the HAREM I dataset, improving the F1-score by 1 point on the selective scenario (5 NE classes) and by 4 points on the total scenario (10 NE classes).

1 Introduction

NER identifies and classifies named entities, but contextual semantics and differing definitions complicate the task and its evaluation. This work applies Portuguese BERT models to NER, comparing feature-based and fine-tuning strategies where annotated resources are scarce.

  • NER identifies text spans mentioning named entities and assigns them predefined categories.Source: §1 · p. 1
  • Contextual semantics, surrounding text, and differing entity definitions make NER and its evaluation challenging.Source: §1 · p. 1
  • Pre-trained language models improve many NLP tasks and reduce the labeled data needed for supervised learning.Source: §1 · p. 1
  • The work evaluates Portuguese BERT models for NER and compares feature-based with fine-tuning training strategies.Source: §1 · p. 1
  • Portuguese NER is a valuable setting because annotated resources are scarce while unlabeled text is abundant.Source: §1 · p. 1

2 Related Work

Prior Portuguese NER research used CRF, local grammars, neural architectures, and contextual embeddings. The proposed method is illustrated as a document-level BERT-to-CRF pipeline using overlapping spans and maximum-context predictions.

  • Earlier Portuguese NER systems combined CRF models with handcrafted features or local grammars.Source: §2 · p. 1
  • Neural NER reduced feature engineering by learning representations, including character-level and word-level features for sequential classification.Source: §2 · pp. 1–2
  • The proposed pipeline tokenizes documents with WordPiece, splits them into overlapping maximum-length spans, and processes each span through BERT and a classifier.Source: Figure 1 · p. 2
  • Subtokens are removed before CRF processing, and tokens with maximum context are concatenated into the final predicted tag sequence.Source: Figure 1 · p. 2
  • Portuguese contextual-embedding systems combined Flair or ELMo representations with BiLSTM-CRF architectures.Source: §2 · p. 2

3 Model

The model combines BERT token representations, a token-level classifier, and a linear-chain CRF for structured NER prediction. It supports frozen feature-based transfer and jointly trained fine-tuning, while using overlapping document spans to preserve context.

3.1 BERT-CRF for NER

  • BERT encodes each token, a classifier projects representations into the tag space, and a linear-chain CRF scores tag transitions.Source: §3.1 · p. 2
  • The CRF transition matrix includes start and end states, and sequence probabilities are trained using the correct tag sequence.Source: §3.1 · pp. 2–3
  • During evaluation, dynamic programming supports the sequence objective and Viterbi decoding selects the most likely tag sequence.Source: §3.1 · p. 3

3.2 Feature-based and Fine-tuning approaches

  • In feature-based transfer, BERT weights remain frozen while a BiLSTM classifier and CRF are trained on BERT representations.Source: §3.2 · p. 3
  • In fine-tuning, a linear classifier is used and all weights, including BERT’s, are updated jointly.Source: §3.2 · p. 3
  • Long documents are split into overlapping spans, and each token’s final prediction comes from the span where it has the most contextual information.Source: §3.3 · p. 3

4 Experiments

The experiments pre-train cased Portuguese BERT Base and Large models using a large, diverse corpus, then evaluate NER training configurations. The setup includes corpus cleaning, whole-word masking, long training runs, and distinct model initializations.

  • The experiments define BERT pre-training and NER training setups, including datasets, training procedures, and hyperparameters.Source: §4 · p. 3
  • Portuguese BERT models are trained in Base and Large sizes with a maximum sequence length of 512 tokens and cased inputs.Source: §4.1 · p. 3
  • The cased 30k-subword vocabulary is built with SentencePiece BPE from 200k Portuguese Wikipedia articles and converted to WordPiece format.Source: §4.1.1 · p. 3
  • brWaC supplies 2.68 billion tokens from 3.53 million documents and provides whole-document coverage, domain diversity, and content quality for pre-training.Source: §4.1.2 · p. 3
  • Pre-processing removes mojibakes and residual HTML tags, producing 17.5GB of raw text.Source: §4.1.2 · p. 3
  • Pre-training runs for 1,000,000 steps with whole-word masking, a 1e-4 learning rate, warmup over 10,000 steps, and linear decay.Source: §4.1.3 · pp. 3–4
  • BERT Base starts from Multilingual BERT Base, while BERT Large starts from English BERT Large and uses different sequence-length and batch-size schedules.Source: §4.1.3 · p. 4

4.2 NER experiments

The experiments evaluate Portuguese NER on HAREM I using two entity-class scenarios, several BERT models, and feature-based or fine-tuning strategies. Dataset ambiguity is resolved into single labels for sequence tagging, and evaluation uses MiniHAREM with CoNLL-style exact-match metrics.

NER datasets

  • HAREM I provides 10 manually annotated entity classes, while experiments use Selective (5 classes) and Total (10 classes) scenarios.The Selective scenario includes Person, Organization, Location, Value, and Date; the Total scenario includes all ten classes.Source: §4.2.1 · p. 4
  • First HAREM is used for training and MiniHAREM for testing.Source: §4.2.1 · p. 4

Dataset preprocessing

  • Ambiguous <ALT> segments are resolved by selecting the alternative with the most named entities, breaking ties by choosing the first.Entities assigned multiple classes receive the first valid class for the scenario.Source: §4.2.2 · p. 4

NER experimental setup

  • Three BERT models are evaluated with IOB2 tagging and a stride of D = 128 tokens for splitting inputs into spans.The models are Multilingual BERT-Base, Portuguese BERT-Base, and Portuguese BERT-Large.Source: §4.2.3 · p. 4
  • Table 2 reports Precision, Recall, and F1-score on MiniHAREM using the CoNLL 2003 exact-match evaluation script.Reported values are averages across runs with different random seeds, and † marks feature-based results.Source: Table 2 · p. 5

Evaluation

  • Invalid IOB2 transitions are removed during evaluation, trading recall for potentially higher precision.Source: §4.2.3 · p. 5

5 Results

Fine-tuned Portuguese BERT-CRF achieves the strongest reported Portuguese NER results, outperforming prior systems across both evaluation scenarios. Results also show precision–recall trade-offs for CRF and post-processing choices, while fine-tuning consistently exceeds feature-based training.

Overall results

  • About 1 F1 point on Selective and 4 points on Total separate Portuguese BERT-CRF from the previous state of the art.The previous state of the art is BiLSTM-CRF+FlairBBP.Source: §5 · p. 5
  • 8.3 absolute F1 points on Total and 7.0 on Selective separate Portuguese BERT-CRF from LSTM-CRF without contextual embeddings.Source: §5 · p. 5

CRF contribution

  • CRF models improve or match simpler variants in overall F1, usually with higher precision but lower recall.Source: §5 · p. 5

Model size and training strategy

  • Portuguese BERT-Large models perform best in both scenarios, but feature-based use degrades relative to smaller variants while remaining above Multilingual BERT.The authors hypothesize that the limited NER dataset size contributes to this behavior.Source: §5 · p. 5
  • Feature-based models perform significantly worse than fine-tuned models.The performance gap exceeds reported English NER values.Source: §5 · p. 5

Post-processing

  • Filtering invalid IOB2 transitions raises average F1 by 1.9 points for feature-based and 1.2 points for fine-tuning approaches.It reduces recall by 0.4 points while increasing precision by 3.5 points on average.Source: §5 · p. 5

6 Conclusion

The paper establishes Portuguese BERT-CRF as a new state of the art on HAREM I by pre-training Portuguese BERT models and fine-tuning them for Portuguese NER. It emphasizes reproducibility and releases models and code, while leaving newer architectures for future work.

Conclusion

  • The work reports a new state of the art on the HAREM I corpora.Source: §6 · p. 6
  • Portuguese BERT models are pre-trained on unlabeled text and then fine-tuned in a BERT-CRF model for Portuguese NER.Source: §6 · p. 6
  • The proposed model outperforms BiLSTM-CRF+FlairBBP despite being pre-trained on substantially less data.Source: §6 · p. 6
  • The authors release code and models to support reproducibility and benchmarking of Portuguese NLP tasks.Source: §6 · p. 6

Future work

  • Experiments with RoBERTa and T5 are left for future work.Source: §6 · p. 6

A.1 SentencePiece to WordPiece conversion

The conversion procedure adapts a SentencePiece vocabulary to BERT’s WordPiece tokenization rules by adding required special tokens and punctuation handling. Tokens containing punctuation are split at those characters.

Vocabulary conversion

  • The generated SentencePiece vocabulary is converted to WordPiece following BERT’s tokenization rules.Source: §A.1 · p. 8
  • The conversion inserts [CLS], [MASK], [SEP], and [UNK], then adds punctuation characters from the Multilingual vocabulary.Source: §A.1 · p. 8
  • SentencePiece tokens containing punctuation are split at those characters after whitespace and punctuation chunking.Source: §A.1 · p. 8

A.2 HAREM Dataset preprocessing example

The HAREM preprocessing example illustrates how one ambiguous annotation can encode alternative named-entity solutions. Applying the stated selection rule chooses the alternative containing the most named entities.

A.2 HAREM Dataset preprocessing example

  • HAREM annotations can represent multiple alternative named-entity solutions within an <ALT> tag.The example uses “|” to separate alternatives.Source: §A.2 · p. 8
  • The example offers one Person entity spanning “Governo de Cavaco Silva.”Source: §A.2 · p. 8
  • The example also offers one Organization entity spanning “Governo de Cavaco Silva.”Source: §A.2 · p. 8
  • A third solution identifies “Governo” as Organization and “Cavaco Silva” as Person, yielding two named entities.Source: §A.2 · p. 8
  • The preprocessing rules select the third solution because it contains the highest number of named entities.Source: §A.2 · p. 8

Related papers

Loading 1909.10649v2…