BERTgrid: Contextualized Embedding for 2D Document Representation and Understanding

1 Introduction

• Document layout can carry crucial semantic information, but conventional NLP discards it and raw-pixel processing must learn text as well as semantics.Serializing a document can lose information needed for extracting tabulated data.Source: §1 · p. 1
• Chargrid and CUTIE preserve document spatial arrangement in a grid, with reported benefits over sequential inputs for tabulated or spatially arranged text.The cited approaches construct a 2D grid of characters or words.Source: §1 · p. 1
• BERTgrid adds contextualized word-piece embeddings to a grid document representation and evaluates it on invoice tables and headers.The embeddings come from a BERT language model pre-trained on unlabeled documents from the target domain.Source: §1 · p. 1
• 65.48% ± 0.58 was achieved, improving on the Chargrid baseline of 61.76% ± 0.72 for invoice information extraction.The authors report significant improvements on an invoice dataset previously described by Katti et al. (2018).Source: §1 · pp. 1–2

2.1 BERTgrid document representation

• The document is serialized line by line, and its word pieces are embedded in the grid using dense contextualized vectors rather than character-level one-hot encodings.The grid is constructed from word-piece positions.Source: §2.1 · p. 2
• BERTgrid assigns contextualized BERT vectors to word-piece positions in a 2D grid, using zero vectors for background.The embedding uses the second-to-last BERT hidden layer; word pieces are associated with bounding boxes.Source: §2.1 · p. 2

2.2 Models

• The extraction pipeline obtains words and positions, builds the BERTgrid, then feeds it to the same fully convolutional encoder-decoder and segmentation and bounding-box tasks used by Katti et al. (2018).Predicted masks and boxes are compared with the document data to obtain extracted strings.Source: §2.2 · p. 2
• [C+BERTgrid] adds a Chargrid branch to the BERTgrid model and merges the branches by adding their hidden representations.The related [Wordgrid] and [C+Wordgrid] variants use non-contextualized word2vec embeddings trained on unlabeled invoices.Source: §2.2 · p. 2

3.1 Data

• The 12k labeled invoices are split 10k/1k/1k, with disjoint vendors across sets; a separate 700k-invoice unlabeled dataset supports BERT pre-training and word2vec embeddings.The invoices span many vendors and mixed languages, mostly English.Source: §3.1 · p. 3

3.2 Metric

• The evaluation measure pools insertions, deletions, and modifications across the test set; it can be negative, while 1 denotes perfect extraction.The measure is described as similar to edit distance.Source: §3.2 · p. 3

3.3 Results and discussion

• 65.48% ± 0.58 mean absolute performance for [C+BERTgrid] is 6.02% (relative) above the baseline; contextualized representations achieve the best extraction scores.[Wordgrid] performs worse on all fields, attributed to out-of-vocabulary words, while [C+Wordgrid] beats the baseline on most fields.Source: Table 1 · p. 4, §3.3 · p. 4
• The authors assume BERTgrid's performance stems from word-piece-level embedding and contextualization; [C+Wordgrid] and [C+BERTgrid] also converge faster than [Chargrid].Words such as “avenue,” “street,” and “drive” can map to approximately the same embedding vector despite differing character-level representations.Source: §3.3 · p. 4
• Pre-training on unlabeled invoices distills invoice-language knowledge into BERT parameters, which the authors say eases downstream extraction.They contrast BERT's capacity to capture complex dependencies with simpler non-contextualized embeddings such as word2vec.Source: §3.3 · p. 4
• BERT receives serialized S rather than document D; the authors identify 2D positional encodings as future work for preserving layout during language-model pre-training and inference.Source: §3.3 · p. 4

• BERTgrid embeds word pieces with contextualized vectors in a 2D grid, using abundant unlabeled data to support downstream tasks such as invoice key-value extraction.The paper reports BERTgrid as superior to Chargrid and Wordgrid baselines.Source: §4 · p. 4