Source-linked AI summary
Chargrid: Towards Understanding 2D Documents
Anoop Raveendra Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, Jean Baptiste Faddoul
TL;DR
Structured-document understanding is difficult because sequential NLP loses layout relationships, while image-based methods emphasize visual content and require text extraction. Chargrid encodes pages as 2D character grids and uses an encoder-decoder pipeline to predict segmentation masks and bounding boxes. On invoice information extraction, chargrid performs well across tasks, while baseline methods are accurate mainly on selected fields.
Problem
Sequential NLP and image-only approaches are limited for structured documents whose semantics depend on both textual content and 2D layout.
Method
Chargrid represents document pages as sparse 2D character grids and applies an encoder-decoder network that predicts segmentation masks and object bounding boxes.
Results
Chargrid performs well across invoice information-extraction tasks, whereas sequential NLP and image-only computer-vision baselines achieve accurate predictions mainly for selected fields.
Takeaways & Limitations
The chargrid representation preserves positioning, size, and alignment so models can capture 2D relationships among characters, words, and larger text units.
Takeaways & Limitations
Chargrid-net training can require up to three days to converge, compared with a few hours for the sequential model.
Abstract
from arXiv · showhide
We introduce a novel type of text representation that preserves the 2D layout of a document. This is achieved by encoding each document page as a two-dimensional grid of characters. Based on this representation, we present a generic document understanding pipeline for structured documents. This pipeline makes use of a fully convolutional encoder-decoder network that predicts a segmentation mask and bounding boxes. We demonstrate its capabilities on an information extraction task from invoices and show that it significantly outperforms approaches based on sequential text or document images.
1 Introduction
Structured documents encode meaning through layout as well as text, exposing limitations in sequential NLP and image-only vision approaches. Chargrid preserves this 2D structure while supporting document understanding and invoice information extraction.
- Structured documents use layout, positioning, and sizing as semantic cues beyond textual content.
- Sequential NLP methods can fall short when word relationships depend on document layout rather than serialized order.
- Image-based approaches preserve 2D visual information but must first extract text before learning document semantics.
- Chargrid represents documents as sparse 2D character grids and formulates understanding as instance-level semantic segmentation with masks and bounding boxes.
- The paper applies chargrid to invoice information extraction and compares it with NLP and computer vision approaches.
2 Related Work
Prior work largely separates sequential language understanding from document layout analysis and visual processing. Chargrid combines these perspectives to address 2D document understanding while retaining spatial structure.
- NLP methods generally process unformatted text as a 1D sequence without intrinsic 2D structure.
- Document analysis focuses on character recognition, localization, binarization, and layout segmentation rather than character- and word-level semantics.
- Related computer vision tasks include scene text recognition, semantic segmentation, and object detection, but typically operate on natural images.
- Yang et al. improved structured-document layout segmentation by appending sentence embeddings to decoder visual features.
- Chargrid combines computer vision, NLP, and document analysis to understand 2D documents while retaining their structure.
3 Document Understanding with Chargrid
Chargrid represents document pages as sparse 2D character grids that preserve spatial layout, then uses a convolutional encoder-decoder to perform semantic and instance segmentation. The architecture combines segmentation and bounding-box prediction to distinguish multiple instances of the same class.
- 3.1 Chargrid: Chargrid maps character boxes onto a page-sized 2D grid, filling each character’s area with its encoded index and empty regions with 0.Character boxes may come from OCR or document layout information, and overlapping corners are assigned using the closest box center.
- 3.1 Chargrid: The representation preserves spatial structure while encoding characters as scalar indices rather than grayscale pixel patterns.Because each character region has a constant value, the grid can be significantly downsampled without losing character information.
- 3.1 Chargrid: Larger characters occupy more character-pixels, implicitly encoding information such as font size.This differs from traditional NLP, where each character is represented by exactly one token.
- 3.1 Chargrid: 1-hot encoding converts the integer chargrid g ∈ N^H×W into a vector representation ˜g ∈ R^H×W×N_C for neural-network input.The vocabulary includes padding/background and unknown-character entries.
- 3.2 Network Architecture: The chargrid-net uses a fully convolutional encoder-decoder with separate semantic-segmentation and bounding-box-regression branches.The encoder uses VGG-type blocks with dilated convolutions, batch normalization, and spatial dropout; decoder branches reverse encoder downsampling through transposed convolutions and lateral connections.
- 3.2 Network Architecture: Training combines three equally contributing losses: segmentation, box-mask, and box-coordinate regression.These use cross entropy, binary cross entropy, and Huber losses respectively, with focal-loss augmentation for both cross-entropy terms.
4 Information Extraction from Invoices
The invoice task extracts header fields and repeated line-items from diverse, multilingual documents without assuming a fixed format. Chargrid encodes pages as 2D character grids, while a segmentation-and-box model supports field classification and line-item grouping.
- 4.1 Data: The extraction target comprises five header fields, three line-item subfields, and background, for nine classes assigned to each invoice character.Header fields are unique, whereas line-items can occur multiple times.
- 4.1 Data: The dataset contains 12k scanned invoices from diverse vendors and languages, split into 10k training, 1k validation, and 1k test invoices with vendor-disjoint sets.The split is designed to estimate generalization to unseen invoice layouts.
- 4.2 Implementation Details: The representation is normalized to a fixed 336x256 resolution through token-space and one-hot-encoding downsampling, with landscape pages squeezed into the target size.The pipeline uses nearest-neighbor interpolation before encoding and bilinear interpolation for the final downsampling.
- 4.2 Implementation Details: Chargrid preserves document layout by representing each page as a sparse 2D character grid, then predicts pixel-level classes and bounding boxes for grouping line-item instances.Bounding boxes span each line-item row while segmentation identifies its column classes.
- 4.3 Evaluation Measure: Evaluation measures extraction work using insertion, deletion, and modification errors pooled across the test set, and the score can be negative when manual extraction would require less work.OCR errors do not affect the measure when identical errors occur in predictions and ground truth.
5 Experiments and Results
On invoice extraction, chargrid-net performs well across fields by exploiting 2D relationships, while image and sequential baselines have narrower strengths. Adding an image encoder provides no benefit at full capacity and reduces accuracy when encoder capacity is divided.
- Chargrid-net matches the sequential approach on single-instance, single-word fields such as Invoice Number, Invoice Amount, and Invoice Date.These fields depend less on 2D structure and can also be approached with serialized-text named entity recognition.
- Chargrid-net significantly outperforms the sequential model on multi-instance or multi-word fields requiring 2D relationships, including line-item descriptions, quantities, and amounts.The sequential model fails to correctly identify these line-item fields and can yield a negative accuracy measure.
- The proposed chargrid models perform consistently well across all eight invoice information-extraction fields compared with sequential and image models.The evaluation uses an 8-class information-extraction problem on invoices.
- Chargrid-net errors include incorrect segmentation and boxes, missed line-item boxes, and failures to separate adjacent multi-row line-items.Some ground-truth annotations are debatable, particularly for line-item descriptions, and peculiar line-item structures can also challenge human experts.
- Chargrid-net performs much better than the image-only model on smaller fields, while the gap is much smaller for larger fields such as Line-item Description.Smaller fields require textual content to be accurately localized, whereas larger fields can be localized using vision alone.
- Chargrid-hybrid-C64 is essentially as accurate as chargrid-net, whereas chargrid-hybrid-C32 has significantly reduced accuracy.The C32 configuration divides encoder capacity between the image and chargrid branches; the added image encoder does not provide additional benefits in this dataset.
- Most discriminative information in this extraction problem comes from the chargrid encoder branch and therefore from the chargrid representation.
6 Discussion
Chargrid models structured documents with a 2D character representation and an encoder-decoder network that captures layout while predicting invoice information. Its accuracy advantages over sequential NLP and image-only vision come with greater computational cost.
- Chargrid captures 2D relationships among characters, words, and larger text units through a character-grid representation.The representation encodes positioning, size, and alignment for textual components.
- An encoder-decoder network predicts accurate segmentation masks and bounding boxes that pinpoint relevant invoice information.The network uses fully convolutional processing and produces both pixel-level labels and grouped object instances.
- Only chargrid performs well across all invoice information-extraction tasks, whereas sequential NLP and image-only vision baselines succeed mainly on selected fields.Sequential models handle fields with discriminative keywords, while image-only vision handles large visual columns but struggles when text understanding is required.
- Chargrid’s accuracy benefits over sequential neural NLP models incur a larger computational cost.The added 2D representation increases total data dimensionality despite fully convolutional parallelization.
- Character-pixels are categorical rather than grayscale, requiring 1-hot encoding and producing a highly sparse representation for segmentation networks.This differs from the pixel representations traditionally used in image segmentation.
7 Conclusion
Chargrid is presented as a generic 2D text representation that supports document understanding beyond the demonstrated invoice information-extraction task. The authors identify other NLP tasks, blended text-image inputs, and alternative representations as follow-up directions.
- Chargrid is a generic representation for 2D text that can support multiple document-understanding tasks.Suggested tasks include document classification, named entity recognition, information extraction, and part-of-speech tagging.
- The invoice information-extraction experiment is presented as an initial demonstration of incorporating 2D document structure into document understanding.
- Future directions include applying chargrid to other NLP tasks, testing other computer-vision algorithms, and replacing character one-hot encoding with word embeddings.The latter representation is termed a wordgrid.