PaperlayerView exact revision on arXiv

Source-linked AI summary

PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks

Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, Rong Xiao

arXiv:2004.07464v3cs.CV

TL;DR

Many KIE systems treat extraction as linear text tagging, leaving visual and other non-sequential document information underused. PICK combines graph learning and graph convolution to integrate document features, and experiments report superior results across scenarios, including a 14.7% overall mEF improvement on medical invoices.

  • Problem

    Many KIE systems cast extraction as linear sequence tagging, leaving visual and non-sequential document information underused despite its relevance to richer semantic representations.

    Source: §I · p. 1

  • Method

    PICK learns a soft adjacency matrix to refine graph relations and uses graph convolution to combine textual, visual, positional, and layout features for KIE.

    Source: §I · p. 2, 1) Graph Learning · p. 4

  • Results

    PICK outperformed the baseline across reported scenarios; on medical invoices, it improved the overall mEF score by 14.7%.

    Source: C. Experimental Results · p. 7, §V · p. 8

  • Takeaways & Limitations

    The results support PICK's capacity for KIE on documents with variable or fixed layouts.

    Source: §V · p. 8

Abstract

from arXiv · show

Computer vision with state-of-the-art deep learning models has achieved huge success in the field of Optical Character Recognition (OCR) including text detection and recognition tasks recently. However, Key Information Extraction (KIE) from documents as the downstream task of OCR, having a large number of use scenarios in real-world, remains a challenge because documents not only have textual features extracting from OCR systems but also have semantic visual features that are not fully exploited and play a critical role in KIE. Too little work has been devoted to efficiently make full use of both textual and visual features of the documents. In this paper, we introduce PICK, a framework that is effective and robust in handling complex documents layout for KIE by combining graph learning with graph convolution operation, yielding a richer semantic representation containing the textual and visual features and global layout without ambiguity. Extensive experiments on real-world datasets have been conducted to show that our method outperforms baselines methods by significant margins. Our code is available at https://github.com/wenwenyu/PICK-pytorch.

I. INTRODUCTION

PICK addresses the challenge of extracting key information from documents by learning graph structure and combining textual and visual document features. Its graph convolution produces richer representations for character-level sequence tagging.

I. INTRODUCTION

  • KIE extracts key fields from OCR documents into structured records, but remains underexplored and challenging because plain-text tagging ignores valuable visual and non-sequential information.Layout and visual features can be crucial to distinguishing an entity such as TOTAL.Source: §I · p. 1
  • Hand-crafted methods rely on text, position, and task-specific rules,, while LayoutLM uses text, layout, and image features but lacks latent segment relationships and requires costly pretraining [6].The introduction also describes sequence-tagging approaches as operating on plain text without visual information or global document layout.Source: §I · p. 1
  • Graph-based methods combine textual and visual information,, but predefined graph edges require extensive human effort, while fully connected self-attention may aggregate noisy, redundant nodes.Choosing effective edge types and adjacency matrices is described as challenging, subjective, and time-consuming for sophisticated document structures.Source: §I · pp. 1–2
  • PICK learns a soft adjacency matrix to refine document graph structure, then propagates text, image, and position features through graph convolution for character-level sequence tagging.The learned representations are passed to a decoder within an encoder-decoder framework.Source: §I · p. 2
  • Extensive experiments on real-world datasets show that PICK outperforms baseline methods by significant margins.Source: §I · p. 2

II. RELATED WORK · III. METHOD · A. Notation

PICK combines textual, visual, and layout information through graph learning and graph convolution for document key information extraction. It encodes document segments, builds graph embeddings, and performs character-level sequence tagging.

II. RELATED WORK

  • Prior KIE methods often emphasize textual features,, [13], omit text or images, [15], or rely on task-specific features,,.Method [6] combines text and layout through pre-training but does not model relationships between document text.Source: §II · p. 2
  • PICK uses spatial graph convolution over neighboring nodes, whereas spectral methods are described as unsuitable for dynamic graph structures, [20],.Prior GCN approaches cited here do not incorporate visual features [22],,, [25],.Source: §II · pp. 2–3
  • Unlike and, PICK uses images and learns graph relationships to refine connectivity, filter useless nodes, and handle complex document layouts.[8] requires predefined task-specific edge types and connectivity; uses full connectivity, which can aggregate redundant information.Source: §II · p. 3

III. METHOD

  • The encoder combines Transformer text embeddings and CNN image embeddings into local representation X, which serves as input to the Graph Module.The text and image segments represent textual and morphological information, respectively.Source: §III · p. 3
  • The Graph Module learns latent node relations and combines bounding-box layout context into graph embeddings; a BiLSTM and CRF then tag the union non-local sentence at character level.This frames extraction as sequence tagging using document layout and global information.Source: §III · p. 3

A. Notation

  • A document D contains N text segments, each represented as characters with image segments and a bounding box; its characters receive sequential IOB labels.T denotes the length of sentence segment s_i.Source: A. Notation · p. 3
  • The document accessory graph is G = (V, R, E): V contains N nodes, R contains relations, and each edge represents a relation from node v_i to v_j.An edge is represented as e_ij = (v_i, α_ij, v_j).Source: A. Notation · p. 3

B. Encoder

PICK’s encoder combines textual and visual features through separate text and image branches, enriching document representations with local context and morphology information before graph processing.

B. Encoder

  • Unlike prior KIE work using only text segments or bounding boxes,, PICK’s encoder uses text and image branches to represent textual content and morphology together.Source: B. Encoder · p. 3
  • The text branch uses a Transformer to capture local textual context, encoding each sentence independently into document text embeddings.Source: B. Encoder · p. 3
  • The image branch uses a CNN implemented with ResNet to capture morphology, encoding image segments individually after resizing them so H × W = T.Source: B. Encoder · p. 4
  • PICK fuses text and image embeddings by element-wise addition, then passes the resulting document node embeddings to the Graph Module after pooling.Source: B. Encoder · p. 4

C. Graph Module

The Graph Module replaces predefined or fully connected graph structures with a learned soft adjacency matrix for modeling graph context. It then uses graph convolution to propagate and aggregate information into new node representations.

C. Graph Module

  • Prior graph methods require predefined task-specific edges, while fully connecting all nodes can aggregate useless and redundant information.The edge scheme in connects nearby horizontally or vertically adjacent text segments but can miss distant latent connections and fail to use all graph nodes.Source: C. Graph Module · p. 4
  • PICK incorporates an improved graph learning-convolutional network to learn a soft adjacency matrix for downstream graph-context modeling.The approach is inspired by.Source: C. Graph Module · p. 4
  • The module learns pairwise node relationship weights, extracts node features with MLP networks using relation embeddings, then applies graph convolution to propagate and aggregate information.The resulting representation is V′.Source: 1) Graph Learning · p. 4

1) Graph Learning:

The graph-learning stage learns a soft adjacency matrix and regularizes it to favor useful node connections while limiting noise and trivial solutions.

1) Graph Learning:

  • LeakyReLU scores and row-wise softmax produce the learnable soft adjacency matrix A.LeakyReLU is used instead of ReLU to address vanishing gradients during training.Source: 1) Graph Learning · p. 4
  • The modified loss based on assigns smaller weights to distant nodes and stronger connections to nearby nodes, discouraging aggregation from noise nodes.The distance effect is enlarged by an exponential operation.Source: 1) Graph Learning · p. 5
  • A sparsity term controls the soft adjacency matrix, with larger γ producing a sparser graph; η trades off the importance of graph nodes.The loss is averaged because the number of nodes varies across documents.Source: 1) Graph Learning · p. 5
  • The graph-learning loss is included as a regularizer in the final loss to prevent the trivial solution wi = 0, as discussed in.The Frobenius-norm-based loss includes the sparsity-control term.Source: 1) Graph Learning · p. 5

2) Graph Convolution:

PICK applies graph convolution to node-edge-node triplets, combining visual features with learned relations to build contextual node representations that encode visual information and global layout for tagging.

Graph Convolution

  • GCN captures global visual information and layout by convolving over node-edge-node triplets (vi, αij, vj), rather than nodes alone, as in.This lets the convolution use both node features and the relation between them.Source: 2) Graph Convolution · p. 5
  • Relation embeddings encode relative node geometry and sentence-length ratios, whose latent importance and text-scale cues can enrich the representation.The geometry includes horizontal and vertical distances, node dimensions, and aspect ratios; height normalization gives affine invariance.Source: 2) Graph Convolution · p. 5
  • Each convolution layer derives hidden pair features from node-edge-node triplets, combining visual features with relation embeddings for downstream aggregation.The hidden features are formed using learnable weights, a bias, and a ReLU activation.Source: 2) Graph Convolution · p. 5
  • Graph convolution updates each node by aggregating pairwise hidden features over the learned adaptive soft adjacency matrix through layer-wise propagation.Each layer uses a layer-specific learnable weight matrix.Source: 2) Graph Convolution · p. 5
  • After L layers, each node embedding contains contextual global-layout and visual information, which is passed to the decoder for tagging.Source: 2) Graph Convolution · p. 5

D. Decoder

The decoder combines sequence packing, graph-node features, bidirectional context, and CRF sequence tagging to extract key information. It trains the network jointly and uses dynamic programming to speed CRF training and decoding.

D. Decoder

  • Union packs variable-length encoder outputs into a document-level sequence and concatenates graph-node embeddings at each timestamp as layout and contextual features for extraction.The packed sequence replaces local text segments as the representation used for sequence tagging.Source: D. Decoder · p. 5
  • A BiLSTM uses past and future context to produce character-level entity emission scores for the packed sequence.The passage defines each emission score as the score for an entity at a character position.Source: D. Decoder · p. 5
  • The CRF [31] assigns sequence probabilities using entity transition scores, trains by minimizing the correct sequence’s negative log-likelihood, and decodes the highest-probability sequence.The start- and end-of-sequence entities are represented as SOS and EOS.Source: D. Decoder · pp. 5–6
  • The whole network is jointly trained using graph-learning and CRF losses, with a tradeoff parameter, while dynamic programming can speed CRF training and decoding.The loss combines LGL and Lcrf; the passage does not provide the tradeoff parameter’s value.Source: D. Decoder · p. 6

IV. EXPERIMENTS · A. Datasets

Experiments use three document datasets with different layouts, field schemas, and annotation resources. Medical Invoice and Train Ticket require added text-box annotations, while SROIE provides bounding boxes and transcripts.

A. Datasets

  • Medical Invoice contains 2,630 images with six key fields: insurance type, Chinese capital total amount, invoice number, social security number, name, and hospital name.It is mainly composed of digits, English characters, and Chinese characters.Source: A. Datasets · p. 6
  • Medical Invoice has variable layouts, illegible text, and out-of-position print, with 2,104 images for training and 526 for testing.Source: A. Datasets · p. 6
  • Train Ticket combines 2k real images and 300k synthetic images proposed in, with eight key fields and a fixed layout affected by background noise and imaging distortions.Its fields include ticket number, starting station, train number, destination station, date, ticket rates, seat category, and name.Source: A. Datasets · p. 6
  • Because Medical Invoice and Train Ticket lack text bounding boxes and transcripts, the authors annotated 400 real and 1,530 synthetic Train Ticket images and used OCR to obtain transcripts.Source: A. Datasets · p. 6
  • For the annotated Train Ticket data, all selected synthetic images and 320 real images were used for training, with the remaining real images reserved for testing.Source: A. Datasets · p. 6
  • SROIE provides 626 training and 347 test receipts with four key fields—company, address, date, and total—and includes text boxes and transcripts.It mainly contains digits and English characters and has a variable layout with a complex structure.Source: A. Datasets · p. 6

B. Implementation Details

This section specifies PICK’s model configuration, dataset-specific label generation and evaluation, and training and inference setup, alongside a BiLSTM-CRF baseline.

Networks Setting

  • PICK adds Transformer text and ResNet50 image embeddings before graph learning/convolution and BiLSTM-CRF decoding,.Transformer output dimension is 512; graph-learning parameters η and γ are 1 and 0.4, respectively, with one graph-convolution layer. The decoder uses a two-layer BiLSTM with hidden size 512, and the decoder loss tradeoff λ is 0.01.Source: B. Implementation Details · p. 6

Baseline method

  • Unlike PICK’s multimodal graph architecture, the baseline uses a two-layer BiLSTM-CRF over text segments concatenated in left-to-right, top-to-bottom order.The baseline architecture is described as validated in prior KIE work,, with BiLSTM hyperparameters similar to PICK’s.Source: B. Implementation Details · pp. 6–7

Evaluation Metrics

  • Medical-invoice, train-ticket, and SROIE performance is evaluated using mean entity recall (mER), mean entity precision (mEP), and mean entity F-1 (mEF) for variable entity counts.These metrics are defined in.Source: B. Implementation Details · p. 6

Label Generation

  • Train-ticket and SROIE labels use annotated entity types converted to IOB format, whereas medical-invoice labels use a heuristic because bbox and OCR-detected boxes may mismatch.For train tickets, OCR transcripts are generated for annotated boxes and enumerated for IOB conversion; SROIE provides boxes and transcripts, so entity types are annotated before the same conversion.Source: B. Implementation Details · p. 6

Implementation

  • PICK is trained from scratch in PyTorch with Adam [49] jointly minimizing CRF and graph-learning losses, using batch size 16 and learning rate 10^-4.Training uses eight NVIDIA Tesla V100 GPUs with 128 GB memory; dropout is 0.1 on the BiLSTM and Transformer encoder, and training runs for 30 epochs of about 35 minutes each.Source: B. Implementation Details · pp. 6–7
  • At inference, PICK predicts each text segment’s most likely entity type without postprocessing or constraint rules, except for SROIE, where a training-data lexicon autocorrects results.Source: B. Implementation Details · p. 7

C. Experimental Results · D. Ablation Studies · V. CONCLUSIONS

PICK improves extraction across the reported document scenarios; ablations highlight the roles of visual segments and graph learning, while the layer study favors shallow graph convolutions.

C. Experimental Results

  • 14.7% improvement in overall mEF was achieved by PICK over the baseline, which it outperformed across all medical-invoice entities.Source: C. Experimental Results · p. 7
  • Invoice Number showed the largest mEF increase, consistent with its distinguishing red-font visual features.Source: C. Experimental Results · p. 7
  • PICK significantly improved over the baseline on train tickets and achieved almost a full mEF score, suggesting it handles fixed-layout extraction well by learning document graph structure.Source: C. Experimental Results · p. 7
  • On SROIE, PICK achieved competitive mEF using only the official training data; LayoutLM [6] also used extra pretraining data and document-class supervision.Together with the invoice and ticket results, this supports robustness across variable- and fixed-layout documents.Source: C. Experimental Results · p. 7

D. Ablation Studies

  • Removing image segments or graph learning lowered scores on both datasets; graph-learning removal especially hurt variable-layout data, while image segments aid disambiguation and graph construction.Source: D. Ablation Studies · p. 7
  • The best graph-convolution results used 1 or 2 layers rather than 3 or 4, motivating task-specific layer counts to limit potential overfitting.The comparison evaluates mEF across datasets.Source: D. Ablation Studies · p. 8, Table IV · p. 8

V. CONCLUSIONS

  • PICK combines textual and visual document features with an improved graph-learning module that refines structure using visually rich context, supporting KIE on complex documents with fixed or variable layouts.Source: §V · p. 8

Related papers

Loading 2004.07464v3…