PaperlayerView exact revision on arXiv

Source-linked AI summary

PICK: Processing Key Information Extraction from Documents using Improved Graph Learning-Convolutional Networks

Wenwen Yu, Ning Lu, Xianbiao Qi, Ping Gong, Rong Xiao

arXiv:2004.07464v3cs.CV

TL;DR

KIE remains challenging because existing systems do not fully exploit documents’ visual and textual information, which is important for complex layouts. PICK combines text and image encoding with learned graph structure, graph convolution, and character-level decoding; experiments report significant improvements over baselines across scenarios.

  • Problem

    KIE is challenging because systems often overlook visual and non-sequential information while extracting key fields from documents.

  • Method

    PICK combines textual, image, positional, and layout features with learned graph structure and graph convolution for character-level KIE.

  • Results

    PICK outperforms baseline methods by significant margins across the reported real-world scenarios.

  • Takeaways & Limitations

    PICK provides a framework for KIE on documents with variable or fixed layouts by refining graph structure from visually rich context.

Abstract

from arXiv · show

Computer vision with state-of-the-art deep learning models has achieved huge success in the field of Optical Character Recognition (OCR) including text detection and recognition tasks recently. However, Key Information Extraction (KIE) from documents as the downstream task of OCR, having a large number of use scenarios in real-world, remains a challenge because documents not only have textual features extracting from OCR systems but also have semantic visual features that are not fully exploited and play a critical role in KIE. Too little work has been devoted to efficiently make full use of both textual and visual features of the documents. In this paper, we introduce PICK, a framework that is effective and robust in handling complex documents layout for KIE by combining graph learning with graph convolution operation, yielding a richer semantic representation containing the textual and visual features and global layout without ambiguity. Extensive experiments on real-world datasets have been conducted to show that our method outperforms baselines methods by significant margins. Our code is available at https://github.com/wenwenyu/PICK-pytorch.

I. INTRODUCTION

KIE remains difficult because sequence-based systems overlook documents’ visual and non-sequential information. PICK addresses this by learning document structure while combining textual, visual, positional, layout, and image features, and experiments report improvements over baselines.

  • Motivation: KIE extracts key fields into structured documents but remains challenging despite advances in OCR.It supports archiving, indexing, and document analytics.
  • Motivation: Sequence-tagging systems often ignore valuable text, position, layout, image, and other non-sequential document information.These features can be crucial for distinguishing entities such as TOTAL in complex layouts.
  • Prior Approaches: Hand-crafted approaches require task-specific knowledge and rules, while predefined graph methods face costly graph-design decisions or redundant node aggregation.The cited approaches use limited features, predefined edge structures, or fully connected graphs with possible noise.
  • PICK: PICK learns a soft adjacency matrix to refine graph context automatically and combines text, image, and position features for richer KIE representations.Graph convolution propagates information between related document nodes before character-level decoding.
  • Contributions: PICK is presented as effective and robust for complex document layouts, using textual, positional, layout, and image features.The contribution is framed around extracting key information without ambiguity.
  • Contributions: The improved graph learning module refines graph structure instead of relying on predefined structure.The paper also reports experiments against baseline methods on real-world datasets.

II. RELATED WORK

Prior KIE work ranges from text-focused extraction to graph-based multimodal modeling, but these approaches have limitations in visual coverage, graph construction, or adaptability. PICK’s architecture combines separate text and image encoders, graph processing, and sequence tagging to incorporate document structure.

  • Existing KIE Methods: Existing KIE research often emphasizes textual features, while some methods use visual features only for image processing.Recent multimodal approaches increasingly rely on graph-based representations.
  • Graph Convolution: Spatial graph convolution operates directly on neighboring node groups, whereas spectral methods rely on graph spectral representations and are less suited to dynamic structures.The paper places its graph convolution in the spatial category.
  • Related Graph Methods: Related graph methods differ from PICK because they may omit images or require task-specific edge types and graph connectivity.PICK instead learns node relationships automatically.
  • PICK Architecture: PICK’s encoder uses Transformer-based text embeddings and CNN-based image embeddings, combining them into local node representations.The text branch captures local textual context, while the image branch captures morphology information.
  • PICK Architecture: Its graph module models latent node relations, layout context from bounding boxes, and non-local, non-sequential document features.The decoder then performs character-level sequence tagging with BiLSTM and CRF components.

B. Encoder

PICK’s encoder builds document representations from two complementary branches: Transformer text embeddings and CNN image embeddings. Element-wise fusion combines these signals into graph-node inputs for downstream graph processing.

  • Encoder Overview: The encoder contains separate branches for text segments and image segments.These branches capture textual and morphological information, respectively.
  • Text Branch: The text branch uses a Transformer encoder to capture local textual context for each sentence.Each sentence is encoded independently into a document-level collection of text embeddings.
  • Image Branch: The image branch uses a CNN to encode each image segment and capture morphology information.The implementation uses ResNet and resizes image segments before encoding them individually.
  • Feature Fusion: Text and image embeddings are fused by element-wise addition to form node representations for the graph module.The fused representation X is pooled into X0 before graph processing.

C. Graph Module

The graph module addresses limitations of predefined and fully connected document graphs by learning a soft adjacency matrix for task-specific context modeling. It then uses graph convolution to propagate information between related nodes.

  • C. Graph Module: Existing graph methods either require predefined task-specific edges or aggregate redundant information through fully connected graphs.The former can miss distant latent connections, while the latter can include useless and redundant node information.
  • C. Graph Module: PICK learns a soft adjacent matrix A to model graph context for downstream tasks instead of predefining edge types.The graph learning module is incorporated into an existing graph architecture to adaptively refine graph structure.
  • C. Graph Module: The graph module first computes pairwise relationship weights between input graph nodes and extracts node features using an MLP with relation embeddings.The resulting graph structure and node features provide inputs for subsequent graph convolution.

1) Graph Learning:

Graph learning constructs a soft adjacency matrix from node representations and relation embeddings, using row-wise softmax normalization and a regularized objective. The objective favors stronger links between nearby nodes while encouraging sparsity and avoiding trivial solutions.

  • 1) Graph Learning:: Graph learning computes adjacency scores with a single-layer neural network and applies LeakyReLU to reduce vanishing-gradient problems.The resulting scores are normalized with softmax across each row of the adjacency matrix.
  • 1) Graph Learning:: Row-wise softmax makes each learned adjacency row nonnegative and normalized.This constrains the soft adjacent matrix to represent weighted pairwise relationships between nodes.
  • 1) Graph Learning:: The modified graph-learning loss encourages smaller weights for distant nodes and stronger connection weights for nearby nodes in representation space.The distance-based term is intended to prevent graph convolution from aggregating noise-node information.
  • 1) Graph Learning:: A sparsity term controls the soft adjacency structure, while regularization prevents the trivial solution wi = 0.The parameter γ controls sparsity, and η controls the importance of graph nodes in the distance-based term.

2) Graph Convolution:

Graph convolution propagates information over learned node relationships and node-edge-node triplets, combining visual features with relation embeddings. After multiple layers, each node receives contextual information about global layout and visual content.

  • 2) Graph Convolution:: The graph convolution network captures global visual and layout information by operating on node-edge-node triplets rather than isolated nodes.Each triplet combines a source node, a relation embedding, and a target node.
  • 2) Graph Convolution:: Relation embeddings encode distances, dimensions, aspect ratios, and sentence-length ratios between document nodes.Sentence length can provide latent importance information and a rough indication of text-segment font size when combined with the image.
  • 2) Graph Convolution:: Hidden triplet features are transformed through convolution layers, and updated node embeddings aggregate information from those features.The layer-wise propagation uses learned graph relationships and nonlinear transformations.
  • 2) Graph Convolution:: Adaptive graph learning supplies the graph structure while convolution layers produce task-specific node embeddings through layer-wise propagation.This separates relationship learning from feature propagation within the graph architecture.
  • 2) Graph Convolution:: After L layers, each node representation contains global layout and visual information and is propagated to the decoder for tagging.The initial graph input is V0 = X0, and the final contextual representation is used by the downstream sequence decoder.

D. Decoder

The decoder converts variable-length encoder outputs into a packed document representation and applies BiLSTM and CRF layers for key-information tagging. CRF training minimizes sequence negative log-likelihood, while decoding searches for the highest-probability entity sequence.

  • D. Decoder: The decoder consists of Union, BiLSTM, and CRF layers for key information extraction.The Union layer packs variable-length encoder outputs into a representation used for sequence tagging.
  • D. Decoder: The packed sequence is processed by a BiLSTM that produces emission scores for each entity at each character position.The output score matrix has one dimension for the number of entity types.
  • D. Decoder: The CRF models transitions between entity labels, including start-of-sequence and end-of-sequence states.Transition scores are combined with emission scores to score complete entity sequences.
  • D. Decoder: CRF training minimizes the negative log-likelihood of the correct entity sequence, and decoding selects the sequence with the highest conditional probability.The probability is defined over all possible entity sequences for the packed input.
  • D. Decoder: The full network is jointly trained with a loss combining graph-learning and CRF objectives.The graph-learning and CRF losses are weighted through a tradeoff parameter λ.
  • D. Decoder: Dynamic programming can improve the speed of the time-consuming CRF training and decoding procedures.The acceleration applies to both the training and decoding phases.

A. Datasets

The study evaluates PICK on medical invoices, train tickets, and SROIE receipts, covering variable and fixed document layouts with OCR-derived text regions.

  • Medical Invoice: Medical Invoice contains 2,630 images and six key text fields, with 2,104 images for training and 526 for testing.The dataset includes digits, English characters, and Chinese characters, with variable layouts and illegible or mispositioned text.
  • Dataset scope: The datasets represent both variable-layout and fixed-layout document extraction scenarios.Medical Invoice is described as variable-layout, while the experiments also include Train Ticket and SROIE datasets.
  • SROIE: SROIE contains 626 training receipts and 347 testing receipts, each annotated with company, address, date, and total fields.The receipts mainly contain digits and English characters.

B. Implementation Details

PICK combines Transformer text embeddings and ResNet50 image embeddings before graph processing, with experiments reported using mEF comparisons and component studies.

  • Encoder: The encoder uses Transformer text embeddings and ResNet50 image embeddings, fusing them by element-wise addition before the graph module and decoder.The Transformer output dimension is d_model = 512.
  • Evaluation: The experimental evaluation reports mEF for medical invoice, SROIE, train ticket, and component-comparison settings.The supplied table descriptions identify mEF as the evaluation metric for the reported comparisons.
  • Training and inference: Training uses 30 epochs, with each epoch taking about 35 minutes.At inference, the model predicts each text segment's most probable entity type without postprocessing except for SROIE, where a training-data lexicon autocorrects results.

C. Experimental Results

PICK outperforms baselines across the evaluated document scenarios, while ablations show that visual features and graph learning contribute to extraction performance and that shallow graph convolution performs best.

  • Medical Invoice: 14.7% improvement in overall mEF was achieved by PICK over the baseline on medical invoices, with gains for all entities.The largest increase occurred for Invoice Number mEF.
  • Benchmark results: PICK shows significant improvement over the baseline on train tickets and achieves competitive mEF results on SROIE using only the official training data.The train-ticket mEF is reported as almost a full score, while the SROIE comparison contrasts PICK with LayoutLM's use of extra pretraining data and document-class supervision.
  • Ablation Studies: Removing image segments lowers performance on both medical invoice and train ticket datasets, indicating that visual features support ambiguous entity extraction.Image segments provide appearance and semantic information such as font colors, backgrounds, and directions.
  • Ablation Studies: Removing graph learning causes large metric reductions on both datasets, especially variable-layout documents.The authors report that graph learning helps handle complex document structures and generalization.
  • Ablation Studies: 1- or 2-layer graph-convolution models outperform 3- or 4-layer models across the evaluated datasets.The paper attributes the deeper-model decline to possible overfitting and recommends task-specific layer counts.
  • Conclusion: PICK shows superior performance across the reported scenarios and supports KIE on documents with variable or fixed layouts.The conclusion presents the method as a perspective on structural information extraction from documents.
Loading 2004.07464v3…