Source-linked AI summary
EATEN: Entity-aware Attention for Single Shot Visual Text Extraction
He guo, Xiameng Qin, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding
TL;DR
Existing entity extraction commonly recognizes all image text before selecting predefined entities with handcrafted rules, and those rules can be sensitive to positional variation. EATEN uses an end-to-end entity-aware network to decode entities directly; the paper reports superior performance across its scenarios, including a passport comparison of 90.8% versus 11.1% for General OCR.
Problem
Conventional entity extraction uses OCR followed by handcrafted rules or layout analysis, and template-sensitive post-processing can produce inaccurate results when entity positions drift.
Source: §I · p. 1Method
EATEN uses a CNN and multiple entity-aware decoders in an end-to-end trainable network, with state transition between decoders and no complicated post-processing.
Source: §I · p. 1, §III · p. 2Results
EATEN shows superior performance in all scenarios; the passport comparison is 90.8% for EATEN versus 11.1% for General OCR.
Source: C. Performance · p. 5, §VI · p. 6Takeaways & Limitations
EATEN directly predicts target entities in one shot and is reported to extract EoIs from images with or without a fixed layout.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Extracting entity from images is a crucial part of many OCR applications, such as entity recognition of cards, invoices, and receipts. Most of the existing works employ classical detection and recognition paradigm. This paper proposes an Entity-aware Attention Text Extraction Network called EATEN, which is an end-to-end trainable system to extract the entities without any post-processing. In the proposed framework, each entity is parsed by its corresponding entity-aware decoder, respectively. Moreover, we innovatively introduce a state transition mechanism which further improves the robustness of entity extraction. In consideration of the absence of public benchmarks, we construct a dataset of almost 0.6 million images in three real-world scenarios (train ticket, passport and business card), which is publicly available at https://github.com/beacandler/EATEN. To the best of our knowledge, EATEN is the first single shot method to extract entities from images. Extensive experiments on these benchmarks demonstrate the state-of-the-art performance of EATEN.
I. INTRODUCTION
Entity extraction targets text belonging to predefined entities, but conventional OCR pipelines first recognize text and then rely on handcrafted rules or layout analysis. EATEN replaces that multi-stage approach with an end-to-end entity-aware network, while the related work distinguishes entity extraction from text-reading methods that do not preserve entity information.
I. INTRODUCTION
- Conventional entity extraction first recognizes image text with OCR, then selects predefined entities through handcrafted rules or layout analysis.For example, passport images may contain Country, Name, and Birthday entities.Source: §I · p. 1
- Rule-based post-processing can be sensitive to template positions, so slight entity-position drift may yield inaccurate extraction.The paper identifies post-processing, rather than detection and recognition ability, as a usual performance constraint.Source: §I · p. 1
- EATEN combines CNN image features with multiple entity-aware decoders, initial-state warm-up, and state transitions to extract entities in one trainable network.The framework is presented as avoiding complicated post-processing and handling spatial transformations and position drift without correction.Source: §I · p. 1
- The paper constructs a dataset of about 0.6 million real and synthetic images spanning train tickets, passports, and business cards.The reported proportions of Chinese, English, and digit EoIs are 40.5%, 21.6%, and 37.8%, respectively.Source: §I · p. 1
II. RELATED WORK
- Prior OCR-free word-spotting approaches predict words but lack entity information, whereas EATEN is designed to extract EoIs for all entities.Earlier entity-extraction pipelines are also described as redundant and complex, with privately established datasets often used for evaluation.Source: §II · p. 2
III. PROPOSED METHOD
EATEN uses a CNN backbone to extract image features and an entity-aware attention network to decode predefined entities. Its decoders are linked through state initialization and transition mechanisms.
III. PROPOSED METHOD
- A CNN-based backbone extracts high-level visual features from each image for the entity-aware attention network.The framework uses Inception v3, with feature maps F of shape H × W × C.Source: §III · p. 2
- Entity-aware decoders learn image entity layouts and decode the content of predefined EoIs.The number of decoders need not equal the number of entities because one decoder can capture one or several EoIs.Source: §III · p. 2
- The network initializes each decoder from the previous decoder’s last state and uses initial-state warm-up to improve attention performance.The paper says these mechanisms are proposed to improve the method’s performance.Source: §III · p. 2
A. Entity-aware Decoder
Each entity-aware decoder attends to image features and recurrently decodes characters. The LSTM combines prior-character and context information, and its output is used with updated context to predict characters.
A. Entity-aware Decoder
- Decoders are generally ordered left-to-right and top-to-bottom, with entities on the same text line assigned to one decoder when appropriate.The paper gives SS/TAN/DS as an example of entities assigned together.Source: A. Entity-aware Decoder · p. 2
- At each decoding step, entity-aware attention obtains a context feature, which is fed with the previously predicted character into an LSTM.The LSTM updates the context feature and predicts the current character.Source: A. Entity-aware Decoder · p. 2, Figure 2 · p. 3
- The LSTM output is combined with updated context to produce the predicted distribution over characters.The character feature is formed by concatenating context c_t with output O_t.Source: A. Entity-aware Decoder · p. 3, Figure 2 · p. 3
- The LSTM update uses the previous character encoding and context together with the prior hidden state.The previous character is represented as a one-hot encoding.Source: A. Entity-aware Decoder · p. 3
B. Initial State Warm-up
The paper introduces a buffer decoding step because the first decoder prediction from constant initial values may be unstable. Its hidden state initializes the first entity-aware decoder.
B. Initial State Warm-up
- The vanilla decoder may fail to converge because its first prediction depends on limited information initialized to constant values.At the first step, only the updated context is available for the prediction, which the authors say is barely sufficient for a successful prediction.Source: B. Initial State Warm-up · p. 3
- A buffer step emits a <WARMUP> token that is discarded, and its hidden state is used as the first decoder’s initial state.The warm-up is introduced to stabilize the first-step prediction.Source: B. Initial State Warm-up · p. 3
- The sample-image figure shows real and synthetic train tickets, alongside synthetic passport and business-card images.Source: Figure 3 · p. 3
C. State Transition
State transition connects neighboring entity-aware decoders because entities often have semantic relations. The preceding decoder’s final state initializes the next decoder, and the paper reports improved performance.
C. State Transition
- Neighboring entities often have strong semantic relations, while the decoders are otherwise independent.State transition is designed to establish those relations between neighboring decoders.Source: C. State Transition · p. 3
- State transition initializes each decoder with the preceding decoder’s final state, integrating semantic information from earlier decoders.The paper reports that this transition improves performance.Source: C. State Transition · p. 3
- The model is trained with maximum likelihood estimation, using a negative log-likelihood loss for each decoder.The overall loss sums the decoder losses across their time steps.Source: C. State Transition · p. 3
IV. DATASET
The synthetic data engine creates scenario-specific text images through text preparation, rendering, transformations, and noise. It preserves aspect ratios and saves each image with its entity labels.
A. Synthetic Data Engine
- Synthetic data offers a way to avoid releasing personal information contained in application images.The paper notes that such information must be erased before public release.Source: A. Synthetic Data Engine · p. 3
- The four-step generation pipeline includes preparing text and rendering entity text onto scenario-specific background templates.The text corpus includes material such as Chinese names and addresses; business-card rendering uses more than 100 templates, including 85 simple backgrounds and pure images with random color.Source: A. Synthetic Data Engine · pp. 3–4
- Synthetic samples are shown for train tickets, passports, and business cards across font rendering, transformation, and noising stages.The stages are arranged from left to right in Fig. 4.Source: Figure 4 · p. 4
- Images are randomly rotated within [-5, +5] degrees, resized by their longer side, and elastically transformed.These transformations are part of the synthetic image pipeline.Source: A. Synthetic Data Engine · p. 4
- The pipeline applies Gaussian noise, blur, average blur, sharpening, brightness, hue, and saturation changes, then resizes or pads images by scenario.Train-ticket images are set to 512 × 512, passport images to 600 × 450, and business-card templates remain 500 × 300.Source: A. Synthetic Data Engine · p. 4
- Text aspect ratios are preserved, images are saved with their entity labels, and test-set entities do not overlap training-set entities.Source: A. Synthetic Data Engine · p. 4
B. Benchmark Datasets
The benchmark datasets cover train tickets, passports, and business cards, with real images included only for train tickets. Passport and business-card evaluation is synthetic-only because of privacy concerns.
Train ticket
- The train-ticket dataset contains 2k real and 300k synthetic images, including 0.4k real test images.The remaining real images are split between training (90%) and validation (10%).Source: B. Benchmark Datasets · p. 4
Passport
- The passport dataset contains 100k synthetic images, with 2k reserved for testing.The remaining images are used for training and validation.Source: B. Benchmark Datasets · p. 4
Business card
- The business-card dataset contains 200k synthetic images, with 2k reserved for testing; entity positions vary and some entities may be absent.Source: B. Benchmark Datasets · p. 4
Benchmark scope
- Passport and business-card results are evaluated only on synthetic data because of privacy concerns.Source: B. Benchmark Datasets · p. 4
V. EXPERIMENTAL RESULTS
The experiments specify training, evaluation, and baseline settings across train-ticket, passport, and business-card scenarios. EATEN uses scenario-specific entity-aware decoders and is compared with OCR-based baselines and a state-transition ablation.
A. Implementation Details
- Training uses SGD with initial learning rate 0.04, decay factor 0.94 every four epochs, momentum 0.9, and gradient-norm clipping at 2.0.The implementation also uses weight decay, label smoothing, and LSTM cell-state clipping; eight Tesla P40 GPUs are used for training.Source: A. Implementation Details · p. 4
B. Experiment Setting
- Train-ticket and passport evaluation uses mean entity accuracy, while business-card evaluation uses mean entity precision, recall, and F-measure because entities may be absent.The business-card F-measure is the harmonic average of precision and recall; when prediction and target are both null, the indicator returns 0.Source: B. Experiment Setting · p. 4
- EATEN uses five decoders for eight train-ticket entities, five for seven passport entities, and nine for ten business-card entities.Entities grouped within a decoder are separated by <EOS> tokens; decoding steps are set according to the maximum character count of the assigned entities.Source: B. Experiment Setting · pp. 4–5
- Table I evaluates train tickets on synthetic-only, real-only, and fused data using mEA; Table II covers passport and business-card results.Source: Table I · p. 5, Table II · p. 5
- The baselines are General OCR with matching rules or templates, Attention OCR adapted to decode EoIs sequentially, and EATEN without state transition.The state-transition-free model is included as an ablation.Source: B. Experiment Setting · p. 5
C. Performance
EATEN outperforms the reported OCR and ablation baselines across train-ticket, passport, and business-card evaluations. The paper also reports faster inference than General OCR and performance across fixed- and flexible-layout scenarios.
C. Performance
- Train-ticket mEA gains are 16.6%, 5.8%, and 4.8% over General OCR, Attention OCR, and EATEN w/o state; passport is 90.8% vs 11.1% for General OCR.Source: C. Performance · p. 5
- EATEN achieves 7×, 12×, and 4× speedups over General OCR across the three scenarios, respectively.The authors also report an advantage over Attention OCR in the last two rows of Table III.Source: Table III · p. 5, C. Performance · p. 5
- EATEN outperforms General OCR by relative 29.6% on business cards.Source: C. Performance · p. 5
- The three scenarios include both fixed layouts and arbitrary business-card entity positions; the authors report room for improvement because results used no parameter tuning or post-processing.Source: C. Performance · p. 5
D. Discussion
The discussion finds that fused real and synthetic training improves results over real-only training, while examples show both common failure conditions and successful extraction under visual distortions.
D. Discussion
- Synthetic-engine training reaches 55.1% mEA, which the authors say shows synthetic images can simulate real images to some degree.Source: D. Discussion · p. 5
- Fused training outperforms real-only training by a large margin across the three reported comparisons.The reported pairs are 95.8% vs. 86.2%, 91.0% vs. 55.4%, and 90.0% vs. 68.1%.Source: D. Discussion · p. 6
- Among 0.4k real-image examples, errors commonly involve challenging lighting, disturbance, or text fusion, while correct extraction also occurs under transformations and position drift.The authors suggest a denoising component could alleviate the common error types.Source: D. Discussion · p. 6, Figure 5 · p. 6
VI. CONCLUSIONS
The conclusion presents EATEN as an end-to-end framework for extracting entities from images and reports strong performance across scenarios, including layouts with and without fixed structure.
VI. CONCLUSIONS
- EATEN is an end-to-end framework for extracting entities from images, evaluated using a dataset spanning three real-world scenarios.Source: §VI · p. 6
- EATEN directly predicts target entities in one shot without bounding-box or full-text annotations.Source: §VI · p. 6
- The conclusion reports superior performance across all scenarios and capacity to extract entities from images with or without a fixed layout.Source: §VI · p. 6