Source-linked AI summary
Image-to-Markup Generation with Coarse-to-Fine Attention
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush
TL;DR
The paper studies converting rendered images into presentational markup by approximately inverting a potentially many-to-one compile function. It proposes attention-based neural OCR with a coarse-to-fine mechanism and evaluates it using a new rendered-mathematics dataset, reporting over 30% exact-match output.
Problem
Image-to-markup generation requires approximately inverting a potentially complicated, ill-specified, many-to-one compile function while preserving content and layout.
Method
The paper uses a visual attention-based model and introduces coarse-to-fine attention, selecting a coarse image region before attending to fine-grained cells.
Results
Over 30% exact match output is reported for the image-to-markup task.
Takeaways & Limitations
The work shows that data-driven models can perform structured text OCR without knowledge of the language, while coarse-to-fine attention reduces attention overhead with only a small accuracy drop.
Takeaways & Limitations
Full attention has decoding complexity O(THW), which can be prohibitive for large images.
Abstract
from arXiv · showhide
We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention.
1. Introduction
The paper applies attention-based neural image-to-text models to structured mathematical markup, introducing a dataset and coarse-to-fine attention for scalable generation. The full system reproduces the same image on more than 75% of real-world test examples and produces over 30% exact-match output.
- Structured markup OCR must capture both textual content and presentational semantics, including subscripts, superscripts, special symbols, and nested fractions.
- Attention-based models are explored for generating correct markup without requiring a textual or visual grammar of the underlying language.The model uses a convolutional network, an attention-based recurrent decoder, and a source encoder adapted to OCR layout.
- CTC-style left-to-right assumptions are not required for neural OCR because general-purpose encoders can provide tracking for accurate attention.
- Coarse-to-fine attention significantly reduces attention overhead while causing only a small drop in accuracy when trained with REINFORCE.
- IM2LATEX-100K is a public dataset of rendered real-world mathematical expressions collected from published articles and paired with markup.
- More than 75% of real-world test examples are reproduced as the same image, while exact-match output exceeds 30%.
2. Problem: Image-to-Markup Generation
Image-to-markup generation converts a rendered image into markup describing its content and layout, approximately inverting an often many-to-one or ill-specified compile function. At test time, outputs are evaluated by rendering them and comparing the resulting image with the input.
- The task converts a rendered source image into target presentational markup that fully describes both content and layout.
- The target is a sequence of markup tokens, while rendering is defined by a possibly unknown, many-to-one compile function.
- Supervised learning approximately inverts the compile function from training pairs whose rendered markup approximately matches the source image.
- At test time, the generated markup is rendered and evaluated against the input image, even when it differs from the ground-truth markup.
3. Model
The model encodes an image into spatially arranged features, enriches each feature-map row with an RNN, and uses an attentive RNN decoder to generate markup tokens. This design preserves layout information and supports non-left-to-right markup order.
- The full-grid encoder supports non-left-to-right ordering in generated markup, unlike the assumptions used by most prior neural OCR systems.
- The encoded grid is supplied to an RNN decoder with visual attention to generate markup as a conditional language model.
- A CNN extracts a spatial feature grid, and row-wise RNNs encode relative layout information across the grid.Trainable row-specific initial hidden states act as positional embeddings for vertical order.
- The row encoder is crucial for performance because it lets the encoder implicitly localize positions within the source image.
4. Attention in Markup Generation
The model uses visual attention over image feature grids to track markup generation, with hierarchical and coarse-to-fine variants that reduce the attention search region and computation.
- Attention mechanism: The decoder uses an attentive context vector to track the image position relevant to each generated markup token.The attention distribution selects cells in an H × W feature grid and supplies the decoder with a context vector.
- Standard Attention: Standard attention computes a distribution over all fine-grid cells at every decoding step, yielding O(THW) decoding complexity.This can become prohibitive for large images.
- Hierarchical Attention: Hierarchical attention first selects relevant coarse-grid cells and then attends to fine cells inside those regions.The coarse grid is imposed over the image so each coarse cell contains a larger region of fine cells.
- Coarse-to-Fine Attention: Coarse-to-fine attention constructs a sparse coarse support before applying fine attention, reducing the number of fine cells considered.The paper explores sparsemax and hard-attention approaches for learning the coarse distribution.
- Coarse-to-Fine Attention: Sparsemax can produce K+ nonzero coarse entries, giving one-step attention complexity O(H′W′ + K+HW′).The paper reports that K+ is suitably small in practice.
- Coarse-to-Fine Attention: Hard coarse attention samples a coarse cell during training and uses an argmax coarse choice at test time.Because sampling makes the objective nondifferentiable, training uses REINFORCE; the resulting gradient estimator is noisy and slow to converge.
5. Dataset Construction
IM2LATEX-100K is a public dataset of rendered real-world LaTeX equations paired with source markup, supplemented by tokenization, normalization, and synthetic handwritten variants.
- Corpus: IM2LATEX-100K contains 103,556 LaTeX mathematical equations paired with rendered images.The formulas were extracted from LaTeX sources of papers in tasks I and II of the 2003 KDD Cup.
- Corpus: The construction pipeline extracts formulas with regular expressions, retains lengths from 40 to 1024 characters, renders compilable formulas, and converts PDFs to PNG images.Formulas that fail to compile are excluded.
- Corpus: The standardized split contains 83,883 training, 9,319 validation, and 10,354 test equations.Formula lengths range from 38 to 997 characters, with mean 118 and median 98.
- Tokenization: Markup is split into minimal meaningful LaTeX tokens rather than individual characters to avoid the longer sequences and greater memory use of character-based models.Tokens include symbols, modifiers, functions, accents, environments, and brackets.
- Normalization: An optional parser-based normalization converts markup to abstract syntax trees and applies safe transformations to remove common spurious ambiguities.Examples include fixing sub-superscript order and transforming matrices to arrays.
- Synthetic Data for Handwriting Recognition: A synthetic handwritten corpus reuses the dataset formulas while replacing individual symbols with randomly selected handwritten symbols from Detexify training data.The paper uses this corpus because labeled handwritten mathematical-expression data are scarce.
6. Experiments
Experiments compare IM2TEX with classical, commercial, neural, and ablated baselines on image-to-LaTeX generation, and also evaluate handwritten-expression recognition across CROHME settings.
- Image-to-LaTeX Experiments: IM2TEX is compared with classical OCR systems, neural baselines, model ablations, and the commercial InftyReader system.InftyReader combines symbol recognition with structural analysis.
- Image-to-LaTeX Experiments: The neural baselines include a captioning model without the row encoder and a CTC-based OCR model designed for natural-image text.The captioning comparison preserves parameter count by increasing CNN filters after removing the row encoder.
- Image-to-LaTeX Experiments: Ablations vary attention structure, including coarse-only, hierarchical, hard reinforcement-learning coarse-to-fine, and sparsemax coarse-to-fine mechanisms.These experiments examine the roles of fine-level features and attention style.
- Handwritten Mathematics: The CROHME 2013 and 2014 comparisons use 8,836 shared training expressions and address a domain designed for stroke-based OCR.The paper converts strokes to images and applies extensions to handle the domain differences.
- Evaluation: Evaluation checks exact rendered-image matches and additional matches after whitespace-column elimination, alongside perplexity and BLEU.Renderer misalignments under 5 pixels are not counted as exact-match errors.
- Training and Decoding: The complete model is trained end-to-end without additional markup-language or generating-process information and uses beam search with beam size 5 at test time.No further hard constraints are employed during generation.
7. Results
On IM2LATEX-100K, IM2TEX achieves over 75% exact image reproduction, while coarse-to-fine variants trade accuracy for fewer attention computations. Pretraining enables comparable performance on CROHME handwriting data, although the model is data hungry and commonly makes font or sizing errors.
- Baseline comparisons: The INFTY system has good text accuracy but poor exact-match image performance, while the neural CTC system performs poorly on this structured markup task.These comparisons support evaluating rendered images rather than text accuracy alone.
- IM2LATEX-100K results: Over 75% of real-world test examples receive markup that reproduces the same image with standard IM2TEX.The experiments also report BLEU and exact-match accuracy, including a whitespace-insensitive image comparison.
- Attention variants: Fine attention is crucial: the coarse-only system causes a large accuracy drop, whereas hierarchical attention preserves high performance.The attention variants are compared using both test accuracy and average coarse and fine lookup counts.
- Attention variants: Hard and sparsemax attention reduce attention lookups at a small accuracy cost, with hard selecting one coarse cell and sparsemax selecting multiple coarse cells.Hard always uses the minimal 4 × 4 = 16 fine lookups, while sparsemax achieves higher accuracy at the cost of multiple coarse selections.
- CROHME handwriting results: With synthetic-handwriting pretraining and CROHME fine-tuning, the models achieve performance comparable to the best systems except MyScript.The synthetic data lacks variation in baselines, font sizes, and other noise common in real data.
- Analysis: At least 16k training examples are needed to reach ≥50% accuracy, and common presentation-affecting errors involve font or sizing choices.The system usually preserves expression structure while making one or two symbol-recognition errors.
8. Conclusion
The paper presents a visual attention-based model and the IM2LATEX-100K dataset for OCR of presentational markup. Its coarse-to-fine attention reduces attention complexity while retaining fine-grained image information.
- Contributions: The work introduces a visual attention-based model for OCR of presentational markup and a public dataset for evaluating the task.IM2LATEX-100K supports image-to-markup experiments on rendered mathematical expressions.
- Evaluation: Figure 4 measures test accuracy (Match) as a function of training set size.The figure provides the comparison needed to assess how training-set scale relates to model accuracy.
- Contributions: Coarse-to-fine attention selects a region from a coarse image view before using the fine-grained cells within it.This design is proposed to reduce attention complexity.
- Evaluation: Figure 5 visualizes typical reconstruction errors by overlaying the gold image in red and the generated image in blue.The color coding makes discrepancies between target and generated renderings visible.