Nougat: Neural Optical Understanding for Academic Documents

1 Introduction

• Line-by-line OCR such as Tesseract misses relationships between characters, treating superscripts and subscripts like surrounding text and losing mathematical structure.This limits extraction of equations from academic documents and, in turn, their machine readability and accessibility.Source: §1 · p. 1
• Nougat is a transformer-based model that converts document-page images into formatted markup text.Its image-based input allows it to process scanned papers and books as well as digital documents.Source: §1 · p. 1
• The authors release a pretrained model that converts PDFs to lightweight markup, along with its code.Source: §1 · p. 1
• The paper introduces a pipeline for creating a dataset that pairs PDFs with source code.Source: §1 · p. 1

• Mathematical-expression recognition progressed from grammar-based methods through convolutional and RNN encoder-decoder models [12] [13] [15] [16] [17] to Transformer-based encoders or decoders [18] [21].Source: §2 · p. 2
• VDU models such as LayoutLM [22] [23] [24] jointly model text and layout, using masked layout prediction to capture spatial relationships among document elements.Source: §2 · p. 2
• GROBID parses born-digital scientific documents to XML focused on bibliographic data, while pdf2htmlEX converts born-digital PDFs to HTML preserving layout; neither recovers mathematical semantics.Source: §2 · p. 2

3 Model

• The end-to-end encoder-decoder model recognizes text implicitly from document images and requires no OCR inputs or modules.It builds on Donut [28].Source: §3 · p. 2
• A Swin Transformer encodes image patches, which a cross-attention decoder converts into tokens autoregressively.The decoder uses the mBART implementation [32] and the scientific-domain tokenizer from Taylor et al. [33].Source: §3 · p. 2

3.1 Setup

• The model uses 96-DPI document images resized and padded to 896 × 672, with a 350M-parameter architecture and decoder sequence length 4096.It is trained for 3 epochs with AdamW [34], effective batch size 192, and a learning rate decaying from 5 · 10^-5 to 7.5 · 10^-6; inference uses greedy decoding.Source: §3.1 · p. 3

3.2 Data Augmentation

• Image transformations simulate scanned-document imperfections, while random ground-truth token replacement significantly reduces collapse into repeating loops.Image augmentations include erosion, dilation, Gaussian noise and blur, bitmap conversion, compression, grid distortion, and elastic transform.Source: §3.2 · p. 3

4 Datasets

• Because no paired PDF-page/source-code dataset was available, the authors built one from open-access arXiv articles, adding PMC for layout diversity and IDL during pretraining.Source: §4 · p. 4
• They collected 1,748,201 arXiv articles, standardized source with LaTeXML, and converted it to markup; PMC was pretraining-only, while IDL provided unformatted OCR text [37] for scanned documents.PMC was limited because its XML does not always provide rich semantic information, and equations and tables may be stored as images.Source: §4 · p. 4

4.1 Splitting the pages

• They pair PDF pages with source paragraphs using TF-IDF bag-of-words and a linear SVM, then fuzzy-match page breaks; pages averaging at least 0.9 across both breaks are included, about 47% overall.Boundary points are chosen by minimizing a measure based on Gini impurity.Source: §4.1 · p. 5

4.2 Ground truth artifacts

• Ground truth may contain LaTeXML artifacts, missing figures or tables, and page splits that include preceding text or cut off words; PMC equations and tables may also be ignored.These issues reduce data quality, but the authors state that the large number of training samples compensates for the small errors.Source: §4.2 · pp. 5–6

5.1 Metrics

• The test-set evaluation reports normalized edit distance, BLEU, METEOR, and F1-score with precision and recall.Normalized edit distance divides character edit distance by the total number of characters; BLEU scores matching n-grams, while METEOR emphasizes recall.Source: §5.1 · p. 6

5.3 Comparison

• Nougat small and base outperform the other approaches and achieve high scores across all reported metrics, with the smaller model on par with base.The comparison is reported on the arXiv test set.Source: Table 1 · p. 7, §5.3 · p. 7

5.2 Text modalities

• Mathematical expressions score lower than plain text, partly because equivalent LaTeX forms and inline math boundaries can differ from ground truth despite visually identical rendering.The evaluation also notes formatting ambiguities and missing text due to inline math; an example page shows the model output rendered back into a PDF while skipping a box around equations.Source: Figure 5 · p. 6, §5 · p. 6, §5.2 · p. 7

5.3 Comparison

• GROBID’s reported results are poor due mainly to missed formulas and bounding-box quality, while embedded PDF text scores better than GROBID because of formatting differences.GROBID output is converted to compatible markup, Unicode formula symbols are replaced with LaTeX commands, and detected formula snippets are sent to LaTeX-OCR.Source: §5.3 · p. 7

5.4 Repetitions during inference

• Repetition collapse occurs on 1.5% of test pages, becomes more frequent out of domain, and generally follows a model mistake that it cannot recover from.The model may repeat one sentence, alternate between sentences, or count its own repetitions.Source: §5.4 · p. 8
• Random-token replacement during training reduces repetitions without observed performance loss, including a 32% decline in failed page conversions for out-of-domain documents.For each training example, a randomly chosen token is replaced with another random token with a fixed probability, continuing until the sampled number exceeds 10%.Source: §5.4 · p. 8
• Inference detects repetition with a logits-variance heuristic using a 15-token sliding window and a 6.75 threshold, checking a last-200-token subset with half the threshold during generation.After generation, the procedure is repeated for the full sequence; a signal below threshold that stays low to the sequence end indicates repetition.Source: §5.4 · pp. 8–9

5.5 Limitations & Future work

• The model works especially well on research-paper-like documents, but is slow and page-by-page processing can cause inconsistencies across a merged document.On an NVIDIA A10G with 24GB VRAM, it processes six pages in parallel and averages 19.5 seconds per batch at about 1,400 tokens, versus GROBID at 10.6 PDF/s. Tests suggest satisfactory performance on other Latin-based languages, but special characters are approximated and non-Latin scripts cause repetitions; resolving repetition collapse remains future work.Source: §5.5 · p. 9

6 Conclusion

• Nougat converts document pages to markup with an end-to-end trainable encoder-decoder transformer that relies solely on rasterized pages, not OCR or embedded text.The work applies visual document understanding to a novel OCR task.Source: §6 · p. 9
• The authors demonstrate an automatic, unsupervised dataset-generation process to train Nougat for scientific-document-to-markup conversion.Source: §6 · p. 9
• Nougat shows potential for extracting text from digital-born PDFs and converting scanned papers and textbooks.Source: §6 · p. 9
• Code for model evaluation, training, and dataset generation is available from the authors.Source: §6 · p. 9

• More than 91.5% of the corpus comes from arXiv, where author-provided LaTeX sources enable selective dataset construction.The LaTeX is left unprocessed, unlike PMC XML, which frequently substitutes equations and tables with images.Source: A Dataset · p. 13

B Examples

• Scanned old textbooks yield sensible text with few errors, but Nougat performs noticeably worse on them than on digital-born documents.The public-domain books came from the Internet Archive11 and Project Gutenberg12.Source: B Examples · p. 13
• An almost illegible exponent n is read as *, while predicting a comma instead of a dot can trigger a repetitive loop [45]; similar problems appear in NASA conference scans [46].Source: B Examples · p. 13, Figure B.1 · p. 14, Figure B.2 · p. 15
• Nougat is robust to artifacts arising when a printed thesis is hand-scanned with a mobile device.Source: B Examples · p. 13, Figure B.3 · p. 16