Source-linked AI summary
Nougat: Neural Optical Understanding for Academic Documents
Lukas Blecher, Guillem Cucurull, Thomas Scialom, Robert Stojnic
TL;DR
Scientific PDFs are difficult to convert into machine-readable text because mathematical expressions lose semantic structure. Nougat addresses this with an image-only encoder–decoder Transformer that converts document pages into formatted markup, supported by an automatically generated dataset. The conclusion reports potential for extracting text from digital-born PDFs and converting scanned papers and textbooks, while limitations remain for repetitions, document structure, languages, and preprocessing artifacts.
Problem
Scientific PDFs are difficult to extract into other formats, particularly because mathematical expressions lose semantic information needed for machine-readable science.
Method
Nougat is an end-to-end encoder–decoder Transformer that converts rasterized document pages into markup without relying on OCR or embedded text representations.
Results
The approach shows potential for extracting text from digital-born PDFs and converting scanned papers and textbooks.
Takeaways & Limitations
The released model, code, and dataset-generation process provide a basis for scientific document-to-markup recognition and future research.
Takeaways & Limitations
The model is trained mainly on English research papers; non-Latin-script documents result in immediate repetitions, and other document types may differ from its training structure.
Abstract
from arXiv · showhide
Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution to enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human-readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.
1 Introduction
Nougat addresses the loss of semantic information when extracting scientific content from PDFs by converting document-page images into formatted markup. The paper contributes a pretrained PDF-to-markup model, a PDF–source-code dataset pipeline, and image-only processing that also supports scanned papers and books.
- Motivation: PDFs make scientific information difficult to extract, especially because mathematical expressions lose semantic information such as relative positioning.Existing OCR engines detect characters and words but do not reliably model relationships among superscripts, subscripts, fractions, exponents, and matrices.
- Approach: Nougat is a transformer-based model that converts images of document pages into formatted markup text.
- Contributions: The paper releases a pretrained model and code capable of converting PDFs to a lightweight markup language.
- Contributions: The authors introduce a pipeline for creating a dataset that pairs PDFs with source code.
- Scope: Because Nougat depends only on page images, it can process scanned papers and books.
2 Related Work
Related work spans OCR, visual document understanding, and document-to-HTML/XML conversion. Nougat differs by targeting scientific documents while avoiding dependence on external OCR or embedded text representations.
- Optical Character Recognition: OCR research includes document digitization, handwriting recognition, scene text recognition, and mathematical-expression recognition, with Transformer architectures replacing earlier convolutional and recurrent components.
- Visual Document Understanding: Visual Document Understanding methods extract information by jointly modeling document text and layout, including spatial relationships among document elements.
- Document Conversion: GROBID and pdf2htmlEX process digital-born PDFs into XML or HTML, but neither recovers the semantic information of mathematical expressions.
3 Model
Nougat uses an end-to-end encoder–decoder Transformer that recognizes document-page images without OCR inputs. Its visual encoder produces latent patch embeddings, while an autoregressive decoder converts them into markup tokens; training uses augmentations that simulate scanned-document variability.
- Architecture: Nougat is an end-to-end encoder–decoder Transformer built on Donut that recognizes text implicitly without OCR-related inputs or modules.
- Encoder: The Swin Transformer encoder converts a resized and padded document image into a sequence of embedded visual patches.
- Decoder: A Transformer decoder uses cross-attention and autoregressive generation to convert encoded image representations into vocabulary logits.
- Data Augmentation: Training applies erosion, dilation, noise, blur, bitmap conversion, compression, grid distortion, and elastic transformations to simulate scanned-document imperfections.
4 Datasets
The authors construct a paired PDF-page/source-markup dataset from arXiv, PMC, and IDL materials, then heuristically align source paragraphs with PDF pages. The resulting ground truth retains semantic markup but includes several documented artifacts and page-splitting errors.
- Dataset sources and preparation: 1,748,201 arXiv articles provide source code and compiled PDFs, normalized through LaTeXML before conversion into lightweight markup.Normalization replaces macros, standardizes whitespace, normalizes tables, and resolves references and citations.
- Dataset sources and preparation: PMC articles are parsed into the same markup format, but limited to pre-training because equations and tables are often stored as images.The PMC XML files do not always provide sufficiently rich semantic information.
- Dataset sources and preparation: IDL documents contribute OCR text without formatting and are used only for pre-training to teach basic recognition of scanned documents.The IDL collection consists of industry-produced documents affecting public health.
- Page alignment: The paired dataset is formed by splitting markup according to PDF page breaks, rasterizing PDF pages, and reinserting removed figures and tables.Figures and tables are temporarily removed because their PDF positions may differ from their source-code positions.
- Page alignment: A TF-IDF bag-of-words model with a linear SVM predicts page numbers for source paragraphs, while recursive Gini-impurity splits identify page boundaries.Fuzzy matching then compares neighboring source text with adjacent PDF-page sentences to refine each predicted boundary and score its accuracy.
- Ground-truth artifacts: LaTeXML artifacts, missing figures or tables, and page-splitting errors can reduce ground-truth quality, although the authors state that many training samples compensate for small errors.Splitting may include text from the previous page or cut off words, especially around invisible formatting characters; PMC display equations and tables may be ignored.
5 Results & Evaluation
Nougat’s evaluation examines document content by modality and reports practical behavior on mathematical pages, repetition detection, and broader document conditions. The results also describe limitations arising from representation ambiguity, repetition collapse, and slow generation.
- Results overview: Nougat converts document-page images into formatted markup, focusing on relevant page content while skipping boxes around equations.A representative page shows model output converted to LaTeX and rendered back into PDF.
- Text modalities: Scientific articles contain plain text, mathematical expressions, and tables, which require separate evaluation because equivalent formulas can have different LaTeX representations.Formatting, notation, and inline-math boundary ambiguities can produce discrepancies even when rendered formulas appear identical.
- Comparison: GROBID results are poor primarily because it misses formulas, while equation accuracy is also affected by the quality of formula bounding boxes.GROBID converts XML output into compatible markup and sends identified formula snippets to LaTeX-OCR.
- Repetitions during inference: 1.5% of test-set pages entered repetitive loops, with higher frequency on out-of-domain documents.The model cannot recover from collapse by itself; repetition may involve repeated or alternating sentences.
- Repetitions during inference: Random token replacement during training was introduced to help the model handle wrongly predicted tokens without reducing observed performance.The perturbation process continues until the newly sampled proportion exceeds a 10% threshold.
- Repetitions during inference: Repetition detection uses logit variance over sliding and end-of-sequence windows, classifying sequences when the signal stays below 6.75.The sliding-window size is B = 15, with an inference-time approximation using the last 200 tokens and half the threshold.
- Limitations & Future work: The model processes 6 pages in parallel on an NVIDIA A10G, averaging 19.5s per batch for approximately 1400 tokens.This is slower than GROBID’s 10.6 PDF/s, although Nougat also handles scanned documents and mathematical expressions.
6 Conclusion
Nougat is an end-to-end encoder-decoder Transformer that converts document pages into markup using only rasterized page images. The authors report potential for processing both digital-born PDFs and scanned papers or textbooks.
- Nougat converts document pages to markup with an end-to-end trainable encoder-decoder Transformer.
- The method relies solely on rasterized page images rather than OCR or embedded text representations.
- The authors describe an automatic, unsupervised dataset-generation process for training scientific document-to-markup conversion.
- The approach is presented as applicable to digital-born PDFs, scanned papers, and textbooks.
- The code for model evaluation, training, and dataset generation is publicly accessible.
A Dataset
The dataset combines arXiv-based source-code pairs with additional PMC and Industry Documents Library material to increase document and layout coverage.
- The dataset composition is summarized in Table A.1.
- The corpus draws primarily on arXiv, where research documents are commonly paired with author-provided LaTeX source.
- PMC articles contribute semantic XML material but are used more sparingly because equations and tables are often represented as images.
B Examples
The examples span scanned historical books, NASA conference pages, scientific documents, and model-internal mathematical or physical formulations. They illustrate both sensible page conversion and the broader technical context surrounding the displayed material.
- Scanned documents: Scanned textbook pages can be converted into sensible text, although performance is noticeably worse than on digital-born documents.
- Scanned documents: An old calculus textbook example contains an exponent error where an almost illegible n is mistaken for ∗.
- Scientific examples: The displayed scientific examples include derivative rules, arithmetic means, pressure problems, and chemical-reaction analysis.
- Scanned documents: The appendix includes an example page from an old calculus textbook and a selection of NASA conference pages from 1970.
- Molecular examples: The molecular examples describe harmonic vibrational modes, cavity coupling, anharmonic interactions, and basis-function expansions.
- Molecular examples: The standard molecular method scales exponentially with coefficient count and is therefore suited only to problems with fewer than 6 degrees of freedom.