Source-linked AI summary
SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion
Ahmed Nassar, Andres Marafioti, Matteo Omenetti, Maksym Lysak, Nikolaos Livathinos, Christoph Auer, Lucas Morin, Rafael Teixeira de Lima, Yusik Kim, A. Said Gurbuz, Michele Dolfi, Miquel Farré, Peter W. J. Staar
TL;DR
Document conversion requires handling varied layouts and multimodal elements while existing solutions rely on large models, handcrafted ensembles, or incomplete open datasets. SmolDocling addresses this with a compact vision-language model that emits unified DocTags and expanded training data, achieving competitive performance against much larger models while retaining known autoregressive output defects.
Problem
Document conversion lacks comprehensive open multimodal datasets, while existing large-model and ensemble approaches have computational, hallucination, or handcrafted-pipeline limitations.
Method
SmolDocling is a 256M-parameter end-to-end VLM trained on expanded document datasets to generate unified DocTags encoding page content, structure, and location.
Results
SmolDocling outperforms GOT, Nougat, and the 27-times-larger Qwen2.5-VL on every full-page transcription metric, while also outperforming Nougat and Qwen2.5-VL on formula recognition.
Takeaways & Limitations
Unified DocTags enable a compact model to compete with substantially larger vision-language models for multi-task document understanding.
Takeaways & Limitations
Typical defects include missing tags, malformed structure, and endless token repetition, which may prevent correct parsing or rendering.
Abstract
from arXiv · showhide
We introduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements in their full context with location. Unlike existing approaches that rely on large foundational models, or ensemble solutions that rely on handcrafted pipelines of multiple specialized models, SmolDocling offers an end-to-end conversion for accurately capturing content, structure and spatial location of document elements in a 256M parameters vision-language model. SmolDocling exhibits robust performance in correctly reproducing document features such as code listings, tables, equations, charts, lists, and more across a diverse range of document types including business documents, academic papers, technical reports, patents, and forms -- significantly extending beyond the commonly observed focus on scientific papers. Additionally, we contribute novel publicly sourced datasets for charts, tables, equations, and code recognition. Experimental results demonstrate that SmolDocling competes with other Vision Language Models that are up to 27 times larger in size, while reducing computational requirements substantially. The model is currently available, datasets will be publicly available soon.
1 Introduction
SmolDocling addresses document conversion challenges caused by variable layouts, opaque PDFs, limited open multimodal datasets, and the cost and hallucination risks of large models. It combines an ultra-compact VLM, unified DocTags, expanded datasets, and evaluation across document features.
- Document conversion is difficult because varied layouts, opaque PDFs, forms, tables, and charts complicate reading order and semantic understanding.
- Large multimodal models offer one-shot conversion but face limited high-quality open datasets, hallucinations, and substantial computational requirements.
- SmolDocling is a 256M-parameter VLM that produces a unified representation of page content, structure, and spatial location.It is reported as 5–10 times smaller than comparable document-understanding VLMs.
- The work adds full-page and task-specific training data covering layout, tables, code, charts, and formulas in natural document context.
- DocTags is introduced as a markup format optimized to represent document content and layout efficiently.
- SmolDocling’s prediction quality is evaluated across several tasks against popular community models.
2 Related Work
Related work spans large multimodal models, compact vision-language architectures, commercial and open-source document systems, ensemble pipelines, and multitask document-understanding models. SmolDocling is positioned among conversion-focused approaches that seek unified structural recognition.
- Large multimodal models extend language models to vision and demonstrate broad capabilities across modalities.
- SmolVLM-256M uses a compact architecture with SigLIP visual features, aggressive pixel shuffling, fewer image hidden states, and special tokens for efficient tokenization.
- Document-understanding systems include commercial services, frontier models, and open-source libraries covering OCR, layout, tables, charts, and equations.
- Ensemble systems conditionally combine specialized models with handcrafted preprocessing, thresholds, matching, and postprocessing logic.
- Multitask models handle several document tasks using shared context and representations, with OCR-reliant and OCR-free approaches.
- SmolDocling is closest to conversion and structural-recognition systems such as Nougat, DocOwl 2, GOT, and Qwen2.5-VL.
3 SmolDocling
SmolDocling converts document-page images into unified DocTags sequences that encode content, structure, and location. Its format separates text from structure, supports nested descriptors, and is trained through progressive alignment and mixed document tasks.
- SmolDocling converts page images into DocTags representing captions, charts, forms, code, equations, tables, lists, headers, footers, headings, and text.
- DocTags encode each element’s type, page location, content, and nested information such as captions, table structure, list items, and classifications.
- SmolVLM-256M combines a SigLIP visual backbone, a 135M language backbone, and pixel shuffling that compresses each 512x512 image patch into 64 visual tokens.
- DocTags use an unambiguous vocabulary that separates textual content from document structure, addressing ambiguity in direct HTML or Markdown conversion.
- OTSL tags encode table structure and content together, including cell spans and header information, while special blocks nest descriptors.
- The model uses one DocTags representation for cropped elements and full pages to maintain consistent visual-semantic alignment.
- Training progressively introduces DocTags, first adapting the non-vision network with the encoder frozen and then training on pretraining and task-specific data.
4 Data
The data program addresses limited unified multimodal document resources with large pretraining corpora, annotated full-page datasets, and task-specific collections for layout, tables, charts, code, equations, and instruction tuning.
- Public document datasets often provide fragmented formats, limited annotations, or insufficient visual diversity across page elements.
- 4.1 Pre-training datasets: DocLayNet-PT contains 1.4M pages with weak annotations for layout, table structure, language, topic, and figure classification in DocTags.
- 4.1 Pre-training datasets: Docmatix adds full multi-page DocTags conversion instructions to 1.3M documents alongside 9.5 million DocVQA questions.
- 4.2 Task-specific Datasets: Layout data combines 60K human-reviewed DocLayNet v2 pages, 63K WordScape pages, and 250K synthetic pages under unified annotations.
- 4.2 Task-specific Datasets: The table task converts public table-structure annotations into OTSL sequences interleaved with cell text.
- 4.2 Task-specific Datasets: The chart dataset contains 2.5 million visually diverse line, pie, bar, and stacked-bar charts generated from 90,000 tables.
- 4.2 Task-specific Datasets: The code dataset contains 9.3M rendered snippets at 120 dpi across 56 programming languages.
- 4.2 Task-specific Datasets: The data program also combines public and custom-rendered formula collections, including approximately 730k public formulas and 4.7 million arXiv formulas.
5 Experiments
SmolDocling is evaluated across text, layout, table, chart, and downstream document-conversion tasks, using standardized comparisons and qualitative inspection. It generally performs strongly against larger models, while retaining identifiable autoregressive output defects.
- Text Recognition: SmolDocling significantly outperforms GOT, Nougat, and 27-times-larger Qwen2.5-VL on every full-page transcription metric.Formula recognition also exceeds Nougat and Qwen2.5-VL while closely matching GOT.
- Layout: SmolDocling outperforms Qwen2.5-VL-7b on layout analysis, although both models remain well below the human baseline.Moderate bounding-box recall and label confusion affect both systems.
- Table Structure Recognition: SmolDocling performs competitively against significantly larger models on table structure recognition, with stronger structure-only TEDS scores than text-inclusive scores.Low-resolution table crops likely limit text transcription because of insufficient training at those resolutions.
- Chart Extraction: Despite its smaller size, SmolDocling achieves a competitive TEDS score on chart-to-table extraction.Variability in ground-truth styles and dataset-specific representations contributes to prediction inconsistencies.
- Qualitative Results: DocTags preserve headers, captions, list nesting, code indentation, and code line breaks in generated document representations.These traits support downstream handling of repetitive page elements and linked document information.
- Qualitative Results: Typical defects include missing location tags, malformed structure, and endless token repetition that can prevent correct parsing or rendering.The single-pass design can avoid some error accumulation found in ensemble pipelines, even when localization is incorrect.
6 Conclusion
The paper presents SmolDocling as a compact document-conversion VLM with rich output representation and supporting datasets. Its results indicate that unified DocTags formats can enable smaller models to compete with much larger systems, while page-element localization remains an area for improvement.
- Conclusion: SmolDocling is an efficient, compact VLM optimized for document conversion with a rich output representation.The paper also presents new datasets in a unified document-conversion format, including code listing transcription.
- Conclusion: Smaller models with unified output formats such as DocTags can compete with significantly larger models in multi-task document understanding.The conclusion identifies page-element localization as requiring further refinement.
A.1 SynthChartNet
The paper describes synthetic datasets and DocTags design choices for representing diverse document content in context. DocTags combine structured markup, nesting, content-specific tags, and normalized page locations.
- SynthChartNet: SynthChartNet contains 2.5 million visually diverse line, pie, bar, and stacked-bar charts rendered with Matplotlib, Seaborn, and Pyecharts.The rendering process uses broad library features while preserving plausible data.
- SynthCodeNet: SynthCodeNet renders code snippets with LaTeX listings and Pygments to produce publication-style and IDE-like visual appearances.The two libraries support visual diversity while preserving plausible code renderings.
- SynthFormulaNet: SynthFormulaNet normalizes extracted equations by filtering tokens, standardizing structure, and simplifying redundant patterns before rendering.Equations are rendered with randomized fonts, sizes, styles, weights, and line spacing to increase visual diversity.
- DocTags: DocTags use XML-like opening and closing tags, standalone instruction tags, and page-break separators within page or multi-page representations.The format is designed to represent document content and layout characteristics explicitly.
- DocTags: DocTags encode page-element positions with four location values on a fixed 0–500 grid proportional to page width and height.Elements can also contain nested tags for captions, tables, lists, code, and other document structures.
- DocTags: The vocabulary includes programming-language tags for 57 supported values and image-class tags covering chart, diagram, document, and scientific-image categories.These tags add semantic classification alongside structural markup.
B Additional Details about the SmolDocling
The additional details include a supported-conversion-instructions table and a qualitative molecule-image comparison. The figure compares outputs from GOT-OCR 2.0, Qwen2.5-VL-72B, and SmolDocling on a simple molecule image.
- Supported Conversion Instructions: Table 6 lists the conversion instructions supported by SmolDocling.
- Qualitative Comparison: Figure 7 compares molecule-image recognition predictions from GOT-OCR 2.0, Qwen2.5-VL-72B, and SmolDocling.The comparison uses a simple molecule image.
- Qualitative Comparison: The molecule-image comparison is qualitative rather than a reported aggregate metric.The supplied figure description identifies the compared systems and input image but does not state an outcome.
C.1 Molecule Image Recognition Experiment
SmolDocling was evaluated for molecule image recognition on both cropped images and full document pages. The experiment shows a substantial gap between its general document recognition and specialized molecular-structure models on detailed reconstruction.
- Training setup: SmolDocling was trained for molecule recognition on cropped images and full document pages.The cropped-image training used MolGrapher-Synthetic-300K, while full-page training used Doc2Mol-80K from PatCID.
- Training setup: MolGrapher-Synthetic-300K contains 300K synthetic samples generated from PubChem structures drawn with RDKit.
- Training setup: Doc2Mol-80K contains 80,000 real United States patent documents published between 2022 and 2024.
- Evaluation: SmolDocling and similar document models capture high-level molecular features but consistently fail to reconstruct detailed molecular structures.The example molecule is described as trivial for specialized models, which accurately recognize its structure.
- Evaluation: Closing the gap may require specialized SMILES vocabulary, explicit atom and bond encoding, and localization tokens for each atom.
C.2 Layout analysis samples
The layout samples compare SmolDocling and Qwen2.5-VL against DocLayNet ground truth across representative page styles. They illustrate layout-handling differences, but the examples are not a generalizable performance measure.
- Comparison setup: Table 7 compares SmolDocling and Qwen2.5-VL layout outputs with DocLayNet ground truth.The examples were selected to represent different layout styles and features.
- Comparison setup: Element-location results are independent of whether document content and structure are reproduced correctly.
- Representative examples: SmolDocling and Qwen2.5-VL handle multi-column pages, although Qwen2.5-VL shows some recall errors.
- Interpretation: The visualizations are representative examples rather than a generalizable measure of performance on similar inputs.