Source-linked AI summary
OmniDocLayout: Towards Diverse Document Layout Generation via Coarse-to-Fine LLM Learning
Hengrui Kang, Zhuangcheng Gu, Zhiyuan Zhao, Zichen Wen, Bin Wang, Weijia Li, Conghui He
TL;DR
Document layout generation remains underexplored, and diverse, complex pages challenge unified layout learning. OmniDocLayout introduces a million-scale dataset and a 0.5B model trained coarse-to-fine. The authors report that it surpasses layout-generation experts and general-purpose LLMs.
Problem
Document layout generation is underexplored, and pages contain many elements with heterogeneous structures that make unified representation learning difficult.
Source: §1 · p. 1Method
OmniDocLayout combines a million-scale dataset with a 0.5B model that learns coarse layout principles from diverse data before fine-grained domain adaptation.
Source: §1 · p. 2, §4.3 · p. 5Results
The authors report that their approach significantly surpasses existing layout-generation experts and powerful general-purpose LLMs.
Source: §6 · p. 8Takeaways & Limitations
The paper concludes that diverse document layouts and staged adaptation offer an approach to complex document-layout generation beyond prior domain limitations.
Source: §4.3 · p. 5, §6 · p. 8
Abstract
from arXiv · showhide
Document AI has advanced rapidly and is attracting increasing attention. Yet, while most efforts have focused on document layout analysis (DLA), its generative counterpart, layout generation, remains underexplored. Distinct from traditional graphic layout design and room layout planning, document layout generation typically involves a larger number of elements per page and exhibits greater structural diversity and complexity. Currently, a major obstacle lies in the scarcity of diverse document layouts: academic papers with Manhattan-style structures dominate existing studies, while open-world genres such as newspapers and magazines remain severely underrepresented. To address this gap, we curate OmniDocLayout-1M, the first million-scale dataset of diverse document layouts, covering six common document types and comprising contemporary layouts collected from multiple sources. Moreover, since existing methods struggle in complex domains and often fail to arrange long sequences coherently, we introduce OmniDocLayout-LLM, a 0.5B model with designed two-stage Coarse-to-Fine learning paradigm:1) learning universal layout principles from our dataset with coarse category definitions, and 2) transferring the knowledge to a specific domain with few fine-grained annotated samples. Extensive experiments demonstrate that our approach achieves strong performance on multiple domains in M$^6$Doc dataset, substantially surpassing both existing layout generation experts and several latest general-purpose LLMs. Our code, dataset, and models will be publicly released.
1. Introduction
Document layout generation remains underexplored despite the complexity and diversity of real-world pages. OmniDocLayout-1M and OmniDocLayout-LLM address the data gap and complex-domain generation with diverse layouts and coarse-to-fine learning.
Motivation
- Document layout generation is less explored than document parsing and must handle more elements per page and greater structural diversity than graphic or room layouts.Heterogeneous document structures make unified representation learning difficult.Source: §1 · p. 1
- Existing public datasets overrepresent academic and outdated layouts, while contemporary complex types such as textbooks remain underrepresented.Datasets with broader contemporary coverage have too few samples for large-scale training.Source: §1 · p. 2
Contributions
- OmniDocLayout-1M is a million-scale dataset covering six common document types, collected through an automated annotation pipeline.The paper presents it as a foundation for training layout-generation models.Source: §1 · p. 2
- OmniDocLayout-LLM uses coarse-to-fine learning and achieves state-of-the-art performance across multiple document domains.It learns layout principles from diverse layouts, then adapts to a specific domain using a small amount of fine-grained annotated data.Source: §1 · p. 2
2. Related Work
Prior document-layout datasets and generation methods span parsing-derived datasets, GANs, transformers, diffusion and flow models, and LLM-based approaches.
Document Layout Dataset
- Layout datasets largely originate from parsing tasks and focus on academic articles; DocLayNet and D4LA cover six and twelve types but contain only on the order of 10^4 pages, most rarely seen in contemporary real-world documents.M6Doc and OmniDocBench each have fewer than 10K samples.Source: §2 · p. 2
Layout Generation
- Earlier generation methods include GANs and transformers, followed by diffusion and flow-based approaches for layout generation.Examples include LayoutGAN++, LayoutTransformer, BLT, LayoutFormer++, LayoutDM, LACE, and LayoutFlow.Source: §2 · pp. 2–3
Large Language Models
- LLM-based methods use HTML completion, chain-of-thought prompting, retrieval, or string-based representations for layout generation.Several approaches remain domain-specific and rely on prompt engineering or retrieval heuristics; LayoutNUWA and LGGPT pursue domain-agnostic approaches.Source: §2 · p. 3
3. OmniDocLayout-1M Dataset
OmniDocLayout-1M responds to limitations in dataset diversity, scale, source timeliness, and annotation efficiency. Its construction and statistical analysis describe a large, balanced, diverse dataset with annotations aligned to reading order.
3.1. Motivation
- Existing document-layout datasets are limited in diversity, volume, source timeliness, or annotation efficiency.Academic datasets tend toward simple Manhattan layouts; diverse datasets have only hundreds to thousands of samples per type, while some sources are outdated or manually annotated.Source: §3.1 · p. 3
3.2. Dataset Construction
- OmniDocLayout-1M draws from 36 public and copyright-clean sources and uses MinerU for automated annotation aligned with natural reading order.Newspaper layouts receive a separate treatment: 1,000 are manually annotated and used to fine-tune DocLayout-YOLO; in a 1,200-page blind study, annotators report at least 92% perceived-quality similarity between model and human annotations.Source: §3.2 · p. 3
3.3. Dataset Statistical Analysis
- OmniDocLayout-1M contains about 1M samples and 48M elements across six balanced document types, with greater feature diversity than PubLayNet.Its element co-occurrence patterns include frequent text and title across types, and prominent formulas in academic content such as textbooks, papers, and educational slides.Source: §3.3 · p. 4, Figure 2 · p. 4
4. Methodology
The method represents layouts as sequences and supports multiple conditioning tasks. Its coarse-to-fine training first learns broad structural priors, then adapts them to domain-specific fine-grained labels.
4.1. Problem Formulation
- A layout is a set of element 5-tuples containing category, box coordinates, width, and height; training proceeds in two stages from diverse coarse-labeled data to a specific fine-labeled domain.The authors report that this staged paradigm lets Stage 2 use typically a few hundred fine-grained samples for effective adaptation.Source: §4.1 · p. 4
4.2. Layout Generation Modeling
- Layout generation is conditional sequence modeling over tokens, with five regimes: unconditional generation, category-conditioned sizing and positioning, position inference, completion, and refinement.The prompt combines document metadata, task-specific conditions, and an instruction; element categories and normalized box coordinates use string-based encoding.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
4.3. Coarse-to-Fine Learning
- Coarse-grained pretraining learns spatial relationships across document types using shared labels; fine-grained adaptation maps those labels to target-domain categories.For example, the coarse class “text” can map to “paragraph,” “lead,” or “ordered list.”Source: §4.3 · pp. 5–6
5. Experiments
Experiments compare OmniDocLayout-LLM with layout-generation experts and general-purpose LLMs, then test model size and the Coarse-to-Fine stages. The model outperforms the compared experts across metrics, while ablations find benefits from both learning stages.
5.1. Experimental Setup
- Fine-grained adaptation uses five M6Doc document types and the original 6:1:3 training, validation, and test split, with evaluation covering similarity and aesthetic consistency.The four metrics are FID, mIoU, Alignment, and Overlap; the base model is Qwen2.5-0.5B-Instruct.Source: §5.1 · p. 6
5.2. Comparison with Layout Experts
- OmniDocLayout-LLM consistently surpasses the compared layout-generation baselines across all metrics, with especially notable gains on mIoU; diffusion models have unsatisfactory FID and LACE still struggles with overlap.The authors attribute diffusion models’ difficulty in low-resource, complex domains to their need for substantially more data and longer convergence.Source: §5.2 · pp. 7–8
5.3. Comparison with General-purpose LLMs
- Among general-purpose LLMs, Claude-3.7-Sonnet performs best; few-shot results improve with more shots but approach an upper bound as input length, API cost, and inference time increase.The zero-shot results show high stochasticity across models; Newspaper is the most challenging type by average FID, while Academic performance is better overall.Source: §5.3 · p. 8
5.4. Ablation Study
- In newspaper ablations, model-size differences are marginal: the 3B model has slightly lower Alignment scores but higher FID in most tasks.The authors suggest that larger models may be insufficiently optimized or overfit, so layout generation does not strictly follow the conventional scaling law.Source: §5.4 · p. 8
- Coarse-only learning improves global organization and alignment over fine-only learning, while the full Coarse + Fine strategy outperforms both single-stage variants.The authors attribute multiple zero mIoU scores in U-Cond and Completion to the metric’s exact label-match requirement, which is often unmet in complex layouts.Source: §5.4 · p. 8
6. Conclusion
The paper presents a million-scale, six-type dataset and a Coarse-to-Fine approach for complex document layout generation. It reports that the approach surpasses existing experts and general-purpose LLMs, while identifying metric inadequacy on complex layouts with limited samples.
6. Conclusion
- OmniDocLayout-1M covers six common document types, and the Coarse-to-Fine approach significantly surpasses existing layout-generation experts and powerful general-purpose LLMs; current metrics remain inadequate for complex layouts with limited samples.The approach learns fundamental layout rules from diverse document types before fine-grained adaptation to a specific complex domain.Source: §6 · p. 8
Supplementary Material
The supplementary material compares layout-generation scenarios and document complexity, and includes dataset curation details and further model analyses.
Supplementary Material
- The appendices cover scenario scope and layout complexity, OmniDocLayout-1M curation, and additional qualitative and quantitative analysis of OmniDocLayout-LLM.Source: Supplementary Material · p. 12
A.1. Comparison Across Multiple Scenario Scopes
- Unlike room layouts [29] and graphic layouts [15], document pages arrange numerous heterogeneous elements under document-type styles, with richer categories, deeper hierarchies, and stricter layout rules.The rules include alignment, reading order, spacing, and style; Figure 5 contrasts examples from all three scenarios.Source: §A.1 · p. 12, Figure 5 · p. 12
A.2. Comparison of Layout Complexity Across Various Documents
- Newspaper pages in M6Doc reach 178 elements maximum and average 73.7 per page, versus PubLayNet’s maximum 59 and average 10.7.Figure 6 compares maximum and average element counts and marks the 25-element limit used by prior methods.Source: Figure 6 · p. 12, §A.2 · p. 12
- M6Doc labels are more fine-grained than PubLayNet’s five element types: Textbook has 42 classes and Academic has 25.The authors describe this evaluation setting as more complex and diverse than prior benchmarks.Source: §A.2 · p. 12
- LayoutNUWA [36] and LGGPT [51] also target diverse, complex document styles, but cover a substantially narrower range of document types.Source: §A.2 · pp. 12–13
B. OmniDocLayout-1M Dataset
The dataset curation pipeline removes duplicate layouts and filters samples using document-specific quality criteria.
B.1. Curation Details
- The curation process applies image-level pHash deduplication followed by layout-level deduplication, retaining one sample at random when pairwise mIoU exceeds 0.95.Images are resized to 32×32 grayscale and compared using normalized Hamming distance with a 0.05 threshold.Source: §B.1 · p. 13
- Filtering removes pages below document-specific element-count thresholds, newspaper pages with fill ratio below 60%, and samples with abnormally high overlap.The minimum is five elements for textbooks, magazines, exams, and academic papers, and ten for newspapers.Source: §B.1 · p. 13
- Alignment score is excluded as a filtering criterion because non-Manhattan document layouts can score highly, and using it would reduce layout diversity.Source: §B.1 · p. 13
B.2. Final Quality Control
A blind comparison of model-generated and human-annotated layouts found high perceptual agreement across 1,200 pages from six document types.
B.2. Final Quality Control
- The study was designed to assess whether using model-generated boxes as training data introduces annotation noise or bias.Evaluators judged whether either annotation was better or both were good, without knowing which was the human ground truth.Source: §B.2 · p. 13
- The evaluation randomly sampled 200 pages per document type and compared human annotations with unprocessed model-generated boxes in randomized, blinded pairs.Four annotators participated; each page was reviewed by two, with disagreements resolved by majority vote.Source: §B.2 · p. 13
B.3. Element-level Statistical Analysis
The analysis finds greater element-level and page-layout diversity in OmniDocLayout-1M than in established academic-focused datasets, and presents annotated examples across six document types.
B.3. Element-level Statistical Analysis
- OmniDocLayout-1M is significantly more diverse than PubLayNet and DocBank across elements per page, occupied area, and element aspect ratios.Figure 8 presents the element-level statistical comparison.Source: Figure 8 · p. 14, §B.3 · p. 14
B.4. Layout Diversity
- OmniDocLayout-1M layouts vary substantially beyond the two-column Manhattan structures typical of PubLayNet and DocBank.The comparison is shown in Figures 9 and 10.Source: §B.4 · p. 14
B.5. More Visual Examples
- The dataset examples show textbook, newspaper, magazine, exam, academic, and slide layouts with MinerU-extracted annotations.The paper describes these annotations as high quality.Source: §B.5 · p. 14
C. OmniDocLayout-LLM Analysis
The analysis describes constraints on general-purpose LLMs for complex document layouts and shows qualitative OmniDocLayout-LLM outputs across five generation tasks and multiple document types.
C.1. Few-shot Performance of General-purpose LLMs
- General-purpose LLMs often produce overlapping layouts in zero-shot use, while limited spatial reasoning and costly long prompts constrain their use for complex layouts.The paper says scaling to 10 or more in-context examples becomes impractical; complex-layout evaluation also requires more than 10,000 USD in API costs.Source: §C.1 · p. 14
C.2. More Generated Layouts
- Across five M6Doc tasks, generated layouts are described as reasonable, aesthetically pleasing, and responsive to task requirements and constraints across document types.Figures 17–19 show examples for textbooks, newspapers, magazines, exams, and academic papers.Source: §C.2 · p. 14, Figure 17 · p. 24, Figure 18 · p. 25