PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Extract Semantic Structure from Documents Using Multimodal Fully Convolutional Neural Network

Xiao Yang, Ersin Yumer, Paul Asente, Mike Kraley, Daniel Kifer, C. Lee Giles

arXiv:1706.02337v1cs.CVcs.LG

TL;DR

Document semantic structure extraction requires pixel-wise recognition of both visual regions and their semantic text roles, while pixel-level labels are costly and existing layouts can be ambiguous. The paper introduces a multimodal fully convolutional network with synthetic pretraining and unsupervised learning on real documents. It reports improved performance from the multimodal approach and unsupervised tasks, including state-of-the-art results on established benchmarks.

  • Problem

    DSSE must distinguish regions by visual appearance and semantic text content, but pixel-wise ground truth is costly and existing datasets lack size and fine-grained labels.

  • Method

    The paper trains a unified multimodal fully convolutional network using text embedding maps, synthetic pixel-labeled documents, and unsupervised reconstruction and consistency tasks.

  • Results

    The multimodal approach and unsupervised tasks improve performance, with results indicating improved state of the art on established benchmarks.

  • Takeaways & Limitations

    The approach simplifies DSSE by simultaneously identifying appearance-based and semantics-based classes and supports document understanding with visual and textual information.

Abstract

from arXiv · show

We present an end-to-end, multimodal, fully convolutional network for extracting semantic structures from document images. We consider document semantic structure extraction as a pixel-wise segmentation task, and propose a unified model that classifies pixels based not only on their visual appearance, as in the traditional page segmentation task, but also on the content of underlying text. Moreover, we propose an efficient synthetic document generation process that we use to generate pretraining data for our network. Once the network is trained on a large set of synthetic documents, we fine-tune the network on unlabeled real documents using a semi-supervised approach. We systematically study the optimum network architecture and show that both our multimodal approach and the synthetic data pretraining significantly boost the performance.

1. Introduction

Document semantic structure extraction assigns semantic roles to document-image regions, but robust pixel-wise labeling requires both visual appearance and textual content. The paper proposes a unified multimodal network, synthetic pixel-labeled pretraining data, and unsupervised training tasks for real documents.

  • The unified MFCN simultaneously identifies appearance-based and semantics-based classes instead of using separate page-segmentation and logical-structure-analysis steps.
  • DSSE labels document-image pixels with roles such as background, figures, tables, paragraphs, headings, lists, and captions.
  • Textual information resolves ambiguities that visual appearance or text alone cannot, such as distinguishing titles, authors, captions, headings, and lists.
  • The network combines an encoder, segmentation decoder, training-only reconstruction decoder, and bridge merging visual and textual representations.
  • Synthetic documents provide large-scale pixel-wise labels, addressing the limited size and fine-grained annotation coverage of existing datasets.
  • The network supports supervised image-and-text training together with unsupervised auxiliary training for better representation learning.

2. Background

Prior document-structure methods rely on visual cues, heuristic features, or multi-stage parsing, leaving semantic context and efficient grouping unresolved. The paper positions multimodal pixel-wise segmentation as an alternative that improves traditional visual segmentation.

  • Page Segmentation: Bottom-up page segmentation groups detected words into lines and paragraphs, but connected-component identification and grouping are time-consuming.
  • Page Segmentation: Neural page-segmentation methods based strictly on visual cues cannot discover the semantic meaning of underlying text.
  • Logical Structure Analysis: Logical-structure methods use location, font, text, grammar, or handcrafted features to label document components such as headings, paragraphs, and lists.
  • Logical Structure Analysis: Handcrafted-feature methods are limited because those features cannot capture highly semantic context.
  • Unsupervised Learning: Unsupervised and weakly supervised approaches reduce annotation demands, while this work’s consistency loss does not require class labels for each bounding box.
  • Language and Vision: The proposed approach uses textual embeddings directly for segmentation and improves traditional segmentation approaches that use only visual cues.

3. Method

The MFCN performs pixel-wise document segmentation using visual features together with sentence-level text representations. It also adds reconstruction and consistency objectives to improve document representations from labeled and unlabeled data.

  • The MFCN jointly predicts appearance-based and semantics-based classes through a unified pixel-wise segmentation model.It combines visual cues with textual representations rather than separating page segmentation and logical structure analysis.
  • Multimodal Fully Convolutional Network: The architecture contains an encoder, main decoder, auxiliary reconstruction decoder, and bridge for merging visual and textual representations.The auxiliary decoder is used only during training.
  • Multimodal Fully Convolutional Network: Dilated convolution blocks expand receptive fields while preserving low-level cues needed for small semantic regions and contextual distinctions such as lists versus paragraphs.The block uses five dilated convolutions with different dilation values.
  • Text Embedding Map: Text embedding maps assign each sentence vector to its corresponding document pixels before concatenation with visual feature responses.Sentence embeddings are formed by averaging word embeddings; pixels outside sentences receive zero vectors.
  • Unsupervised Tasks: The reconstruction task reconstructs encoder feature maps, while the consistency task encourages similar representations within bounding-box regions.Consistency loss is evaluated in the main decoder alongside supervised segmentation loss and encourages intra-region consistency.

4. Synthetic Document Data

The paper addresses scarce fine-grained pixel annotations with a synthetic document engine that creates large-scale labeled training data from varied document elements and layouts.

  • Existing datasets contain only a few hundred to a few thousand pages and provide limited label granularity for semantic document segmentation.The target includes classes such as section headings, lists, and figure and table captions.
  • The synthetic engine generates large-scale documents with pixel-wise annotations for paragraphs, figures, tables, captions, section headings, and lists.It uses LaTeX source files with randomized single-, double-, or triple-column layouts.
  • Text content is sampled by semantic role, including Wikipedia sentences for paragraphs and contents-block phrases for section headings.List items come from the same Wikipedia page, while captions use associated image captions or web image titles.
  • The engine increases layout complexity by replacing elements in 271 varied documents with generated paragraphs, figures, tables, captions, headings, or lists.
  • 135,000 document images comprise the synthetic dataset.Examples include synthetic documents, raw segmentations, and optionally post-processed results.

5. Implementation Details

Implementation uses a convolutional encoder-decoder with training-time auxiliary reconstruction, standardized image preprocessing, alternating synthetic and real batches, OCR-based text embeddings, and optional PDF post-processing.

  • All convolutional layers use 3 × 3 kernels and stride 1, while encoder pooling and decoder unpooling use 2 × 2 kernels.Batch normalization follows each convolution and precedes nonlinear functions.
  • During semi-supervised training, synthetic batches activate classification and unsupervised losses, whereas real batches activate only unsupervised losses.Inputs undergo per-channel mean subtraction and resizing so the longer side is below 384 pixels.
  • Class weights for per-pixel classification are adjusted according to each class’s total pixel count in the training set.
  • Each word is represented by a 128-dimensional skip-gram embedding trained on the 2016 English Wikipedia dump, with Tesseract providing OCR.Out-of-dictionary embeddings follow Bojanowski et al.
  • Post-processing: Optional PDF post-processing averages class probabilities within candidate element boxes and assigns each box its most likely label.

6. Experiments

Experiments evaluate datasets, architecture choices, multimodal text input, unsupervised objectives, and comparisons with prior methods. The results support unpooling, dilated blocks, textual information, and consistency learning as useful components.

  • Evaluation setup: The evaluation uses ICDAR2015, SectLabel, and DSSE-200, with pixel-wise IoU as the primary metric.DSSE-200 contains both appearance-based and semantics-based labels, while ICDAR2015 and SectLabel emphasize different evaluation settings.
  • Architecture ablation: 61.4% mean IoU is achieved by the plain encoder-decoder baseline on DSSE-200.This baseline uses convolutional encoding, fully convolutional decoding, and bilinear interpolation.
  • Architecture ablation: 65.4% mean IoU follows from adding skip connections, a 4% improvement over the base model, with larger gains for captions.Skip connections particularly improve performance on small objects.
  • Architecture ablation: 71.2% mean IoU results when bilinear upsampling is replaced with unpooling, compared with 65.4% for the skip-connection model.The authors attribute the improvement to retaining pooled location indexes during decoding.
  • Architecture ablation: Model5 with a dilated block outperforms Models3 and 4 across all classes, whereas Model4 is on par with or worse than Model3.The comparison keeps parameter counts similar by adjusting output channels.
  • Textual information: Text embeddings improve textual-class accuracy by 1.1% for section headings, 0.1% for captions, 1.7% for lists, and 2.2% for paragraphs.Using real text on synthetic documents improves mean IoU by 6.4%, while OCR text improves it by 2.6%; the DSSE-200 improvement is 0.3%.
  • Unsupervised objectives: The consistency task improves mean IoU by 1.9%, compared with 0.6% for reconstruction.Both tasks are added through semi-supervised fine-tuning on unlabeled real documents.
  • Comparisons with prior art: On ICDAR2015, binary MFCN obtains 94.5%, 91.0%, and 77.1% IoU for non-text, text, and figure regions, respectively.On SectLabel, it improves F1 for section heading, caption, and list over the cited prior method: 0.919 vs 0.916, 0.893 vs 0.781, and 0.793 vs 0.712.

7. Conclusion

The paper concludes that its multimodal MFCN, synthetic data, and unsupervised auxiliary tasks improve document semantic structure extraction. It reports improved state-of-the-art results and releases large-scale synthetic and benchmark datasets.

  • Conclusion: The MFCN combines visual and textual information for document semantic structure extraction.The model predicts document structure using both modalities.
  • Conclusion: Synthetic document generation provides per-pixel ground truth for a 135,000-page dataset.The dataset is publicly provided alongside DSSE-200.
  • Conclusion: Unsupervised auxiliary tasks improve performance by using unlabeled real documents for representation learning.The conclusion specifically identifies these tasks as performance-improving components.
  • Conclusion: The reported results improve the state of the art on previously established benchmarks.The paper also introduces DSSE-200 as a new benchmark dataset.

A. Synthetic Document Data

The synthetic document pipeline generates labeled PDFs by arranging document elements in LaTeX layouts and by replacing elements in varied real-document layouts.

  • Generation methods: LaTeX source files randomly arrange paragraphs, figures, tables, captions, section headings, and lists with the textblock environment.Compiling the files produces single-, double-, or triple-column PDFs.
  • Generation methods: Algorithm 1 selects a document type, repeatedly chooses elements and examples, appends LaTeX code, and compiles the result into a PDF.The loop continues while space remains on the page.
  • Element selection: Candidate content includes MS COCO, academic-style and web-searched figures; web-searched tables; Wikipedia sentences, headings, and list items.List items are sampled from the same Wikipedia page for each list.
  • Element selection: Captions use associated MS COCO captions or image-search titles.The image-search title is taken from the span with class name “irc pt”.
  • Generation methods: A second method starts from 271 varied, complicated layouts and randomly replaces each element with generated document components.Figure 8 shows examples from these layouts.

B. Visualizing the Segmentation Results

The model visualizes each output pixel by assigning it the color of its most likely class, weighted by that class probability.

  • Segmentation visualization: Each output pixel receives the color associated with its most likely class label.The color is then weighted by the probability assigned to that label.

C. Post-processing

The optional PDF post-processing step refines segmentation masks by aggregating class probabilities within candidate bounding boxes and assigning each box its most likely label. Candidate boxes are organized hierarchically so parent boxes are processed before child boxes, while existing labels are preserved.

  • C. Post-processing: Candidate bounding boxes are obtained from Adobe Acrobat auto-tagging results and organized as TextRun, TextLine, Paragraph, or Container nodes.The procedure ignores semantic meanings associated with the boxes.
  • C. Post-processing: The algorithm initializes every output location as background before iterating through candidate boxes.Each box then contributes its selected class label to eligible locations.
  • C. Post-processing: For each candidate box, the method averages pixel-wise class probabilities and selects the class with the highest average probability.The selected label is assigned to pixels inside the box during mask refinement.
  • C. Post-processing: Synthetic-document generation uses natural images, academic-style figures, symbols, graphic drawings, and tables as visual content.These examples are identified as sources used in generating synthetic documents.
  • C. Post-processing: Complicated-layout documents are labeled by regions that are randomly replaced with paragraphs, figures, tables, captions, section headings, or lists.The replacements follow the procedure described in the referenced appendix section.

D. Additional Visualization Results

Additional visualizations show synthetic and real documents alongside their segmentation outputs. The examples include candidate bounding boxes, raw outputs, post-processed masks, and multiple semantic region labels.

  • D. Additional Visualization Results: Figures 9 and 10 visualize synthetic documents together with candidate boxes, raw segmentation outputs, and post-processed segmentations.The displayed labels include figures, tables, section headings, captions, lists, and paragraphs.
  • D. Additional Visualization Results: Figure 11 provides additional examples of real documents and their corresponding segmentations.The supplied caption identifies figure and table labels among the segmentation colors.
Loading 1706.02337v1…