PaperlayerView exact revision on arXiv

Source-linked AI summary

Synth-JDoc: Synthesizing a Japanese Document Image Dataset for OCR with Diverse Layouts and Embedded Images

Keito Sasagawa, Shuhei Kurita, Daisuke Kawahara

arXiv:2608.28248v1cs.CVcs.CL

TL;DR

LVLMs need better OCR data for Japanese documents, particularly vertically written text, but manual annotation is costly and existing synthesis methods lack complex layouts. Synth-JDoc generates text-derived document images with HTML and CSS, embedded images, and degradation, and it most effectively improves OCR on vertically written Japanese documents. Its main limitations are that the evaluation did not adequately test noise robustness and the pipeline lacks NSFW filtering.

  • Problem

    Existing Japanese OCR datasets and synthesis methods do not adequately support complex vertically written layouts, while manual construction is costly and OCR extraction introduces errors.

  • Method

    Synth-JDoc synthesizes Japanese document images from text-derived elements using HTML and CSS, embedding generated images and applying noise and degradation.

  • Results

    Synth-JDoc most effectively improves OCR capabilities on real-world Japanese document images featuring vertically written text compared with the evaluated baseline settings.

  • Takeaways & Limitations

    Text-based HTML/CSS synthesis provides a dataset approach for training models on Japanese documents with diverse layouts and vertically written text.

  • Takeaways & Limitations

    The pipeline lacks NSFW filtering, and the relatively clean VJRODa test set could not adequately evaluate the effectiveness of noise application.

Abstract

from arXiv · show

The ability of Large Vision Language Models (LVLMs) to read text within document images is crucial, as it enables various applications such as Document Visual Question Answering. To enhance the text-reading capabilities of LVLMs, high-quality OCR datasets are essential. This need is particularly critical for Japanese documents, which often feature vertically written text alongside horizontally written text. Current LVLMs demonstrate considerably lower performance on vertically written Japanese text than on horizontally written text, necessitating specialized OCR datasets to bridge this gap. However, manually constructing OCR datasets is expensive and difficult to scale. Alternatively, constructing datasets by extracting text from existing document images using OCR models introduces challenges, such as text recognition errors and the prerequisite of sourcing document images. To address these issues, we construct an OCR dataset by synthesizing document images directly from text. Leveraging HTML and CSS, we generate multi-column documents that incorporate both vertical and horizontal writing styles. Furthermore, to ensure the visual realism of the documents, we embed images generated by text-to-image models within the layout. Additionally, to foster model robustness, we apply noise and degradation filters to the synthesized document images. In our experiments, we compared the performance of models fine-tuned on our synthetic dataset against baselines fine-tuned on synthetic datasets from prior work and those generated by a high-performance text-to-image model. Evaluation results demonstrate that our synthetic dataset is the most effective approach for improving LVLM performance on reading vertically written Japanese text. Our dataset and code are publicly available (https://github.com/llm-jp/synth-jdoc).

1 Introduction

Synth-JDoc addresses the lack of scalable OCR datasets for complex Japanese documents by synthesizing images with vertical and horizontal text, multi-column layouts, and embedded images. Experiments show that fine-tuning on Synth-JDoc most effectively improves OCR for real-world vertically written Japanese documents.

  • Motivation: LVLM document understanding requires reliable OCR, especially for Japanese documents combining horizontal and vertical writing.Vertical Japanese text is read top-to-bottom, with lines progressing right-to-left; multi-column reading order proceeds from uppermost to lowest columns.
  • Motivation: Manual OCR annotation is costly and difficult to scale, while OCR-based extraction introduces recognition errors and requires collecting document images.
  • Research gap: Previous document-image synthesis methods do not support complex Japanese layouts with vertical text and inserted images.
  • Approach: Synth-JDoc synthesizes documents from prepared text, titles, and selected image-caption pairs using HTML and CSS, with horizontal or vertical multi-column layouts.Noise is applied to a portion of the images to enhance model robustness.
  • Results: Fine-tuning on Synth-JDoc most effectively improves OCR capabilities on real-world Japanese document images featuring vertically written text.The comparison includes original models, prior synthetic datasets, and images generated by a high-performance text-to-image model.

2 Related Work

Prior work shows that text-to-image synthesis can avoid document-image collection and OCR errors, but existing methods lack realistic complex Japanese layouts. Japanese OCR resources therefore remain limited, motivating Synth-JDoc.

  • Synthetic document synthesis: Text-driven document synthesis avoids collecting document images and avoids OCR errors in ground-truth text.
  • Synthetic document synthesis: Genalog, DocCreator, and SynthDoG provide document synthesis or degradation capabilities, but their layouts remain limited for this task.Genalog supports horizontal templates, DocCreator extracts elements from real documents, and SynthDoG produces Japanese layouts described as unnatural.
  • Research gap: Existing methods cannot synthesize documents combining vertically written text, multiple columns, and inserted images.
  • Japanese document resources: VJRODa is a real-world OCR test dataset for vertically written Japanese documents containing 100 image-text pairs.
  • Japanese document resources: SynthDoG-ja has unnatural layouts, while JSSODa supports one- to four-column vertical and horizontal text but lacks inserted images, leaving comprehensive Japanese OCR training data scarce.

3 Synth-JDoc: Synthetic Japanese Document Dataset

Synth-JDoc constructs Japanese document images from text-derived elements with HTML and CSS, supporting diverse writing directions, column layouts, embedded images, and degradation. The final filtered dataset contains 17,970 images.

  • Document synthesis: HTML and CSS control layouts with horizontal or vertical writing, one to four columns, and embedded images confined to a column or spanning all columns.Orientation and column counts follow JSSODa, with the eight combinations uniformly distributed.
  • Element preparation: Synth-JDoc generates document elements from text, including paragraphs, titles, images, and image captions, before constructing document images.Images are generated from paragraph-related prompts, captions are produced by an LVLM, and titles are generated from the full text.
  • Noise application: The pipeline applies scan-simulated noise and Augraphy degradation to improve OCR robustness.Scan simulation includes paper noise, geometric distortion, shadows, vignetting, and blur; Augraphy uses Ink, Paper, and Post phases.
  • Dataset composition: 40% clean, 30% scan-simulated, and 30% Augraphy-processed images form the training set.
  • Dataset composition: 17,970 images remain after manual filtering for synthesis errors and potential copyright concerns.

4 Experiments

The experiments fine-tune five Japanese-capable open-source LVLMs and compare Synth-JDoc with JSSODa and a text-to-image baseline under matched text, configurations, and training volume.

  • Experimental setup: Five open-source LVLMs capable of reading Japanese text are fine-tuned with prompts requesting all text in each image.The models are Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3-8B, InternVL3.5-8B, and Gemma 3 12B IT.
  • Baselines: JSSODa provides a comparable synthetic baseline with one- to four-column horizontal and vertical documents but no inserted images.
  • Baselines: The second baseline uses Nano Banana Pro to generate documents from the same text, writing direction, column count, titles, and figure captions as Synth-JDoc.
  • Experimental setup: Training comparisons keep text distributions, fine-tuning configurations, and training-data volume consistent across datasets.

Evaluation Settings

The evaluation uses VJRODa, a real-world Japanese document OCR dataset, and scores outputs with CER and BLEU under raw and repetition-removed settings.

  • Dataset and decoding: Models are evaluated on VJRODa, which contains 100 real-world document images paired with their corresponding texts.Texts are generated with greedy decoding and prompts identical to those used during fine-tuning.
  • Metrics: Performance is measured using Character Error Rate and BLEU after character-level tokenization and Unicode preprocessing.CER divides edit distance by the number of ground-truth characters and multiplies by 100; BLEU uses SacreBLEU.
  • Output settings: Scores are reported for Raw Output and Remove Repetition configurations to account for repeated-string generation by LVLMs.The latter removes trailing repetitive strings before scoring.

4.2 Results

Synth-JDoc generally produces the strongest OCR results on VJRODa, whereas Nano Banana Pro training degrades performance and Gemma 3 shows no improvement.

  • Results: Under both evaluation settings, Synth-JDoc fine-tuning achieves the best CER and BLEU scores for every model except Gemma 3.The result is reported for real-world documents with complex layouts, including vertically written Japanese text.
  • Results: Nano Banana Pro training degrades all models’ scores relative to their original versions, except Qwen2.5-VL’s BLEU score.The authors attribute this degradation to distorted characters and unsuccessful vertical multi-column layout generation.
  • Results: Table 1 identifies JSSODa, Nano Banana Pro, and Synth-JDoc as alternative fine-tuning datasets, with some scores cited from prior work.The table reports results on VJRODa.
  • Results: Synth-JDoc fine-tuning does not improve Gemma 3 performance.The authors hypothesize that fixed 1:1 image resizing prevents effective learning from Synth-JDoc’s varied resolutions.

4.3 Analysis

A Qwen3-VL case study shows Synth-JDoc fine-tuning prevents omission of text from a vertically structured document and improves CER.

  • Case study: The original Qwen3-VL-8B-Instruct skips text in the second column from the top, while the Synth-JDoc-fine-tuned model does not.The comparison uses a VJRODa document image.
  • Case study: The original output jumps from “A1” in the first section directly to “Q2” in the third, whereas Synth-JDoc preserves the intervening text.This behavior results in an improved CER score for the fine-tuned model.
  • Case study: Models fine-tuned on JSSODa and Nano Banana Pro additionally exhibit repetitive string outputs alongside character recognition errors.The case study contrasts these outputs with the Synth-JDoc-fine-tuned model.

Japanese Document Images Generated by Nano Banana Pro

Nano Banana Pro generates realistic-looking document images from layout prompts, but struggles with distorted text and vertically written multi-column layouts.

  • Visual realism: The generated images have realistic visual details such as wooden-desk backgrounds and natural page folds.Realistic appearance is identified as an advantage of Nano Banana Pro outputs.
  • Generation limitations: Nano Banana Pro frequently produces distorted characters and text that differs from the prompt.It also fails to generate multi-column documents with vertically written text as instructed.
  • Generation limitations: The instructed vertical 2-column layout instead becomes two separate vertically written pages.The authors connect these limitations to the lack of performance improvement from Nano Banana Pro training.
  • Generated layouts: The top example follows a horizontal 3-column prompt, while the bottom example follows a vertical 2-column prompt.These examples illustrate document images generated by Nano Banana Pro.

4.4 Ablation Study

The ablation study evaluates whether embedded images and their captions contribute to Synth-JDoc’s effectiveness. Models trained on the complete dataset outperform models trained with these elements masked, supporting their role in visual diversity and realism.

  • Table 2 compares complete Synth-JDoc with datasets masking embedded images, captions, or both.The masked variants are evaluated for Qwen3-VL-8B and InternVL3.5-VL-8B.
  • For both Qwen3-VL-8B and InternVL3.5-VL-8B, complete Synth-JDoc performs better than datasets with masked images or captions.
  • The authors attribute the advantage to embedded images and captions representing more visually diverse and realistic documents.

5 Limitations

The paper identifies limitations in layout complexity, decoding behavior, test-set noise, and content safety. These constraints define boundaries for interpreting the current dataset and evaluation.

  • 5.1 Layout Diversity: Synth-JDoc supports vertical and horizontal multi-column layouts but not documents with more complex reading orders, such as newspapers.Handling such documents remains future work.
  • 5.2 Decoding Method: Greedy decoding caused repeated output strings that can obscure models’ fundamental OCR capabilities.The authors suggest that decoding-hyperparameter adjustments may suppress this behavior.
  • 5.3 Diverse Test Set: The clean VJRODa test images prevented adequate evaluation of Synth-JDoc’s noise application.Constructing and evaluating on a noisy test dataset remains future work.
  • 5.4 Removing NSFW Content: The construction pipeline lacks NSFW filtering, so unwanted content may enter during text preparation, generation, or captioning.The paper identifies filtering text and images at each stage as future work.

6 Conclusion

The paper concludes that Synth-JDoc uses prepared text, titles, images, captions, HTML/CSS layouts, and noise to create synthetic Japanese document images. Models trained on it improve OCR for vertically written Japanese documents more effectively than models trained on baseline datasets.

  • Synth-JDoc synthesizes Japanese document images from text, titles, images, and captions using HTML and CSS.
  • Noise is applied to generated document images to support construction of more robust models.
  • Models trained on Synth-JDoc improve OCR capabilities on vertically written Japanese document images more effectively than models trained on other baseline datasets.
  • Future work will explore document-image synthesis incorporating complex elements such as graphs and tables.
Loading 2608.28248v1…