ImgMT: Wikipedia-based Dataset For Evaluating Text-in-Image Translation

1 Introduction

• Existing resources cover at most 14–16 languages, while localized Wikipedia illustrations encode text separately from graphics and can supply aligned multilingual examples.The shared visuals with translated labels make SVGs a source for text-in-image translation data.Source: §1 · p. 1
• 35,612 aligned image pairs across 129 languages, more than 20 writing systems, and 4,886 language pairs make IMGMT a multilingual text-in-image translation benchmark.The pairs are grouped into 1,855 sets of visually distinct SVG images.Source: §1 · pp. 1–2
• IMGMT targets diagrams, maps, and infographics, where short, information-dense labels and specialized terminology make translation—not distorted or occluded text—the central challenge.Its labels often include named entities from scientific and educational domains.Source: §1 · p. 2
• The evaluation framework measures source-side text-region detection and OCR accuracy, then scores target-side translations with standard machine-translation metrics.The paper focuses on TIT; evaluating IIMT is left for future work.Source: §1 · p. 2
• The authors benchmark several vision-language models and a dedicated TIT system from English into many target languages, and one system into English to study source-writing-system effects.The framework supports both Text Image Translation, which outputs translated text, and In-Image Machine Translation, which renders it back into the image.Source: §1 · pp. 1–2
• The paper also tests whether images improve multimodal translation by comparing full-image input with layout-preserving text-only images and plain text.The comparison is designed to distinguish visual grounding from cues such as layout and line segmentation.Source: §1 · p. 2

2.1 Datasets for text-in-image translation

• Many existing datasets cover few languages—the broadest report 14–16—and often lack target-language images, limiting reverse-direction evaluation.Resources include synthetic text-on-background corpora, authentic scene-text datasets, e-commerce imagery, and document-page translation.Source: §2.1 · pp. 2–3
• IMGMT differs by targeting vector-graphic diagrams, maps, and infographics, preserving layouts across localized SVG variants, and spanning 129 languages and over 20 writing systems.Its 4,886 language pairs support multilingual and multi-script text-in-image translation beyond prior benchmark coverage.Source: §2.1 · p. 3

2.2 Effect of Image in Multimodal Translation

• Prior multimodal translation systems often showed marginal gains over text-only baselines, and adversarial evaluations found many insensitive to unrelated or shuffled images.Other work found visual grounding more useful when text is degraded or ambiguous, and questioned whether some reported benefits reflected evaluation protocols.Source: §2.2 · p. 3
• For diagrams, maps, and infographics, visual structure may help locate text or link labels to graphics, but may add little once extracted text is self-contained.The paper expects visual context to matter mainly for ambiguous labels, abbreviations, or domain-specific terms whose meaning depends on the graphic.Source: §2.2 · p. 4

3.1 Basic Statistics

• 35,612 aligned image pairs across 1,855 image groups cover 129 languages, more than 20 writing systems, and 4,886 language pairs.The test split contains 3,575 pairs, including 548 (15%) involving English.Source: §3.1 · p. 4
• Strings average 9.29 characters; about 60% are proper names, while English, Spanish, and French are common and some languages occur in only four pairs.LLMs agreed on labels for 210,644 strings (89.4%); well-formed sentences account for about 0.5%.Source: §3.1 · p. 4

3.2 Processing Pipeline

• The pipeline crawls Wikimedia Commons variants, filters and groups visually similar SVGs, then aligns text using language tags, structural identifiers, or spatial overlap.Spatial overlap is used most often because internal identifiers are frequently mismatched, missing, or incomplete.Source: §3.2 · pp. 5–6
• An 80:10:10 split groups visually related images together to prevent cross-split leakage, without explicitly balancing languages or writing systems.Text boxes are normalized across image groups using the smallest font size; alignment methods are balanced across splits.Source: §3.2 · p. 6

3.3 Manual Evaluation of the Dataset Collection Method

• In a 100-pair manual evaluation, 13% of pairs were too different for TIT, 16% had minor graphical differences, and the remainder were visually identical.BBA precision remained above 91%, while recall was around 83%, with most of the gap attributed to text-field detection; this should inform interpretation of absolute results.Source: §3.3 · pp. 6–7

Text detection and OCR evaluation

• Source OCR is evaluated by comparing predicted text with gold text using Character Error Rate (CER).When boxes are available, BBA measures area overlap and BBC measures block-level retrieval; overlapping predicted boxes are merged before scoring.Source: §4 · p. 7

Translation evaluation

• Translated text is scored with chrF and the wmt22-comet-da model of COMET.Source: §4 · p. 7

Text-based alignment

• Without predicted boxes, the Hungarian algorithm matches text blocks using text dissimilarity and a spatial reading-order penalty, rejecting pairs above task-specific thresholds.CER supplies OCR dissimilarity; translation alignment uses one minus chrF or COMET quality.Source: §4 · p. 7

5 Experimental Setup

• English-source TIT experiments compare several multimodal large language models and AnyTrans using a shared zero-shot prompt for text detection, transcription, boxes, and translation.AnyTrans uses PaddleOCR and DeepSeek-V4-Flash-Thinking; its image-rendering stage is not used in these experiments.Source: §5 · p. 8
• Only images with valid structured model outputs are scored, so reported metrics cover a subset of English-source test images rather than the full test set.Source: §5 · p. 8

Effect of source script on TIT performance

• GPT-5.6 Luna is evaluated with English as the target language to compare translation quality across source writing systems; MMT conditions vary whether full images, text-only images, or no image are supplied.The MMT comparison holds extracted source text constant and excludes AnyTrans.Source: Effect of source script on TIT… · p. 8

TIT from English

• Qwen3.5's worse OCR and localization coincide with nearly unchanged translation quality, whereas Claude's translation performance is similar to the other models on gold SVG text.Claude's large gap across source- and target-side metrics indicates detection and recognition errors can become a major bottleneck when severe.Source: §6 · pp. 8–9
• Kimi K3 leads English-source TIT with chrF (61.3) and COMET (0.834), while GPT-5.6 Luna scores slightly higher on BBA and BBC localization.Claude Haiku 4.5 lags across metrics, with CER (0.822).Source: §6 · p. 8, Table 4 · p. 9

Effect of source script

• Latin improves over the overall mean across metrics, while Cyrillic gains in translation quality despite slightly worse OCR; Georgian and Chinese show similar patterns with smaller samples.Table 5 reports absolute scores and per-script deltas relative to the overall mean.Source: Table 5 · p. 9, §6 · p. 9

TIT from English

• AnyTrans attains the lowest CER (0.161) with competitive localization, but translation quality trails Kimi K3, Qwen3.5, and GPT-5.6 Luna.Its translation quality remains well above Claude Haiku 4.5.Source: §6 · p. 9

Effect of source script

• Dravidian and Bengali-Assamese degrade across most metrics, while Arabic has better CER but worse localization and translation; script-level metrics are only weakly coupled.Results for very small script groups warrant caution.Source: §6 · p. 9

Effects of visual modality on translation quality

• Across four models, the Image condition is never best and yields little overall change; No image or Text-only image is strongest depending on the model.The strongest condition is No image for Claude Haiku 4.5 and Qwen3.5, and Text-only image for GPT-5.6 Luna and Kimi K3.Source: Table 6 · p. 9, §6 · p. 9

7 Conclusion

• IMGMT is a dataset for evaluating text-in-image translation, and its multi-level protocol combines complementary metrics at different granularities.The experiments cover several MLLMs and one dedicated system in English→X and X→English settings.Source: §7 · p. 10
• The experiments show that source language and writing system matter for TIT quality, while adding the source image yields little benefit and can degrade performance in the tested setup.The authors conclude that IMGMT is not well suited to isolate visual-information contributions in MMT.Source: §7 · p. 10
• The paper leaves benchmarking In-image Machine Translation, which renders the translation into the image, for future work.The experiments focused mainly on TIT.Source: §7 · p. 10

Data and evaluation

• SVG text represented as vector paths is omitted, and elements without explicit IDs or queryable bounding boxes are excluded, potentially biasing evaluation such as by overestimating recall.The dataset also does not capture all text present in source images.Source: Limitations · p. 10
• Language-pair coverage is imbalanced, with high-resource languages dominating and some low-resource or regional languages appearing in fewer than 10 pairs.The dataset is built from publicly available Wikimedia data, so some tested MLLMs may have encountered benchmark material during pretraining.Source: Limitations · p. 10

Experiments

• Evaluation is limited to zero-shot English→X and X→English directions on the test split, leaving roughly 30,000 training and development pairs unevaluated.The paper also does not benchmark the IIMT setup.Source: Limitations · p. 10

A Dataset Structure

• Figure 4 is identified as the overall dataset directory layout.The passage supplies the figure title but no further description of the layout.Source: Figure 4 · p. 13
• Figure 5 is identified as the JSON structure.The passage supplies the figure title but no further description of the JSON structure.Source: Figure 5 · p. 13

• Table 7 lists each language’s writing system, translation-pair total, number of unique paired target languages, and top paired counterparts.Source: Table 7 · p. 16

• Figure 6 is a screenshot of the manual annotation tool.Source: Figure 6 · p. 17