VISTA-Bench: Benchmarking Multilingual Image Translation with Image-Specific Rubrics

1 INTRODUCTION

• Existing benchmarks vary in language and scenario coverage, while aggregate similarity scores and general judgments may not reveal whether image-specific requirements are met.Visual-dependence annotations and localization metrics provide complementary information but do not explicitly specify meanings, relations, and acceptable interpretations to preserve.Source: §1 · p. 1
• VISTA-Bench contains 2,228 images spanning 22 languages, 10 major domains, and more than 100 fine-grained scenarios.Construction jointly considers source-language and scenario coverage through sampling across language–scenario groups.Source: §1 · p. 1
• Model-assisted annotation with human verification establishes source text, semantic blocks, and multilingual reference translations.Annotators merge fragments that jointly express a semantic unit while keeping unrelated content separate.Source: §1 · p. 2
• Image-specific rubrics specify required meanings, semantic relations, acceptable expressions, and disallowed errors, yielding separate V, T, and K scores.Each image receives criteria for visual understanding, translation quality, and applicable knowledge-dependent interpretation; target-language-specific requirements are added when needed.Source: §1 · p. 2
• The evaluation covers 16 mainstream models, including 12 multimodal models and four text-input models, across 22 languages and 10 domains.The paper also analyzes translation quality, visual understanding, knowledge use, language and domain variation, dataset-size sensitivity, and judge–human agreement.Source: §1 · p. 2

2.1 TEXT-IMAGE MACHINE TRANSLATION

• Text-image machine translation translates text embedded in images, with prior approaches addressing vision–text alignment and contextual interpretation of fragmented text.PEIT bridges the vision–text modality gap in end-to-end translation, while AnyTrans incorporates textual and visual context.Source: §2.1 · p. 2

2.2 MULTILINGUAL IMAGE TRANSLATION BENCHMARKS

• Existing multilingual benchmarks vary in source–target language coverage and scenario emphasis, while evaluation uses automatic metrics, multimodal judging, or case-specific rubrics.VISTA-Bench combines multilingual, multi-scenario coverage with rubrics specifying required meanings, semantic relations, and acceptable translation variants.Source: §2.2 · p. 2

3 BENCHMARK CONSTRUCTION AND EVALUATION

• VISTA-Bench comprises 2,228 images spanning 22 languages, 10 major domains, and 100+ fine-grained scenarios; its annotation inventory distinguishes two Chinese scripts, giving 23.Source: §3 · p. 2

3.1 LANGUAGE AND SCENARIO COVERAGE

• Sampling uses nonempty source-language–scenario groups, then adds images across scenarios within each language until the target sample size is reached.All selected images undergo human review.Source: §3.1 · pp. 3–4

3.2 HUMAN SEMANTIC-BLOCK MERGING

• 42.7% is the mean per-language rate of text or structural revision across ten selected languages.The rate counts images with at least one revision once; structural revisions exclude coordinate-only edits to matched blocks.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 REFERENCE AND RUBRIC ANNOTATION

• Image-specific rubrics assess visual understanding, translation quality, and knowledge-dependent interpretation where applicable, with shared and target-language-specific criteria.Criteria carry expected, accepted, and forbidden content, and importance labels determine their aggregation weights.Source: §3.3 · pp. 4–5

3.4 EVALUATION PROTOCOL

• Scores use met-item proportions unless a forbidden error zeros the criterion, then aggregate criteria into separate importance-weighted 0–100 V, T, and K scores.T is the primary translation measure; V and K are complementary diagnostics, and dimensions with no applicable criteria are marked not applicable.Source: §3.4 · p. 5

4.1 MODELS

• The 16-model evaluation includes 12 image-input multimodal models and four text-input baselines; text-input runs use human-annotated layouts rather than original images.The reported text-input configurations are four named baselines; the text-input results do not claim other configurations.Source: §4.1 · pp. 5–6

4.2 EVALUATION SCORES

• T is the primary 0–100 score, while V and K are separate complementary diagnostics; image-input global scores average directions equally, and K includes only applicable criteria.The table’s Overall column denotes global T, not a composite of T, V, and K.Source: §4.2 · p. 6

4.3 DOMAIN-LEVEL PERFORMANCE

• 77.63 global T puts GPT-6 Astra first among image-input models, leading eight of ten domains; Gemini 3.8 Flash leads Healthcare and Doubao Seed 2.1 Pro leads Finance.With human-organized source text, HY-MT2 30B-A3B scores 72.82 globally versus GPT-6 Astra’s 77.63, but exceeds image-input leaders in Products (78.02 vs. 76.11) and Healthcare (81.67 vs. 79.52), while trailing in Finance (64.56 vs. 80.20).Source: Table 2 · p. 6, §4.3 · pp. 6–7

4.4 LANGUAGE-LEVEL PERFORMANCE

• Burmese is the lowest-scoring source language for all six profiled models, while GPT-6 Astra scores highest among them on 19 of 22 languages.Dutch is strongest for Gemini 3.8 Flash, Claude Sonnet 5, Qwen3.8 27B, and GLM-5.3 Flash; Czech is strongest for GPT-6 Astra and DeepSeek Flash. The three China-developed models exceed their own averages on Chinese, while GPT-6 Astra, Gemini 3.8 Flash, and Claude Sonnet 5 fall 2.20–3.95 points below their averages.Source: §4.4 · pp. 7–8

4.5 DIMENSION-LEVEL ANALYSIS

• T co-varies strongly with V and K across the 12 models, with Pearson correlations of 0.9876 and 0.9874, respectively.Claude Sonnet 5 and Qwen3.8 27B have similar V scores (67.58 vs. 68.02), but Claude scores higher in K (84.38 vs. 78.24) and T (69.43 vs. 61.11).Source: Figure 5 · p. 8, §4.5 · p. 8

4.6 DATASET SIZE ABLATION

• The 2,228-image selected set retains the ranking of all four models, with T scores 0.5–1.5 points lower and an MAE of 0.88 than on the full pool.It retains about one fifth of the 10,920-image pool, spanning source-language and fine-grained scenario groups while reducing translation and judging requests.Source: §4.6 · pp. 8–9, Table 3 · p. 8

4.7 JUDGE ROBUSTNESS AND HUMAN AGREEMENT

• Both automatic judges preserve the human ordering of all four models (Spearman ρ = 1.0000), but all eight automatic scores exceed their human references.Qwen3.6 and Qwen3.8 Max have Pearson correlations of 0.9992 and 0.9976 with Human, with MAEs of 1.75 and 1.82 points. Switching judges changes scores by 1.25 points on average and at most 2.34 points without changing rankings.Source: §4.7 · p. 9, Figure 6 · p. 9

5 CONCLUSION

• The 2,228-image benchmark spans 22 languages and 10 domains, with coverage-oriented sampling, human-verified semantic blocks, and multilingual references.Its rubrics specify required meanings, semantic relations, and acceptable variants, and score translation quality, visual understanding, and knowledge use separately.Source: §5 · p. 9
• Evaluations cover 12 multimodal models and four text-input baselines, revealing language and domain disparities and generally aligned translation, visual-understanding, and knowledge-use scores.The compact subset preserves the tested models’ ranking while reducing evaluation workload; rubric-guided judges closely match human model-level assessments.Source: §5 · p. 9
• Generative models assisted source-text recognition, semantic grouping, reference translations, rubric drafting, and evaluation; human annotators verified and revised annotation drafts.The authors reviewed adopted AI-assisted content and take responsibility for the final text, claims, and artifacts.Source: AI USE STATEMENT · p. 10
• Image files will be shared only where redistribution is permitted; otherwise, the authors will provide source references and annotations they are permitted to release.Data sharing follows source licenses and permissions.Source: REPRODUCIBILITY STATEMENT · p. 10

A ANNOTATION TEAM AND QUALITY CONTROL

• Nine team members, including two responsible for quality control, conducted annotation and verification; the team included annotators with linguistics degrees.Their work covered source text, semantic blocks, multilingual references, and image-specific rubrics.Source: A ANNOTATION TEAM AND QUALITY CONTROL · p. 13
• For some languages, fluent target-language reviewers were unavailable, so reviewers familiar with related languages used dictionaries and AI-assisted recognition.The authors state this assisted procedure is not equivalent to independent fluent-specialist review and does not establish uniform annotation quality across languages.Source: A ANNOTATION TEAM AND QUALITY CONTROL · p. 13
• Two verification rounds combined annotator crosschecks and quality-control review of accuracy, completeness, grouping, reading order, translation fidelity, and rubric consistency.Potential omissions or inconsistencies were revisited against source material and corrected.Source: A ANNOTATION TEAM AND QUALITY CONTROL · p. 13

A.1 SOURCE-LANGUAGE DISTRIBUTION

• Figure 7 depicts source-image counts across 22 languages, with sector areas proportional to counts and simplified and traditional Chinese combined.Each of the 2,228 images is counted once, and the legend gives absolute counts.Source: Figure 7 · p. 13, §A.1 · p. 13

B.1 EXAMPLE AND INTERPRETATION

• The English medicine-package example has six source blocks and demonstrates grouping, translation, and knowledge-dependent interpretation in an English-to-French task.The example concerns packaging text, not medical advice; the source image retains all package text after cropping blank surroundings.Source: §B.1 · p. 14, Table B1 · p. 14

B.2 COMPLETE CRITERIA AND TOLERANCES

• The French example’s shared rubric separates translation quality from knowledge use, accepting natural wording and variants while requiring the location to match Wang Noi’s verified identity.No target-language-specific criteria are included in this record.Source: §B.2 · pp. 15–16, Table B2 · p. 15, Table B3 · p. 17

B.3 RUBRIC DRAFTING TEMPLATE

• The drafting template takes an image, source blocks, target languages, verified references, and annotations as inputs, then specifies shared criteria and tolerances.Criteria describe required meanings rather than strings every target language must copy; language-specific criteria are added only when shared criteria are insufficient.Source: §B.3 · p. 18

B.4 RUBRIC-BASED JUDGE TEMPLATE

• The judge template checks expected and forbidden items against supplied criteria, evaluates semantic equivalence, and returns item-level judgments without scores.Deterministic code applies forbidden-item vetoes, computes expected-item fractions, and aggregates dimensions independently; the templates are not verified verbatim prompts for reported runs.Source: §B.4 · p. 19

C SUPPLEMENTARY EXPERIMENTS

• Supplementary results extend the domain, language, and dimension analyses reported in Sections 4.3–4.5.Source: C SUPPLEMENTARY EXPERIMENTS · p. 20

• Domain-level T averages available script-level scores within each natural-language pair, then equally averages pair scores over available source languages.Same-language directions are excluded, and available source-language sets can differ across targets and domains.Source: §C.1 · p. 20
• GPT-6 Astra leads in 116 of 220 target-language–domain combinations, but Products-to-English favors Gemini 3.8 Flash (76.21 vs 74.65), while Chinese favors GPT-6 Astra (77.82 vs 73.37).Source: Table C1 · p. 20, Table C22 · p. 27, §C.1 · p. 27
• For Japanese, Qwen3.8 Max leads Products at 80.05 and Finance at 84.17, while GPT-6 Astra leads Healthcare at 83.76.Source: Table C20 · p. 26, §C.1 · p. 27

• Each image-input model has a complete source-to-target T matrix.Source: §C.2 · p. 28
• Matrix scores use a 0–100 scale, with source languages in rows and target languages in columns.Source: §C.2 · p. 28
• Same-language diagonal cells are omitted from the matrices.Source: §C.2 · p. 28

• Across models within each source language, Pearson correlations range from 0.9277 to 0.9917 for T–V and 0.8635 to 0.9932 for T–K.Source: §C.3 · p. 36
• Within GPT-6 Astra, source-language-mean correlations are 0.6820 for T–V and 0.7274 for T–K, below the cross-model correlations.Source: §C.3 · p. 36
• Burmese has the lowest T for 11 of 12 models, the lowest K for 11, and the lowest V for 10; GPT-6 Astra scores 51.22 in T, 51.29 in K, and 66.22 in V.These scores identify shared difficulty but do not isolate error sources or exclude annotation and judge limitations.Source: §C.3 · p. 36