PaperlayerView exact revision on arXiv

Source-linked AI summary

VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

Zhi Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Yun-Nung Chen

arXiv:2503.10427v2cs.CL

TL;DR

Traditional Chinese VLM capabilities remain under-tested because existing multimodal benchmarks largely focus on English or Simplified Chinese, despite distinct linguistic and Taiwanese cultural challenges. VisTW introduces exam-style and open-dialogue benchmarks for these contexts. The authors report performance gaps between Traditional- and Simplified-Chinese-trained models and a correlation between structured and free-form performance.

  • Problem

    Comprehensive VLM evaluation in Traditional Chinese that covers both structured reasoning and open-ended generation in Taiwanese cultural contexts is lacking.

    Source: §2.2 · p. 3

  • Method

    The paper constructs VisTW-MCQ from manually curated Taiwanese exam questions and VisTW-Dialogue from manually captured real-life images paired with culturally grounded questions.

    Source: §3 · p. 3, §4 · pp. 4–5

  • Results

    The authors report that Traditional-Chinese-trained VLMs underperform Simplified-Chinese counterparts on Traditional Chinese visual content, alongside a strong correlation between structured and free-form task performance.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The benchmarks and analyses provide evaluation of Traditional Chinese visual content and insights into model scaling and image-resolution sensitivity for practical deployment.

    Source: §7 · p. 11

    Multiple-choice evaluation does not reflect the mostly free-form question-answering format of real-world VLM use.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.

1 Introduction

Vision-language models remain largely untested on Traditional Chinese visual content, where script complexity and Taiwan-specific cultural context pose distinct evaluation challenges. VisTW addresses this gap with complementary exam-style and open-dialogue benchmarks.

1 Introduction

  • Traditional Chinese VLM capabilities remain largely untested because multimodal benchmarks predominantly focus on English or Simplified Chinese.Traditional Chinese is widely used in Taiwan and Hong Kong.Source: §1 · p. 1
  • Traditional characters may not map directly to Simplified Chinese, while Taiwan-specific signs, labels, and documents add contextual challenges.These content types are not typically addressed by Western-centric or Simplified-Chinese-oriented datasets.Source: §1 · p. 1

1.1 Our Contributions

  • VisTW-MCQ tests interpretative reasoning and subject knowledge through image-based, exam-style questions across 21 academic subjects.The benchmark contains 3,795 test questions.Source: §1.1 · p. 2
  • VisTW-Dialogue evaluates free-form Traditional Chinese responses to images depicting everyday Taiwanese contexts, including landmarks, cultural practices, and daily life.It comprises 131 culturally grounded dialogue pairs.Source: §1.1 · p. 2
  • The paper also contributes a VLM-as-judge framework with human correlation ρ = 0.8466 and analyzes performance gaps, model scaling, and image-resolution sensitivity across 31 models.The evaluation framework includes a calibration approach intended to ensure continuity across model generations.Source: §1.1 · pp. 2–3

2 Related Work

Existing visual benchmarks cover multimodal reasoning and generation but remain English-centric, while Traditional Chinese benchmarks largely lack multimodal evaluation. VisTW targets the resulting gap in culturally relevant Taiwanese visual-language assessment.

2.1 English-based Multimodal Benchmarks

2.2 Traditional Chinese and Multilingual Benchmarks

  • TMMLU (Hsu et al., 2023) and TMMLU+ provide Traditional Chinese exam questions but lack multimodal components.TMMBench, released with Breeze2 (Hsu et al., 2025), covers Taiwan-related visual questions, but its details and dataset were not publicly released as of the paper’s writing.Source: §2.2 · p. 3
  • Other multilingual benchmarks do not fill the Taiwan-specific gap: M3Exam (Zhang et al., 2023) lacks Traditional Chinese script and Taiwan-specific content, while JMMU and CMMU have stated scope constraints.JMMU (Onohara et al., 2024) has only 30 questions per subject; CMMU (He et al., 2024) has limited public access and insufficient cultural context.Source: §2.2 · p. 3
  • The review identifies no comprehensive Traditional Chinese VLM benchmark spanning both structured reasoning and open-ended generation in culturally relevant Taiwanese contexts.VisTW-MCQ and VisTW-Dialogue are introduced to address this need.Source: §2.2 · p. 3

3 VisTW-MCQ : Visual Multi-Choice Question Answering

VisTW-MCQ is a visual multiple-choice benchmark built from Taiwanese examinations and designed to assess VLM reasoning across subjects. Its evaluation uses zero-shot reasoning prompts, with answer extraction to address response-format failures.

3 VisTW-MCQ : Visual Multi-Choice Question Answering

  • VisTW-MCQ draws on Taiwanese examination questions spanning primary and secondary education through specialized undergraduate courses.The benchmark focuses on questions requiring visual comprehension, including medical diagnostics, geometry, circuit design, and chemistry.Source: §3 · p. 3
  • Questions and images were manually extracted from public exam papers from 2013–2024 and independently reviewed for accuracy, clarity, and completeness.Answer choices were randomly shuffled to maintain an approximately 25% chance baseline for guessing.Source: §3 · p. 3
  • VisTW-MCQ spans 21 subjects, with at least 100 test questions per subject, plus five few-shot development examples and ten validation questions.Table 1 compares VisTW-MCQ with CMMU, JMMMU, and ALM-Bench; for ALM-Bench, only its Traditional Chinese subset is compared.Source: §3 · p. 4, Table 1 · p. 4

3.1 Evaluation Method

  • Evaluation uses zero-shot Chain-of-Thought prompting, asks models to provide reasoning and an answer choice, and averages scores across all 21 subjects.An auxiliary gpt-4o-mini parser extracts and validates answers when models fail to follow the requested format.Source: §3.1 · p. 4
  • The authors note that multiple-choice evaluation does not reflect the mostly free-form question-answering format of real-world VLM use.They introduce a separate benchmark to address this limitation.Source: §3.1 · p. 4

4 VisTW-Dialogue: Visual Free Form Dialogue Benchmark

VisTW-Dialogue evaluates open-ended VLM responses to images and questions rooted in Taiwanese everyday contexts. Its manually collected benchmark contains 131 image-question pairs, and its automated judging approach is assessed against human ratings.

4 VisTW-Dialogue: Visual Free Form Dialogue Benchmark

  • VisTW-Dialogue is designed to reflect open-ended Traditional Chinese interactions between users and VLMs rather than structured question-answering formats.Its scenarios are intended to represent real user experiences.Source: §4 · p. 4
  • Researchers captured images and wrote challenging questions about real-life Taiwanese scenarios, testing spatial reasoning and local cultural knowledge.The researchers intended the images to be unseen during VLM training.Source: §4 · p. 4
  • Collection and quality control used a Discord server and bot, with answer provision, peer review, emoji moderation, and removal of submissions receiving more than two downvotes.The bot also supported quality-control monitoring and experimentation with rating prompts.Source: §4 · pp. 4–5
  • 131 unique image-question pairs with ground-truth answers were collected between July 2024 and February 2025.Source: §4 · p. 5

4.1 Automatic Evaluation : VLM as a Judge

  • The automated judge scores responses using the question, image, and ground-truth answer, with a prompt that specifies scoring criteria.The prompt asks the judge to explain its assessment before assigning a numerical score from 0 to 10.Source: §4.1 · p. 5
  • Judging reliability was assessed by comparing model-generated ratings with the average ratings of three human annotators across four selected models.The comparison covered several prominent VLMs, including Gemini, Claude, GPT-4o, and Qwen models.Source: §4.1 · p. 5
  • Averaging scores from all VLM judges produced the strongest human-score correlation, reaching 0.8466 without images and 0.8463 with images.Including images in evaluation prompts generally improved model-human score correlations; Gemini 2.0 Flash was selected as primary judge given the results and future cost considerations.Source: §4.1 · p. 5
  • Automated judge scores showed no substantial response-length bias, with Spearman’s ρ = 0.26789 (p < 0.001).The authors state that human-annotated data are available to calibrate newer VLMs against human preference.Source: §4.1 · p. 5

5 Benchmark Results

The evaluation finds a strong association between VisTW-MCQ and VisTW-Dialogue performance, alongside substantial variation in how models perform across tasks. Smaller models retain more capability on multiple-choice questions than on open-ended dialogue, while the evaluated Traditional Chinese models lag similarly scaled Simplified Chinese counterparts.

5 Benchmark Results

  • Spearman’s ρ = 0.7746 and Kendall’s τ = 0.6086 (both p < 0.001) indicate strong correlation between VisTW-Dialogue and VisTW-MCQ performance.The evaluation spans closed and open-weight VLMs.Source: §5 · p. 6
  • Breeze2-3B and Breeze2-8B perform worse than similarly scaled Simplified Chinese counterparts InternVL2.5-4B and InternVL2-8B variants.Source: §5 · p. 6
  • InternVL2-1B scores 0.2689 versus 0.3431 for InternVL2-8B on VisTW-MCQ, a 21.6% reduction despite 87.5% fewer parameters, while dialogue scores are 2.13 versus 3.45.InternVL2 performance improves consistently from 1B to 8B on both metrics; the passage characterizes smaller models as struggling more with open-ended multimodal tasks.Source: §5 · p. 6

6 Analysis

The analysis examines resolution sensitivity, the contribution of visual input, cross-benchmark relationships, judge-score calibration, and reasoning prompts. It reports resolution- and task-dependent patterns, a broad association with other exam benchmarks, improved judge alignment after calibration, and model-dependent effects of Chain of Thought prompting.

6.1 Sensitivity of Image Resolution

  • VisTW-MCQ performance remains consistent across upscaled and downscaled images, while VisTW-Dialogue performance stabilizes at about half resolution.Gemini-2.0-Flash-Lite shows slight improvement with increased resolution, whereas Gemini-1.5-Flash-8B shows minor declines.Source: §6.1 · pp. 6–7, Figure 2 · p. 8

6.2 Comparison between With and Without Vision Cues

  • Withholding images consistently reduces scores for all four tested models on both datasets.The image-free comparison is intended to measure the contribution of linguistic knowledge without visual information.Source: §6.2 · pp. 7–8, Table 4 · p. 8

6.3 Correlations with Other Benchmarks

  • Models that score well on MMMU or CMMU generally score higher on VisTW-MCQ, though deviations indicate the benchmarks are not fully interchangeable.The comparison uses a selected subset of models because of computational and financial constraints.Source: §6.3 · p. 8, Figure 3 · p. 9

6.4 Contingency Plan When the Gemini-2.0-Flash Is Retired

  • After calibration, Qwen2.5-VL’s Spearman correlation with human score and ensemble scores rises from 0.7993 to 0.8070.A linear mapping based on five-vote distributions aligns Qwen2.5-VL scores with Gemini-2.0-Flash; the analysis presents this as a strategy for continuity if the original judge becomes unavailable.Source: §6.4 · pp. 8–9, Table 5 · p. 10

6.5 Direct Answer vs CoT in VisTW-MCQ

  • CoT yields 2–10% improvement for high-performing models, but lower-performing VLMs degrade; Gemini-2.0-Flash-001 improves by 13.9%.The authors hypothesize that weaker vision processing may introduce hallucinations during extended reasoning.Source: §6.5 · pp. 9–10, Table 6 · p. 10

7 Conclusion

The conclusion presents VisTW-MCQ and VisTW-Dialogue as benchmarks for evaluating VLMs in Traditional Chinese. Its analysis finds performance gaps across language settings and a strong relationship between structured and free-form performance.

7 Conclusion

  • VLMs trained specifically on Traditional Chinese still significantly underperform Simplified Chinese counterparts on visual content in Traditional Chinese contexts.The analysis also reports a strong correlation between structured and free-form task performance, suggesting consistent capabilities across evaluation formats that measure different aspects.Source: §7 · pp. 10–11
  • The analysis of model scaling and image resolution provides insights into practical deployment, while future work should improve Traditional Chinese performance and broaden cultural coverage in Taiwan.Source: §7 · p. 11

Ethics Statement

The paper describes safeguards for collecting and publishing dialogue images and exam questions, while noting that its Taiwan-wide contributions are concentrated in Taipei.

Ethics Statement

  • VisTai-Dialogue images were collected by researchers with contributors’ consent for public use, and submissions containing specified personally identifying information were prohibited and filtered through community moderation.Prohibited content included names, faces of non-public figures, Wi-Fi passwords, and residential addresses.Source: Ethics Statement · p. 11
  • VisTai-MCQ uses publicly available official exam questions from 2013–2024, with academic integrity and intellectual property named as considerations.Source: Ethics Statement · p. 11

B Creating VisTW-Dialogue on Discord

B.1 Human Annotation Process

The annotation process uses a Discord bot and community discussion to submit images, establish or revise standard answers, and evaluate model responses. The accompanying material points to subject coverage and image characteristics for VisTW-MCQ.

B.1 Human Annotation Process

  • Annotators submit an image, question, and ground-truth answer through the Discord bot; a discussion thread supports adding or revising the standard answer before two-model scoring is requested.The bot captures text following “ground truth:” and allows repeated updates; typing “validate result” requests evaluations.Source: §B.1 · p. 14, Figure 5 · p. 15
  • Image uploads entail waiving ownership rights and agreeing to make the photos public.Source: §B.1 · p. 14

C.1 Details of subjects for VisTW-MCQ

  • Table 7 presents VisTW-MCQ subjects by broad category and question count.The supplied caption identifies the table as an overview of subjects with VisTW-MCQ splits.Source: §C.1 · p. 16, Table 7 · p. 16

C.2 VisTW-MCQ

  • VisTW-MCQ images are described as mostly wider than tall, with most images below one million pixels.Source: §C.2 · p. 16

C.3 VisTW-Dialogue

VisTW-Dialogue images span broader aspect ratios and resolutions than the VisTW-MCQ images, with many Dialogue images near square and most described as wider than tall.

C.3 VisTW-Dialogue

  • Dialogue images have a broader aspect-ratio spread than MCQ images, with many near square; the accompanying description says most Dialogue images are wider than tall.Source: §C.3 · p. 16, Figure 6 · p. 17
  • Dialogue images span a broader range of resolutions than MCQ images, and most Dialogue images are described as exceeding one million pixels.Source: §C.3 · p. 16, Figure 7 · p. 17

D Benchmarked VLM Details

The section describes the evaluated models and the prompting and scoring procedures used for VisTW-Dialogue and VisTW-MCQ. It also frames whether visual context makes exam questions harder, without reporting an answer in the supplied passages.

D Benchmarked VLM Details

  • The evaluated-model overview distinguishes dense and MoE architectures and marks undisclosed closed-source model sizes with “-”; open-weight models use local GPUs or Open Router, while closed-weight models use official APIs.Local GPU resources (3090) are used for models under 11B parameters; larger open-weight models are accessed through Open Router.Source: Table 8 · p. 18, D Benchmarked VLM Details · p. 18

E.1 VisTW-Dialogue Judgment Prompt

E.2 VisTW-MCQ Prompt

F Does exams with figure harder than normal exam questions?

Loading 2503.10427v2…