PaperlayerView exact revision on arXiv

Source-linked AI summary

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi, Md Adnan Arefeen

arXiv:2609.12509v1cs.CV

TL;DR

Bengali multimodal geometry evaluation is limited by scarce benchmarks that pair diagrams with questions. ChitraMiti introduces synthetic and textbook-derived benchmarks with a three-phase protocol; across five models, description-only and diagram-plus-description performance are statistically equivalent, yet models remain weak at cross-modal verification.

  • Problem

    Multimodal mathematical reasoning benchmarks are scarce for low-resource languages such as Bengali, where existing resources are mainly text-only.

    Source: §1 · p. 1

  • Method

    The paper introduces ChitraMiti-12.8k and NCTB-500 and evaluates models with diagram-only, diagram-plus-description, and description-only inputs.

    Source: §1 · p. 1, §10 · p. 8

  • Results

    Across five evaluated models, structured descriptions are statistically validated as a reference representation, while VLMs rely heavily on relational text and remain weak at checking it against diagrams.

    Source: §9 · p. 7, §10 · p. 8

  • Takeaways & Limitations

    The benchmarks and protocol support controlled study of Bengali multimodal geometry reasoning and whether model answers remain grounded in visual evidence.

    Source: §9 · pp. 7–8

    Pretraining exposure to public NCTB questions cannot be ruled out, and adversarial-swap and ablation diagnostics are evaluated only on ChitraMiti-1k.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems that require reading a diagram and a question together. We introduce ChitraMiti-12.8k, a synthetic benchmark of 12,874 Bengali planar geometry problems paired with structured 15-attribute descriptions, and NCTB-500, a complementary set of 500 diagrams manually extracted from Bengali school textbooks. Using a three-phase protocol that separates diagram-only, diagram-plus-description, and description-only inputs, we show across five open-weight and closed-source VLMs that description-only performance is statistically indistinguishable from diagram-plus-description performance, establishing structured descriptions as a sufficient textual proxy for controlled evaluation. Despite this, models remain poor at cross-modal verification, frequently misled by a swapped spatial relation even when they answer the unmodified item correctly. We further evaluate supervised adaptation on ChitraMiti-12.8k, finding that fine-tuning improves performance on both ChitraMiti-1k and NCTB-500, although a substantial gap to the strongest zero-shot model remains. Together, ChitraMiti-12.8k, NCTB-500, and our evaluation protocol offer a standardized way to study Bengali multimodal geometry reasoning and, more broadly, whether VLMs actually check their text against what they see. Our dataset and code are publicly available on Hugging Face at https://huggingface.co/datasets/RaiyanKhaan/ChitraMiti.

1 Introduction

ChitraMiti addresses the shortage of Bengali resources for multimodal geometry by pairing synthetic problems with textbook diagrams and evaluating models under controlled input conditions. The experiments show that models rely on relational text and can still fail to verify it against diagrams.

1 Introduction

  • The three-phase protocol compares diagram-only, diagram-plus-description, and description-only inputs, with TOST equivalence testing across five VLMs and both benchmarks.The protocol supports testing whether models use visual evidence and whether they check spatial relations against the diagram.Source: §1 · pp. 1–2
  • ChitraMiti-12.8k contains 12,874 Bengali planar geometry problems with verified 15-attribute descriptions, while NCTB-500 contains 500 manually extracted textbook diagram-question pairs.The structured descriptions have κ = 0.81 inter-annotator agreement; NCTB-500 covers official Bengali textbooks for Classes 6–10.Source: §1 · p. 1
  • VLM accuracy drops by 2.2% to 12.0% when spatial relations are swapped, and models rely heavily on relational text, with ablation gaps exceeding 12 points.Even the strongest model is misled by swapped text on the majority of items, so success on aligned inputs does not guarantee cross-modal verification.Source: §1 · p. 2
  • Fine-tuning Qwen-3.5 4B raises in-distribution accuracy from 12.7% to 34.2% and NCTB-500 test performance from 9.6% to 23.6%.Source: §1 · p. 2

2 Related Work

  • Existing Bengali math resources are mainly text-only, and BanglaMATH evaluates text-based reasoning without geometric diagrams or cross-modal consistency evaluation (Prama et al., 2025).More broadly, multimodal mathematical benchmarks are commonly built in English, and few provide a standardized evaluation protocol.Source: §1 · p. 1, §2 · p. 2
  • Prior diagnostic studies often measure textual reliance by removing visual input, whereas ChitraMiti also tests whether models detect contradictions between text and images.Related work on multimodal inconsistency and knowledge conflict has not focused on Bengali geometric reasoning with controlled structured spatial descriptions (Zhu et al., 2024).Source: §2 · p. 2

3 Datasets

The paper constructs two Bengali planar-geometry benchmarks through a shared pipeline and uses a common description schema. Its evaluation protocol compares diagram-only, diagram-plus-description, and description-only inputs across the synthetic and textbook-derived sets.

3 Datasets

  • Both benchmarks use a shared pipeline of source selection, topic filtering, description generation, and human verification, with a common 15-attribute schema.ChitraMiti questions are translated into Bengali, while NCTB-500 retains the original Bengali textbook questions; descriptions are checked against source images.Source: §3 · p. 2

3.1 ChitraMiti-12.8k

  • ChitraMiti-12.8k samples 12,874 planar-geometry items from DeepVision-103k after filtering, with composite geometry making up 53.8% of the dataset.A stratified audit of 200 items found a 3.0% correction rate; agreement was κ = 0.81 for descriptions and κ = 0.78 for translations.Source: §3.1 · p. 3
  • The ChitraMiti split has 11,231 training, 643 validation, and 1,000 test items, with train and test drawn from disjoint source IDs and fewer than 0.1% candidate near-duplicates removed.The split is described as suitable for zero-shot evaluation and supervised transfer experiments.Source: §3.1 · p. 3

3.2 NCTB-500

  • NCTB-500 comprises 500 manually identified diagram-question pairs from official Bengali-medium geometry textbooks for Classes 6–10.It preserves scanned classroom material without automated OCR of diagram content; the set covers planar geometry categories across five grade levels and is 78.6% hard.Source: §3.2 · p. 3

4 Evaluation Protocol

  • The same evaluation settings are applied to ChitraMiti and NCTB-500 to compare synthetic and textbook-derived data.Phase A uses the diagram and question, Phase B adds the structured description, and Phase C retains the description and question without the image; B-versus-C comparison isolates the visual contribution beyond the shared text.Source: §4 · p. 3, §4.1 · pp. 3–4

4.2 Experimental Settings

  • Five VLMs are evaluated zero-shot with identical prompts and deterministic decoding, and Phase B versus Phase C is tested using TOST with α = 0.05 and a ±5% equivalence margin.Answers are scored first by symbolic normalization with SymPy, then by an LLM judge when symbolic matching fails; GPT-4o-mini outputs use Gemini 2.5 Flash as judge.Source: §4.2 · p. 4

5 Benchmark Validation

Across the synthetic and textbook benchmarks, diagram-plus-description and description-only performance are similar, while diagram-only performance is generally lower. Formal equivalence tests confirm that the description is a suitable textual proxy for the controlled diagnostics that follow.

5.1 Evaluation on ChitraMiti (1k Test Split)

  • On ChitraMiti-1k, Phase B and Phase C are closely matched for all five models, while Phase A is lower for most; Gemma-4-26B scores 34.0% in both B and C versus 28.7% in A.Qwen3-VL-8B-Instruct scores 29.8% in B, 29.1% in C, and 27.0% in A.Source: Table 3 · p. 4, §5.1 · p. 4

5.2 Validation on NCTB-500

  • On NCTB-500, Phase B and Phase C again achieve similar accuracy across all five models, while Phase A is lower for most.The matching pattern across the textbook and synthetic benchmarks suggests the B–C equivalence is not specific to ChitraMiti-1k.Source: §5.2 · p. 4, Table 4 · p. 5

5.3 Equivalence Between Phase B and Phase

  • Across five models and both benchmarks, Phase B−Phase C differences fall within the ±5% margin, and all TOST tests reject non-equivalence at α = 0.05.The largest difference is 3.4% for Gemini 2.5 Flash on NCTB-500; this supports using the description as a textual proxy without implying diagrams are redundant.Source: §5.3 · p. 5

6 Diagnostics

The diagnostics test which parts of structured descriptions support geometry solving and whether models check those descriptions against diagrams. They find relational information matters more than numerical values alone, while spatial-relation contradictions often mislead models.

6.1 Ablation: Relational vs. Numerical Content

  • 29.6% vs. 26.5% for Gemma-4-26B and 19.2% vs. 16.7% for Qwen3-VL-8B-Instruct: relational-only descriptions outperform values-only descriptions.The same direction holds for all five models, though the gap varies; numerical values still help, but contribute less than relational structure.Source: §6.1 · pp. 5–6, Table 7 · p. 6

6.2 Cross-Modal Verification Under Contradictory Descriptions

  • The ablation identifies which description content models use, but not whether they verify its consistency with the diagram; the contradiction test changes one relation while leaving the question, values, and image unchanged.The test uses Phase B on a 200-item stratified subset of ChitraMiti-1k.Source: §6.2 · p. 6
  • A spatial-relation swap drops accuracy by 2.2–12.0%, and even the strongest model is misled on most contradicted items.The pattern holds across all five tested open- and closed-weight architectures.Source: §6.2 · p. 6

6.3 Qualitative Case Studies

  • Qualitative examples show descriptions can compensate for visual-perception failures, but models also miss adversarial contradictions and struggle with multi-step geometry.Representative examples are in Figure 3 (Appendix H).Source: §6.3 · p. 6

7 Error Analysis

Error analysis finds that NCTB-500 failures are widespread across models, with hard problems dominating the benchmark. On ChitraMiti-1k, composite multi-step problems contribute the largest share of errors.

7 Error Analysis

  • 10.2% of NCTB-500 problems are solved by all five models, 65.0% are missed by every model, and 24.8% show mixed agreement.The authors interpret the pattern as indicating benchmark difficulty rather than architecture-specific weaknesses.Source: §7 · pp. 6–7
  • Hard problems make up 78.6% of NCTB-500 and dominate the overall error profile, while both Gemma-4-26B and Qwen3-VL-8B-Instruct do markedly better on easy and medium items.This pattern holds in the difficulty-stratified results.Source: §7 · p. 7
  • Composite items constitute 53.8% of ChitraMiti-12.8k and 45.9% of its 1k test split, contributing the largest share of errors alongside difficult coordinate-geometry and quadrilateral problems.The analysis attributes failures mainly to integrating multiple geometric relationships rather than isolated perception errors.Source: §7 · p. 7

8 Fine-Tuning

Fine-tuning Qwen3.5-4B on ChitraMiti-12.8k improves accuracy on both ChitraMiti-1k and textbook-derived NCTB-500, though it does not reach the strongest zero-shot model. The discussion and conclusion frame the benchmarks and protocol as tools for studying visual grounding in Bengali geometry.

8 Fine-Tuning

  • 34.2% vs. 12.7% on ChitraMiti-1k and 23.6% vs. 9.6% on NCTB-500: fine-tuning improves in-distribution and out-of-distribution accuracy.About 65% of the ChitraMiti-1k gain transfers to NCTB-500, but fine-tuned Qwen3.5-4B's 23.6% on NCTB-500 remains below zero-shot Gemma-4-26B's 35.0%.Source: §8 · p. 7, Table 9 · p. 7

9 Discussion

  • The synthetic benchmark supports controlled experiments, ablations, and fine-tuning, while NCTB-500 checks whether conclusions hold on Bengali textbook diagrams.The transfer gains suggest ChitraMiti captures reasoning patterns rather than synthetic-construction artifacts; the authors conclude that answering correctly is not enough without grounding in visual evidence.Source: §9 · pp. 7–8

10 Conclusion

  • Across five models, structured descriptions are a statistically validated reference representation, and the framework reveals reliance on explicit relations alongside weak diagram verification.The conclusion also reports that fine-tuning improves transfer to textbook problems in NCTB-500.Source: §10 · p. 8

Limitations

The limitations constrain the scope of the fine-tuning and diagnostic findings, and identify scoring, benchmark, and evaluation-design boundaries. The paper also specifies data-use restrictions and study disclosures.

Limitations

  • Fine-tuning covers one open-weight model, Qwen3.5-4B; whether its gains transfer to other architectures, larger models, or proprietary systems is untested.The authors attribute this scope to resource constraints.Source: Limitations · p. 8
  • Absolute accuracy is low throughout, and NCTB-500’s 500-item size widens its confidence intervals.Predictions that cannot be symbolically parsed are assessed for semantic equivalence by an LLM judge.Source: Limitations · p. 8
  • Possible pretraining exposure to public NCTB textbook questions could inflate Phase C accuracy; ablation and adversarial-swap diagnostics are limited to ChitraMiti-1k.The synthetic ChitraMiti-1k diagrams limit this exposure risk, and Phase B ≈ Phase C there suggests equivalence is not solely a memorization artifact.Source: Limitations · p. 8
  • The TOST equivalence conclusion depends on the predefined ±5% margin, so a different margin could change the equivalence decision.Appendix F also reports results under a stricter ±3% margin.Source: Limitations · p. 8
  • The adversarial metric measures accuracy drops under contradictory descriptions, not explicit contradiction detection; the authors call for forced-choice verification tasks.Such tasks would ask models to flag text-image inconsistencies directly.Source: Limitations · p. 8
  • All experiments are restricted to Bengali planar geometry.Source: Limitations · p. 8

Ethical Considerations

  • NCTB-500 images are released for non-commercial research only; users are advised to consult legal counsel before commercial use.The paper says this is not a legal determination and releases derived annotations and metadata under CC BY 4.0.Source: Ethical Considerations · p. 8

Ethical Considerations; Use of AI Assistants

  • Human evaluation participants were recruited through the authors’ academic and research networks; AI writing tools were used only for writing clarity.The authors state that experimental design, ideas, validation, and conclusions were their own and that they are accountable for the paper.Source: Ethical Considerations · p. 8, Use of AI Assistants · p. 8

A 15-Attribute Schema: Full Specification

The appendix specifies the structured schema, supplies supporting dataset and model details, and documents representative data and evaluation prompts. It also clarifies how NCTB-500 categories relate to the benchmark’s own taxonomy.

A 15-Attribute Schema: Full Specification

B Extended Dataset and Model Statistics

C Per-Category Results on NCTB-500

D Representative Dataset Item

E System Prompts

  • The prompts define diagram-only, diagram-plus-description, and description-only evaluation inputs, while dataset generation follows the 15-attribute schema.Evaluation prompts were used verbatim across models, varying only the phase-specific input; the appendix also includes scoring and translation prompts.Source: E System Prompts · pp. 10–11

F Additional Statistical and Error Validation

The appendix details the equivalence-testing implementation and reports confidence intervals and stratified results. Equivalence generally persists across the reported strata, with specific category exceptions.

F Additional Statistical and Error Validation

G Additional Error Analysis

The appendix adds cross-model error-agreement and difficulty breakdown tables, alongside figure examples of spatial-contradiction and composite-problem failures.

G Additional Error Analysis

H Additional Qualitative Case Studies

The case studies highlight three recurring failure modes, while the data cards document the datasets’ sources, intended uses, and limitations.

H Additional Qualitative Case Studies

  • Table 20 groups three failure modes: text rescuing a failed visual read, unverified reliance on swapped text, and failure on multi-step composite reasoning.Figure 3 illustrates the adversarial swap mechanism and a representative composite failure.Source: H Additional Qualitative Case Studies · p. 13, Table 20 · p. 14

I Dataset Documentation and Data Card

  • ChitraMiti-12.8k contains 12,874 diagrams filtered from DeepVision-103k, with structured descriptions and annotations licensed CC BY 4.0.The source images are released under MIT; the benchmark is intended for Bengali geometry reasoning and multimodal-model fine-tuning.Source: I Dataset Documentation and Data Card · p. 13
  • Compound diagrams with dense topological references may require multi-step reasoning beyond the capabilities of current 4B–26B models.The data card points to the main-text Limitations for further discussion.Source: I Dataset Documentation and Data Card · pp. 13–14
  • NCTB-500 comprises manually cropped diagrams from Classes 6–10 Bengali geometry textbooks, intended for held-out evaluation rather than pretraining or commercial deployment.Its limitations include small size, possible pretraining exposure, heuristic difficulty labels, and source-copyright constraints.Source: I Dataset Documentation and Data Card · p. 14

Related papers

Loading 2609.12509v1…