PaperlayerView exact revision on arXiv

Source-linked AI summary

BanglaMATH : A Bangla benchmark dataset for testing LLM mathematical reasoning at grades 6, 7, and 8

Tabia Tanzin Prama, Christopher M. Danforth, Peter Sheridan Dodds

arXiv:2510.12836v1cs.CY

TL;DR

Existing mathematical-reasoning benchmarks are concentrated in English, leaving Bangla underexplored despite widespread educational use of LLMs. The paper introduces BanglaMATH, a 1.7k-problem Bangla benchmark with grade and reasoning annotations, and evaluates commercial and open-source models. Gemini 2.5 Flash and DeepSeek V3 reach accuracy ≥80% across three grades but remain vulnerable to distractions and perform better after translation into English.

  • Problem

    Most mathematical-reasoning benchmarks are English-focused, while mathematical reasoning in Bangla remains largely unexplored.

    Source: §1 · p. 1

  • Method

    The paper constructs BanglaMATH from Bangla elementary-school materials and evaluates LLM accuracy using annotated answers, grades, and reasoning complexity, including robustness and translation tests.

    Source: §1 · p. 1, §3.2 · p. 5, §4.4 · p. 6

  • Results

    Gemini 2.5 Flash and DeepSeek V3 achieve accuracy ≥80% across three elementary grades, are misled by irrelevant information, and improve by approximately 6.5% after English translation.

    Source: §1 · p. 2, §4.5 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    BanglaMATH provides a benchmark for advancing non-English mathematical reasoning research and evaluating cross-linguistic reasoning capabilities.

    Source: §6 · p. 8

    Budget constraints and restricted API access prevented evaluation of all available LLMs.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) have tremendous potential to play a key role in supporting mathematical reasoning, with growing use in education and AI research. However, most existing benchmarks are limited to English, creating a significant gap for low-resource languages. For example, Bangla is spoken by nearly 250 million people who would collectively benefit from LLMs capable of native fluency. To address this, we present BanglaMATH, a dataset of 1.7k Bangla math word problems across topics such as Arithmetic, Algebra, Geometry, and Logical Reasoning, sourced from Bangla elementary school workbooks and annotated with details like grade level and number of reasoning steps. We have designed BanglaMATH to evaluate the mathematical capabilities of both commercial and open-source LLMs in Bangla, and we find that Gemini 2.5 Flash and DeepSeek V3 are the only models to achieve strong performance, with $\ge$ 80\% accuracy across three elementary school grades. Furthermore, we assess the robustness and language bias of these top-performing LLMs by augmenting the original problems with distracting information, and translating the problems into English. We show that both LLMs fail to maintain robustness and exhibit significant performance bias in Bangla. Our study underlines current limitations of LLMs in handling arithmetic and mathematical reasoning in low-resource languages, and highlights the need for further research on multilingual and equitable mathematical understanding. Dataset link: \href{https://github.com/TabiaTanzin/BanglaMATH-A-Bangla-benchmark-dataset-for-testing-LLM-mathematical-reasoning-at-grades-6-7-and-8.git}{https://github.com/BanglaMATH}

1 Introduction

Bangla mathematical reasoning remains underexplored despite the importance of mathematical competence for LLMs and the dominance of English-language benchmarks. BanglaMATH addresses this gap by evaluating models across grade levels and finds strong headline performance alongside complexity, robustness, and language-bias weaknesses.

  • English-dominated benchmarks leave mathematical reasoning in Bangla largely unexplored despite its relevance to education, science, and industry.Source: §1 · p. 1
  • BanglaMATH provides 1.7k elementary-level Bangla math problems with metadata supporting evaluations that resemble grade-specific human exams.The dataset includes problems from Bangladesh and is designed specifically to evaluate mathematical reasoning in Bangla.Source: §1 · pp. 1–2
  • Gemini 2.5 Flash and DeepSeek V3 achieve accuracy ≥80% across all three elementary school grade levels.The models are evaluated across arithmetic and reasoning complexity, with accuracy decreasing as complexity increases.Source: §1 · p. 2
  • Both top-performing models are easily misled by irrelevant information added to the math problems.The robustness evaluation tests whether models preserve correct reasoning when contextually related but solution-irrelevant details are introduced.Source: §1 · p. 2
  • English translation improves both models’ accuracy by ≥6.5%, revealing performance disparity between English and Bangla reasoning.This result is presented as evidence of language bias affecting low-resource-language mathematical reasoning.Source: §1 · p. 2

2 BanglaMATH Dataset

BanglaMATH is a diverse Bangla mathematics benchmark built from elementary-school materials and designed for fine-grained evaluation. Its collection and annotation process supports analysis by topic, grade, answer type, reasoning complexity, and numerical representation.

2 BanglaMATH Dataset

  • BanglaMATH covers Arithmetic, Algebra, Geometry, and Logical Reasoning to assess general-purpose mathematical reasoning in Bangla.Source: §2 · p. 2

2.1 Data Collection

  • The dataset compiles elementary-school problems from exercise books, quizzes, and exams, then applies transcription, deduplication, cleaning, and human validation.Source: §2.1 · p. 2
  • Each problem is annotated with grade level, answer, reasoning steps, explanations, and digit count for fine-grained analysis.Source: §2.1 · p. 2
  • The dataset’s statistics summarize average problem length, reasoning steps, and digits involved in solutions.Source: Table 1 · p. 2
  • 90.37% of problems require a single-word numerical answer, while 9.63% require descriptive answers; reasoning processes are not provided.Source: §2.1 · p. 2
  • Reasoning-step count serves as a proxy for problem complexity, although a small fraction of items permit multiple valid step counts.Source: §2.1 · pp. 2–3
  • A dataset example contrasts a correct human explanation with an incorrect ChatGPT-4o response.Source: Figure 1 · p. 3
  • Digit count is computed from the maximum base-10 length of numbers in each problem and answer, ignoring decimal points, percent signs, and slashes.Source: §2.1 · p. 3

3 Experimental Setup

The experiments evaluate several commercial and open-source LLMs on BanglaMATH using zero-shot Bangla prompts and accuracy-based answer verification. The setup includes annotated problem samples and compares models differing in architecture and access method.

3.1 Models

  • The study examines LLaMA 4 Maverick, Gemini 2.5 Flash, and Grok 3 through Meta.AI 1, Google AI Studio or Vertex AI, and official website access, respectively.The models differ in architecture, size, and access methods, with some available via APIs and others through open-source model weights.Source: §3.1 · pp. 3–5
  • Table 2 presents sample problems with English translations and human annotations for answers, reasoning steps, and maximum digit counts.The English translations are not part of the dataset.Source: Table 2 · p. 4

3.2 Evaluation Procedure

  • The evaluation uses zero-shot prompts containing each problem in its original Bangla text without additional context, examples, or instructions.This design is intended to measure baseline capabilities in an out-of-the-box deployment setting.Source: §3.2 · p. 5, Figure 2 · p. 5
  • Accuracy is computed by comparing extracted numerical answers with ground truth, with manual verification for formatting inconsistencies and semantic equivalence in descriptive responses.Exact numerical matching is used for single-word numerical answers, while descriptive answers are judged for meaning.Source: §3.2 · p. 5

4 Result and Analysis

Across grade level, arithmetic complexity, reasoning complexity, robustness, and language, LLM performance declines with greater difficulty or distraction, while Gemini 2.5 Flash and DeepSeek V3 are the most consistent models. Translation into English improves performance, exposing a Bangla-language disparity.

4.1 Grade level accuracy

  • Accuracy declines as grade level increases, while DeepSeek V3 and Gemini 2.5 Flash achieve accuracy exceeding 80% across all question types.LLaMA 4 performs strongly on grade 6 but struggles with grades 7 and 8. Grock 3 exceeds 70% on grades 6 and 7 but fails on grade 8, while GPT-4 remains below 70% across all grades.Source: §4.1 · pp. 5–6

4.2 Arithmetic Complexity

  • As the number of digits increases, accuracy drops for every model, with five-digit problems producing approximately 75% accuracy for DeepSeek-V3 and Gemini 2.5 Flash and 58.13% for GPT-4.For zero- to two-digit problems, DeepSeek-V3, Gemini 2.5 Flash, and LLaMA 4 exceed 80%, whereas Grok 3 and GPT-4 fall below 70% beyond two digits.Source: §4.2 · p. 6, Figure 4 · p. 6

4.3 Reasoning Complexity

  • Accuracy exceeds 70% for problems requiring one or two reasoning steps but falls below 50% when four or more steps are required.Reasoning complexity is approximated by the manually annotated number of steps needed to solve each problem.Source: §4.3 · p. 6, Figure 5 · p. 6

4.4 Robustness

  • Adding irrelevant numerical distractors reduces both top models’ accuracy by approximately 20% after two distractors, with DeepSeek V3 degrading more beyond two.The distractor set contains 60 examples, with four modified versions per original problem and one to four contextually integrated distractors.Source: §4.4 · pp. 6–7, Figure 6 · p. 7

4.5 Language Bias

  • Translating Bangla problems into English improves both models’ accuracy by approximately 6.5%, while Gemini 2.5 Flash outperforms DeepSeek V3 in both languages.The authors interpret this accuracy gain as evidence that the models reason more effectively in English than in Bangla.Source: §4.5 · p. 7, Figure 7 · p. 7

5 Related Works

Prior mathematical reasoning benchmarks are predominantly English-based, while multilingual alternatives often target complex or broader evaluations. Elementary-level Chinese datasets provide closer comparators, but comparable resources may lack public release or annotations tailored to LLM evaluation.

5 Related Works

  • Most established math datasets are English-based, including MATH, GSM8K, SVAMP, and several traditional math word problem repositories.The MATH dataset targets seven subjects and is difficult even for humans, with reported accuracy of only 40%.Source: §5 · pp. 7–8
  • Chinese and Hindi alternatives span general-purpose evaluations and elementary-level word problems, with CMATH and K6 organized by grade level similarly to BanglaMATH.These datasets range from middle school to college level in some cases, while CMATH and K6 focus on elementary school problems.Source: §5 · p. 8
  • CMATH contains 1.7K workbook and exam problems and K6 contains 600 institution-collected problems, but neither dataset was publicly released for empirical comparison.APE210K is much larger, with 210K Chinese problems, yet its test set lacks annotations specific to LLM evaluation.Source: §5 · p. 8

6 Conclusion

BanglaMATH is introduced as a fine-grained benchmark for evaluating LLM mathematical reasoning on elementary-level Bangla word problems. The results show declining performance with increasing grade, arithmetic, and reasoning complexity, alongside vulnerability to distractions and a Bangla–English performance gap.

  • BanglaMATH is a benchmark for fine-grained evaluation of LLMs on elementary-level mathematical word problems in Bangla.Source: §6 · p. 8
  • Performance declines as grade level and arithmetic or reasoning complexity increase, making the dataset challenging even for state-of-the-art LLMs.Source: §6 · p. 8
  • Top-performing models such as Gemini 2.5 Flash and DeepSeek V3 struggle when problems contain irrelevant or distracting information.Source: §6 · p. 8
  • Performance improves when Bangla problems are translated into English, indicating a language bias against reasoning in low-resource Bangla.Source: §6 · p. 8

7 Limitations

The study’s evaluation is constrained by limited model coverage caused by budget constraints and restricted API access. The authors therefore call for broader evaluation as more open-source and multilingual models become available.

  • Budget constraints and restricted API access prevented evaluation of all available LLMs.Source: §7 · p. 8
  • Future work should include a broader range of models, particularly newly accessible open-source and multilingual LLMs.Source: §7 · p. 8

8 Ethics Statement

BanglaMATH uses publicly available elementary-school materials and involves no human subjects or sensitive data. Annotation relies on the authors’ judgment, so some labels may vary by interpretation.

  • All dataset problems come from publicly available elementary-school materials, and no human subjects or sensitive data were involved.Source: §8 · p. 8
  • Annotations rely solely on the authors’ judgment, so labels such as solution-step counts may vary by individual interpretation.Source: §8 · p. 8

A.1 Model Responses Under Distractor Augmentation

The distractor-augmentation analysis compares the original toy-car problem with variants containing irrelevant information and records responses from Gemini 2.5 Flash and DeepSeek V3. The examples indicate that some added details are irrelevant to the underlying number logic.

  • Table 3 compares model responses for the original problem and versions containing one to four irrelevant distractors.Source: Table 3 · p. 11
  • The supplied variants report answers of Mintu 6 and Sentu 10, or Mintu 10 and Sentu 14.Source: §A.1 · p. 12
  • Price and total cost are irrelevant to the toy-car logic when the number conditions remain the same.Source: §A.1 · p. 12
  • Playing time does not affect the toy-car answer because the solution depends on the number relationships alone.Source: §A.1 · pp. 12–13

A.2 LLM Response

The section presents example LLM responses to BanglaMATH problems, including an insufficient-information judgment and varied toy-car answers. It also includes onion-cutting problems and examples where all LLMs answered incorrectly.

  • One response marks a problem as having insufficient information because its given information cannot confirm a solution.Source: §A.2 · p. 13
  • The reported toy-car responses include Mintu: 6 and Sentu: 14; another response states that Sentu had 10 toy cars and Mintu had 14 toy cars.Source: §A.2 · pp. 13–14
  • All LLMs provide incorrect answers for the example problems identified in Table 4.Source: Table 4 · p. 13
  • The toy-car problem asks for the original numbers of cars after two conditional exchanges between Mintu and Sentu.Source: §A.2 · p. 14
  • The section also includes a wedding problem asking how many onions a chef and assistant cut and how long they worked under production-rate and timing conditions.Source: §A.2 · p. 14

Related papers

Loading 2510.12836v1…