PaperlayerView exact revision on arXiv

Source-linked AI summary

PhysicsMate: A Curriculum-Grounded Bengali Benchmark for Secondary Physics QA with Small-Model Adaptation

Rashid Azraf Jahin, Saadman Sajid, Khan Raiyan Ibne Reza, Sumaiya Tabassum Nimi

arXiv:2610.00664v1cs.CL

TL;DR

Bengali secondary education lacks curriculum-grounded STEM QA resources, while physics problems demand precise terminology, units, and derivations. PhysicsMate builds a curriculum-linked Bengali physics benchmark and adapts Qwen3 models at three sizes. Accuracy gains rise with model size, and the 4B model is quantized for offline local inference.

  • Problem

    Bengali lacks secondary-physics QA resources linked to the national curriculum, and physics answers require accurate laws, definitions, SI units, and numerical forms.

    Source: §I · p. 1

  • Method

    PhysicsMate builds a Bengali QA benchmark from the NCTB Grade 9–10 syllabus using a multi-relational knowledge graph, then applies a unified LoRA recipe to three Qwen3 model sizes.

    Source: §I · p. 1, A. Models and Adaptation · p. 4

  • Results

    +5.5, +15.0, and +23.3 percentage points accuracy for adapted 0.6B, 1.7B, and 4B models, respectively.

    Source: §I · p. 1, §VI · p. 5

  • Takeaways & Limitations

    The 4B model is quantized as an offline binary for local inference where connectivity is limited.

    Source: §VI · p. 5

    Adaptation benefits structured curricular knowledge more than entity-level knowledge, leaving the latter a weaker area of improvement.

    Source: §VI · p. 5

Abstract

from arXiv · show

Bengali secondary education lacks curriculum-grounded benchmarks for STEM question-solving, and general-purpose language models struggle with the precise terminology, unit conventions, and derivations that physics problems demand. We introduce PhysicsMate, a benchmark of 1834 question-answer pairs built from the National Curriculum and Textbook Board (NCTB) Grade 9-10 physics syllabus and grounded in a multi-relational knowledge graph of 1760 nodes and 2600 edges across ten ontological types. We Low-Rank Adapt at 0.6B, 1.7B, and 4B parameters, with a unified recipe and demonstrate a significant increase in closed-book accuracy in all scales (+5.5, +15.0, and +23.3 percentage points). A node-type analysis shows that the most benefited by adaptation is the structured curricular knowledge, which consists of physical quantities and named laws, while the least benefited is the loosely specified entity-level knowledge. The 4B model has been adapted and quantized to a small offline binary that can be used for local inference in resource constrained environments and offers a viable path to curriculum aligned physics support in environments with limited connectivity and hardware.

I. INTRODUCTION

Bengali secondary physics lacks curriculum-linked QA resources, while solving its problems requires precise physical relationships, SI units, and derivations. PhysicsMate formalizes the NCTB syllabus as a knowledge graph and benchmark, then studies small-model adaptation.

Motivation

  • Existing Bengali benchmarks do not organize secondary-physics questions around a national curriculum, leaving no resource for controlled category-level evaluation.The cited resources address other topics, broader topical coverage, or physics above the Grade 9–10 syllabus.Source: §I · p. 1
  • Secondary-physics QA requires applying laws, expressing definitions and dimensions in SI units, and producing standard-form numerical answers; general multilingual models often struggle with these demands.The paper characterizes the gap as ontological as well as linguistic.Source: §I · p. 1

Contributions

  • PhysicsMate builds 1,834 Bengali QA pairs from a 13-chapter NCTB syllabus, linked to a graph with 1,760 nodes across 10 types and 2,600 directed edges.The pairs are partitioned into training, validation, and held-out test sets.Source: §I · p. 1
  • Adaptation gains increase with model capacity: 5.5, 15.0, and 23.3 percentage points for 0.6B, 1.7B, and 4B Qwen3 models, respectively.The models use a unified LoRA recipe.Source: §I · p. 1, Figure 1 · p. 2
  • Adaptation benefits structured quantities, laws, and definitions most, while loosely specified entity-level knowledge benefits least.The authors present this node-type analysis as an empirical basis for targeted future improvement.Source: §I · p. 2

II. RELATED WORK

Prior Bengali science and physics QA resources differ from PhysicsMate in curriculum alignment and educational level. Related work also covers compact models, parameter-efficient adaptation, and structured QA, but PhysicsMate uses a curriculum-defined ontology for benchmark construction and category-level evaluation.

A. Bengali Educational and Science QA

  • PhysicsMate targets Grade 9–10 physics and maps each question to a curriculum-defined ontology, unlike cited resources covering broader topics or Higher Secondary Certificate physics.BEnQA, BnMMLU [3], and P6Jiggasha are described as lacking this specific combination of level and curriculum structure.Source: A. Bengali Educational and Science QA · p. 2

B. Efficient Adaptation and Structured QA

  • PhysicsMate combines small-model adaptation with an ontology built from the curriculum, using that structure to construct and partition its benchmark and support category-level evaluation.Prior work discussed includes LoRA, KGfor-PhysicsQA [9], and a curriculum-materials ontology [10].Source: B. Efficient Adaptation and Structured… · p. 2

III. THE PHYSICSMATE BENCHMARK

The benchmark corpus is derived from the official NCTB Grade 9–10 physics textbook and preserves curriculum concepts, notation, units, examples, and source links. Its semantic units are extracted using overlapping page context and organized by topic.

A. Curriculum Corpus and Semantic Chunking

  • The source is the official NCTB Grade 9–10 physics textbook, spanning 13 chapters and 359 pages, converted page by page into normalized Markdown.The conversion preserves text design, mathematical expressions, and headings while removing immediate PDF-extraction formatting noise.Source: A. Curriculum Corpus and Semantic… · p. 2
  • Three-page windows with two-page overlaps are used to extract self-contained curricular knowledge units under a predefined ontology schema.The shared context allows units spanning multiple pages to be captured.Source: A. Curriculum Corpus and Semantic… · p. 2
  • The knowledge-graph node types span declarative recall, procedural application, visual interpretation, and curricular scaffolding.These levels correspond respectively to definitions and laws; formulas and examples; figures; and HSC topics.Source: Table I · p. 3
  • The 811 semantic units preserve full curricular concepts, mathematical notation, SI-unit conventions, worked examples, and figure references, and feed graph creation and benchmark generation.Each unit retains its topic association and textbook source.Source: A. Curriculum Corpus and Semantic… · p. 3

B. Knowledge Graph Architecture

PhysicsMate represents extracted curriculum records in a multirelational graph whose typed links encode curricular hierarchy, applicability, visual grounding, examples, dependencies, and provenance. The graph provides a structured curricular index for category-level benchmark analysis.

B. Knowledge Graph Architecture

  • Relations encode curricular hierarchy, law and formula applicability, figure-concept links, example links, dependencies, and provenance.The named relations include part_of, applies_to, illustrates, example_of, depends_on, and derived_from.Source: B. Knowledge Graph Architecture · p. 3
  • The graph contains 1,760 nodes across 10 ontological types and 2,600 directed edges spanning 30 relation types.The paper summarizes node and relation statistics in Table I.Source: B. Knowledge Graph Architecture · p. 3
  • Each graph record has an ontology type, a self-contained Bengali description, source-page links, mathematical expressions, physical units, and typed relations to other nodes.The records retain provenance to the source textbook pages.Source: B. Knowledge Graph Architecture · p. 3, Table II · p. 3

C. QA Construction and Partitioning

PhysicsMate links each standard-Bengali QA pair to a knowledge node and textbook semantic unit across the Grade IX–X NCTB physics syllabus. Its textbook-grounded extraction and fixed schema preserve curricular terminology and conventions, while partition checks prevent question-ID overlap.

C. QA Construction and Partitioning

  • Each standard-Bengali question–answer pair links to a knowledge node and originating textbook semantic unit across the full Class IX–X physics syllabus.This curricular provenance traces each pair to a textbook passage and its ontology-typed representation.Source: C. QA Construction and Partitioning · p. 3
  • The benchmark contains 1,834 pairs, split by chapter into 1,374 training, 185 validation, and 275 held-out test questions.Chapter 1 contributes the most items (302), and Chapter 12 the fewest (82).Source: C. QA Construction and Partitioning · p. 3

D. Data Provenance and Quality Control

  • The NCTB textbook is the source of truth: benchmark items derive from specific passages rather than general knowledge.Answers are expected to follow the source curriculum’s terminology, formulas, and unit conventions.Source: D. Data Provenance and Quality Control · p. 3
  • A fixed schema represents each physical law by its NCTB-designated name and formula, limiting variation in equivalent curricular facts.Question identifiers were checked across all three partitions to ensure no overlap.Source: D. Data Provenance and Quality Control · p. 3

E. Dataset Release and Documentation

The paper describes release of the knowledge graph and QA pairs with documentation and provenance metadata. It also evaluates three Qwen3 model sizes using the same LoRA adaptation recipe.

E. Dataset Release and Documentation

  • The multi-relational knowledge graph and all 1,834 QA pairs are released with a Dataset Card documenting extraction, ontology, provenance, and license.The release’s evaluation splits match those used in the paper, and records link to textbook chapter, node type, and mathematical expressions.Source: E. Dataset Release and Documentation · p. 4

A. Models and Adaptation

  • Qwen3 models at 0.6B, 1.7B, and 4B parameters are adapted using the same LoRA recipe, with only adapters updated.LoRA uses rank 16, scaling factor 32, and dropout 0.05; base weights are loaded in 4-bit NF4 quantization.Source: A. Models and Adaptation · p. 4
  • The fixed adaptation configuration makes model capacity the key variable across the three evaluated scales.Training runs for three epochs with 8-bit AdamW at a learning rate of 2 × 10^-4 under cosine decay.Source: A. Models and Adaptation · p. 4

B. Evaluation

On the 275-question held-out test set, LoRA fine-tuning improves every reported metric at every model scale. The evaluation uses a Bengali LLM judge for accuracy alongside Token F1 and BERTScore, while cloud baselines have higher raw accuracy but lower lexical and semantic scores than adapted 4B.

B. Evaluation

  • Accuracy is judged by Gemini-2.5-Flash-Lite on 275 held-out questions, alongside Token F1 and multilingual BERTScore.The judge marks an answer correct only if it conveys the gold reference’s key fact; the authors did not cross-validate it against an independent model or human-annotated subset.Source: B. Evaluation · p. 4, Table III · p. 4

A. Main Benchmark

  • Fine-tuning improves every metric at every scale on the held-out test set, including accuracy, Token F1, and BERTScore.Qwen3-0.6B rises from no correct answers to 5.5% accuracy; 1.7B rises from 10.5% to 25.5%, and 4B gains 23.3 percentage points over baseline.Source: Table III · p. 4, A. Main Benchmark · p. 4
  • Cloud baselines score 69.5% and 82.9% raw accuracy, but adapted 4B scores higher on Token F1 and BERTScore.The paper attributes this metric gap to cloud models’ longer, more explanatory responses compared with shorter NCTB-standard phrases.Source: A. Main Benchmark · p. 4

B. Capacity-Dependent Adaptation

Adaptation gains increase with model capacity across the three tested scales and all three reported metrics. The authors caution that three data points cannot establish a scaling law.

B. Capacity-Dependent Adaptation

  • Figure 2 presents adaptation gains as growing with model capacity.The described trend is observed across accuracy, Token F1, and BERTScore.Source: B. Capacity-Dependent Adaptation · p. 4
  • 50.9% improvement in fine-tuned accuracy, from 25.5% to 50.4%, accompanies the increase from 1.7B to 4B.The paper reports a monotonic capacity trend across accuracy, Token F1, and BERTScore.Source: B. Capacity-Dependent Adaptation · p. 4
  • The observed monotonic gains across three model sizes are not enough to prove a scaling law.The authors describe the pattern as suggesting that larger models use the same LoRA capacity more effectively for domain-specific structure.Source: B. Capacity-Dependent Adaptation · p. 4

C. Node-Type Diagnostics

Adaptation benefits Qwen3-4B Token F1 unevenly across knowledge-node types: quantities, laws, and definitions gain more than entities. The ordering aligns with how canonically specified these facts are in the NCTB textbook.

C. Node-Type Diagnostics

  • 93.9% is the largest relative Token F1 gain for quantity nodes, followed by 56.6% for scientific laws and 56.1% for definitions.Source: C. Node-Type Diagnostics · p. 4
  • 8.7% is the smallest relative Token F1 gain, for entity nodes.Source: C. Node-Type Diagnostics · p. 4
  • Quantities, laws, and definitions have canonical Bengali textbook phrasings, whereas entity mentions are loosely specified and often allow multiple valid surface forms.The passage attributes the stronger adaptation of the former to their concrete, structured form.Source: C. Node-Type Diagnostics · p. 4

D. Qualitative Analysis

The qualitative analysis places increasing accuracy gains alongside examples of adaptation correcting base-model errors. Its cases illustrate corrections for an out-of-curriculum optics answer and an incomplete circuit derivation.

D. Qualitative Analysis

  • Accuracy gains rise monotonically with model size: +5.5, +15.0, and +23.3 percentage points from 0.6B to 4B parameters.Token F1 and BERTScore also show consistent gains as model size grows.Source: Figure 2 · p. 5
  • Table IV reports Qwen3-4B Token F1 by knowledge-node type on 275 test questions and identifies structured knowledge types as benefiting most from adaptation.Source: Table IV · p. 5
  • In the optics case, adaptation replaces an out-of-curriculum spatial rule with the correct three adjectives; in the circuits case, it supplies P = I2R after the base model omits the algebraic step.The cases illustrate failure modes corrected by adaptation.Source: D. Qualitative Analysis · pp. 4–5

VI. DISCUSSION, LIMITATIONS, AND CONCLUSION

Adaptation gains increase with model capacity and concentrate in canonical curricular knowledge, while the paper also identifies evaluation and deployment boundaries. PhysicsMate provides diagnostic evidence and an offline 4B model, with broader curriculum applications proposed rather than established.

VI. DISCUSSION, LIMITATIONS, AND CONCLUSION

  • 23.3 percentage points at 4B versus 5.5 percentage points at 0.6B: accuracy gains increased with capacity under the same LoRA configuration.The authors attribute the difference to larger models offering more useful representational space for a given domain structure.Source: §VI · p. 5
  • The adapted 4B model reached 50.9% accuracy versus 82.9% for frontier models, so it remains far from frontier-model accuracy.The model was quantized to an offline binary for local inference where connectivity is limited.Source: §VI · p. 5
  • 93.9% for physical quantities and 56.6% for scientific laws were the largest Token F1 gains, while entity-level knowledge showed the smallest improvement at 8.7%.The authors characterize the pattern as stronger adaptation for canonical, structured knowledge than for loosely specified entities.Source: Figure 3 · p. 5, §VI · p. 5
  • PhysicsMate is limited to NCTB Grade 9–10 physics, while its authors describe extending the framework to adjacent disciplines and neighboring educational systems.The proposed applications rely on structural relations rather than language-specific idioms.Source: §VI · p. 5
  • Evaluation is text-only, excludes the knowledge graph’s 218 figures as visual inputs, and uses LLM-judged accuracy without independent or human cross-checking.The held-out split is disjoint by question ID, but the authors do not claim broader contamination analysis.Source: §VI · pp. 5–6
  • The authors caution against unmoderated student deployment, citing remaining error in the best offline model and possible benchmark inaccuracies or stylistic mismatch.They recommend teacher review and note that offline use still requires a capable device.Source: §VI · p. 6
Loading 2610.00664v1…