PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark for LLM's Understanding of Middle School and High School Science Topics

Noah L. Schroeder, Yessy Eka Ambarwati, Yuji Zhang, ChengXiang Zhai

arXiv:2609.32020v1cs.AIcs.CYcs.HC

TL;DR

Educators lack robust, standards-aligned evidence for choosing LLMs for K-12 science. This paper builds NGSS-aligned middle- and high-school benchmarks and evaluates open-weight models, finding that size did not consistently predict performance and that reviewed items were not perfectly aligned.

  • Problem

    Existing benchmarks largely assess general language or advanced scientific reasoning, leaving limited evidence about LLM performance on science content encountered in K-12 schools.

    Source: §1 · p. 2

  • Method

    The study developed NGSS-aligned middle- and high-school benchmarks using synthetic question generation and multi-judge validation, then evaluated open-weight models and reviewed items for standards alignment.

    Source: §1 · p. 2, §3.1.1 · p. 6, §3.1.3 · p. 6

  • Results

    Model size did not correspond to higher secondary-science scores; several smaller open-weight models performed strongly, while 236 of 240 reviewed items (98.3%) aligned with their intended NGSS standards.

    Source: §4.3 · p. 16, §5.3 · p. 19

  • Takeaways & Limitations

    Model selection in education should be intentional, and future benchmarks should connect content with pedagogical knowledge and assess multi-turn conversations.

    Source: §4.3 · p. 16, §7 · p. 21

    Only a sample of benchmark items underwent human review, so broader validation could review more items and add validity analyses.

    Source: §6.1 · p. 20

Abstract

from arXiv · show

Large language models (LLMs) are increasingly integrated into educational settings, yet educators lack robust, standards-aligned tools to evaluate their effectiveness in K-12 science contexts. Existing benchmarks predominantly assess general language or advanced scientific reasoning, leaving a critical gap in understanding LLMs' performance on content directly relevant to secondary science curricula. To address this gap, we developed a comprehensive NGSS-aligned benchmark for both middle and high school science using a rigorous synthetic data pipeline, multi-judge validation, and item-level psychometric analysis. Nine open-weight LLMs were systematically evaluated using this benchmark, indicating that several smaller, locally deployable models achieved high accuracy across diverse science domains and question types. Our findings indicate that model size did not consistently predict performance, emphasizing the importance of intentional model selection for educational deployment. We then incorporated a human reviewer into the loop, reviewing the items generated by the LLMs for alignment with NGSS standards. The human review indicated that synthetically generated items were not in perfect alignment with the NGSS standards, indicating the benefits of human-in-the-loop item development, the need to explore the intersection of content and pedagogical knowledge, and the need to extend benchmarks to evaluate LLMs' capacity for interactive, evidence-based feedback in educational scenarios.

1. Introduction

Educators need evidence to choose LLMs for specific educational contexts because performance varies across domains, tasks, and reasoning demands. This study develops and evaluates an NGSS-aligned benchmark for middle and high school science.

1. Introduction

  • LLM performance can vary substantially across domains, tasks, and reasoning demands, complicating model selection for educational contexts.LLMs are increasingly incorporated into educational tools and workflows.Source: §1 · p. 2
  • Existing benchmarks largely target general language, coding, reasoning, or advanced science, leaving limited evidence about LLM performance on school science content.This leaves educators without a simple tool for choosing models for particular educational situations.Source: §1 · p. 2
  • The NGSS define science learning through performance expectations integrating disciplinary core ideas, science and engineering practices, and crosscutting concepts.Their broad adoption in K-12 science education makes them a basis for evaluating standards-aligned questions.Source: §1 · p. 2
  • The study develops and evaluates the first NGSS-aligned benchmark for middle and high school science.It aims to provide empirical evidence about LLM capabilities and limitations in science education.Source: §1 · p. 2

2. Literature Review

The literature describes practical reasons for evaluating models in education, but existing benchmarks offer limited coverage of standards-aligned content knowledge. The present study builds on benchmark measurement, item-quality evaluation, synthetic data, and LLM judging to address that gap.

2.1 The State of LLMs in Education

  • Open-weight models can be hosted locally, giving educators control over student data, though deployment practicality depends on model size and computational requirements.The cited discussion also notes potential fine-tuning and post-training opportunities.Source: §2.1 · pp. 2–3

2.2 Benchmarking as Educational Measurement

  • Benchmark scores support inferences about performance only insofar as tasks represent the intended domain; this study treats scores as observed performance, not underlying competence.Fine-grained and summary results are reported because area-level patterns may be obscured by aggregate scores.Source: §2.2 · pp. 3–4

2.3 Benchmark Quality and Item-Level Evaluation

  • Item difficulty and discrimination help assess whether a benchmark is too easy, too difficult, or insufficiently differentiating among models.Difficulty reflects the proportion answering correctly; discrimination measures differentiation between high- and low-performing respondents.Source: §2.3 · p. 4

2.4 Existing Educational Benchmarks for LLMs

  • Existing educational LLM benchmarks cover content knowledge or pedagogical abilities, but content-focused work is relatively sparse compared with teaching-focused benchmarks.Examples include computer-science concept inventories and E-EVAL's coverage of Chinese K-12 subjects.Source: §2.4 · p. 4

2.5 Synthetic Data Generation and LLM-as-a-Judge

  • Synthetic data generation and LLM judges can scale benchmark construction and evaluation, but concerns about data quality, validity, and evaluator reliability motivate human validation.This study uses LLMs for both generation and judging and incorporates a human-in-the-loop process.Source: §2.5 · p. 5

2.6 The Present Study

  • The study addresses the gap with NGSS-aligned benchmarks, item-level and question-type analyses, a synthetic-data pipeline, and evaluation of nine open-weight LLMs.It also examines human validation of whether items align with NGSS standards.Source: §2.6 · pp. 5–6

3. Part I

Part I builds NGSS-aligned middle- and high-school science datasets through synthetic item generation, quality filtering, and multi-judge validation. The resulting datasets form a benchmark infrastructure for studying model and item performance.

3.1.1 Dataset Generation

  • Sonnet 4.6 Thinking generated question–answer pairs across 12 NGSS science areas per school level, balanced among recall, conceptual understanding, and applied reasoning.The item counts were 34/33/33 by cognitive level for middle school and 33/33/34 for high school.Source: §3.1.1 · p. 6

3.1.2. Item Quality Requirements

  • Automated quality gates removed near-duplicates, enforced school-level reading ranges, and checked item appropriateness.The duplicate threshold was rapidfuzz token-set ratio ≥ 88; reading targets were grades 5.0–9.0 for middle school and 5.0–13.0 for high school.Source: §3.1.2 · p. 6

3.1.3 Item Validation

  • Three independent LLM judges assessed question accuracy and standard alignment, and items with judge disagreement or unanimous rejection were removed.Judges were Nemotron 3 Super 120b a12, Gemini Flash 3, and OSS 120b.Source: §3.1.3 · p. 6

3.2.1 Dataset Generation

  • The pipeline rejected and regenerated 4,433 candidate middle-school questions and 1,770 high-school items..Source: §3.2.1 · p. 7

3.2.2 Item Validation

  • The judges disagreed on 45 middle-school items and 44 high-school items, which were dropped; the final pipeline supports studying model scores and item performance.The study describes the resulting benchmark as a comprehensive ground-truth dataset built through quality gates for duplicates, reading level, wording, and standard agreement.Source: §3.2.2 · p. 7, §3.3 · p. 8

3.2.3 Final Dataset

  • The final benchmark contains 1,078 middle-school items and 1,150 high-school items.Table 1 presents the distribution of questions across NGSS areas.Source: §3.2.3 · p. 7, Table 1 · p. 8

4. Part II

Part II evaluates nine open-weight models on the datasets developed in Part I using a shared zero-shot protocol. It scores model responses and examines both overall accuracy and item-level performance.

4.1.2 LLMs Evaluated

  • Nine open-weight models from six industry labs answered every benchmark question three times.The evaluated models ranged from Llama 3.1 8b to OSS-120b and included multiple model families and architectures.Source: §4.1.2 · p. 9

4.1.3 Prompting Method

  • All models received shared middle-school or high-school prompts in a standardized zero-shot protocol, with no per-model tuning or specialized prompting..Source: §4.1.3 · p. 9

4.1.4 Evaluation Procedures

  • A single judge scored recall responses as correct or incorrect and conceptual and applied responses on a 0, 0.5, or 1 rubric.Empty, errored, and refused responses received zero and were tracked separately as refusal rates; macro and micro accuracy were both reported.Source: §4.1.4 · p. 9

4.1.5 Single-shot and Ensemble Methods

  • Single-shot accuracy uses the first run, while ensemble accuracy uses majority voting across three runs as a measure of self-consistency.Accuracies are proportions between 0 and 1.Source: §4.1.5 · p. 9

4.1.6 Item-level Performance

  • Item-level analysis evaluated difficulty and discrimination and classified items as trivial when at least 95% answered correctly or broken when at most 5% did..Source: §4.1.6 · p. 9

4.2 Results

Across both benchmarks, OSS 120b performed best and most tested models scored above 90%; stronger overall models generally performed consistently across question types and standards. Item-level analysis also found high mean difficulty, low discrimination, and sometimes more than 90% trivial questions.

4.2.1 RQ1: How well do popular open-weight LLMs understand secondary science topics across various item types?

  • OSS 120b performed best across both benchmarks, and most tested models scored above 90% on each; majority-vote ensembles raised scores by 1% to 5% without degrading any model’s score.Table 2 presents the overall benchmark leaderboard.Source: §4.2.1 · p. 10, Table 2 · p. 11
  • The most accurate overall models tended to score consistently well across item types.Table 3 reports accuracy by question type for single-shot and ensemble methods.Source: §4.2.1 · p. 11, Table 3 · p. 12
  • OSS-20b achieved one-shot accuracy of 94% on middle school and 96% on high school, while Nemotron 3 Nano 30b a3 achieved 97% on both benchmarks.The models’ accuracy differences were not statistically significant at high school but were significant at middle school, where Nemotron scored better; Nemotron used fewer tokens on average in both benchmarks.Source: §4.2.1 · pp. 11–12
  • Models that scored well overall tended to score well across content areas, while weaker models such as Llama 3.1 8b and Granite 3.3 Instruct varied notably across areas.Table 4 reports ensemble scores by model and content standard area.Source: §4.2.1 · p. 13, Table 4 · p. 13

4.2.2 RQ2: How well did the individual items in the benchmark function?

  • Item-level analysis found generally high mean difficulty and low item discrimination, with some standard–question-type groups containing more than 90% trivial questions.One broken question occurred in MS-LS2 Recall at the middle school level.Source: §4.2.2 · p. 14, Table 5 · p. 14

4.3 Discussion

The results suggest that model size did not determine performance: smaller models performed well, including across NGSS areas and item types. However, because the benchmark is fully synthetic and includes many trivial items, whether its questions are too easy remains unresolved.

4.3 Discussion

  • OSS-20b and Nemotron 3 Nano 30b a3 performed well across metrics and outperformed significantly larger models, while strong models generally did well across NGSS areas and item types.The authors conclude that model size did not correspond to higher scores and model choice matters in educational contexts.Source: §4.3 · p. 16
  • The fully synthetic benchmark includes many trivial questions and generally high item difficulty with low discrimination, leaving open whether questions are too easy or models find secondary science relatively easy.The authors propose future comparisons with human-curated items.Source: §4.3 · p. 16
  • Small, locally deployed open-weight models provided competitive NGSS-aligned science knowledge, including beyond recall questions.The authors frame model selection in educational contexts as an intentional decision-making process.Source: §4.3 · p. 16

5. Part III

Part III used a former K–12 science teacher’s review of sampled benchmark items to assess NGSS alignment. Most reviewed items aligned, but the review also surfaced omissions in scientific practices, item formats, and standard coverage.

5.1 Methods

  • The review covered 240 question–answer pairs, and a science teacher with 10 years of grades 7–12 teaching experience assessed alignment and answer correctness.The sample included 120 pairs from each school-level dataset and was distributed across NGSS areas and question types.Source: §5.1 · p. 17

5.2 Results

  • 236 of 240 reviewed items (98.3%) aligned with their intended NGSS standards; four misaligned items often omitted a key performance-expectation component.The four questions generally addressed related content but missed a primary relationship, scientific practice, or representation emphasized by the standard.Source: §5.2 · p. 17, Table 6 · p. 18
  • The reviewer found no graphical elements or data-interpretation tasks, and mathematical reasoning was usually expressed in text rather than formulas or quantitative representations.These patterns suggest that some scientific practices were underrepresented in the reviewed sample.Source: §5.2 · p. 17
  • The review sample omitted items associated with several middle- and high-school standards, but those omissions do not establish that the full benchmark lacked them.Part III examined only a subset of benchmark items.Source: §5.2 · pp. 17–18

5.3 Discussion

  • The findings support the benchmark’s validity for NGSS content expectations, while the misaligned items show that high coverage does not guarantee complete alignment with learning objectives.The misalignments tended to retain related content while missing scientific practices or conceptual relationships central to the performance expectation.Source: §5.3 · p. 19
  • Future benchmark iterations could add graphical representations, data interpretation, and mathematical representations to better reflect the range of NGSS practices.The discussion notes that some performance standards require evidence use, data analysis, visual interpretation, or mathematical reasoning.Source: §5.3 · p. 19

6. General Discussion

The discussion frames the benchmark as an initial, domain-specific reference rather than a broadly generalizable measure of model competence. It identifies validation and human curation as research needs, while noting that some small open-weight models may be practical options for educators.

6. General Discussion

  • The benchmark is a replicable pipeline, but its results indicate performance in specific domains rather than broadly generalizable model competence.The authors position it as a starting point for further experimentation and comparisons.Source: §6 · p. 20, §6.1 · p. 20

6.1 Limitations and Implications for Research

  • Only a sample of benchmark items received human review, and the authors recommend broader review by more experts and additional validity analyses.They also note generally poor item-level metrics, which they suspect may reflect high model scores rather than inherent item problems.Source: §6.1 · p. 20
  • Future work could test whether human review and editing improve item-level metrics or change conclusions about models’ secondary-science understanding.The authors leave the value added by human involvement as an open research question.Source: §6.1 · pp. 20–21
  • The authors identify a need to benchmark the intersection of content and pedagogical knowledge, including multi-turn feedback aligned with evidence-based strategies and scaffolding.They describe such interactive evaluation as a possible gold standard for educational LLM benchmarks.Source: §6.1 · p. 21

6.2 Implications for Practice

  • OSS-20b and Nemotron 3 Nano 30b a3 are presented as plausible options for educators, while model guardrails remain outside the benchmark’s scope.The benchmark can also provide a basis for comparing base-model performance with training improvements.Source: §6.2 · p. 21

7. Conclusion

Evidence for selecting specific models in educational settings has been sparse, but several small, consumer-hardware-deployable open-weight models perform strongly across middle- and high-school NGSS areas.

7. Conclusion

  • Several small open-weight LLMs deployable on consumer hardware perform strongly across middle- and high-school NGSS areas.The pipeline also provides a foundation for further testing and specialized-model development.Source: §7 · p. 21
  • Evidence supporting the choice of specific LLMs for educational settings has been sparse or non-existent.Source: §7 · p. 21
  • The authors call for benchmarks bridging content and pedagogical knowledge, especially in multi-turn conversations.Source: §7 · p. 21

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

The authors disclose using AI tools to transcribe dictated notes and draft some sections, and identify their funding sources.

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process

Related papers

Loading 2609.32020v1…