PaperlayerView exact revision on arXiv

Source-linked AI summary

K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs

Hao Liang, Qihan Lin, Zhaoyang Han, Xiaochen Ma, Zhen Hao Wong, Meiyi Qiang, Linzhuang Sun, Wentao Zhang

arXiv:2605.09635v3cs.CL

TL;DR

Existing K–12 benchmarks do not test whether models understand how curriculum knowledge is structured and visually presented. This paper builds a curriculum knowledge graph and derives a benchmark and training corpus from it; its training experiments report gains across text-only and multimodal educational benchmarks.

  • Problem

    Existing K–12 benchmarks do not probe structural and visual understanding of curriculum knowledge, and existing training data do not explicitly teach it.

    Source: §1 · p. 3

  • Method

    The authors build K12-KGraph from official Chinese K–12 textbooks and derive K12-Bench for evaluation and K12-Train for graph-guided supervised fine-tuning.

    Source: §1 · p. 3

  • Results

    At approximately 2,300 samples, K12-Train-Text outperforms equally sized subsets of eight mainstream SFT corpora on GaokaoBench and EduEval, while K12-Train-Full outperforms larger datasets on three multimodal benchmarks.

    Source: §6 · p. 14

  • Takeaways & Limitations

    Current LLMs lack robust curriculum cognition despite strong factual recall, while textual and visual supervision are complementary.

    Source: §6 · p. 14

Abstract

from arXiv · show

Large language models are increasingly used in K-12 education, but existing benchmarks mainly test exam question answering rather than understanding how curriculum knowledge is structured and visually presented. We call this capability curriculum cognition. It covers prerequisite chains, concept taxonomies, experiment-concept links, pedagogical sequencing, and visual grounding. We introduce K12-KGraph, a curriculum-aligned knowledge graph extracted from official People's Education Press textbooks in mathematics, physics, chemistry, and biology across primary, middle, and high school. It contains nine node types and fourteen relation types covering curriculum structure and visual grounding. From this graph, we derive K12-Bench, a 23,640-question multi-select benchmark with five task families: Ground, Prereq, Neighbor, Evidence, and Locate. We also build K12-Train, a graph-guided supervised fine-tuning corpus of 7,335 samples, including 2,267 text-only QA pairs and 5,068 multimodal VQA pairs. On K12-Bench, Gemini-3-Flash achieves only 57 percent exact match and Gemma-4-31B-IT reaches 46 percent, with Prereq and Neighbor being the hardest tasks. Our training experiments show that domain-specific supervision can reduce this gap. Under a matched 2,300-sample budget, K12-Train-Text consistently outperforms equally sized subsets of eight mainstream instruction-tuning corpora on GaokaoBench and EduEval. For vision-language models, K12-Train-Full achieves the best overall results on Gaokao-MM, MDK12-medium, and K12Vista among all compared training configurations, despite using fewer samples than the full DataFlow and WizardLM baselines. It also surpasses both text-only and multimodal-only variants, showing that textual and visual supervision are complementary. We release the graph, benchmark, training data, and complete construction pipeline.

1 Introduction

The paper identifies curriculum cognition—understanding curriculum structure and textbook visuals—as a gap in existing K–12 benchmarks and training data. It introduces a textbook-derived knowledge graph that supports both a benchmark and graph-guided training data.

1 Introduction

  • The authors define curriculum cognition as understanding prerequisite order, links between experiments and concepts, and where ideas appear in textbooks—capabilities current K–12 benchmarks do not probe.Textbook figures can communicate concepts, provide visual evidence for relations, or supply information needed to solve problems; the paper identifies the lack of explicit benchmark and training coverage as a bottleneck.Source: §1 · p. 3
  • K12-KGraph is a heterogeneous graph from official Chinese textbooks spanning mathematics, physics, chemistry, and biology across primary, middle, and high school.Its textual component models curriculum structure with seven node types and relations for taxonomy, prerequisites, association, verification, assessment, location, and order; its multimodal component adds Figure and VisualElement nodes and relations for visual grounding.Source: §1 · p. 3
  • A single graph supports evaluation through neighborhood traversal and training by rendering node properties and edge semantics into QA pairs.The work also describes a reproducible LLM-based extraction and hierarchical-merge pipeline with DAG validation for taxonomic and prerequisite relations.Source: §1 · p. 3
  • Under a matched 2,300-sample budget, K12-Train-Text consistently outperforms eight instruction-tuning corpora across two base models on GaokaoBench and EduEval.On GaokaoBench, the reported gains are +24.1/+32.4 over the strongest SFT baseline and +114.6/+221.0 over official instruction-tuned variants; K12-Train comprises 2,267 text-only and 5,068 multimodal samples.Source: §1 · p. 3
  • K12-Bench contains 23,640 graph-derived multi-select questions across Ground, Prereq, Neighbor, Evidence, and Locate; Gemini-3-Flash reaches 57% exact match and Gemma-4-31B-IT reaches 46%.The five families probe structural curriculum understanding, including prerequisite reasoning, neighbor recommendation, experiment evidence chains, and cross-chapter indexing.Source: §1 · p. 3
  • K12-Train-Full achieves the best overall results on Gaokao-MM, MDK12-Bench, and K12Vista, surpassing full DataFlow and WizardLM baselines and both K12-Train variants with fewer samples.K12-Train-Full uses 7,335 samples, compared with 10,000 for DataFlow and 142,759 for WizardLM; its results show textual and visual supervision are complementary.Source: §1 · p. 4

2 Related Work

Prior educational benchmarks test domain-question answering but not curriculum structure. Existing educational knowledge graphs and instruction-tuning datasets also leave gaps in curriculum alignment and structured grounding.

K–12 and education benchmarks

  • Existing Chinese K–12 and multimodal benchmarks assess domain-question answering, but none systematically test prerequisite dependencies, concept taxonomies, or pedagogical sequencing.The benchmarks span multiple subjects, educational capabilities, and visual reasoning, while CK12 focuses on holistic cognition rather than structural curriculum understanding.Source: §2 · p. 4

Educational knowledge graphs

  • Prior educational knowledge graphs support knowledge tracing or prerequisite discovery, but typically lack multi-subject alignment to official K–12 curricula and grounding in textbook figures.Existing graphs generally represent textbook knowledge as textual entities and relations; no prior work built a large-scale Chinese K–12 graph for both benchmarking and training LLMs.Source: §2 · p. 4

Instruction tuning datasets

  • General-purpose instruction-tuning datasets lack educational knowledge, while Self-Instruct variants do not use structured sources; KG-guided synthesis grounds training data in curriculum structure.The cited general-purpose datasets include OpenHermes, UltraChat, WizardLM, Tulu-3, and SmolTalk.Source: §2 · p. 5

3 K12-KGraph

K12-KGraph uses a heterogeneous schema for textual and multimodal K–12 curriculum knowledge, with nine node types and fourteen directed edge types. Its construction pipeline extracts, reconciles, and validates textbook graphs.

3.1 Schema Design

  • The schema defines nine node types and fourteen directed edge types for textual and multimodal K–12 education content.Concepts can store textbook-worded definitions and importance levels, while Experiment nodes can record instruments, process, phenomena, and conclusions.Source: §3.1 · p. 5

3.2 Construction Pipeline

  • Five stages turn textbook PDFs into section-level graphs, merge and reconcile entities, then detect and manually resolve cycles in taxonomy and prerequisite edges.The pipeline uses MinerU for OCR parsing, prompts GPT-5.2 to extract nodes and edges with textbook evidence or confidence scores, and applies expert verification and review during quality control.Source: §3.2 · pp. 5–6

3.3 Graph Statistics

  • Graph summaries focus on six pedagogically salient content node types plus Book, omitting Chapter, Section, and ubiquitous container relations.Every content node is anchored to a textbook location, while sections and chapters follow a fixed hierarchy guaranteed by the schema.Source: §3.3 · p. 6

4 Benchmark and Training Data from K12-KGraph

K12-KGraph yields a benchmark for evaluating curriculum cognition and training data for educational LLMs. Both resources derive from graph structure, so each sample traces to a specific subgraph for controllable difficulty, coverage, and factual correctness.

4.1 K12-Bench: Benchmark Construction

  • Five multi-select task families—Ground, Prereq, Neighbor, Evidence, and Locate—test knowledge grounding, prerequisites, concept relations, experiment evidence, and cross-chapter indexing.Questions and distractors are graph-derived, while models receive no graph context; candidate sets are filtered for near-duplicates and synonymous distractors.Source: §4.1 · p. 7
  • 23,640 graph-derived items include 98.4% fully correct items in a spot-check; K12-KGraph was human-verified with Fleiss’ κ = 0.84 overall.Because benchmark correctness depends on the underlying graph, graph errors can be localized to specific edges.Source: §4.1 · p. 7

4.2 K12-Train: KG-Guided Data Synthesis

  • K12-Train synthesizes node-grounded and relation-grounded QA with Qwen3-235B-A22B [29], while deterministic templates generate exercise-assessment QA from tests_concept and tests_skill edges.Node prompts teach content from typed properties; relation prompts target edge semantics, and deterministic exercise templates avoid LLM fabrication of those relations.Source: §4.2 · pp. 7–8
  • K12-Train comprises 7,335 pairs: 2,267 text-only and 5,068 multimodal; source-balanced sampling and edge-confidence selection address pool skew and balance supervision.The authors report that these modestly sized, structurally grounded samples are highly effective for educational SFT.Source: §4.2 · p. 8

5 Experiments and Results

The experiments benchmark models’ curriculum-structure understanding on K12-Bench and test whether K12-Train improves educational performance after text-only and multimodal fine-tuning.

5.1 Experimental Settings

  • The experiments compare zero-shot models on K12-Bench with text-only and multimodal fine-tuning on downstream educational benchmarks.The K12-Bench evaluation covers five open-source models and five proprietary APIs; text-only datasets are matched at approximately 2,300 samples, while multimodal experiments include K12-Train variants and full DataFlow and WizardLM datasets.Source: §5 · p. 8, §5.1 · pp. 8–9

5.2 Benchmarking LLMs on K12-Bench

  • Gemini-3-Flash achieves 57.1% overall exact match on K12-Bench, while Gemma-4-31B-IT reaches 46.4%; overall F1 peaks at about 73%.Models fail to identify the complete answer set in more than half of questions; LLaMA-3-8B-Instruct’s 7.2% EM is close to random guessing at 6.7%.Source: Table 3 · p. 9, §5.2 · pp. 10–11
  • Prereq and Neighbor are the hardest K12-Bench tasks, with Gemini-3-Flash below 35% EM on both, while Ground and Evidence are relatively easier.Top-model F1 exceeds 75% on Ground and 72% on Evidence; the paper suggests their associations are more explicit in pretraining materials.Source: §5.2 · p. 11

5.3.1 Text-Only Fine-Tuning Results

  • K12-Train-Text leads the matched-budget SFT configurations on GaokaoBench and EduEval across both evaluated base models.On GaokaoBench, it scores 1009.96 total on Qwen3-4B-Base and 625.49 on Llama3.1-8B-Base; on EduEval, its averages are 66.76 and 40.90, respectively.Source: Table 4 · p. 10, §5.3.1 · pp. 11–12, Table 5 · p. 11

5.3.2 Multimodal Fine-Tuning Results

  • K12-Train-Full scores 39.9% on Gaokao-MM and 52.94 on MDK12-Bench, leading all evaluated configurations on both benchmarks.It surpasses K12-Train-Text and K12-Train-MM on both, with gains distributed across subjects rather than driven by a single subject.Source: §5.3.2 · pp. 12–13, Table 6 · p. 12, Table 7 · p. 12
  • On K12Vista, K12-Train-Full is the only fine-tuned configuration to exceed the base model, achieving the best average score of 79.95.It leads across multiple-choice, fill-in-the-blank, and open-ended QA; the passage attributes the advantage over text-only and multimodal-only variants to combining both supervision sources.Source: §5.3.2 · p. 13, Table 8 · p. 13

5.3.3 Analysis

  • The analysis attributes K12-Train’s gains to explicit relation-level grounding and training data aligned with the official curriculum’s pedagogical coherence.Despite being synthesized from math, physics, chemistry, and biology subgraphs, it scores 120.18 on Chinese (+3.33 over DataFlow) and 132.00 on Humanities Math (+4.08), and matches the top tier in History, Geography, and Politics—subjects without in-domain supervision.Source: §5.3.3 · p. 13
  • K12-Train-Full achieves the best overall result on all three multimodal benchmarks and outperforms both text-only and multimodal-only training.The results support complementarity: text supervision strengthens curriculum knowledge and relation-level reasoning, while multimodal QA grounds these capabilities in figures and diagrams.Source: Table 6 · p. 12, Table 7 · p. 12, §5.3.3 · p. 13

6 Conclusion · Appendix · A K12-KGraph Construction Details

K12-KGraph, K12-Bench, and K12-Train support evaluating curriculum cognition and training educational language and vision-language models. The reported findings show weak curriculum cognition in current LLMs and benefits from curriculum-grounded, complementary text and visual supervision.

6 Conclusion

  • The paper introduces K12-KGraph from official Chinese K–12 textbooks, alongside K12-Bench for evaluating curriculum cognition and K12-Train for KG-guided SFT.Source: §6 · p. 14
  • K12-Train-Full outperforms larger general-purpose datasets and both text-only and multimodal-only variants across three multimodal educational benchmarks.The conclusion describes these results as evidence that textual and visual supervision are complementary.Source: §6 · p. 14
  • With approximately 2,300 samples, K12-Train-Text outperforms equally sized subsets of eight mainstream SFT corpora on GaokaoBench and EduEval.The conclusion characterizes this as sample-efficient KG-guided synthesis.Source: §6 · p. 14
  • A strong open-source model achieves 46% EM on K12-Bench, while current LLMs lack robust curriculum cognition despite strong factual recall.Source: §6 · p. 14

A.1 Node and Edge Attribute Specification … A.3 Prompt for KG Extraction

K12-KGraph’s appendix specifies its ontology and source-processing setup, then describes a staged extraction workflow that builds a textual graph before adding conservatively grounded multimodal information. The prompts constrain outputs to textbook-supported text and visible image evidence.

A.1 Node and Edge Attribute Specification

  • The ontology defines nine node types and fourteen edge types, with detailed node attributes constrained to verifiable textbook or visible-image evidence.Visual descriptions and localization must be supported by image content, textual evidence must quote aligned textbook text verbatim, and inferred visual relations require rationale and confidence.Source: §A.1 · p. 18

A.2 Source Documents and Preprocessing Setup

  • The graph draws on official PEP mathematics, physics, chemistry, and biology textbooks across primary, middle, and high school, preprocessed with MinerU’s default pipeline.The PDFs are converted to structured text without custom parsing rules or modifications.Source: §A.2 · p. 18, SINGLE-TURN USER PROMPT FOR TEXTUAL… · p. 19

SINGLE-TURN USER PROMPT FOR TEXTUAL K12-KGRAPH EXTRACTION

  • Textual extraction reads textbook sections and returns schema-defined nodes and edges as JSON; its full single-user-message template incorporates the detailed schema.The prompt directs the model to extract structured knowledge from the provided section text.Source: SINGLE-TURN USER PROMPT FOR TEXTUAL… · p. 19, §A.3 · p. 19

A.3 Prompt for KG Extraction

  • Image-level analysis records verbatim aligned textbook evidence and visible content, retaining images only when they have an explicit instructional connection.The prompt rejects decoration and images whose instructional function or stable link to a knowledge object, problem context, or learning task cannot be determined.Source: §A.3 · p. 20
  • Retained figures receive one of six pedagogical roles, while extraction identifies only instructional visual elements that can genuinely be localized.The roles are explaining a concept, introducing a real-life context, demonstrating an experiment, assisting problem solving, summarizing and consolidating knowledge, or presenting data.Source: §A.3 · p. 21
  • Localization returns boxes only for supplied visual elements and links only to supplied candidate nodes or edges, with uncertain links omitted.Boxes use normalized 0–1000 coordinates; unlocalizable elements receive a null box and zero confidence.Source: §A.3 · p. 22
  • Multimodal relations require direct image evidence: supporting an edge has a high threshold, and an image is required by an exercise only when its removal blocks understanding or a key reasoning step.Each relation rationale must identify visible supporting content, and its confidence reflects the strength of that evidence.Source: §A.3 · pp. 22–23
  • Textual extraction precedes a three-step multimodal phase that filters images, classifies retained figures and visual elements, then localizes them and links them to the textual graph.The multimodal phase uses the images, section text, and previously extracted textual graph.Source: §A.3 · p. 19, Figure 4 · p. 23

B K12-Bench Construction Details and Evaluation Protocol … PROMPT FOR FILTERING K12-BENCH CANDIDATE DISTRACTORS

K12-Bench distractors are built through layered, graph-driven sampling followed by rule-based and pedagogical filtering, with task-specific pools designed to keep options plausible yet unambiguously incorrect. A conservative teacher-style LLM filter retains only candidates judged valid distractors.

B.1 Distractor Pool and Structural Sampling Rules

  • Distractors follow a near-to-far pipeline: graph-derived gold answers are paired with structurally relevant candidates, then filtered, deduplicated, and stably ordered for option sampling.Candidate pools expand to broader curriculum contexts only when local neighborhoods are insufficient.Source: §B.1 · p. 23
  • Candidate pools expand from graph-local neighborhoods to curriculum scopes, using sibling structures and an undirected graph of relates_to, is_a, and prerequisites_for; initial pools contain about 8–20 raw candidates.Broader layers are used only when earlier layers are insufficient, and previously selected candidates are removed.Source: §B.1 · p. 24
  • Within each layer, candidates are ranked by BAAI/bge-smallzh-v1.5 [32] using the average of question similarity and maximum similarity to any gold answer.Node names are used as representations, except Exercise nodes preferentially use their stems.Source: §B.1 · p. 24
  • Sampling is task-specific: Ground, Prereq, Neighbor, Evidence, and Locate use distinct graph neighborhoods and curriculum pools tailored to each query’s answer structure.For example, Neighbor distractors come from the outer ring around direct relates_to and is_a answers, while Prereq pools distinguish prerequisite from successor queries.Source: §B.1 · p. 24
  • Rule-based filters remove gold answers, name-level duplicates, and candidates too closely related to the gold set before pedagogical filtering.The filters target candidates that could remain acceptable under task semantics or a looser interpretation.Source: §B.1 · p. 24

B.2 Prompt for Pedagogical Filtering

  • A conservative K–12 teacher-style LLM evaluates each candidate independently against the question and gold answers, removing trivially implausible or potentially acceptable options.The filter is intended to ensure pedagogical validity, not to improve semantic similarity.Source: §B.2 · p. 24

PROMPT FOR FILTERING K12-BENCH CANDIDATE DISTRACTORS

B.3 Benchmark Composition Statistics · B.4 Answering Prompt and Decoding Rules · B.5 Baseline EM/F1 Computation for the Random Predictor

These appendices describe K12-Bench’s composition and answer-label balancing, its deterministic prompt and decoding protocol, and the instance-level scoring and random-predictor baseline.

B.3 Benchmark Composition Statistics

  • Subject coverage is broadly balanced across task families, but Evidence contains no mathematics items because mathematics lacks Experiment nodes needed for verifies queries.Figure 6 shows subject distributions and answer cardinalities; Table 11 reports per-subtask counts and probed graph relations, with four options in every multi-select task.Source: §B.3 · p. 25, Figure 6 · p. 26, Table 11 · p. 26
  • Single-answer items dominate several task families, especially Locate, while construction balances feasible correct-option counts k ∈{1, 2, 3} and label combinations.This preserves intrinsic task-family differences while avoiding systematic bias toward particular answer letters or label combinations.Source: §B.3 · pp. 25–26

B.4 Answering Prompt and Decoding Rules

  • The evaluation uses a fixed system prompt requiring option labels and a user prompt with the question and four options, with decoding set to temperature 0.0, top_p 1.0, and max_tokens 32.Raw outputs are retained verbatim; parsed predictions are uppercased, deduplicated, sorted, and used for evaluation. Figure 7 presents the prompt.Source: §B.4 · p. 26, Figure 7 · p. 27
  • The parser normalizes common delimiter variants and extracts labels without semantic repair; empty responses or API failures are flagged and retried until valid.The pipeline retains both the raw model output and the normalized parsed prediction.Source: §B.4 · pp. 26–27

B.5 Baseline EM/F1 Computation for the Random Predictor

  • F1 is aggregated as instance-level macro F1, averaging per-instance scores and weighting task averages by instance count rather than pooling label counts.This keeps F1 on the same per-instance footing as exact match and avoids up-weighting items with more correct labels; F1 is zero when both precision and recall are zero.Source: §B.5 · pp. 27–28
  • The random-guess baseline samples uniformly from the 15 non-empty subsets of {A, B, C, D}, and its expected scores depend only on gold-set cardinality.Symmetric balancing of label positions makes the specific gold letters irrelevant to these expectations.Source: §B.5 · p. 28

B.6 Illustrative Example of KG-Grounded Resource Derivation … 3. Preferred Answer Structures

The pipeline derives benchmark and training resources from relation-specific local subgraphs, preserving graph queries for evaluation and reformulating them as instruction–response pairs for supervision. Its QA prompts tailor questions and answers to node attributes or typed relations while enforcing grade-appropriate, logically rigorous, graph-grounded content.

B.6 Illustrative Example of KG-Grounded Resource Derivation

  • K12-Bench preserves the original graph query form, while K12-Train reformulates the same relation-specific subgraph structure into instruction–response pairs.Both resources are generated by selecting a target relation, extracting its local subgraph, and applying task-specific structural constraints.Source: §B.6 · p. 28

C.1 Prompt for QA Synthesis

  • Node-level prompts target key node attributes, while edge-level prompts require reasoning about the relation; both use separate templates and graph-grounded style constraints.Prompts are sent as single user messages, and node properties are cropped to fields relevant to the target QA type.Source: §C.1 · p. 28

SINGLE-TURN PROMPT FOR NODE-LEVEL QA SYNTHESIS

SINGLE-TURN PROMPT FOR EDGE-LEVEL QA SYNTHESIS

  • Edge-level examples must contain exactly one question and answer, stay grounded in input properties, and explain why the edge holds rather than restating endpoint names.The edge prompt supplies the typed relation, source and target names, and properties.Source: SINGLE-TURN PROMPT FOR EDGE-LEVEL QA… · p. 30, Figure 9 · p. 31

2. Relation-Specific Prompting Goals

  • Relation-specific goals tailor questions across is_a, prerequisites_for, relates_to, verifies, illustrates, refers_to, requires_figure, and supports_edge.They elicit explanations of taxonomy, learning order, conceptual connections, experimental evidence, visual meaning, localized elements, figure-dependent exercises, and visual support for an edge.Source: §2 · p. 30

3. Preferred Answer Structures

  • Preferred answer structures require explanations linking definitions, prerequisite knowledge, concept relations, or experimental observations to the target claim.For visual relations, answers connect the figure’s overall content or an element’s visible features to the target; figure-dependent exercise answers identify information the figure supplies.Source: §3 · pp. 30–31

C.2 Training Configuration … D.2 KG Validation: Structural Checks and Human Verification

Text-only baseline comparisons use fixed-size, distribution-preserving subsamples, while K12-KGraph is validated through automatic structural checks and full human verification. K12-Bench and K12-Train receive targeted spot-checks because their correctness is grounded in the validated graph.

C.2 Training Configuration

  • Text-only SFT uses full-parameter training and multimodal SFT uses LoRA, with identical hyperparameters shared within each experiment group.Unlisted hyperparameters follow the training framework defaults.Source: §C.2 · p. 31, Table 12 · p. 32, Table 13 · p. 33

C.3 Baseline Subsampling and Fairness Controls

  • Each text-only baseline is subsampled to 2,300 instances, using stratification when metadata is available to preserve dataset distributions and control for data volume.Pilot experiments found similar performance trends across random samples from the same dataset.Source: §C.3 · p. 31

D Validation and Quality Assurance

  • K12-KGraph receives automatic structural checks and full human verification, while K12-Bench and K12-Train receive targeted manual spot-checks for consistency with the validated graph.Benchmark answers and distractors are deterministically derived from graph structure; minor training-data imperfections are tolerated when data remain factually grounded and pedagogically useful.Source: §D.1 · p. 32

D.1 Validation Strategy, Annotator Background, and Ethics

  • Twelve experienced K–12 practitioners—three subject specialists for each of four disciplines—conduct validation across primary, middle, and high school levels.Annotators are compensated at or above local minimum wage, and the data come from public educational materials without sensitive personal information.Source: §D.1 · p. 32

D.2 KG Validation: Structural Checks and Human Verification

  • The is_a and prerequisites_- subgraphs are checked for cycles and structural inconsistencies, which subject annotators review and resolve according to curriculum correctness.Ambiguous cases are discussed to maintain consistency with standard curriculum progression.Source: §D.2 · p. 32
  • Every graph node and edge is independently annotated by three same-subject annotators, with disagreements jointly reviewed to reach a final consensus label.Consensus decisions retain, modify, or remove the item.Source: §D.2 · pp. 32–33
  • Visual relations and bounding boxes are retained only above confidence thresholds of 0.85, 0.95, and 0.9, respectively, with unsupported figures and unused visual elements pruned.Agreement is higher for structurally explicit relations than semantically nuanced ones; the adjudicated graph is described as achieving high precision.Source: §D.2 · p. 33

D.3 Spot-Check Validation of K12-Bench and K12-Train · E Extended Results and Sanity Checks

Stratified manual spot checks found high correctness in both K12-Bench and K12-Train, with most identified issues limited to surface-level wording or distractor quality rather than semantic errors.

D.3 Spot-Check Validation of K12-Bench and K12-Train

  • 98.4% of sampled K12-Bench instances and 96.9% of sampled K12-Train instances were judged fully correct.Most errors involved phrasing ambiguity or borderline distractor quality, rather than semantic or factual errors.Source: §D.3 · p. 33
  • The review covered 15% of K12-Bench instances stratified by task family, subject, and grade, plus 10% of K12-Train QA pairs across construction categories.Benchmark checks compared questions, gold answers, distractors, and graph structure; training checks assessed factual consistency, pedagogical appropriateness, and linguistic clarity.Source: §D.3 · p. 33
  • Across both resources, identified issues were primarily surface-level rather than semantic, supporting the validated knowledge graph as a basis for benchmark construction and QA synthesis.Examples included phrasing problems and distractor similarity.Source: §D.3 · p. 33
  • Table 14 reports pre-adjudication Fleiss’ κ across subjects and node/edge types; Table 15 reports per-subtask K12-Bench EM and option-level F1 in percent.The supplied captions do not provide the tables’ numerical values.Source: Table 14 · p. 34, Table 15 · p. 34

E.1 Full Benchmark Results · E.2 Stability Across Random Seeds · E.3 Overlap and Leakage Analysis

The appendix provides fine-grained K12-Bench breakdowns, checks training-result stability across random seeds, and analyzes potential leakage into external evaluations. The analyses report minor seed variation and negligible lexical overlap, with no duplicated or near-duplicated sampled question-answer pairs found.

E.2 Stability Across Random Seeds

  • For GaokaoBench and EduEval, three-seed validation used 20% stratified subsamples that preserved each subtask’s distribution.The SFT process was repeated with seeds 42, 123, and 2026; full multi-seed evaluation was omitted due to computational cost.Source: §E.2 · p. 34, Table 17 · p. 35
  • The sampled evaluation results showed only minor variation across seeds, indicating that performance gains were stable across random initializations.Table 17 reports performance variation across the three seeds on the stratified subset.Source: §E.2 · p. 34, Table 17 · p. 35

E.1 Full Benchmark Results

  • Table 16 reports K12-Bench performance by curriculum subject using exact match and instance-level macro F1.It complements Table 15’s per-subtask breakdown; the passages do not provide table cell values.Source: §E.1 · p. 34, Table 16 · p. 35

E.3 Overlap and Leakage Analysis

  • K12-Train draws on K–12 textbooks, while GaokaoBench comprises standardized high-stakes exam questions and EduEval has independently constructed evaluation objectives.The sources are institutionally and functionally distinct.Source: §E.3 · p. 35
  • K12-Train emphasizes textbook-grounded definitions, procedures, and relation explanations, unlike the problem-solving and applied reasoning assessed by GaokaoBench and EduEval.The paper presents this difference in content form and objective as further reducing the likelihood of overlap.Source: §E.3 · p. 35
  • N-gram analysis found negligible lexical overlap, and manual inspection found no duplicated or near-duplicated question-answer pairs between K12-Train and the evaluation benchmarks.The authors conclude that K12-Train introduces no measurable leakage into the benchmarks.Source: §E.3 · p. 35

Related papers

Loading 2605.09635v3…