PaperlayerView exact revision on arXiv

Source-linked AI summary

From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation

Yuan An, Emily Wang, Benjamin Wang, Ruhma Hashmi

arXiv:2608.21668v1cs.AIcs.HC

TL;DR

Prompt-based LLM student simulation fails to reliably reflect target mastery because models often decide answers using their built-in capabilities. This paper moves outcome sampling into a Stochastic Student Knowledge Graph, then uses an LLM only to narrate the sampled result, producing mastery-aligned and diagnostically labeled responses.

  • Problem

    The open problem is controlling LLM-generated student responses so simulated accuracy, localized knowledge gaps, and error causes reflect the intended mastery profile.

    Source: §I · pp. 1–2

  • Method

    The method samples mastery over curriculum-derived chains of required triples to determine outcomes and invokes an LLM afterward to generate a consistent first-person rationale.

    Source: B. Simulation sketch · p. 5, §XIII · p. 10

  • Results

    96.8-100% accuracy across profiles occurs under direct prompting, whereas explicit stochastic sampling produces the monotone accuracy gradient and skill-specific error patterns required for faithful simulation.

    Source: §I · p. 2, §XI · p. 9

  • Takeaways & Limitations

    Separating outcome decisions from narration enables reproducible synthetic responses with skill-specific diagnostic labels and supports downstream knowledge-tracing research.

    Source: B. Simulation sketch · p. 5, §XI · p. 9, §XIII · p. 10

    The evaluation establishes a monotone performance gradient but not how closely simulated accuracy matches real student performance, and the graph construction requires substantial human effort.

    Source: §XII · pp. 9–10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to simulate students at different mastery levels. These simulations can generate synthetic training data and stress-test tutoring systems. However, common prompt-based approaches leave the answer decision to the LLM, which tends to perform according to its built-in capabilities even when instructed to simulate a student with low mastery. As a result, these approaches may have difficulty distinguishing students with low and high levels of mastery. We demonstrate this limitation using 379 College Board-calibrated SAT Algebra items and five archetypal mastery profiles. Three LLMs from three vendors (Gemini 3.1 Flash Lite, Claude Haiku 4.5, and GPT-5.4-mini) achieve 96.8-100% accuracy across all profiles. To address this limitation, we introduce a method grounded in a Stochastic Student Knowledge Graph (SSKG). A curriculum knowledge graph (CKG) is extracted from an open algebra textbook, and each SAT solution is decomposed into a chain of required triples. The SSKG assigns a mastery probability to each triple, which is sampled to determine question correctness. An LLM then generates a first-person rationale consistent with the outcome. The simulation reduces accuracy to 44.1-85.2% across profiles and produces a clear monotone mastery gradient.

I. INTRODUCTION

Prompt-based LLM student simulation fails to enforce mastery differences: models answer SAT Algebra items with near-ceiling accuracy across profiles. The paper proposes SSKG grounding, which separates outcome sampling from LLM narration and targets monotone, skill-specific simulation.

  • Synthetic student responses support knowledge-tracing training, tutoring-system stress tests, and assessment-item piloting when real classroom data is costly or difficult to collect.Source: §I · p. 1
  • 96.8-100% accuracy across five mastery profiles shows that three LLMs perform near the ceiling regardless of the intended student profile.The benchmark contains 379 SAT Algebra multiple-choice items and includes profiles ranging from near-expert to severely gapped.Source: §I · p. 2
  • The SSKG method represents mastery probabilities over curriculum triples and samples each question’s required triple chain before generating a rationale.Source: Figure 1 · p. 1, §I · p. 2
  • The four cumulative ablation arms add sampling, retrieval/execution decomposition, taxonomy-conditioned weighting, and distractor routing as distinct mechanisms.The design enables gains to be attributed to specific mechanisms.Source: §I · p. 2
  • The paper contributes a human-guided curriculum knowledge graph, stratified SSKG profiles, theoretical and practical foundations, and a mechanism-level ablation study.Source: §I · p. 2

B. Stochastic and Probabilistic Knowledge Graphs

Prior student-simulation methods control mastery through prompts, model behavior, or external architectures. The paper situates SSKG grounding among probabilistic knowledge representations and contrasts it with prior control strategies.

  • Probabilistic ontology and knowledge-graph approaches attach degrees of belief or confidence to facts rather than treating all assertions as certain.Source: B. Stochastic and Probabilistic… · pp. 2–3
  • Prior LLM student-simulation methods place mastery control in prompts, model weights or logits, or an external symbolic architecture.Source: D. LLM-Driven Student Simulation · p. 3
  • Prompt-level studies report compressed or insufficiently controlled performance, including strong models outperforming average real students at every prompted NAEP grade.Source: D. LLM-Driven Student Simulation · p. 3
  • PS2 preserves monotone accuracy ordering through logit interpolation, but its proficiency dial remains a single scalar interpolation ratio between two models.Source: D. LLM-Driven Student Simulation · p. 3
  • BEAGLE uses architectural control with metacognitive states, Bayesian Knowledge Tracing, flaw injection, and separate strategist/executor components in open-ended Python problem solving.Its domain is disjoint from this paper’s single-shot multiple-choice setting.Source: D. LLM-Driven Student Simulation · p. 3

III. ILLUSTRATIVE EXAMPLE

The illustrative SAT Algebra item is represented through curriculum triples, a required solution chain, and a distractor map linking localized failures to wrong choices. Student profiles assign mastery probabilities to these triples.

III. ILLUSTRATIVE EXAMPLE

  • The running item contains three wrong choices generated by two independently identifiable errors: an inverted unit rate and a mistranslated right-hand side.Source: §III · p. 4

A. Formal Definitions

  • A curriculum knowledge graph is an ordered set of triples annotated with chapter-and-section positions from the source algebra textbook.Source: A. Formal Definitions · p. 4
  • A triple chain is an ordered sequence of distinct curriculum triples required to derive an item’s solution, supplemented by gap steps absent from the graph.Source: A. Formal Definitions · p. 4
  • The distractor map assigns a wrong choice to a mapped triple when that single step fails; unmapped failures fall back to uniform sampling among incorrect choices.Source: A. Formal Definitions · p. 4
  • Unit-conversion failures at T149/T150 route to B, while sentence-to-equation failures at T170–T172 route to C; compound errors such as choice A remain unmapped.Source: A. Formal Definitions · p. 4, Figure 4 · p. 4
  • An SSKG assigns each CKG triple a mastery probability, with a profile represented by the full mastery vector and a fixed execution rate for known-gap steps.Source: A. Formal Definitions · pp. 4–5

B. Simulation sketch

SSKG simulation separates deciding a student’s outcome from narrating it: analytic sampling resolves mastery and errors first, then an LLM produces a consistent first-person rationale.

  • The analytic procedure samples an item’s required triple chain against a profile’s mastery vector without invoking an LLM to decide correctness.Source: B. Simulation sketch · p. 5
  • After the outcome and any specific wrong choice are determined, the LLM narrates a first-person rationale consistent with that result.Source: B. Simulation sketch · p. 5
  • This design separates deciding from narrating to support faithful student simulation.Source: B. Simulation sketch · p. 5

V. CONSTRUCTING CKG

The CKG is extracted from an open algebra textbook and represents assessment solutions as reviewed chains of fine-grained triples, including procedural and relational knowledge.

  • 864 triples span ten textbook chapters, with each triple tagged by its source chapter and section.Source: §V · p. 5
  • The CKG distinguishes definitional, procedural, and relational predicates across 163 distinct predicates.These categories cover 461 definitional, 302 procedural, and 101 relational triples.Source: §V · p. 5
  • Procedure nodes represent precondition facts for procedures such as finding a line’s x-intercept.Source: A. Concept and Procedure nodes · p. 5, Figure 5 · p. 5
  • Each assessment item is mapped to a sequential chain of required triples identified by matching its official rationale to CKG-supported derivation steps.The illustrative item includes triples supporting unit conversion and translating an English sentence into an equation.Source: A. AI Annotation · p. 5
  • The illustrative item contains a known gap because the CKG approximates rate composition through unit-conversion triples without stating it as a general principle.Source: A. AI Annotation · pp. 5–6

B. Human verification

Human–AI collaboration verifies item chains and distractor mappings, while five SSKG profiles model different overall and region-specific mastery patterns.

Human verification

  • Each candidate triple chain is human-reviewed for derivational validity, completeness, and absence of unnecessary triples.Source: B. Human verification · p. 6
  • Wrong-answer mappings are inferred from official rationales and distractor values, then verified by re-deriving each distractor from a deliberately wrong triple.Source: A. Annotation · p. 6, B. Verification · p. 6
  • Unverified distractor mappings are discarded, with unmapped choices routed through uniform-random wrong-answer selection.Source: B. Verification · p. 6

Student profiles

  • Five archetypal profiles P1–P5 are represented as concrete SSKGs with per-triple mastery probabilities and profile-specific execution rates.Source: §VIII · p. 6
  • P1–P3 vary uniformly from near-expert to struggling, whereas P4 and P5 place intermediate mastery gaps in early versus later curriculum chapters.The profiles are designed to produce complementary regional error patterns.Source: §VIII · p. 6

IX. THE SIMULATION

The simulation separates stochastic answer determination from LLM narration, using knowledge-tracing and cognitive-diagnosis ideas to produce traceable, skill-sensitive outcomes.

Two-stage architecture

Theoretical foundations

  • Retrieval and execution are modeled separately, distinguishing missing knowledge from misapplication of knowledge that is available.A profile-independent slip probability is set to 0.05.Source: B. Theoretical foundations · pp. 6–7
  • The method treats CKG triples as cognitive-diagnosis attributes and requires all active chain steps for correctness, with distractor routing determining specific wrong choices.Source: B. Theoretical foundations · p. 7
  • Definitional triples are removed from the active correctness set, while procedural and relational triples retain their discriminating role.Source: B. Theoretical foundations · p. 7

C. The four-arm ablation ladder

The four-arm ablation ladder adds sampling, retrieval/execution decomposition, taxonomy-based weighting, and distractor routing cumulatively under matched random conditions.

C. The four-arm ablation ladder

X. EVALUATION

The evaluation compares direct prompting with SSKG-based simulation using four questions covering monotonicity, skill specificity, diagnosability, and ablation attribution.

  • The evaluation compares Method 1, where an LLM jointly decides the answer and rationale from a profile description, with Method 2, where correctness is analytically determined before narration.Source: §X · p. 8
  • Q1 tests whether accuracy decreases from high- to low-mastery profiles.Source: A. Experimental design · p. 8
  • Q2 tests whether structured-gap profiles fail on items whose required chains pass through their weak curriculum bands.Source: A. Experimental design · p. 8
  • Q3 tests whether wrong answers can be traced to specific low-mastery triples and whether triple mastery predicts errors on required items.Source: A. Experimental design · p. 8
  • Q4 attributes any Method 2 gain to successive ablation rungs.Source: A. Experimental design · p. 8

B. Datasets

The evaluation uses 379 SAT Algebra items, an 864-triple curriculum graph, five profiles, and three LLMs, with Method 2 correctness determined independently of the narrator. Direct prompting remains nearly perfect across profiles, whereas SSKG-based arms separate them substantially.

Datasets

  • The datasets contain 379 SAT Algebra items, an 864-triple curriculum knowledge graph, and five student profiles.Source: B. Datasets · p. 8

Evaluation setup

  • Method 1 was evaluated on 1,895 item-profile pairs for each of three LLMs, while each Method 2 arm was evaluated on 1,895 records.Source: B. Datasets · p. 8
  • Method 2 accuracy is LLM-invariant because the sampled trace, rather than the narrator, determines correctness.Source: B. Datasets · p. 8

Results

2) Skill specificity (Q2):

The skill-specificity analysis tests structured gaps across chain-footprint categories and examines how ablation mechanisms affect accuracy and diagnostics.

Skill specificity

Difficulty check

Diagnosability

  • Triple mastery correlates with profile error rate at Spearman ρ = −0.459 (p≈10−28), while the bottleneck triple correlates with item outcome at ρ = 0.318.Source: 3) Diagnosability (Q3) · p. 9

Attribution

  • The b→c taxonomy-awareness rung increases accuracy by 14.0 −19.5% across profiles, while c→d leaves accuracy unchanged at 0.0% and contributes only to diagnosis.Source: 4) Attribution (Q4) · p. 9

XI. DISCUSSION

The SSKG method produces more faithful student simulations than direct prompting by restoring mastery-ordered accuracy, skill-specific errors, and diagnostic labels. Ablations identify taxonomy-conditioned weighting as the largest improvement, while key limitations remain around manual construction, real-data validation, and comparisons with existing methods.

Results

  • Direct prompting reaches a high-accuracy ceiling across vendors regardless of target mastery, whereas stochastic curriculum-graph sampling produces the required monotone accuracy gradient and skill-specific errors.The contrast is reported across three vendors and five archetypal profiles for 379 SAT Algebra items.Source: §XI · p. 9

Practical implications

  • Every wrong answer receives a first_failed_triple_id and failure_kind by construction, supplying diagnostic information for knowledge-tracing research.Stage 1 requires no LLM calls, so response corpora can be generated at scale and reproducibly, with LLM calls reserved for narration.Source: §XI · p. 9, §XIII · p. 10

Limitations

  • The method’s current limitations include substantial human effort to build CKGs, solution chains, and distractor maps.The paper identifies reducing this manual workload through more automated methods as important future work.Source: §XII · pp. 9–10
  • The evaluation establishes a monotone mastery gradient but not how closely simulated accuracy matches real student performance.The authors call for grounding and validating simulations against real student response data.Source: §XII · p. 10

Future work

  • The work still lacks direct comparisons with existing student-simulation methods and calls for validation against external IRT statistics and real student data.Future work also includes extending the framework for knowledge-tracing models trained on by-construction diagnostic labels.Source: §XII · p. 10, §XIII · p. 10

Ablation findings

  • Taxonomy-conditioned item weighting provides the largest ablation improvement, while distractor routing leaves accuracy unchanged but improves the diagnostic value of incorrect answers.Knowledge-tracing decomposition mainly moves high-mastery profiles away from an unrealistic near-ceiling; effects on lower-mastery profiles are smaller.Source: §XI · p. 9, §XIII · p. 10

Related papers

This paper cites

Loading 2608.21668v1…