Source-linked AI summary
From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation
Yuan An, Emily Wang, Benjamin Wang, Ruhma Hashmi
TL;DR
Prompt-based LLM student simulation fails to reliably reflect target mastery because models often decide answers using their built-in capabilities. This paper moves outcome sampling into a Stochastic Student Knowledge Graph, then uses an LLM only to narrate the sampled result, producing mastery-aligned and diagnostically labeled responses.
Problem
The open problem is controlling LLM-generated student responses so simulated accuracy, localized knowledge gaps, and error causes reflect the intended mastery profile.
Method
The method samples mastery over curriculum-derived chains of required triples to determine outcomes and invokes an LLM afterward to generate a consistent first-person rationale.
Results
96.8-100% accuracy across profiles occurs under direct prompting, whereas explicit stochastic sampling produces the monotone accuracy gradient and skill-specific error patterns required for faithful simulation.
Takeaways & Limitations
Separating outcome decisions from narration enables reproducible synthetic responses with skill-specific diagnostic labels and supports downstream knowledge-tracing research.
Takeaways & Limitations
The evaluation establishes a monotone performance gradient but not how closely simulated accuracy matches real student performance, and the graph construction requires substantial human effort.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to simulate students at different mastery levels. These simulations can generate synthetic training data and stress-test tutoring systems. However, common prompt-based approaches leave the answer decision to the LLM, which tends to perform according to its built-in capabilities even when instructed to simulate a student with low mastery. As a result, these approaches may have difficulty distinguishing students with low and high levels of mastery. We demonstrate this limitation using 379 College Board-calibrated SAT Algebra items and five archetypal mastery profiles. Three LLMs from three vendors (Gemini 3.1 Flash Lite, Claude Haiku 4.5, and GPT-5.4-mini) achieve 96.8-100% accuracy across all profiles. To address this limitation, we introduce a method grounded in a Stochastic Student Knowledge Graph (SSKG). A curriculum knowledge graph (CKG) is extracted from an open algebra textbook, and each SAT solution is decomposed into a chain of required triples. The SSKG assigns a mastery probability to each triple, which is sampled to determine question correctness. An LLM then generates a first-person rationale consistent with the outcome. The simulation reduces accuracy to 44.1-85.2% across profiles and produces a clear monotone mastery gradient.
I. INTRODUCTION
Prompt-based LLM student simulation fails to enforce mastery differences: models answer SAT Algebra items with near-ceiling accuracy across profiles. The paper proposes SSKG grounding, which separates outcome sampling from LLM narration and targets monotone, skill-specific simulation.
- Synthetic student responses support knowledge-tracing training, tutoring-system stress tests, and assessment-item piloting when real classroom data is costly or difficult to collect.
- 96.8-100% accuracy across five mastery profiles shows that three LLMs perform near the ceiling regardless of the intended student profile.The benchmark contains 379 SAT Algebra multiple-choice items and includes profiles ranging from near-expert to severely gapped.
- The SSKG method represents mastery probabilities over curriculum triples and samples each question’s required triple chain before generating a rationale.
- The four cumulative ablation arms add sampling, retrieval/execution decomposition, taxonomy-conditioned weighting, and distractor routing as distinct mechanisms.The design enables gains to be attributed to specific mechanisms.
- The paper contributes a human-guided curriculum knowledge graph, stratified SSKG profiles, theoretical and practical foundations, and a mechanism-level ablation study.
B. Stochastic and Probabilistic Knowledge Graphs
Prior student-simulation methods control mastery through prompts, model behavior, or external architectures. The paper situates SSKG grounding among probabilistic knowledge representations and contrasts it with prior control strategies.
- Probabilistic ontology and knowledge-graph approaches attach degrees of belief or confidence to facts rather than treating all assertions as certain.
- Prior LLM student-simulation methods place mastery control in prompts, model weights or logits, or an external symbolic architecture.
- Prompt-level studies report compressed or insufficiently controlled performance, including strong models outperforming average real students at every prompted NAEP grade.
- PS2 preserves monotone accuracy ordering through logit interpolation, but its proficiency dial remains a single scalar interpolation ratio between two models.
- BEAGLE uses architectural control with metacognitive states, Bayesian Knowledge Tracing, flaw injection, and separate strategist/executor components in open-ended Python problem solving.Its domain is disjoint from this paper’s single-shot multiple-choice setting.
III. ILLUSTRATIVE EXAMPLE
The illustrative SAT Algebra item is represented through curriculum triples, a required solution chain, and a distractor map linking localized failures to wrong choices. Student profiles assign mastery probabilities to these triples.
- III. ILLUSTRATIVE EXAMPLE: The running item contains three wrong choices generated by two independently identifiable errors: an inverted unit rate and a mistranslated right-hand side.
- A. Formal Definitions: A curriculum knowledge graph is an ordered set of triples annotated with chapter-and-section positions from the source algebra textbook.
- A. Formal Definitions: A triple chain is an ordered sequence of distinct curriculum triples required to derive an item’s solution, supplemented by gap steps absent from the graph.
- A. Formal Definitions: The distractor map assigns a wrong choice to a mapped triple when that single step fails; unmapped failures fall back to uniform sampling among incorrect choices.
- A. Formal Definitions: Unit-conversion failures at T149/T150 route to B, while sentence-to-equation failures at T170–T172 route to C; compound errors such as choice A remain unmapped.
- A. Formal Definitions: An SSKG assigns each CKG triple a mastery probability, with a profile represented by the full mastery vector and a fixed execution rate for known-gap steps.
B. Simulation sketch
SSKG simulation separates deciding a student’s outcome from narrating it: analytic sampling resolves mastery and errors first, then an LLM produces a consistent first-person rationale.
- The analytic procedure samples an item’s required triple chain against a profile’s mastery vector without invoking an LLM to decide correctness.
- After the outcome and any specific wrong choice are determined, the LLM narrates a first-person rationale consistent with that result.
- This design separates deciding from narrating to support faithful student simulation.
V. CONSTRUCTING CKG
The CKG is extracted from an open algebra textbook and represents assessment solutions as reviewed chains of fine-grained triples, including procedural and relational knowledge.
- 864 triples span ten textbook chapters, with each triple tagged by its source chapter and section.
- The CKG distinguishes definitional, procedural, and relational predicates across 163 distinct predicates.These categories cover 461 definitional, 302 procedural, and 101 relational triples.
- Procedure nodes represent precondition facts for procedures such as finding a line’s x-intercept.
- Each assessment item is mapped to a sequential chain of required triples identified by matching its official rationale to CKG-supported derivation steps.The illustrative item includes triples supporting unit conversion and translating an English sentence into an equation.
- The illustrative item contains a known gap because the CKG approximates rate composition through unit-conversion triples without stating it as a general principle.
B. Human verification
Human–AI collaboration verifies item chains and distractor mappings, while five SSKG profiles model different overall and region-specific mastery patterns.
- Human verification: Each candidate triple chain is human-reviewed for derivational validity, completeness, and absence of unnecessary triples.
- Human verification: Wrong-answer mappings are inferred from official rationales and distractor values, then verified by re-deriving each distractor from a deliberately wrong triple.
- Human verification: Unverified distractor mappings are discarded, with unmapped choices routed through uniform-random wrong-answer selection.
- Student profiles: Five archetypal profiles P1–P5 are represented as concrete SSKGs with per-triple mastery probabilities and profile-specific execution rates.
- Student profiles: P1–P3 vary uniformly from near-expert to struggling, whereas P4 and P5 place intermediate mastery gaps in early versus later curriculum chapters.The profiles are designed to produce complementary regional error patterns.
IX. THE SIMULATION
The simulation separates stochastic answer determination from LLM narration, using knowledge-tracing and cognitive-diagnosis ideas to produce traceable, skill-sensitive outcomes.
- Two-stage architecture: The simulation has two stages: stochastic sampling determines the outcome and an LLM narrates a first-person rationale consistent with it.
- Two-stage architecture: Stage 1 records a seed, successful and failed chain steps, failure types, and—when incorrect—the selected answer choice.
- Theoretical foundations: Retrieval and execution are modeled separately, distinguishing missing knowledge from misapplication of knowledge that is available.A profile-independent slip probability is set to 0.05.
- Theoretical foundations: The method treats CKG triples as cognitive-diagnosis attributes and requires all active chain steps for correctness, with distractor routing determining specific wrong choices.
- Theoretical foundations: Definitional triples are removed from the active correctness set, while procedural and relational triples retain their discriminating role.
C. The four-arm ablation ladder
The four-arm ablation ladder adds sampling, retrieval/execution decomposition, taxonomy-based weighting, and distractor routing cumulatively under matched random conditions.
- C. The four-arm ablation ladder: Four cumulative experimental arms each add one mechanism to the preceding SSKG simulation arm.
- C. The four-arm ablation ladder: Common random numbers reuse identical seeded draws across arms for each item–profile pair, enabling comparisons under the same random conditions.
- C. The four-arm ablation ladder: In arms a and b, any failed solution-chain step makes the answer incorrect without distinguishing failure type.
- C. The four-arm ablation ladder: Arm c records a procedural failure as an execution error while preserving its effect on correctness.
- C. The four-arm ablation ladder: Arm d links an execution error at T149 or T150 to distractor B, 25/4 m = 5, rather than sampling randomly among incorrect choices.
- C. The four-arm ablation ladder: The generated rationale is required to describe the specific unit-conversion mistake carried through to the selected distractor.
- C. The four-arm ablation ladder: A definitional failure at T19 affects narration but does not make the answer incorrect in arms c or d.
X. EVALUATION
The evaluation compares direct prompting with SSKG-based simulation using four questions covering monotonicity, skill specificity, diagnosability, and ablation attribution.
- The evaluation compares Method 1, where an LLM jointly decides the answer and rationale from a profile description, with Method 2, where correctness is analytically determined before narration.
- Q1 tests whether accuracy decreases from high- to low-mastery profiles.
- Q2 tests whether structured-gap profiles fail on items whose required chains pass through their weak curriculum bands.
- Q3 tests whether wrong answers can be traced to specific low-mastery triples and whether triple mastery predicts errors on required items.
- Q4 attributes any Method 2 gain to successive ablation rungs.
B. Datasets
The evaluation uses 379 SAT Algebra items, an 864-triple curriculum graph, five profiles, and three LLMs, with Method 2 correctness determined independently of the narrator. Direct prompting remains nearly perfect across profiles, whereas SSKG-based arms separate them substantially.
- Datasets: The datasets contain 379 SAT Algebra items, an 864-triple curriculum knowledge graph, and five student profiles.
- Evaluation setup: Method 1 was evaluated on 1,895 item-profile pairs for each of three LLMs, while each Method 2 arm was evaluated on 1,895 records.
- Evaluation setup: Method 2 accuracy is LLM-invariant because the sampled trace, rather than the narrator, determines correctness.
- Results: Method 1 achieves 96.8-100% accuracy across every profile, with individual P1–P5 accuracies of 100.0%, 99.7%, 96.8%, 100.0%, and 100.0%.
- Results: Claude Haiku 4.5 and GPT-5.4-mini are flatter still at 99.2–99.7% on every profile under Method 1.
- Results: Every Method 2 arm spans a 30–55% accuracy range across profiles, while arm-d spans 44.1-85.2%.
2) Skill specificity (Q2):
The skill-specificity analysis tests structured gaps across chain-footprint categories and examines how ablation mechanisms affect accuracy and diagnostics.
- Skill specificity: The 379 items are partitioned into 140 early-only, 143 mixed, and 96 later-only chains to test whether P4 and P5 fail in their weak bands.
- Skill specificity: Under arm-d, P4 is weaker on early and mixed items than on later-only items by 35.7%.
- Skill specificity: Under arm-d, P5 reaches 70.0% on early items, 48.3% on mixed items, and 62.5% on later items.
- Difficulty check: Arm-d yields a monotone Easy > Medium > Hard slope for P3: 48.3% > 40.8% > 39.2%.
- Diagnosability: Triple mastery correlates with profile error rate at Spearman ρ = −0.459 (p≈10−28), while the bottleneck triple correlates with item outcome at ρ = 0.318.
- Attribution: The b→c taxonomy-awareness rung increases accuracy by 14.0 −19.5% across profiles, while c→d leaves accuracy unchanged at 0.0% and contributes only to diagnosis.
XI. DISCUSSION
The SSKG method produces more faithful student simulations than direct prompting by restoring mastery-ordered accuracy, skill-specific errors, and diagnostic labels. Ablations identify taxonomy-conditioned weighting as the largest improvement, while key limitations remain around manual construction, real-data validation, and comparisons with existing methods.
- Results: Direct prompting reaches a high-accuracy ceiling across vendors regardless of target mastery, whereas stochastic curriculum-graph sampling produces the required monotone accuracy gradient and skill-specific errors.The contrast is reported across three vendors and five archetypal profiles for 379 SAT Algebra items.
- Practical implications: Every wrong answer receives a first_failed_triple_id and failure_kind by construction, supplying diagnostic information for knowledge-tracing research.Stage 1 requires no LLM calls, so response corpora can be generated at scale and reproducibly, with LLM calls reserved for narration.
- Limitations: The method’s current limitations include substantial human effort to build CKGs, solution chains, and distractor maps.The paper identifies reducing this manual workload through more automated methods as important future work.
- Limitations: The evaluation establishes a monotone mastery gradient but not how closely simulated accuracy matches real student performance.The authors call for grounding and validating simulations against real student response data.
- Future work: The work still lacks direct comparisons with existing student-simulation methods and calls for validation against external IRT statistics and real student data.Future work also includes extending the framework for knowledge-tracing models trained on by-construction diagnostic labels.
- Ablation findings: Taxonomy-conditioned item weighting provides the largest ablation improvement, while distractor routing leaves accuracy unchanged but improves the diagnostic value of incorrect answers.Knowledge-tracing decomposition mainly moves high-mastery profiles away from an unrealistic near-ceiling; effects on lower-mastery profiles are smaller.