PaperlayerView exact revision on arXiv

Source-linked AI summary

Embedding-Based Educational Taxonomies

Kia Karbasi, Kaveh Karbasi, Gregory Pottie

arXiv:2610.33694v1cs.CLcs.AIcs.LG

TL;DR

This paper asks whether semantic embeddings can construct useful alternatives to expert-developed educational taxonomies, which organize questions into knowledge components for educational applications. It clusters math-question embeddings, compares the resulting taxonomies with existing ones, and evaluates them in DKT. The induced taxonomies show substantial but incomplete correspondence with the originals and yield higher test AUC in both datasets, though the best reported gains are selected from tested combinations and evaluated on new students answering questions used to build the taxonomy.

  • Problem

    Expert-developed taxonomies organize educational content, while computational alternatives derived from content embeddings have not been established here as comparable structures for downstream knowledge tracing.

    Source: §1 · pp. 1–2

  • Method

    The study embeds math-question text, reduces and clusters the representations into taxonomies, compares them with original KC mappings, and evaluates relabeled sequences in DKT.

    Source: §1 · p. 2

  • Results

    The best tested induced taxonomies improve test AUC by .0474 in ASSISTments 2012 and .0272 in ASSISTments 2017, while closer agreement with the original is not consistently associated with higher AUC.

    Source: §5 · p. 7, §7 · p. 8

  • Takeaways & Limitations

    Embedding-based taxonomies can complement expert-developed systems by offering alternative groupings for expert review, but their educational value still requires expert review and evidence from students.

    Source: §5 · p. 7, §7 · p. 8

    The reported gain is the best test-AUC result across tested combinations, and test students answer questions used to build the taxonomy rather than entirely new questions.

    Source: §8 · p. 9

Abstract

from arXiv · show

Educational taxonomies organize educational content, such as exercise problems, into concepts, which are also called skills or knowledge components. These taxonomies define learning objectives in educational applications such as intelligent tutoring systems. Developed through human expertise and iterative refinement, they also provide fundamental structure for curricula, assessment, and models of student learning. Recent advances in semantic text embeddings make it possible to construct alternative educational taxonomies computationally from the content itself. In this paper, we investigate this possibility by clustering embeddings of math questions to construct embedding-based educational taxonomies. We compare these taxonomies with the existing Common Core State Standards (CCSS)-based knowledge-component taxonomies in two ASSISTments datasets and evaluate their utility in Deep Knowledge Tracing, a downstream task that relies on taxonomic mappings of questions to knowledge components. The embedding-based taxonomies exhibit substantial but incomplete correspondence with the existing taxonomies, revealing both preserved and alternative organizations of educational content. When used for knowledge tracing, they improve predictive performance by approximately 3-5 AUC percentage points. These results demonstrate the potential of semantic embeddings for automatically constructing educational taxonomies that both recover structure present in human-developed taxonomies and provide useful alternative representations for downstream educational modeling.

1 Introduction

Educational taxonomies organize questions into knowledge components that structure curricula, assessment, and student-learning models. This paper constructs text-embedding-based taxonomies of math questions, compares them with expert-developed taxonomies, and tests them in knowledge tracing.

1 Introduction

  • These taxonomies matter because question-to-KC mappings organize learning objectives and provide structure for curricula, assessment, educational applications, and models of student learning.Expert-developed systems encode decisions about which distinctions matter, but are not direct evidence that knowledge has one uniquely correct organization.Source: §1 · p. 1
  • Text embeddings represent question text numerically, allowing language and meaning similarities to be compared and nearby questions to be clustered into categories.Recent advances in embeddings make it possible to organize educational content using patterns in the content itself.Source: §1 · pp. 1–2
  • The paper clusters semantic embeddings of math-question text to create alternative mappings from questions to knowledge components and compares those mappings with the original taxonomies.It also evaluates their utility in a downstream task that relies on question-to-KC mappings.Source: §1 · p. 2
  • The study asks how closely induced and original taxonomies correspond, how correspondence changes with category count, and which taxonomy yields higher DKT AUC.DKT uses question-response sequences and assigned KC labels to estimate student knowledge and predict later performance.Source: §1 · pp. 1–2

2 Related Work

The related work frames expert taxonomies as useful educational structures rather than uniquely correct accounts of knowledge. It positions embedding-based clustering as an alternative to approaches that add text features while retaining existing skill labels.

2.1 Educational Taxonomies and Knowledge Components

  • Expert-developed knowledge components connect related tasks to proposed units of knowledge, but do not directly measure how students think or establish one uniquely correct grouping.Learning Factors Analysis uses student performance to evaluate and refine expert KC models; this study instead constructs alternative categories from question content.Source: §2.1 · p. 2

2.2 Embedding-Based Clustering and Automated KC Discovery

  • Prior clustering studies construct educational categories from course or question content, compare them with expert categories, and evaluate them using AFM or knowledge-tracing models.SMART and KCluster do not use ASSISTments or DKT; Moon et al. evaluate extracted KCs with knowledge-tracing models including DKT.Source: §2.2 · pp. 2–3
  • This study compares induced and original taxonomies across tested PCA dimensions and category counts, evaluates induced mappings in DKT, and examines whether closer matches tend to have higher AUC.The core methodology was developed independently of SMART and KCluster, which the authors identified as related approaches during later literature review.Source: §2.2 · p. 3

2.3 Knowledge Tracing and Text-Aware Student Modeling

  • Unlike text-aware student models that retain existing skill labels, this approach uses embeddings first to define alternative question categories, then trains DKT with those labels.The mapping is assessed by its correspondence to the original taxonomy and the test AUC achieved by DKT.Source: §2.3 · p. 3

3 Method

The method builds one-level question-to-category mappings from text and evaluates them by taxonomy comparison and DKT prediction. Student records, responses, sequence order, and evaluation splits are held fixed when original KC labels are replaced.

3 Method

  • The method derives a one-level mapping from question text, then replaces dataset-provided KC labels in DKT while leaving student-answer records and their order unchanged.The relabeling illustration distinguishes the conceptual mastery depiction from the actual evaluation procedure.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Data and Building the Taxonomies

  • The analyses use text-linked subsets of ASSISTments 2012 and 2017; the 2012 subset has 32,792 questions and 162 KCs, while the 2017 subset has 638 questions and 68 KCs.The 2017 placeholder category noskill covers 142 of the 638 text-linked questions (22%).Source: §3.1 · p. 3
  • Cleaned problem-body text is embedded, reduced with PCA, and clustered by k-means; original KC labels and student responses are not used to construct the taxonomies.Each question receives one label in a flat taxonomy, built from all linked question texts before the student split; held-out test students are not unseen questions.Source: §3.1 · p. 3

3.2 Taxonomy Comparison

  • Taxonomy comparison uses measures of category concentration, grouping similarity, and category sizes, interpreted together because scores can change with the number of induced categories.The measures describe how categories stay together, split, or mix; educational usefulness requires separate evidence.Source: §3.2 · p. 4

3.3 DKT Evaluation

  • For DKT, the study replaces question-to-KC mappings while keeping questions, responses, and sequence positions fixed; changing K can also change the number of labels and associated model parameters.Separate Bayesian searches select DKT runs by validation AUC; test AUC is then reported, with 20% of students reserved for final testing.Source: §3.3 · p. 5
  • The association analysis averages structural scores and test AUC across two embedding models at each of 48 PCA/K settings, then uses Spearman correlations with adjustment for PCA/K rank patterns and multiple comparisons.The analysis tests associations, not causal effects.Source: §3.3 · p. 5

4 Results

The induced taxonomies preserve some structure of the original KC taxonomies, but correspondence varies across datasets and category counts. DKT test AUC is higher with the best tested induced taxonomy in both datasets, while closer taxonomy agreement is not consistently associated with higher AUC.

4.1 RQ1: Comparing the Original and Induced Taxonomies

  • Homogeneity, completeness, and NMI are .930, .717, and .809 for ASSISTments 2012, versus .681, .509, and .582 for ASSISTments 2017 at the reported settings.Neither setting has the highest ARI or NMI across all tested settings, so the DKT settings are not simply those with the closest match.Source: §4.1 · p. 5
  • Large original 2012 categories are often distributed across several induced categories, potentially reflecting finer distinctions, recurring wording or templates, or use of more categories.The analysis does not establish whether these divisions matter for teaching.Source: §4.1 · p. 5

4.2 RQ2: Effect of the Number of Categories

  • As K increases, 2012 homogeneity generally rises while completeness falls; in 2017 both averages generally rise, while inverse purity and pairwise recall decline.The measures capture different effects of increasing category count and do not indicate improvement in every aspect of the match.Source: §4.2 · p. 5

4.3 RQ3: Taxonomies and DKT Performance

  • After accounting for PCA dimension and K and correcting for 15 comparisons, no 2012 correlations between structural measures and AUC remain statistically significant.In 2017, several closer-agreement measures are associated with lower AUC, while larger variation of information is associated with higher AUC; these correlations do not establish causation.Source: §4.3 · pp. 5–6
  • .0474 in ASSISTments 2012 and .0272 in ASSISTments 2017 are the gains in test AUC for the best observed induced taxonomies over the original-taxonomy baselines.For 2012, test AUC rises from .7367 to .7841; for 2017, it rises from .7131 to .7403. Text-embedding-3-large exceeds the original baseline at all 48 tested PCA/K settings in each dataset.Source: §4.3 · p. 5, Table 2 · p. 6

5 Discussion

The induced taxonomies align more closely with the original KC structure in ASSISTments 2012 than in 2017, but the analysis cannot determine what explains their differences. They improve DKT prediction in both datasets, while the results do not establish educational value or which taxonomy properties drive performance.

5 Discussion

  • Question-text patterns align more closely with the original KC structure in ASSISTments 2012, while original categories are split across induced categories in both datasets.The splits may reflect mathematical content, recurring wording, or the use of more induced categories; the analysis cannot distinguish these explanations.Source: §5 · p. 6
  • Figure 3 shows homogeneity rising while completeness falls in 2012, whereas both generally rise in 2017 as K increases.In 2017, inverse purity and pairwise recall decline over the same range, so the measures capture different aspects of correspondence.Source: Figure 3 · p. 7
  • Test AUC exceeds the original-taxonomy baseline at every displayed PCA/K setting for text-embedding-3-large in both datasets.Figure 4 shows the setting-level comparison; the original 2017 taxonomy includes the nonsubstantive noskill placeholder, and its effect on the AUC difference is not isolated.Source: Figure 4 · p. 7, §5 · p. 7
  • The observed AUC difference compares each full induced mapping with the original mapping, and the experiment does not isolate the contribution of the 2017 noskill redistribution.The original 2017 noskill category covers 142 of 638 questions (22%) despite grouping by absence of an assigned KC, not shared content.Source: §5 · pp. 6–7
  • AUC measures prediction rather than educational value, and these results do not identify which taxonomy property drives the gain.The paper treats induced taxonomies as complements to expert-developed systems; judging whether their distinctions matter for teaching requires evidence from educators, student behavior, and solution steps.Source: §5 · pp. 7–8

6 Future Work

Future work proposes testing induced taxonomies on richer datasets and in additional KC-based educational tasks. These settings could support multiple labels or hierarchical relations among categories.

6 Future Work

  • Future work proposes datasets combining student interactions, question text, and more detailed KC structures, including XES3G5M and DBE-KT22.XES3G5M contains 5.5 million responses and DBE-KT22 provides university database-course text and response data.Source: §6 · p. 8
  • These datasets could support induced taxonomies with multiple labels per question or hierarchical relations among categories.The current ASSISTments datasets would remain useful for evaluating models that explicitly support these richer structures.Source: §6 · p. 8
  • The paper proposes evaluating induced taxonomies in other KC-based tasks, including progress measurement, identifying student strengths and weaknesses, and recommending practice questions.The cited examples include measuring progress with practice, identifying relative strengths and weaknesses, and recommending the next practice question.Source: §6 · p. 8

7 Conclusion

The paper constructs taxonomies by embedding math-question text and compares them with the original KC taxonomies in ASSISTments 2012 and 2017. The induced taxonomies preserve some original structure while improving DKT test AUC, but educational value still requires review and student evidence.

7 Conclusion

  • Embedding-based taxonomies preserve much, but not all, of the original structure, with closer agreement in 2012 than in 2017.Closer agreement with the original taxonomy is not consistently associated with higher AUC across tested settings.Source: §7 · p. 8
  • DKT test AUC improves by .0474 in 2012 and .0272 in 2017 with the induced taxonomy having the highest observed test AUC.These gains distinguish predictive usefulness from similarity to the expert-developed taxonomy.Source: §7 · p. 8
  • Embedding-based taxonomies offer a promising alternative for student models, but their educational value still requires expert review and evidence from students.The conclusion separates similarity to an expert taxonomy from usefulness for prediction.Source: §7 · p. 8

8 Limitations

The limitations constrain the findings to text-linked ASSISTments questions and to the tested taxonomy-construction and DKT settings. The comparison and evaluation do not establish the instructional value of induced categories.

8 Limitations

  • Findings cover only ASSISTments 2012 and 2017 questions with linked text from one platform, so they may not extend to other educational settings.The examples also show that converting web pages to plain text can remove mathematical layout and diagrams; images are not supplied to the embedding models.Source: §8 · p. 8, A Question Text Inputs and Examples · p. 10
  • Each question receives one induced category in a flat taxonomy, which may group by wording, formatting, answer type, or source rather than mathematical knowledge.The method does not allow multiple KCs per question, represent broad-to-narrow KC structure, or model student reasoning.Source: §8 · p. 8
  • Agreement scores measure grouping similarity, not educational value; without solution-step analysis, boundary inspection, or educator judgments, instructional value remains unknown.Purity and homogeneity can rise simply as the number of categories increases, and overall scores do not identify which splits or mergers matter for instruction.Source: §8 · p. 8
  • Results may depend on the two embedding models, single clustering method and seed, tested PCA dimensions and K values, and the distribution of question templates and cluster sizes.PCA is the only dimensionality-reduction method tested, and the study does not directly benchmark against SMART or KCluster.Source: §8 · p. 8
  • The DKT evaluation uses one architecture, tunes it separately for each taxonomy, and lacks size-matched random-label baselines to separate grouping effects from category counts, label frequencies, or model size.The best induced AUC is selected across tested embedding, PCA, and K combinations using test AUC; test students answer questions used to build the taxonomy, not entirely new questions.Source: §8 · pp. 8–9

A Question Text Inputs and Examples

  • Question-text examples use cleaned problem-body text without grade information, original category labels, or images, and plain-text conversion can remove mathematical layout and diagrams.Examples span elementary through high-school mathematics, and dataset grade labels do not necessarily match students’ enrolled grades.Source: A Question Text Inputs and Examples · p. 10

B Taxonomy and DKT Settings

The appendix specifies how question embeddings are reduced and clustered into induced labels, then describes how DKT settings are selected and evaluated under shared data-splitting and training conditions.

B Taxonomy and DKT Settings

  • The two embedding models produce default vectors of 1,536 and 3,072 dimensions, with requested outputs of 768 and 1,536 dimensions before PCA and k-means assignment.PCA dimensions tested are 10, 20, 40, 60, 80, 100, 120, and 160; K ranges from 50 to 300 for 2012 and 20 to 150 for 2017.Source: B Taxonomy and DKT Settings · pp. 10–11
  • Each question’s PCA-reduced vector is assigned the label of its nearest k-means centroid, with one induced category label per question.The clustering seed is 42; each setting uses 10 initializations and at most 300 iterations.Source: B Taxonomy and DKT Settings · pp. 10–11
  • The relabeling replaces each response’s original KC label with the induced question label while retaining its question and correctness response.Induced labels are written to skill_id for 2012 and as text in skill for 2017, then pyKT renumbers them as consecutive concept IDs.Source: B Taxonomy and DKT Settings · p. 11
  • For each baseline or induced taxonomy, a separate Bayesian search maximizes validation AUC, selects the completed run with the highest validation score, and reports its test AUC.The embedding model, PCA dimension, and K stay fixed outside each search; 2012 includes seed among search settings, while 2017 fixes it at 42.Source: B Taxonomy and DKT Settings · p. 11
  • All searches share student splits, fold, training procedure, candidate settings, trial count, and selection rule, with an approximately 64/16/20 train-validation-test split.Training uses Adam, batch size 256, at most 200 epochs, and sequence length 200, stopping after 30 epochs without a validation AUC improvement of at least .001.Source: B Taxonomy and DKT Settings · p. 11

C Measures for Comparing the Taxonomies

The appendix compares original and induced taxonomies with multiple agreement measures and separate cluster-size summaries. Because category count can shift scores in different directions, the measures are interpreted together.

C Measures for Comparing the Taxonomies

  • At the two reported PCA/K settings, average cluster-size spread is .591 for 2012 and .447 for 2017, while the largest-cluster question shares average 1.28% and 2.51%, respectively.No model puts more than 3% of questions in its largest cluster; mean cluster sizes are 62.45 in 2012 and 6.38 in 2017.Source: C Measures for Comparing the Taxonomies · pp. 11–12
  • Homogeneity asks whether induced categories mostly contain one original category, while completeness asks whether each original category remains concentrated in one induced category.The score set should be interpreted together because increasing K can raise some measures and lower others.Source: Table 3 · p. 11, Table 5 · p. 12
  • Agreement is summarized with ARI, AMI, NMI, FM, and VI across 48 settings averaged over two embedding models, with lower VI and higher values of the other measures indicating closer matches.NMI and V-measure have the same values and reach their highest values at the same setting; the passage does not provide the numeric table cells.Source: Table 4 · p. 12
  • Figure 5 maps the 25 largest original 2017 categories across induced categories at text-embedding-3-large, PCA=10, and K=100, showing counts and within-category shares.Only the three induced categories containing the most questions from each original category are shown; the remainder are combined as Other, and category identifiers are arbitrary.Source: Figure 5 · p. 13
  • Across both datasets, increasing K raises homogeneity, global purity, and pairwise precision, but lowers global inverse purity and pairwise recall.In 2012, completeness falls and VI rises; in 2017, completeness rises and VI falls.Source: Table 6 · p. 13
  • Figure 6 separates ARI and NMI by embedding model and dataset, while Figure 7 plots homogeneity and completeness across K for each model/PCA combination.In 2012, homogeneity generally rises as completeness falls; in 2017 both generally rise, even as global inverse purity and pairwise recall fall from the smallest to largest K.Source: Figure 6 · p. 14, Figure 7 · p. 15

D Relation Between Taxonomy Agreement and AUC

The analysis tests whether closer taxonomy agreement is associated with higher test AUC across matched PCA/K settings. It also removes rank patterns linked to PCA dimension and K and corrects for multiple comparisons.

D Relation Between Taxonomy Agreement and AUC

  • The study compares taxonomy-agreement scores and test AUC across 48 matched PCA/K settings per dataset, averaging each measure across the two embedding models.Three additional 2017 AUC settings are excluded because they lack matching taxonomy-comparison scores.Source: D Relation Between Taxonomy Agreement… · p. 16
  • Spearman correlations are computed before and after removing rank patterns linked to PCA dimension and K, with Benjamini–Hochberg correction applied separately by dataset and analysis stage.The correction covers 15 measures.Source: D Relation Between Taxonomy Agreement… · p. 16
  • Figure 8 displays correlation coefficients, and Table 7 marks significance after correction; for VI, a positive coefficient means greater taxonomy difference is associated with higher test AUC.“After” in Figure 8 denotes removal of rank patterns linked to PCA dimension and K.Source: Figure 8 · p. 16, Table 7 · p. 17
Loading 2610.33694v1…