Source-linked AI summary
ProIQA: A Process-Based Framework for Fine-Grained Math Item Quality Assessment
Junkai Tong, Mingjia Li, Haoran Chen, Yaoyu Jiang, Hanjie Ge, Yixuan Wang, Hong Qian
TL;DR
Fine-grained IQA remains difficult because existing methods rely on unscalable reviews or stem-focused metrics that miss mathematical reasoning. ProIQA combines stem semantics with structured reasoning trees in a unified assessment framework, reporting gains across concept, difficulty, and competency tasks.
Problem
Existing IQA relies on unscalable manual reviews or shallow stem-based methods that do not capture the reasoning process in mathematical problem-solving.
Source: §I · pp. 1–2Method
ProIQA augments items with LLM-generated hierarchical reasoning trees, encodes their dependencies with GNNs, and fuses solving and stem representations across three dimensions.
Source: §I · p. 2, 2) Deep View · p. 4Results
ProIQA reports average gains over the second-best algorithms of 7.5% in concept assessment, 6.3% in difficulty estimation, and 19.5% in competency assessment.
Source: §I · p. 2Takeaways & Limitations
The framework shares a process-enriched representation across concepts, difficulty, and competencies while preserving task-specific learning objectives.
Source: §I · p. 2Automated verification does not ensure reasoning trees are pedagogically representative, and the evaluation uses small subsets and IRT-estimated labels.
Source: §VI · p. 9
Abstract
from arXiv · showhide
Automatic Item Generation (AIG) is pivotal for personalized education, yet guaranteeing the pedagogical value of generated items remains a bottleneck. Existing Item Quality Assessment (IQA) methods typically rely on unscalable manual reviews or shallow stem-based metrics, failing to capture the reasoning process required for mathematical problem-solving. To bridge this gap, this paper proposes Process-based Item Quality Assessment (ProIQA), a process-aware framework for fine-grained quality assessment of math items. We first formulate IQA across three heterogeneous dimensions, including knowledge concepts, difficulty, and disciplinary competencies, under a unified process-aware perspective. Based on this formulation, we construct a process-enhanced IQA resource by augmenting original item data with structured reasoning trees derived from raw solutions. Technically, ProIQA leverages Large Language Modelsto construct hierarchical reasoning trees and employs Graph Neural Networks (GNN) to encode their topological dependencies and procedural semantics. The resulting solving representation is fused with stem semantics through a dual-view (``Stem + Solving'') architecture, enabling comprehensive assessment across learning objectives. Extensive experiments on K12 mathematical datasets show that ProIQA effectively captures process-oriented features, offering a scalable data-driven solution for evaluating AIG outputs in intelligent education systems.
I. INTRODUCTION
The introduction frames fine-grained IQA as a bottleneck for AIG and argues that stem-only assessment misses mathematical solving processes relevant to item quality. ProIQA combines stem and reasoning-tree representations across three assessment dimensions.
I. INTRODUCTION
- Manual IQA is unscalable, while automated methods often rely on shallow metrics or stem text and struggle with complex open-ended math problems.The introduction identifies manual reviews as labor-intensive, time-consuming, and prone to subjective inconsistency.Source: §I · p. 1
- Experts simulate solutions to identify invoked concepts, required reasoning steps, and mathematical thinking, signals linked to concepts, difficulty, and competencies.The paper argues stem-only methods omit this solving path and may therefore fail to match expert judgment.Source: §I · p. 1
- The paper identifies reasoning-tree construction and procedural-semantic encoding as technical challenges for process-oriented assessment.Mathematical reasoning is hierarchical, and representing its flow and dependencies requires more than standard text encoders.Source: §I · pp. 1–2
- ProIQA uses LLMs to build hierarchical reasoning trees, GNNs to encode their dependencies, and dual-view fusion to combine solving and stem semantics.The framework preserves task-specific objectives for knowledge concepts, difficulty, and disciplinary competencies.Source: §I · p. 2
- The paper formulates IQA across three dimensions and augments item data with structured reasoning trees derived from raw solutions.The resulting process-enhanced K12 resource is presented as reusable for educational data mining and IQA research.Source: §I · p. 2
- ProIQA reports average gains over the second-best algorithms of 7.5% in concept assessment, 6.3% in difficulty estimation, and 19.5% in competency assessment.The authors say experiments across diverse K12 mathematical datasets include comparisons with supervised baselines and advanced LLMs.Source: §I · p. 2
II. RELATED WORK
The related work describes IQA methods spanning human review, response-based approaches, and content-based models, while identifying gaps in automated coverage and process modeling. The formulation section sets up item inputs and a unified process-aware assessment framework.
A. Automated Item Generation
- AIG has shifted toward flexible LLM-based generation, but quality assurance remains a bottleneck because generated content may contain hallucinations or logical flaws.Prior evaluation includes human review and shallow automated metrics such as N-gram, ROUGE-L, and readability scores.Source: A. Automated Item Generation · p. 2
B. Item Quality Assessment
- Prior automated IQA often centers on MCQ difficulty and stem-level classification, leaving solution logic and multi-step reasoning under-modeled.The paper presents integration of process-level information into quality prediction as an open question.Source: B. Item Quality Assessment · p. 2
III. PROBLEM FORMULATION OF IQA
- The formulation targets knowledge coverage, difficulty calibration, and cognitive depth as pedagogical requirements for generated items.The section presents these tasks as a quality-assurance mechanism for AIG systems.Source: §III · p. 2
A. Notations
- The paper defines each evaluated item as a generated math item represented by its textual stem, then maps it to labels across three assessment dimensions.The dimensions are knowledge concepts, difficulty, and disciplinary competencies.Source: A. Notations · p. 3
B. Assessment Tasks
The assessment tasks align learning objectives with the distinct structures of concept matching, difficulty estimation, and competency classification. Difficulty combines broad level classification with fine-grained pairwise comparison.
B. Assessment Tasks
- The unified formulation distinguishes concept coverage, difficulty, and competencies, and notes that relative pairwise comparisons suit difficulty better than absolute regression in adaptive testing.The paper presents task-specific learning objectives as a reusable template for IQA research.Source: B. Assessment Tasks · p. 3
1) Dimension 1 (Concept):
- Concept assessment is formulated as binary item-concept matching, allowing validation against candidate concepts from dynamic lists.A positive prediction indicates that an item aligns with the queried concept.Source: 1) Dimension 1 (Concept) · p. 3
2) Dimension 2 (Difficulty):
- Difficulty assessment combines classification into broad levels with pairwise ranking of item difficulty rather than absolute-value regression.The pairwise labels compare IRT-estimated difficulty parameters, with a margin of 0.5 distinguishing non-ties.Source: 2) Dimension 2 (Difficulty) · p. 4
3) Dimension 3 (Competency):
Competency assessment classifies mathematical items by cognitive domain, while ProIQA models assessment through stem and solving-process views. Its reasoning-tree construction decomposes problems and checks generated trees before downstream use.
3) Dimension 3 (Competency):
- Competency assessment assigns each item to one TIMSS cognitive domain: Knowing, Applying, or Reasoning.The paper frames this classification as distinguishing rote recall from logic and critical thinking.Source: 3) Dimension 3 (Competency) · p. 4
IV. THE PROPOSED PROIQA
- ProIQA has three components: reasoning-tree construction, dual-view representation learning, and task-specific assessment of concepts, difficulty, and competencies.The approach models both surface text and procedural logic to simulate expert evaluation of an item's solution process.Source: §IV · p. 4
A. Reasoning Tree Construction
- An LLM recursively decomposes item stems into interconnected subproblems, with generated trees checked for structural integrity, logical connectivity, and mathematical correctness.Items are unified into constructed-response format before tree generation; each node contains subproblem text and its solution rationale.Source: A. Reasoning Tree Construction · p. 4
1) Surface View:
- A frozen pretrained language model encodes the item stem using the last valid token's hidden state, capturing stated entities, quantities, and constraints but not the solution process.Source: 1) Surface View · p. 4
2) Deep View:
The deep view encodes each reasoning-tree node from its subproblem and rationale, then propagates those features across dependencies to form a tree-level solving representation.
2) Deep View:
- Each node’s subproblem and rationale are PLM-encoded, with their embeddings concatenated into an initial node representation.The hidden state of the last token is used as the sentence embedding.Source: 2) Deep View · p. 4
- A multi-layer GIN aggregates neighboring node features to encode reasoning-step dependencies and distinguish graph structures.After L propagation layers, nodes integrate global structural information.Source: 2) Deep View · p. 4
- Summation pooling produces the tree-level process representation, preserving the magnitude of reasoning complexity.Source: 2) Deep View · p. 5
C. Multi-Dimensional Assessment
The assessment heads use stem and process representations to assess concepts, difficulty, and competencies. The experiments frame comparisons across three assessment tasks and datasets spanning multiple K12 mathematics contexts.
C. Multi-Dimensional Assessment
- Concept assessment concatenates stem, process, and candidate-concept representations, then predicts concept relevance with a thresholded binary classifier.Encoding concept text directly lets the model adapt to dynamic concept sets.Source: C. Multi-Dimensional Assessment · p. 5
- Difficulty assessment uses the fused stem-and-process representation for both continuous difficulty prediction and difficulty-level classification.Source: C. Multi-Dimensional Assessment · p. 5
- Competency assessment predicts categories such as Knowing, Applying, and Reasoning from the process representation, which can be derived directly from solutions when stems are unavailable.Source: C. Multi-Dimensional Assessment · p. 5
V. EXPERIMENT
- The experiment poses questions about comparative performance, process-representation contributions, hyperparameter robustness, and joint assessment.The study also describes a unified case study across the three tasks.Source: §V · p. 5
- Table II reports verification accuracy of constructed reasoning trees across datasets.Source: Table II · p. 5
- The selected datasets span elementary arithmetic, secondary symbolic reasoning, and international competency benchmarks across the K12 mathematics spectrum.Table I reports dataset statistics for the three IQA subtasks.Source: §V · p. 5, Table I · p. 5
A. To RQ1: Multi-Dimensional Assessment Performance
Across concept, difficulty, and competency assessment, ProIQA outperforms the reported baselines, with task-specific results across multiple datasets. The evaluation also reports reasoning-tree validation and a shared cross-validation setup.
A. To RQ1: Multi-Dimensional Assessment Performance
- The experiments curate six datasets across the three tasks and use an 80%–20% train-test split with 5-fold cross-validation, reporting average results.Source: A. To RQ1: Multi-Dimensional… · pp. 5–6
- Reasoning-tree verification accuracy ranges from 90.38% on XES-1600 to 97.80% on Algebra across the four datasets with item stems.The trees are validated through LLM-based automated quality assurance.Source: A. To RQ1: Multi-Dimensional… · p. 6
- On both concept datasets, ProIQA achieves the highest F1 scores, and its reasoning trees expose concept-application steps for alignment with target labels.Concept assessment is multi-label classification; the target concepts are the first two levels of the XES3G5M knowledge routes.Source: A. To RQ1: Multi-Dimensional… · pp. 5–7
- ProIQA outperforms the reported baselines across concept, difficulty, and competency assessment; its process-only variant achieves state-of-the-art competency accuracy.The comparison covers the three assessment tasks, while the competency result uses solution text without reasoning-tree construction or GNN encoding.Source: A. To RQ1: Multi-Dimensional… · pp. 6–7, Table IV · p. 7
- On both difficulty datasets, ProIQA outperforms all baselines, with the highest WACC on Algebra and superior pairwise accuracy on XES-1500.WACC gives partial credit to adjacent-level predictions, while pairwise accuracy evaluates relative difficulty judgments.Source: A. To RQ1: Multi-Dimensional… · pp. 6–7
- For competency assessment, ProIQA’s process-only variant achieves state-of-the-art accuracy without reasoning-tree construction or GNN encoding, as TIMSS lacks item stems; solution-process complexity and logic alone suffice for assessing higher-order competencies.Processes are categorized into Knowing, Applying, and Reasoning.Source: A. To RQ1: Multi-Dimensional… · pp. 6–7
B. To RQ2: Ablation Study
The ablations indicate that fine-grained IQA benefits from solution-process information, hierarchical tree structure, and explicit modeling of relationships among tree nodes.
B. To RQ2: Ablation Study
- Using only item stems yields the lowest performance, indicating that surface text alone is insufficient for fine-grained IQA.The ablation contrasts stem-only input with variants that include solution-process information.Source: B. To RQ2: Ablation Study · p. 7
- Tree-based ProIQA outperforms the linear-solution variant, showing that hierarchical structure captures logical relationships more discriminatively than unstructured solution text.Source: B. To RQ2: Ablation Study · p. 7
- The full-text representation of a reasoning tree degrades performance because it introduces noise instead of extracting core process information.This is the “w/o Graph” variant.Source: B. To RQ2: Ablation Study · p. 7
- Pooling tree nodes without explicit interaction modeling underperforms ProIQA, which models inter-node relationships to integrate global reasoning signals.The comparison is between “w/o GNN” and the full ProIQA system.Source: B. To RQ2: Ablation Study · p. 7
C. To RQ3: Stability Analysis
The stability analysis tests GNN depth and base embedding choice on concept assessment using XES-500. F1 remains stable across depths, while embedding-optimized models outperform a general math model.
C. To RQ3: Stability Analysis
- The analysis uses XES-500 concept assessment as a representative case because of its balanced scale and well-defined concept taxonomy.It examines GNN depth and base embedding model as central process-representation design choices.Source: C. To RQ3: Stability Analysis · p. 7
- F1 scores vary by at most 1.5% from the mean across 1–4 GNN layers on XES-500 concept assessment.Figure 2 reports the depth comparison.Source: C. To RQ3: Stability Analysis · p. 8, Figure 2 · p. 8
- Embedding-optimized models outperform MetaMath-7B, while Qwen3-Embed and Llama-Embed achieve near-identical F1 scores.Figure 3 compares the base models on XES-500 concept assessment.Source: C. To RQ3: Stability Analysis · p. 8, Figure 3 · p. 8
D. To RQ4: Three-dimensional applicability
On a two-item example, ProIQA matches ground truth across concepts, competencies, and relative difficulty, while a zero-shot LLM baseline errs on each dimension. Qualitative analyses identify concept-label drift and difficulty errors tied to labels and tree structure.
D. To RQ4: Three-dimensional applicability
- On the item pair, ProIQA matches ground truth across all three dimensions, while the LLM baseline misclassifies concepts and reverses competency and difficulty judgments.The pair comprises a candle problem and a square perimeter-and-area problem; experts supplied the missing competency annotations.Source: D. To RQ4: Three-dimensional… · p. 8
E. Qualitative Error Analysis
- Concept confusions tend to remain within the same macro-category, indicating preserved hierarchical knowledge structure despite drift between related sub-concepts.A separate error pattern arises when the generated reasoning tree follows a mathematically valid path that maps to a different concept label than the ground truth.Source: E. Qualitative Error Analysis · pp. 8–9
- Difficulty errors cluster at adjacent levels, while annotation noise and flattened reasoning trees can also drive apparent or substantial misclassification.A completing-the-square item labeled Level 4 was rated low difficulty; for Algebra item 341, a shallow tree placed the full solution at the root and ProIQA predicted Level 2 for a Level 5 item.Source: E. Qualitative Error Analysis · p. 9
F. Time Complexity Analysis
ProIQA’s computational complexity is comparable to T-IRT and lower than R2DE, while the discussion identifies scope choices and limitations that qualify its current evidence. It also outlines possible extensions beyond K12 mathematics.
F. Time Complexity Analysis
- ProIQA and T-IRT have comparable complexity, both substantially lower than R2DE, with ProIQA’s accuracy gains achieved without substantial additional overhead.ProIQA’s dominant cost is GNN encoding; T-IRT fine-tunes BERT, while R2DE uses TF-IDF features and a random forest.Source: F. Time Complexity Analysis · p. 9
VI. DISCUSSION
- The process-representation paradigm could extend beyond K12 mathematics to domains with decomposable solution paths and to additional quality dimensions.The paper names physics, chemistry, biology, and programming, as well as item clarity and normativity, as possible extensions.Source: §VI · p. 9
- The task formulation uses difficulty alone because the paper treats it as population-invariant, whereas discrimination estimates from a single-grade, single-platform cohort are less reliable general-purpose labels.Source: §VI · p. 9
- Current evidence is bounded by limitations in tree representativeness and structure, frozen language-model representations, and benchmark coverage and scale.Over 90% of trees pass logical-coherence verification, but pedagogical representativeness is not guaranteed; tree structures may miss cross-branch analogy or iterative refinement. TIMSS lacks item stems, limiting competency assessment to ProIQA’s process view, and small subsets with IRT-estimated labels motivate larger expert-annotated benchmarks.Source: §VI · p. 9