SRJudge: Empowering Large Language Models with Selective Reasoning for Fine-Grained Knowledge Concept Tagging

1 Introduction

• Knowledge concepts support educational tasks such as knowledge tracing, cognitive diagnosis, and intelligent recommendation, and accurate indexing requires interpreting questions and solutions.The paper frames tagging as identifying appropriate concepts from semantic and logical information in educational content.Source: §1 · p. 1
• SLMs often include the correct concept in their top-K shortlist but have limited top-1 accuracy, while LLMs still struggle with large candidate sets and similar concepts.The paper attributes the SLM top-1 limitation to semantic similarity among top-K candidates.Source: §1 · pp. 1–2
• SRJudge consistently outperforms existing methods on datasets from three disciplines, supporting selective reasoning with reinforcement learning for concept tagging.The paper also reports constructing S_Bio and S_Phy for method evaluation.Source: §1 · p. 2
• SRJudge first narrows concepts to a shortlist with a fine-tuned BERT Selector, then uses a lightweight LLM Reasoner and an LLM Judger to assess recommendations and reasons.The Reasoner is trained with pruning-based optimization and a task-specific reward that includes a dynamic position reward.Source: §1 · p. 2

2.1 Traditional Knowledge Concept Tagging

• Traditional tagging uses feature engineering, contextual models, and graph structures, but has limited capacity to capture effective features and reason about similar concepts in fine-grained settings.Examples include SVM, RNN, graph-based methods, and BERT-based feature fusion.Source: §2.1 · p. 2

2.2 LLM-based Tagging

• LLM-based tagging explores prompting, chain-of-thought, and multi-agent collaboration, while prompt reasoning may not align with human preferences and fine-tuned LLMs can be costly.Prior approaches also report human preference for LLM-generated concepts and improved consistency through independent agent subproblems.Source: §2.2 · pp. 2–3

3.1 Problem Statement

• Knowledge concept tagging maps an exercise to a predicted label from the set of candidate concepts.The candidate set’s size is represented as |Φ|.Source: §3.1 · p. 3

3.2 Selector for Scope Reduction

• A fine-tuned lightweight Selector ranks concepts by predicted probability and passes the top K as the Reasoner’s reduced decision space.The Selector uses continued masked-language-model pretraining and supervised fine-tuning.Source: §3.2 · p. 3

3.3 Reasoner for Concept Recommendation

• GRPO trains a lightweight LLM to choose a concept and justify it from the Selector’s K candidates, rewarding output format, correctness, and shortlist position.The Reasoner treats the shortlist as its decision space and produces both a recommended concept and reasons.Source: §3.3 · pp. 3–4
• A dynamic position reward favors correct answers ranked below first, while pruning removes low-absolute-advantage completions before policy updates.The position reward applies when the correct answer ranks 2nd through Kth; pruning retains completions that meet an absolute-advantage threshold.Source: §3.3 · p. 4

3.4 Judger for Overall Assessment

• A frozen LLM Judger evaluates the Selector’s and Reasoner’s predictions and the Reasoner’s explanations to make the final decision.Its prompt includes the exercise, candidate concepts, both model predictions, and the Reasoner’s rationale.Source: §3.4 · pp. 4–5

4.1 Datasets and Baseline Methods

• Experiments use S Math, S Bio, and newly constructed S Bio and S Phy datasets, with stratified 8:1:1 splits and comparisons spanning SLM, LLM, and task-specific baselines.The datasets include 23,596, 15,669, and 32,138 unlabeled instances for U Math, U Bio, and U Phy, respectively.Source: §4.1 · p. 5

4.2 Overall Results and Analysis

• SRJudge achieves the best F1 across S Math, S Bio, and S Phy—0.7602, 0.6987, and 0.6643, versus LGCEL’s 0.7311, 0.6840, and 0.6476.The comparison reports the leading baseline’s F1 scores for each dataset alongside SRJudge’s.Source: §4.2 · p. 5, Table 3 · p. 6

4.3 Ablation Study

• Ablations report the best performance for the full framework; adding the Reasoner improves F1 by 2.02%, 0.96%, and 1.92% on S Math, S Bio, and S Phy, while the Judger further boosts all three.The Selector’s continued pretraining is credited with improving domain-specific contextual and terminology capture; larger Judgers also enhance final performance.Source: §4.3 · p. 6, Table 4 · p. 7
• Dynamic position rewards outperform no position reward, help prevent overfitting to Selector outputs, encourage exploration of other candidate positions, and improve RL efficiency.The dynamic reward also yields additional improvements over a static reward as training progresses.Source: §4.3 · p. 6, Table 5 · p. 7
• Across pruning rates, F1 first rises and then declines while remaining above no pruning, whereas training time decreases as pruning increases.The authors attribute the efficiency gains to filtering low-quality completions with marginal positive advantages, reducing noisy feedback and gradient updates.Source: Table 6 · p. 7, §4.3 · p. 7

4.4 Analysis on Hard-to-Distinguish Samples

• On hard-to-distinguish samples, average F1 improvements are 8.84% on S Math, 2.31% on S Bio, and 3.24% on S Phy; gains are relatively limited on the more complex biology and physics concepts.Difficulty is proxied by Selector category F1 below 0.5, with five intervals from [0, 0.1] through [0.4, 0.5].Source: Figure 3 · p. 7, §4.4 · p. 7

• SRJudge enhances LLMs with reinforcement-learning-based selective reasoning to distinguish fine-grained knowledge concepts among numerous candidates.Source: §5 · p. 7
• The paper constructs two high-quality datasets to support follow-up research.Source: §5 · p. 7
• Experiments across datasets demonstrate SRJudge’s effectiveness for knowledge concept tagging.Source: §5 · p. 7
• The authors describe their work as the first to customize reinforcement learning for improving LLM reasoning over fine-grained knowledge concepts.Source: §5 · p. 7

• The authors report no ethical issues.Source: Ethical Statement · p. 8