Source-linked AI summary
A Dual-Dimensional LLM Framework for Automated Item Incidental Content Similarity Analysis in Large-Scale Assessments
Jing Huang, Jihong Zhang, Hua-Hua Chang
TL;DR
Large-scale automatic item generation creates a need to detect repetitive construct-irrelevant content that conventional similarity metrics do not fully represent. This paper proposes an LLM-based dual-dimensional AISA framework combining structural decomposition and semantic relatedness, and validates it in CAT. LLM similarity aligned more closely with psychometrically meaningful local dependencies than BLEU and cosine similarity.
Problem
Existing similarity methods do not jointly capture the structural and semantic dimensions of incidental content redundancy, limiting fine-grained measurement for large-scale assessment.
Method
The study combines LLM-based structured decomposition and semantic relatedness into a weighted similarity framework and applies it to cluster-constrained CAT item selection.
Results
r=0.50 was observed for LLM similarity against residual correlations, compared with essentially zero for BLEU and cosine similarity.
Takeaways & Limitations
LLM-powered AISA can support interpretable incidental-content analysis and content-aware item selection in CAT.
Takeaways & Limitations
Current LLM-assisted item-analysis approaches treat generated components holistically rather than directly distinguishing structural and semantic incidental-content dimensions.
Abstract
from arXiv · showhide
The rapid expansion of large-scale assessments and the growing adoption of automatic item generation have intensified concerns about incidental content redundancy, where construct-irrelevant elements such as wording or contextual framing become unintentionally repetitive across items. Traditional similarity metrics like BLEU or cosine similarity, often fail to capture the nuanced structural and semantic layers that drive perceived redundancy simultaneously. This study proposes a dual-dimensional framework for Automated Item Similarity Analysis (AISA) powered by Large Language Models (LLMs), operationalizing similarity through Structured Decomposition and Semantic Relatedness. Psychometric validation indicates that LLM-derived metrics align more closely with indicators of construct-irrelevant local dependence and yield more coherent item parameter groupings than traditional text-based measures. The framework is further evaluated through its application in Computerized Adaptive Testing (CAT). Simulations reveal that incorporating LLM-based similarity constraints into item selection improves estimation stability and reduces bias with minimal efficiency trade-offs, outperforming constraints based on conventional metrics. These findings highlight the potential of LLM-powered AISA to support scalable bank curation, content-aware test assembly, and experience-sensitive adaptive testing across diverse assessment contexts.
Introduction
The shift toward LLM-supported automatic item generation increases the need to detect incidental content redundancy at scale. The proposed framework combines structural and semantic analysis to support item-bank curation and CAT while preserving content diversity.
- Motivation: Automatic item generation increases item quantity but introduces quality-assurance challenges involving repetitive construct-irrelevant scenarios, phrasing, and linguistic structures.Such redundancy may contribute to practice effects, test-taking fatigue, and threats to score-interpretation accuracy.
- Limitations of existing metrics: BLEU, ROUGE, and METEOR capture lexical overlap, whereas cosine similarity and related contextual metrics capture broader relationships but do not jointly assess structure and semantics.Existing methods therefore provide fragmented coverage of incidental content.
- LLM opportunity: LLMs offer contextual understanding that can separate surface form from deeper meaning and evaluate semantic coherence and logical structure at scale.This makes them a candidate alternative to human review for comprehensive item similarity analysis.
- Proposed contribution: The study proposes an LLM-powered framework that quantifies item similarity through integrated structural and semantic analysis.The framework is intended for automatic item-generation quality control, item-pool diagnostics, and test assembly.
- CAT application: In CAT, LLM-derived similarity is integrated into cluster-constrained item selection to pursue psychometric precision and content diversity together.This application extends beyond categorical content balancing and exposure control by incorporating semantic relationships between items.
Background
Incidental content shapes item presentation without defining the target construct, yet its repetition can affect examinee experience and psychometric quality. Existing automated measures offer either narrow lexical or opaque semantic views, motivating a multidimensional LLM framework evaluated in CAT.
- Incidental content: Incidental content shapes presentation and contextual framing rather than the target construct, and can include scenarios, item formats, sentence structures, and language complexity.It is distinct from targeted content and inappropriate content such as cultural or gender bias.
- Test-taker experience: Repeated incidental features such as money scenarios or family contexts may produce cognitive fatigue, reduced novelty, and perceived redundancy during testing.These effects are especially relevant where contextual variety is important, including reading comprehension.
- Psychometric implications: High semantic similarity can compromise measurement quality through associations with lower discrimination, locally correlated residuals, and unintended item-context error.These findings connect incidental overlap to both item functioning and model fit.
- CAT challenge: CAT makes redundancy control more difficult because real-time algorithms prioritize psychometric efficiency and may select semantically repetitive items without explicit safeguards.Unlike linear forms, CAT cannot rely solely on manual curation and distribution of items.
- Need for automation: Expert judgment, crowdsourcing, ablation studies, and post hoc response analyses are informative but labor-intensive and difficult to scale for large item pools.AISA addresses this need by assigning similarity scores directly from item content for curation and CAT constraints.
- Prior AISA methods: Lexical metrics detect near-duplicate text but remain structurally narrow, while embedding-based measures identify conceptual proximity yet conflate style with meaning and lack interpretable structural measures.The latter also remain inconsistent for fine-grained logical relationships in test items.
- Research gap: Current LLM-assisted approaches handle decomposition and similarity judgment but treat components holistically rather than distinguishing structural and semantic dimensions of incidental content.Consequently, they do not directly provide the fine-grained multidimensional measurement needed to manage redundancy.
- Study focus: This study proposes an interpretable multidimensional LLM framework and evaluates its measurement and operational utility in CAT through questions progressing from conceptualization to validation and application.The CAT setting is treated as a representative, methodologically demanding context.
Methods
The method represents incidental similarity through structural decomposition and semantic relatedness, combines them with domain-defined weights, and uses the resulting scores for validation and CAT item-selection constraints. The empirical testbed is the ECR short form, analyzed with psychometric models, clustering, and similarity comparisons.
- Framework definition: Incidental content is operationalized as construct-irrelevant but appropriate content that shapes item presentation and contextual framing rather than the target construct.The framework treats this content as multidimensional rather than monolithic.
- Structured decomposition: Structured decomposition captures item format, sentence structure, and other surface-level organization patterns that may increase perceived redundancy.It focuses on how content is organized and expressed.
- Semantic relatedness: Semantic relatedness captures overlapping themes or contexts, lexical meaning proximity, and shared connotations or emotional tone.These dimensions assess conceptual overlap beyond surface wording.
- Composite similarity: S_ij = w_1S1_ij + w_2S2_ij combines structured and semantic similarity, with domain-defined weights satisfying w_1 + w_2 = 1.The weighting can emphasize presentation diversity, semantic coverage, or a balance of both.
- LLM scoring: The LLM generates dimension-level scores from 0-10 and justifications for item pairs, producing S_ij, S1_ij, and S2_ij ranges of 0 to 30.The study sets w_1 = 0.3 and w_2 = 0.7, while allowing adjustment by application.
- Empirical testbed: The ECR short form provides a 36-item Likert-type empirical testbed with 51,491 participants, reduced to 35,278 valid participants after cleaning.The data preparation removed out-of-range ages and incomplete responses and recoded reverse-scored items.
- Psychometric modeling: A graded response model estimated item parameters, with discrimination constrained to α≥0.1 and thresholds constrained to b∈[-10, 10].Estimation used MHRM, and latent traits were obtained by EAP estimation.
- Validation: LLM similarity was validated against BLEU and cosine similarity using residual correlations and coherence of psychometric characteristics within similarity-based clusters.The validation examines both criterion-related correspondence and interpretability of item groupings.
Results
The LLM framework separates item similarity into structural decomposition and semantic relatedness, revealing both general alignment and meaningful differences across item pairs. Its similarity-based clusters show stronger psychometric coherence and support CAT selection with lower bias and greater stability than conventional constraints.
- Dual-dimensional similarity: The total LLM similarity score correlated strongly with semantic relatedness (r=0.98) and moderately with structured decomposition (r=0.76), reflecting the study’s weighting scheme.The total score ranged from 6 to 26.60 (M=15.41, SD=4.02), with weights w1 = 0.3 and w2 = 0.7.
- Dual-dimensional similarity: S1 and S2 were generally aligned but retained distinct information, with a Spearman correlation of 0.66 and most pairwise differences within [-5,5].Larger discrepancies identified item pairs whose surface structure and deeper semantic content diverged.
- Dual-dimensional similarity: For Items 13 and 30, structural similarity was high (S1 = 16) but semantic relatedness was low (S2 = 6) because their relationship concerns, emotions, and desired outcomes differed.Both items shared declarative format and an “I [emotion verb] when [partner situation]” structure, but had minimal lexical overlap and opposing relational perspectives.
- Psychometric validation: LLM-derived similarity aligned more strongly with residual correlations (r=0.50) than BLEU (r=-0.10) or cosine similarity (r=0.06).The result indicates closer alignment with psychometrically meaningful local item dependencies and incidental content similarity.
- Psychometric validation: LLM-based clustering produced balanced, differentiated groups, with within-cluster similarity means of 19.2 to 22.0 versus a global mean of 15.41.BLEU- and cosine-based approaches instead assigned a large proportion of items to one dominant cluster.
- CAT evaluation: The LLM-based SC-MFI condition achieved the lowest average bias (M=0.029), while its RMSE was higher (M=0.115) but more concentrated across replications.Unconstrained MFI had the lowest mean RMSE (M=0.106), whereas BLEU- and cosine-constrained methods had intermediate RMSE levels with wider dispersion.
Discussion and Conclusions
The study frames incidental content similarity as a dual-dimensional construct and demonstrates an LLM-powered framework for scalable monitoring, item-bank curation, content-aware assembly, and CAT. Its psychometric and simulation evidence supports stronger alignment and more stable adaptive testing, while validation remains limited by model, item-bank, prompt, and weighting choices.
- Discussion and Conclusions: The framework distinguishes surface-level structural similarity from deeper semantic relatedness, addressing the multifaceted nature of incidental content.
- Discussion and Conclusions: LLM-generated similarity scores showed meaningful alignment between structural and semantic dimensions while preserving substantive differentiation.The correlation between structural and semantic similarity was r= 0.66.
- Discussion and Conclusions: LLM-based similarity supported psychometrically coherent item clusters, with stronger correspondence to incidental local dependencies than BLEU and cosine.The LLM-based clusters also showed coherent patterns in item discrimination and threshold parameters.
- Discussion and Conclusions: The framework enables scalable, fine-grained monitoring for item-bank curation and quality control, including detection of superficially cloned items.The S1 and S2 metrics can be applied during pre-testing to identify and filter redundant items.
- Discussion and Conclusions: In CAT, LLM-based similarity constraints achieved the lowest bias and the most stable RMSE distribution, with a modest efficiency trade-off.Unconstrained MFI produced the lowest average RMSE, while BLEU- and cosine-based constraints were less stable.
- Discussion and Conclusions: Validation used one LLM and one ECR item bank, while the prompt structure and theoretically chosen weights may not generalize optimally across models, domains, or contexts.The authors call for cross-model comparisons and alternative prompt or weighting schemes.
Appendix
The appendix contains the ECR item list, item-parameter and clustering results, and comparison figures for BLEU and cosine similarity.
- Appendix: Table I lists the items of the ECR scale.
- Appendix: Table II presents item parameters and clustering results.
- Appendix: Figures I and II show hierarchical cluster results for BLEU and cosine similarity, respectively.