PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

Jinsong Chen, Shi-Ting Chen

arXiv:2609.31976v1cs.CL

TL;DR

The paper asks what semantic structure contextual scores can establish about assessment items and how that representation relates to behavioral measurement. It applies partially specified factor models to contextual scores and benchmarks resulting diagnostic constructions against student responses. The featured construction has a persistent seven-group structure, while comparisons favor a general dimension alongside group associations; a matched 2PL outperforms all tested diagnostic models on AIC and BIC.

  • Problem

    The paper asks what semantic structure contextual scores establish about assessment items and what additional evidence connects that representation to behavioral measurement.

    Source: §1 · pp. 2, 4

  • Method

    The study models corpus-derived item similarities with partially specified factor procedures and compares initial Q constructions and response-informed revisions using student responses.

    Source: §2.2 · p. 6

  • Results

    The featured construction has a persistent seven-group structure and comparisons favor a general dimension alongside group associations, while the matched 2PL has lower AIC and BIC than every tested CDM.

    Source: §6 · pp. 33–34

  • Takeaways & Limitations

    The findings support a conditional semantic representation while limiting direct diagnostic interpretation of selected semantic associations.

    Source: §3.2 · p. 15, §6 · p. 33

    The graded map is conditional on the score matrix and model specification, and its inclusion support is not a response-calibrated cognitive-requirement probability.

    Source: §3.2 · p. 15

Abstract

from arXiv · show

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.

1 Introduction

The paper investigates what contextual-score representations establish about assessment-item semantics and what evidence connects them to student responses. It applies partially specified factor models to describe semantic associations while distinguishing them from behavioral requirements.

1 Introduction

  • Contextual scores measure item similarity to corpus-derived reference words, and their factors describe corpus-conditioned item semantics rather than examinee abilities.The paper examines whether this representation supports structural investigation and behavioral measurement.Source: §1 · p. 2
  • Partial framework specification, factor-count persistence, general-versus-group comparisons, and signed loadings make semantic item–factor associations more explicitly interpretable.The procedure distinguishes persistence across counts from selecting a best-fitting count and describes graded associations with model-conditional inclusion support.Source: §1 · p. 2
  • The study tests whether binary projections of semantic associations can serve as diagnostic Q-matrices, comparing initial constructions and response-informed Hull-PVAF revisions with the official framework.A direct four-domain PCFA projection provides a simpler comparator to the seven-group projection.Source: §1 · p. 3
  • Student responses do not construct contextual scores, factors, markers, or initial binary maps; they enter only the explicitly labeled Hull revisions.This separation evaluates the behavioral relevance of a response-free representation and the changes made by empirical revision.Source: §1 · p. 3
  • Favorable response comparisons can support validity for a specified use, but do not establish every selected semantic association as a cognitive requirement.The paper treats the graded map as an output in its own right and considers response use a narrower application.Source: §1 · pp. 3–4
  • Using semantic structure to inform response parameters requires an empirically learned connection, a prospective application for a larger item bank.The present study evaluates semantic structure and the narrower direct-Q use.Source: §1 · p. 4

2 Data and Contextual Scores

The study analyzes contextual scores for TIMSS mathematics items separately from booklet-sampled student responses. The corpus-conditioned score matrix supports factor analysis, but its computed word rows do not have the sampling interpretation of student observations.

2.1 TIMSS Items and Student Responses

  • The dataset contains 40 text-only scored units representing 36 base items, with content and cognitive frameworks supplying distinct substantive starting points.Students answered only subsets under a booklet design, and response coverage differs across analyses.Source: §2.1 · p. 5
  • Table 1 distinguishes the contextual-score matrix from response matrices and their analysis-specific samples; responses do not enter score construction, factor estimation, or initial Q construction.The common Q benchmark requires at least two observed focal responses per student.Source: Table 1 · p. 5, §2.1 · p. 5

2.2 Contextual-Score Construction and Interpretation

  • Frequent corpus words index rows and TIMSS items index columns, allowing item relations to be studied through a shared reference vocabulary.The featured reference corpus is MathQA; the word profiles are computed, corpus-conditioned representations rather than sampled persons.Source: §2.2 · p. 6
  • Qwen3-Embedding-8B produces the featured score from item embeddings and averaged word-occurrence embeddings; equal weighting across containing documents prevents repeated occurrences in one document from dominating.The resulting matrix has 1,864 word rows and 40 item columns and is standardized by column before factor analysis.Source: §2.2 · pp. 6–7
  • Corpus choice and retained vocabulary shape the geometry of item comparisons, so stricter word-frequency cutoffs test sensitivity rather than establish a universally optimal threshold.A corpus change can alter the modeled item relations.Source: §2.2 · p. 7
  • Word rows share an encoder and corpus and are not independent sampled persons, so conventional fit indices and BIC are descriptive under a working independent-row likelihood.Participation-ratio rank is a spectral concentration index, not an effective sample size; Step 2 emphasizes the repeated direction of fixed contrasts rather than their magnitude as a Bayes factor.Source: §2.2 · pp. 8–9

3 Analytical Approach

The two-step procedure first tests whether factor counts yield stable, persistent structures, then compares correlated groups with a general-plus-groups representation. Signed loadings and model-conditional inclusion support form a graded semantic map, whose interpretation and sensitivity are evaluated separately from its binary response-model projection.

3 Analytical Approach

  • Step 1 searches factor counts for structures that persist under direct comparison with wider solutions; Step 2 fixes the retained count and compares correlated factors with an anchored bifactor representation.The bifactor has a general dimension and orthogonal group factors; the resulting signed loadings and inclusion support define the graded map.Source: §3 · p. 9, §3.2 · p. 14

3.1 Identifying Persistent Structure

  • The featured Step-1 specification crosses content or cognitive backbones, corpus, anchor design, and depth, with candidate-count ranges and an anchor ladder informed partly by preliminary stability.The featured cell is MathQA, content, anchor-only, depth four; the ladder is not independently selected or outcome-blind validated.Source: §3.1 · p. 11

3.2 Comparing Factor Representations

  • Count evidence and persistence are evaluated separately, while Step 2 compares four marker definitions at the retained count and labels its BIC-based marker choice data-adaptive.The graded map’s PIPs are conditional on the score matrix, loading design, prior, and fitted representation; imposed markers have no estimated inclusion probability.Source: §3.1 · p. 12, §3.2 · pp. 13–15

3.3 Estimation and Evaluation

  • The binary map fixes present markers and thresholds free-cell PIPs at .50, then projects the graded map for response-model evaluation rather than establishing diagnostic requirements.The selected covariance uses a variational plug-in rather than a maximum-likelihood refit; mean-field approximation may understate uncertainty, and the reported computational evidence does not measure the complete two-step procedure.Source: §3.2 · p. 15, §3.3 · pp. 16–17

4 Semantic Structure of the Items

The featured construction yields a persistent seven-group structure, and comparisons favor representing broad shared variation with a general dimension alongside more specific group associations. The resulting semantic map is interpretable but conditional: some associations vary with specification, and the anchored representation does not lead every fit criterion.

4.1 Persistence Across Factor Counts

  • Direct comparisons show the original seven-column structure can remain recognizable in wider solutions even when adjacent transitions suggest weaker persistence.Persistence asks whether the original structure is retained, not whether each intermediate enlargement is equally stable.Source: §4.1 · p. 18, Table 3 · p. 19
  • At depth four, all three profiles return the same persistent seven-factor solution, while count support is qualified and other constructions can yield persistent structures with uncertain counts.The working-BIC path supports the count, whereas the ELBO path favors a wider model; the featured agreement is distinctive, and profiles share data and models.Source: §4.1 · pp. 18–20, Table 4 · p. 19

4.2 General and Group Dimensions

  • Every examined group definition favors a bifactor over its correlated-factor counterpart under working BIC, supporting an explicit general dimension alongside group associations.The direction recurs across anchor depths and marker assignments, but the contrast size depends on the definition.Source: §4.2 · p. 20, Table 5 · p. 21
  • The anchored bifactor improves on a single-factor reference, but the unrestricted model has lower working BIC while the anchored model has smaller standardized covariance residuals.The criteria emphasize different aspects of approximation; the anchored procedure supplies substantive orientation and explicit inclusion decisions, not optimality on every fit measure.Source: Table 6 · p. 22, §4.2 · p. 22

4.3 Interpreting Item–Factor Associations

  • The general factor is positive across items and has the largest share of squared loading mass, while group columns retain more specific semantic relations.Squared loading mass is a property of the loading matrix, not observed variance explained or students’ general mathematics ability.Source: §4.3 · pp. 22–23, Table 7 · p. 23
  • Item examples range from a recurring freely estimated algebra association to cross-domain augmentation, a specification-sensitive association, and an imposed marker.The coordinate-pair pattern is interpretable but does not establish a necessary or sufficient response attribute; the subtraction-pattern item retains its official number classification while adding algebra association. Consecutive-integers membership varies across definitions, and the staff-count marker is specified rather than discovered.Source: §4.3 · p. 25

4.4 Stability and Sensitivity

  • The broad general-plus-groups organization agrees more strongly than individual group associations, which include recurring, unselected, and definition-sensitive cells.Group comparisons require alignment, but alignment does not make patterns invariant; the support comparison counts only cells free under every definition.Source: §4.4 · p. 26
  • Fixed-design refits mostly preserve the representation under keyword and subsample changes, while normalization and prior changes affect some groups; these checks do not retest the full count search.The featured specification and anchor ladder were developed using application information, so the broad organization is conditional on this bank and construction; transfer requires choices made before examining a new bank.Source: §4.4 · p. 27

5 Response Evaluation

The response benchmark evaluates initial and Hull-PVAF-revised Q matrices under higher-order and saturated attribute distributions. The matched 2PL ranks best overall by AIC and BIC, while the preferred CDM depends on the criterion.

5 Response Evaluation

  • The tested benchmark concerns direct use of semantic associations as binary requirements; it does not establish whether other uses of contextual scores succeed.Full-sample comparisons do not test held-out prediction, true-Q recovery, or individual mastery accuracy, and Hull search cost is omitted from conventional criteria.Source: §5 · pp. 27–28, §5.1 · p. 30

5.1 Q-Matrix Constructions and Comparison Design

  • The initial maps represent official content domains, seven semantic groups, or a four-factor PCFA that can augment but not remove official memberships.The semantic projection excludes the general factor; selected associations do not themselves establish cognitive requirements.Source: §5.1 · p. 28, Table 10 · p. 29
  • Each initial Q receives a common Hull-PVAF revision, then both Q stages are fitted under two attribute distributions, yielding 12 planned conditions.The same revised Q is used under both distributions; failed or unstable branches remain unavailable rather than receiving ad hoc repairs.Source: §5.1 · pp. 28–29

5.2 Comparative Results and Their Implications

  • All planned CDM conditions converged, and every Hull branch reached a stable revised Q, with only small revisions to the maps.The saturated semantic-Q fits required the longer iteration allowance; differences in attribute count and membership density remained.Source: Table 10 · p. 29, §5.2 · p. 31
  • The 2PL’s AIC and BIC are 55.70 and 524.00 lower than the respective CDM minima, and it has the lowest values overall.This comparison uses the common response sample; the criteria condition on the realized Q and omit Hull search cost.Source: §5.2 · pp. 30–31, Table 11 · p. 31
  • Within CDMs, BIC favors the official framework and AIC its direct PCFA augmentation; the seven-group Q is less competitive than the PCFA Q.The semantic Qs have higher BIC and AIC than the framework Qs in all four matched comparisons.Source: §5.2 · pp. 31–32
  • Q-origin rankings stay unchanged across higher-order and saturated distributions, while within-origin BIC preferences differ by Q and stage.Hull lowers BIC in every matched comparison, but saturated revised semantic Q has worse AIC than its initial version.Source: §5.2 · p. 32
  • Response-driven edits changed some official memberships, but these fit decisions are not expert confirmation of item requirements.Examples include moving M032132 from data and chance to algebra and removing official memberships for two PCFA items.Source: §5.2 · p. 32

6 Discussion

The discussion presents contextual scores as an inspectable semantic representation whose structure depends on construction choices. Response comparisons support only conditional use, not direct diagnostic interpretation or demonstrated mastery accuracy.

6 Discussion

  • The contribution is an explicit structural investigation of item semantics that separates graded associations from response-model requirements.It combines contextual scores, partial substantive specification, distinctions between count and persistence, and interpretable associations.Source: §6 · p. 33
  • The featured construction has a persistent group structure, while comparisons consistently favor a general dimension alongside group associations.Persistence across counts and organization of common versus specific variation answer different questions.Source: §6 · p. 33
  • The anchored bifactor has smaller covariance residuals, but the unrestricted reference has better working BIC; neither establishes improved attribute recovery.Anchoring contributes substantive orientation and explicit selection, not optimization of every covariance-fit measure.Source: §6 · p. 33
  • Item interpretations distinguish recurring associations, cross-domain augmentation, sensitive secondary associations, and imposed markers rather than treating them as equivalent evidence.General profiles are more consistent than individual group coordinates, and matching resolves some—but not all—apparent sensitivity.Source: §6 · p. 34
  • Construction sensitivity remains a boundary: the featured specification and anchor ladder were development-informed, without independent replication or a chance-calibrated guarantee against favorable selection.Fixed-design frequency and metric refits do not establish that a new count sweep would recover the same structure.Source: §6 · p. 34
  • The response benchmark favors the official framework by BIC and its PCFA augmentation by AIC among CDMs, but the matched 2PL beats every tested CDM on both criteria.These comparisons do not measure mastery accuracy or instructional benefit, and the joint Q, attribute, response-model, and coverage choices remain unresolved.Source: §6 · p. 34
  • The evidence is conditional on a small, text-only item subset and shared encoder and corpus; missing responses, boundary estimates, and incomplete encoder provenance further limit transfer.Computed word rows are not an independent respondent sample, and working comparisons and variational PIPs remain construction-conditional.Source: §6 · p. 35
  • A prospective calibration study could learn links from contextual representations to response-derived parameters, but requires a sufficiently large, varied calibrated item bank and independent evaluation.Predictions or priors could be updated as responses accumulate; more students answering the same few items cannot replace item-level information.Source: §6 · pp. 35–36

Data and Code Availability

The archive supports numerical verification and documents what is needed for re-estimation, but several underlying data and model materials are not redistributed.

Data and Code Availability

  • The archive includes aggregate results, Q matrices, analysis settings, verification and re-estimation scripts, and checksums.Its verifier checks deposited numerical results.Source: Data and Code Availability · p. 36
  • Re-estimating the response models requires acquiring and preparing the external TIMSS data according to the documented instructions.Re-estimating reference text models also requires the specified R packages.Source: Data and Code Availability · p. 36
  • Respondent-level responses, identifiers, person scores, response-containing fitted models, corpus documents, encoder weights, and released item text are not redistributed.Source: Data and Code Availability · p. 37

Appendix A Data and Estimation Details

The appendix documents item inventory, contextual-score construction, response-data characteristics, and estimation settings. It also distinguishes partial timing records from the complete analysis workflow.

Appendix A Data and Estimation Details

  • The featured vocabulary retains lemmas appearing in at least 20 documents and no more than 80% of the processed MathQA corpus, which contains 29,837 problems; the encoder produces 4,096-dimensional vectors.Saved item lengths were 25–126 tokens, so no item was truncated.Source: Appendix A · p. 39
  • The contextual-score matrix is supplied, but the encoder revision was not commit-pinned and the original word-occurrence run’s hardware and complete settings were not retained.The archive does not supply the extraction code.Source: Appendix A · p. 39
  • Response-profile summaries account for booklet structure: 75.8% of focal response cells are structurally missing, so one panel summarizes eight focal-item booklets.Its profile measures are not tests of stochastic row independence or effective sample size.Source: Appendix A · p. 40
  • The response benchmark uses GDINA and cdmTools controls, while the 2PL uses mirt with 61 quadrature points and a maximum of 1,000 cycles.The saturated semantic-Q fits required the longer iteration cap; complete attempt and termination records are deposited.Source: Appendix A · pp. 40–41
  • Recorded timing covers selected sweeps, not the complete anchored two-step procedure and response analyses.The depth-two anchor-only sweep took 14.18–287.94 seconds per candidate and 950.05 seconds overall; a separate unanchored grid covered 10 fits.Source: Appendix A · p. 41

Appendix B Structural Specifications and Checks

Appendix B records the structural-specification sweep, inherited framework anchors, and rules for adding group markers. The featured count-path display gives the numerical basis for a persistence example, while the supplied passages do not state its cell values.

Appendix B Structural Specifications and Checks

  • Table 15 retains every examined specification, and the featured count-path display supplies the numerical basis for the main text’s count-versus-persistence example.The separately normalized 20%-gain path counts are not raw optima.Source: Appendix B · p. 41, Table 15 · p. 41, Table 16 · p. 42

B.1 Framework anchors

  • The inherited official-framework assignments are imposed associations, not estimated high-PIP discoveries; geometry and reasoning anchor sets are capped by their four and five items.The same assignments apply across both corpora and anchor modes.Source: §B.1 · p. 42, Table 17 · p. 42

B.2 Added-group markers

  • Added-group markers follow either a within-column absolute-loading ranking that permits reuse or a global maximum-loading assignment that makes added slots disjoint.Exact ties are resolved deterministically; disjointness applies to marker assignment, not fitted loading support.Source: §B.2 · pp. 42–43, Table 18 · p. 42

B.3 Alignment and identification checks

The checks characterize the selected covariance approximation and compare it with alternative item representations. They are local or descriptive checks, not evidence of global identification or diagnostic benefit.

B.3 Alignment and identification checks

  • 173 parameters span a 40 + 93 + 40 decomposition, and the 820-by-173 covariance Jacobian has full numerical rank at the stated tolerance.All definition-specific Step-2 fits have equal nominal and rank counts; these local checks are not global identification proofs.Source: §B.3 · p. 43
  • The continuous unthresholded plug-in covariance has smaller observed-minus-implied residuals than the hard-selected covariance, across summaries of 780 unique item pairs.The summaries use unique off-diagonal pairs, unlike package SRMR, which includes diagonal entries; they do not justify conventional respondent-sample cutoffs.Source: §B.3 · p. 43, Table 19 · p. 43

B.4 Descriptive EFA and embedding-clustering references

  • The nine-factor EFA reference is selected from a different sweep and is not count-matched to the seven-group model; embedding clusterings instead use k=4 and k=8.Those clusterings use L2-normalized raw embeddings; alignment statistics concern official content labels, not student responses.Source: §B.4 · pp. 43–44, Table 20 · p. 44

Appendix C Complete Semantic Maps

The appendix presents graded semantic item–factor maps and a binary projection, then distinguishes imposed framework memberships from additions selected using contextual scores. The projection omits loading direction and does not encode cognitive requirements.

Appendix C Complete Semantic Maps

  • The graded map combines signed loadings and raw group PIPs; its binary projection selects imposed cells or free cells with raw PIP at least .50 and excludes the general column.Of 93 selected group cells, 19 have negative loadings, illustrating the direction lost in positive binary entries.Source: Appendix C · pp. 45–46, Table 21 · p. 45, Table 22 · p. 46

D.1 Q constructions and empirical revisions

  • The initial framework-guided Q projection contains 40 imposed official memberships and 47 contextual-score-selected additions, with no student responses entering that projection.A zero denotes nonselection under the fit, not absence of a cognitive requirement.Source: §D.1 · p. 47, Table 23 · p. 47
  • Hull edits are listed relative to each initial Q, and the PCFA initial additions have deposited signed loadings and PIPs; all 40 official memberships were retained before response revision.The table identifies the edits but the supplied passage does not specify their individual values.Source: §D.1 · p. 47, Table 24 · p. 47

D.2 Diagnostic eligibility and response-model comparisons

The response comparisons favor the matched unidimensional 2PL over additional higher-order CDMs, while text-based projections vary by criterion. Eligibility checks and auxiliary mapping analyses further constrain diagnostic interpretation.

D.2 Diagnostic eligibility and response-model comparisons

  • The adopted Q fails the exclusive-item screen because S6 and S7 lack exclusive indicators, so the application stops its diagnostic claim.Row coverage is a prespecified practical rule; passing both screens would still not prove full diagnostic identifiability.Source: §D.2 · p. 48, Table 25 · p. 48
  • The matched 2PL has lower AIC and BIC than every additional higher-order CDM; within CDMs, AIC favors free-slope G-DINA while BIC favors A-CDM.Boundary slopes affect even the official framework, and the monotonic-constrained fit remains less competitive by both criteria.Source: §D.2 · pp. 48–49, Table 26 · p. 49
  • The adopted placement beats all 10 row-permuted matrices, but its plus-one descriptive tail is 1/11 = .091 and its advantage combines official anchors with semantic markers.That advantage is insufficient for the joint projection–model combination to earn its complexity over the 2PL.Source: §D.2 · p. 49, Table 27 · p. 50
  • AIC favors the depth-two disjoint projection and BIC favors depth-four overlap, so response results do not reselect the adopted text-side definition.The structural witness is not a certification of the fitted response model or diagnosis under the sparse design; relevant conditions remain unverified.Source: §D.2 · pp. 49–50

D.3 Residual dependence

  • The continuous semantic geometry has only a weak association with response residual dependence; removing shared-stem bundles increases the association, but it remains small.The text predictor is held fixed in these comparisons.Source: §D.3 · pp. 50–51, Table 28 · p. 51

D.4 Grouped item-parameter prediction

The grouped ridge check finds little recoverable linear signal from either contextual scores or raw embeddings for the evaluated item parameters. Only contextual-score prediction of difficulty has a small positive mean.

D.4 Grouped item-parameter prediction

  • The nested ridge regressions predict 2PL difficulty and log-discrimination from contextual scores or raw embedding coordinates, using grouped folds that keep shared-stem items together.Twenty fixed grouped 10-fold assignments quantify split variation; performance is held-out predictive R^2_pred.Source: §D.4 · p. 51
  • Only contextual-score prediction of difficulty has a positive mean; raw embeddings perform worse than the mean reference for difficulty, and both representations perform worse for log-discrimination.Leave-one-base-item-group-out results are similar, and the check does not distinguish limited item count from a weak, diffuse, nonlinear, or representation-specific relationship.Source: §D.4 · p. 51, Table 29 · p. 51
  • Calibration standard-error ratios are .9819 for difficulty and .9227 for log-discrimination, making reported standard errors alone unlikely to explain weak prediction under the stated assumptions.The ratios do not rule out misspecification or other noise, and the small item-level training sample remains a limitation.Source: §D.4 · p. 51
  • Response parameters never construct or select a text-side factor or mapping.Source: §D.4 · p. 51
Loading 2609.31976v1…