The Computational Assessment Stack: An Architecture for AI-Native Measurement

1 Introduction: From Capabilities to Architecture

• Separate prose artifacts leave links across assessment stages to human interpretation, motivating a shared representation for AI-native measurement.The paper frames the gap around connecting constructs to generated content, psychometric calibration, score inference, and reported claims.Source: §1 · p. 1
• A construct is computational when a system can evaluate, execute, and enforce it programmatically, making it a machine-readable test blueprint.The representation is symbolic and supports coverage queries, blueprint validation, targeted item generation, and version control over dependent validity claims.Source: §1 · p. 2
• As agentic AI takes on more lifecycle work, explicit, machine-readable constructs, task models, and blueprints give agents queryable requirements and accountability.Provenance records authorship, version, grounding evidence, and empirical status for measurement claims.Source: §1 · p. 2

1.2 The Computational Assessment Stack

• The CAS places a computational construct representation beneath content generation, computational psychometrics, and agentic orchestration under human governance.Its governing principle is that consequential measurement decisions should have explicit computational representations before automation.Source: §1.2 · p. 2
• The base combines a construct graph, an assessment/evidence graph, and their alignment; each upper layer reads from and writes evidence back to it.Generation returns items with validated construct links, psychometrics derives substantive structure from the representation, and orchestration keeps reported claims traceable to foundational evidence.Source: §1.2 · pp. 2–3, Figure 1 · p. 3

2 What Already Exists: The Lineage of Each Layer

• Earlier formal assessment work addresses individual layers or links between them, but the paper identifies no persistent representation shared across the full lifecycle.The CAS claims a combination of lifecycle persistence, versioned provenance, modular revision, and representation uncertainty.Source: §2 · p. 3, §2.7 · pp. 4–5

2.1 Design and validity frameworks: ECD, PADI, e-ECD, and Kane’s validity approach

• ECD and Kane connect claims, evidence, and validation warrants, while PADI encodes ECD patterns as structured objects but lacks a persistent operational representation.Such an object could be versioned, queried, and enforced by a running system.Source: §2.1 · p. 3

2.2 Machine-actionable skill structures: knowledge space theory and learning maps

• Knowledge space theory and learning maps support operational skill structures, but their bespoke designs lack modularity and representation uncertainty.The cited language-domain graph guides LLM tutoring but is not versioned or linked to a measurement model.Source: §2.2 · p. 4

2.3 Diagnostic structures: Q-matrices, attribute hierarchies, and diagnostic classification models

• The CAS generalizes Q-matrices and attribute hierarchies as views of a richer construct graph rather than hand-written tables.The Q-matrix links items to required skills, while attribute hierarchies and hierarchical diagnostic models add prerequisite relations.Source: §2.3 · p. 4

2.4 Constrained generation: cognitive design systems and automatic item generation

• Cognitive design systems and automatic item generation use formal templates to constrain item families; the CAS extends constraints across families and through calibration and reporting.Graph-constrained generation can target new subskills without new templates and enforce global blueprint coverage; pool admission remains a separate governed step.Source: §2.4 · p. 4

2.5 Feature-explained parameters: explanatory IRT

• Explanatory IRT models item parameters as functions of item properties; the CAS proposal explains them through items’ positions in a structured construct graph.The extension is representational rather than a new statistical model.Source: §2.5 · p. 4

2.6 Computational psychometrics

• Computational psychometrics combines data-driven methods with theory-driven psychometric modeling; in the CAS, it names the layer housing those models.The paper also uses the term for the broader discipline.Source: §2.6 · p. 4

2.7 What is missing

• The paper defines its novelty as combining five capabilities, culminating in a persistent, modular, provenance-bearing construct representation across the lifecycle that treats representation as uncertain.Prior frameworks reach different capability levels, but the paper claims none combines the full set.Source: §2.7 · pp. 4–5

3.1 Definition

• The representation combines entities, attributes, and typed relations with constraints, provenance, and versions, making both its content and governance explicit.The first three properties describe the representation’s content; the latter three distinguish it from static specifications and formal predecessors.Source: §3.1 · p. 5
• A graph cannot capture everything conveyed by prose or expert knowledge, and some constructs may be harder to represent; its simplifications are explicit, inspectable, and revisable.Omissions can be named, debated, and added module by module.Source: §3.1 · p. 5

3.2 Psychometric consequences

• Its entities, attributes, and relations make coverage and gap analyses computational; constraints enable screening, provenance supports audit, and versions link claims to the representations and models behind them.The authors note these capabilities do not themselves constitute validity evidence; validation remains to be done.Source: §3.2 · pp. 5–6

3.3 Two graphs, one relation

• The construct graph GC represents what is measured, while the assessment/evidence graph GA represents what is administered, observed, and inferred; alignment A links elements across the graphs.Alignment links may represent construct alignment or operational conformance; the Q-matrix is a slice of construct-alignment links.Source: §3.3 · p. 6
• Alignment links begin as asserted claims and can receive empirical status as data confirm or contest them; graph-comparison metrics require reliability, sensitivity, and population-invariance checks before interpretation.The proposed similarity functions may combine node coverage, relational and prerequisite-path agreement, and content-category distributions.Source: §3.3 · p. 6

3.4 Design decisions

• The construct must be stored, queried, checked, and versioned within assessment infrastructure, but the architecture does not depend on RDF, property graphs, or relational databases.Candidate authoring procedures include expert decomposition and LLM-extracted decomposition, and should be documented in provenance.Source: §3.4 · pp. 6–7
• Finer-grained representations are more informative but harder to link to standard statistical models; the paper takes no position on optimal grain size.The modular architecture makes this trade-off explicit, inspectable, and revisable.Source: §3.4 · p. 7

4.1 The ossification problem

• Formalizing constructs as graphs risks hardening provisional claims, hiding alternative decompositions, and causing low-level revisions to invalidate dependent items, calibrations, and reports.Human approval alone does not make revision affordable; without architectural design, the representation may become less flexible than prose.Source: §4.1 · p. 7

4.2 A modular architecture

• The four stages run from a stable core ontology through framework decompositions and operationalizations to empirical assessment modules, with dependencies flowing downward.Stage 3 contains the assessment/evidence graph and empirical status of alignment links; it instantiates the upper stages without modifying them.Source: §4.2 · pp. 7–8
• Coexisting modules preserve alternative decompositions, while traceability ties claims to module versions, alignment links, and evidence; replacing a Stage 2 module flags only dependent links and assessment objects for re-validation.The cost of revision is bounded by the stage where it occurs.Source: §4.2 · p. 8

5 The Q-Matrix as a View of the Alignment

• The Q-matrix is the slice of alignment A pairing items with subskills; the graph generalizes its representational scope rather than replacing it.A Q-matrix can be derived as a view alongside prerequisite hierarchies and item-to-domain mappings, but this generalization is representational, not statistical.Source: §5 · p. 8
• The representation treats item–subskill and item–domain links as cross-graph claims, prerequisite structure as within-construct relations, and item–evidence links as within-assessment relations.Each relation carries provenance and a version, and the mapping from graph relations to statistical models must be declared explicitly.Source: §5 · p. 8

5.1 Hybrid construction: expert elicitation refined by data

• Experts propose theory- and curriculum-grounded structure, while response data can test graph edges and alignment links; data-contested claims trigger expert review rather than silent revision.Data-driven structures may reflect population and teaching sequences as well as constructs.Source: §5.1 · pp. 8–9

6 Layer 2: Computational Content Generation

• Item generation queries the construct graph for target descriptors, subskills, and permitted task features, then supplies that subgraph as context to a generator.The generator may be an LLM, an agentic workflow, or a template-based script.Source: §6 · p. 9
• The proposed pipeline runs structural screening and semantic evaluation automatically before human review, followed by piloting, calibration, and admission to the item pool.The ordering is a design proposal; operational programs already implement parts of it.Source: §6 · p. 9
• The construct graph constrains admission, not generation: production may be fast and exploratory, but admission is validated, documented, and follows review, piloting, and calibration.Generation and admission can use different models, agents, and governance.Source: §6 · pp. 9–10
• Graph-linked admission records make it possible to query which construct claims supported an item later showing DIF or misfit and whether other items admitted under those claims share the problem.The graph makes those questions simple queries rather than time-consuming document-trail searches.Source: §6 · p. 10

7.1 Graph-constrained psychometric models

• A hard graph constraint fixes an item’s discrimination at zero on excluded dimensions, while a soft constraint treats graph assignments as priors that response data can challenge.Under either formulation, data estimate permitted discriminations; item difficulty remains freely estimated unless modeled from graph-derived properties.Source: §7.1 · pp. 10–11

7.2 The graph as an explanatory representation

• Graph-explanatory IRT uses construct-graph representations to predict discrimination and difficulty parameters, which then enter an IRT response model.For newly generated items, these predictions should remain provisional or serve as informative priors until response evidence accumulates.Source: §7.2 · pp. 11–12
• The proposed extension uses a structured, versioned construct graph as the explanatory basis for item parameters rather than relying exclusively on manually engineered item-feature vectors.The paper proposes comparing four models—conventional IRT, feature-explanatory, features-plus-graph, and graph-only—to test whether the graph adds explanatory value.Source: §7.2 · p. 12, §7.3 · p. 12

7.3 A falsifying comparison: four models

• The principal proposed comparison is graph-only M3 against features-plus-graph M2; comparable out-of-sample performance would support the graph capturing much relevant information in conventional features, but not every construct-relevant property.If M3 underperforms M2, the gap could reflect missing graph information, construct-irrelevant features, or limitations of the mapping function.Source: §7.3 · p. 13

• Agents may propose changes to the construct graph and alignment, but human approval is mandatory for changes to what is measured and what counts as measuring it.Source: §8 · p. 13
• Agents propose versioned module updates across stages, which require appropriate human approval before becoming operational.Source: §8 · p. 13
• Every reported claim must be reconstructible through links to the active representation versions, measurement models, item administrations, and decision rules.Source: §8 · p. 13

• Construct-representation uncertainty includes uncertainty over competing competency decompositions and framework interpretations, alongside uncertainty about individual graph relationships and whole modules.Source: §9 · p. 13
• When admissible modules coexist, claims can be reported per module or tested across modules with sensitivity analysis; conclusions should be limited to the detail supported consistently.For example, a shared overall proficiency classification may be defensible even when subskill-level claims require qualification.Source: §9 · pp. 13–14
• Combining module predictions is legitimate only when the claim has the same meaning under each module and a documented mapping to a common claim is available.Source: §9 · p. 14
• Predictive stacking weights reflect modules’ contributions to out-of-sample prediction, not the probability that a module is correct; mixture weights cannot represent decompositions nobody proposed.Source: §9 · pp. 14–15
• Different frameworks cannot be combined directly when they define different constructs; comparison requires a validated crosswalk, and developing such mappings remains open.Representation invariance across populations and language versions is a distinct question from measurement invariance, with fairness implications.Source: §9 · p. 15

10.1 Step 1: Source descriptors

• The case study starts from illustrative paraphrases of two B1 Reading descriptors and distinguishes those official descriptors from the computational interpretation built from them.A working system would reference official descriptor identifiers; the example's paraphrases are for illustration.Source: §10 · p. 15, §10.1 · pp. 15–16

10.2 Step 2: Decomposition as a Stage 1 framework module

• The panel interprets two illustrative descriptors as requiring three subskills and a prerequisite relation, while another panel could encode a different decomposition as a coexisting module.The interpretation belongs to the panel, not the CEFR; choosing among coexisting modules is a substantive research question.Source: §10.2 · pp. 16–17

10.3 Step 3: Operationalization as a Stage 2 module

• A Stage 2 module makes the decomposition operational through permissible stimulus types, task templates, and admission constraints that are mechanically enforced at screening.Unlike hand-checking a sample, the proposed system checks every candidate item automatically.Source: §10.3 · p. 17

10.4 Step 4: A blueprint coverage query

• A standing coverage query identifies S3 as a gap when the pool has no admitted items for it, prompting targeted generation with the relevant graph substructure.The request carries the subskill definition, prerequisite, permissible text types, and task template.Source: §10.4 · p. 17

10.5 Step 5: Generation and screening

• I003 is automatically rejected before human review because it lacks a descriptor alignment and uses a text type not permitted at B1.The rejection record names the violated constraints and the module versions in force.Source: §10.5 · p. 17, Table 3 · p. 18

10.6 Step 6: Piloting and calibration

• The illustrative calibration uses a larger hypothetical pool because three items alone could not identify a three-dimensional model.The displayed calibration outputs are invented for illustration, not estimated from response data.Source: §10.6 · p. 18, Table 4 · p. 19

10.7 Step 7: Evidence, decision, and a reported claim with provenance

• A reported subskill claim is a decision supported by a posterior probability and reporting rule, and its provenance makes it auditable rather than precise.The example reports a posterior probability of 0.87 against a rule threshold of 0.80; the credible interval's lower end is 0.51.Source: §10.7 · p. 18, Table 5 · p. 19

10.8 What the example shows

• The representation links a blueprint gap query, mechanically screened items, calibration, and a reported claim to the relevant modules, evidence, and versions.The example demonstrates these capabilities but does not perform the psychometric work; review and validation remain necessary.Source: §10.8 · p. 18

11.1 Fairness considerations

• Formal representations make some biases auditable but can introduce risks in ontology and decomposition, so fairness evaluation should inspect the construct-representation layer as well as response-to-score outcomes.Culturally specific decompositions can embed bias before items are generated.Source: §11.1 · p. 19
• For cross-language adaptation, structural equivalence can be assessed before calibration, but graph structure and measurement parameters may diverge and require joint evaluation.The framework distinguishes structural, semantic, and measurement equivalence.Source: §11.1 · p. 19

11.2 Validation agenda and open problems

• The proposed validation agenda asks whether independent experts agree on decompositions and whether competing modules yield different alignment, prediction, fit, or classification outcomes.Agreement is not correctness; indistinguishable results make module choice less consequential, while differing results require justification.Source: §11.2 · pp. 19–20
• Static graphs may poorly represent interactive, collaborative, dynamic, or process constructs, which may require temporal and interaction structure.The paper also notes that no general framework yet evaluates alternative decompositions and validates a module rather than a whole graph.Source: §11.2 · p. 20
• The agenda calls for tests of graph-derived alignment, psychometric structure, diagnostic traceability, and generation quality rather than treating these outcomes as established.For graph-constrained generation, the proposed comparison includes unconstrained, prompt-constrained, and graph-constrained systems.Source: §11.2 · p. 20
• The graph-to-parameter mapping may introduce systematic error distinct from the graph, leaving its trade-off among interpretability, flexibility, and misspecification risk unresolved.Source: §11.2 · p. 20

• CAS unifies an AI-native measurement system around a foundational, machine-readable construct representation.The architecture integrates modular knowledge graphs with content generation, explanatory psychometrics, and agentic orchestration under human governance.Source: §12 · p. 20
• Its architectural contributions include a persistent, versioned representation across the lifecycle, modularity, and a formal treatment of representation uncertainty.The paper frames computational representations connecting construct theory, task design, evidence, inference, and reporting—and their validation—as a next frontier.Source: §12 · p. 20

• The appendix's RDF/Turtle encoding is illustrative, uses placeholder namespaces, and covers example objects rather than a complete operational schema.The blocks express subject–relation–object statements and add no content beyond the body tables.Source: Appendix: Machine-Level Encoding of… · p. 23
• The encoding includes a framework module with three proposed subskills and a prerequisite relation, plus a provenance record for the module.Source: Appendix: Machine-Level Encoding of… · p. 23
• The screening example expresses constraints C1–C4 with SHACL shapes, but checking whether a target belongs to an active module also requires a SPARQL query.Running the shape against I003 produces two violation reports, for C2 and C3.Source: Appendix: Machine-Level Encoding of… · pp. 23–24
• The appendix's evidence record stores hypothetical posterior values, a decision rule, model, source items, and construct and operationalization versions.Source: Appendix: Machine-Level Encoding of… · p. 24