PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use

Daniele Veri'

arXiv:2609.15624v1cs.HCcs.AI

TL;DR

Earlier AI-literacy frameworks did not test the added operational decisions required by tool-using assistants, and no focal instrument covers the full proposed agent-control set. The paper reviews measures and explores subjective–objective associations; the direct-correlation estimate is small and uncertain, while the proposed workplace battery remains untested.

  • Problem

    Earlier AI-literacy frameworks did not test tool-using assistants’ added decisions about access, state changes, execution, and evidence, while no focal instrument covers the full proposed agent-control set.

    Source: §1 · p. 1, §7 · p. 15

  • Method

    The paper synthesizes a small selected evidence base, pools three direct subjective–objective correlations, and proposes a layered workplace battery with non-compensatory rules.

    Source: §7 · p. 15, §9 · p. 16

  • Results

    The three direct correlations yielded r = .055 with a Hartung-Knapp interval including zero, and the focal corpus lacks an individual instrument covering the full agent-control set.

    Source: §9 · p. 16

  • Takeaways & Limitations

    The proposed battery keeps objective literacy, epistemic oversight, reliance behavior, and situated operational performance separate, with non-compensatory rules as one candidate approach.

    Source: §9 · p. 16

    The evidence base is small and selected, the largest study’s weight depends on an unresolved reporting discrepancy, and the proposed thresholds and transfer remain untested.

    Source: §7 · p. 15

Abstract

from arXiv · show

Researchers assessing competent generative-AI use at work must choose among self-reports, objective tests, and measures of oversight and reliance. We conducted a structured, seeded review of 24 focal empirical publications, starting from the 2024 COSMIN-based review and adding a targeted update through 17 August 2026. We grouped the measures into four domains: knowledge and use, epistemic oversight, reliance calibration, and operational control of tool-using agents. In an exploratory meta-analysis, we pooled three direct subjective-objective correlations from one research program (REML r = .055; Hartung-Knapp 95% CI [-.047, .156]; combined reported N = 2,765). We could not resolve a discrepancy between the largest study's reported correlation and p-value, leaving its weight uncertain. Adding a synthetic mean of 12 cross-factor correlations from a fourth study gave r = .079 (95% CI [-.025, .181]). This sensitivity analysis concerns a broader comparison. From this small evidence base, we cannot establish a population correlation, validate workplace cutoffs, or justify substituting self-ratings for performance scores. We identified tests of foundation knowledge (AICOS-S and GLAT) and measures of verification, reliance, trust, and dependency. We found no validated individual-level instrument in the focal corpus that tests the full combination of agent scope, permissions, recovery, state isolation, independent review, and evidence-based closure; some cover subsets. We propose a four-layer workplace battery with non-compensatory decision rules, but have not tested its thresholds or whether it improves on other assessment approaches.

1 Introduction

Competent GenAI use now involves operational decisions that earlier AI-literacy frameworks did not test, alongside established knowledge, evaluation, use, and ethics domains. The paper asks which instruments cover these competencies and how subjective ratings align with objective performance.

Introduction

  • Earlier AI-literacy frameworks cover recognizing AI, understanding mechanisms, using applications, evaluating outputs, and reasoning about ethics [Long and Magerko, 2020, Ng et al., 2021].The paper argues that tool-using assistants add decisions about access, state changes, execution, and evidence.Source: §1 · p. 1
  • Operational GenAI use can fail through unsupported acceptance, excessive tool access, continuing after a plan changes, faulty-state repair, or unverified completion claims.These behaviors span AI literacy, critical thinking, reliance, human factors, and professional task competence, whose measurement boundaries remain loose.Source: §1 · p. 1
  • Lintner [2024] identified 16 AI-literacy scales across 22 validation or revalidation studies; 13 were self-reports and three performance-based.Structural validity and internal consistency had the strongest support, while several other measurement properties were assessed less consistently; cross-cultural validity and measurement error were unexamined.Source: §1 · p. 2
  • Measures published in 2025–2026 expanded coverage to GenAI knowledge, critical thinking, reliance, information practices, trust, and dependency, complicating construct selection.Examples include GLAT, AICOS, and its short form, plus measures of oversight and adjacent constructs.Source: §1 · p. 2
  • The paper asks which validated instruments assess literacy competencies, whether subjective assessments align with objective performance, and which agent-supervision competencies lack validated individual-level measures.It treats the 2024 review as a historical baseline and limits quantitative synthesis to same-sample perceived-versus-objective associations.Source: §1 · p. 2

2 Conceptual frame

The paper organizes measures into four complementary domains, distinguished by what they assess rather than as levels on a single proficiency continuum. The domains span foundation knowledge through operational control of tool-using agents.

2.1 Four measurement domains

  • Knowledge and use covers AI recognition, conceptual and procedural knowledge, application, evaluation, ethics, and GenAI-specific capabilities, measured by self-reports and performance tests.Examples include AILS, SNAIL, and MAILS, alongside AILIT, GLAT, AICOS, and other objective tests.Source: §2.1 · p. 2
  • Epistemic oversight concerns checking provenance and factual support, separating evidence from inference, noticing missing support, and calibrating confidence in outputs.Measures include the Critical Thinking in AI Use Scale, AILIS, and an objective epistemic-knowledge dimension.Source: §2.1 · p. 3
  • Reliance calibration concerns how users distribute cognitive work; reliance behavior, trust, and dependency are related but distinct targets.Trust measures predict willingness to rely but are not competence scores, while dependency represents a maladaptive pattern rather than skilled reliance.Source: §2.1 · p. 3
  • Operational control covers scope and permissions, plan review, recovery choices, independent review, state isolation, and closure with execution evidence.Human-agent teaming metrics and realistic workplace scenarios address subsets, but neither source provides a validated individual placement scale for the full behavior set.Source: §2.1 · p. 3
  • The domains should not be collapsed into one continuum: people can combine strong AI knowledge, verification, and familiar-workflow control in different ways.The paper distinguishes its measurement-claim framework from curricular frameworks, whose dimensions need not conflict.Source: §2.1 · p. 3

3 Methods

The methods combine a seeded review and structured coding with a deliberately narrow synthesis of directly reported subjective–objective correlations. The update is not a de novo database-wide systematic search, and the effect dataset is incomplete.

3.1 Review design

  • The review used Lintner’s registered PRISMA/COSMIN review as its historical baseline and added a targeted update through 17 August 2026, not a de novo database-wide search.The update inspected publisher records, arXiv, and citation links, but the authors make no claim of complete post-June 2024 capture or a reproducible database-wide protocol.Source: §3.1 · pp. 3–4

3.2 Eligibility and coding

  • The focal synthesis included 24 empirical publications selected for instrument development or validation, relevant adjacent measures, direct modality comparisons, or situated-work assessment.The authors coded instrument and sample characteristics, psychometric and external evidence, and relevance to the four-domain framework; they did not conduct a new COSMIN appraisal.Source: §3.2 · pp. 4–5

3.3 Quantitative synthesis

  • Three directly reported same-sample subjective–objective correlations came from one research program, all using the MAILS family as the subjective comparator.The reported pairs were r = .21 for MAILS and the Hornberger test in N = 120, r = .04 for AICOS and MAILS in N = 514, and r = .05 for AICOS-S and short MAILS in N = 2,131. The AICOS-S article’s reported r and p-value conflict, so its weight depends on using the overall sample size.Source: §3.3 · p. 6
  • The Zhang et al. [2026] extension averaged 12 cross-factor correlations in Fisher-z space, a broader comparison than a direct total-score correlation.The component correlations share participants and factors; the composite variance used an assumed common dependence, and does not make the composite equivalent to a total-score effect.Source: §3.3 · pp. 6–7

3.4 Limits of effect availability and extraction

  • The quantitative dataset contains extracted effects rather than all eligible effects, and incomplete extraction limits its size.Additional candidates, including a professional-population comparison, were not fully extracted; the authors do not classify those records as studies lacking reported coefficients. For GLAT, the reported regression does not supply the zero-order correlation needed for pooling, which would require a correlation matrix or further author data.Source: §3.4 · p. 7

3.5 Why reliability coefficients were not meta-analyzed

  • Reliability coefficients were not pooled because the instruments use heterogeneous formats and report different reliability statistics, making a pooled alpha incoherent.The authors instead report reliability evidence at instrument level and reserve pooling for effects with a shared interpretation.Source: §3.5 · p. 7

4 Results

The review finds limited alignment between subjective and objective AI-literacy scores, while new measures assess knowledge, oversight, and reliance. It identifies operational decisions around tool-using agents that existing validated individual-level instruments do not test as a full set.

4.1 Post-2024 instrument development has expanded objective measurement

  • AICOS-S offers a 12-item objective test with adult norms, while GLAT scores predicted GenAI-supported task performance better than perceived ChatGPT proficiency.For AICOS-S, the total score is the defensible interpretation because its six short subscales had weak reliabilities.Source: §4.1 · pp. 7–8

4.2 Self-report remains useful when the construct is subjective

  • Self-report measures remain useful for perceived capability, confidence profiling, needs analysis, and perceived change, but their interpretation shifts when used as proficiency gates.The measures include broad AI-literacy scales and a GenAI-specific confidence measure.Source: §4.2 · p. 8

4.3 Exploratory meta-analysis: perceived versus demonstrated AI literacy

  • REML pooled the three direct subjective-objective correlations at r = .055 (Hartung-Knapp 95% CI [-.047, .156]), with intervals including zero.The expanded four-effect sensitivity was r = .079 (Hartung-Knapp 95% CI [-.025, .181]); it adds a cross-factor composite, not a direct total-score correlation.Source: §4.3 · pp. 8–9

4.4 Critical oversight has become measurable

  • The Critical Thinking in AI Use Scale links self-reported verification, motivation, and reflection to observed verification strategies and claim judgments.CAILS embeds critical oversight alongside knowledge and ethics, but does not yet provide criterion evidence against observed verification behavior.Source: §4.4 · p. 10

4.5 Reliance, trust, and dependency should remain separate

  • Reliance behavior, trust calibration, and dependency are distinct targets; dependency was associated with lower task performance and critical thinking.The reliance scale describes behavior during problem solving, whereas a trust score alone does not establish good judgment.Source: §4.5 · pp. 10–11

4.6 The operational-agent measurement gap

  • No validated individual-level instrument in the focal corpus tests the full set of agent-control behaviors, which the authors label provisionally as agentic operational competence.The proposed behaviors include bounding permissions, replanning after new evidence, recovery choices, independent review, state isolation, and evidence-based closure.Source: §4.6 · pp. 11–12
  • Prior work offers related but partial foundations: delegation frameworks address authority and accountability, while team and occupational assessments do not supply a general normed individual scale.The reviewed sources motivate candidate content but do not validate the full competence construct or its scoring.Source: §4.6 · p. 11

5 Cross-instrument synthesis

The cross-instrument map separates content coverage from evidence of demonstrated skill and validation quality. Measures address distinct targets, but none covers the full four-domain framework, and assessment mode should match the claim being made.

Interpreting coverage ratings

  • Table 1 ratings describe content coverage, not psychometric quality or suitability for certification.A strong content rating does not itself establish demonstrated skill or a validated placement decision.Source: §5 · p. 12

Reliance and operational-control qualifications

  • GLAT, AILS, SNAIL, and MAILS receive only partial or weak reliance coverage, while trust and dependency are distinct from skilled reliance.Team or workplace measures also do not establish full individual agent-control coverage.Source: §5 · p. 12

Cross-instrument coverage

  • No instrument in the focal corpus covers all four domains, and some cover only subsets of operational control.The coverage map does not show that combining measures improves workplace decisions.Source: §5 · pp. 12–13

Matching measurement mode to claim

  • Self-efficacy, demonstrated knowledge, and execution under constraints call for different measures; problems arise when one mode stands in for another.A self-efficacy scale targets confidence, a performance test targets demonstrated knowledge, and a role-specific scenario targets decisions under constraints.Source: §5 · p. 13

Operational evidence

  • Operational assessment should score decisions and traces as well as final answers, including source locators, permissions, state transitions, tests, and recovery decisions.A correct answer can still accompany a process that breaches an information boundary, bypasses a check, or changes an unauthorized artifact.Source: §5 · p. 13

6 Implications for workplace placement and training evaluation

The paper proposes a four-layer workplace battery and non-compensatory decision rules, while treating both as designs for local validation rather than validated placement tools. It also outlines situated scenarios and the evidence needed before standardization.

6.1 A proposed layered battery

  • The proposed four-layer battery keeps measurement targets separate, but is a candidate design rather than a validated placement protocol.No study tested its thresholds or superiority over alternative scoring approaches.Source: §6.1 · p. 13, Figure 4 · p. 14
  • AICOS-S is a brief adult foundation-test candidate, while GLAT offers GenAI-focused assessment with criterion evidence in higher-education samples.AICOS-S has 12 objectively scored items and German-speaking adult norms; neither instrument establishes workplace pass thresholds.Source: §6.1 · p. 13
  • Epistemic oversight can combine the Critical Thinking in AI Use Scale or job-relevant AILIS with a short verification task when demonstrated performance matters.The Critical Thinking scale's criterion task links scores to fact-checking behavior.Source: §6.1 · p. 13
  • The GenAI Reliance Behaviors Scale describes reliance patterns after problem-solving tasks, while S-TIAS measures trust calibration as a separate research target.S-TIAS should remain separate from proficiency.Source: §6.1 · p. 13
  • A situated operational challenge should reproduce role-relevant decisions such as permissions, plan review, test evidence, recovery, and independent review.Software, research, QA, and operations tasks require different action surfaces; neither cited example establishes a validated general test of the full behavior set.Source: §6.1 · p. 13
  • Separate results may help distinguish training needs, but questionnaire or knowledge-test success does not establish success on an operational challenge involving authority or state.The proposed battery has not shown improved placement accuracy or training outcomes.Source: §6.1 · p. 14

6.2 Non-compensatory rules as a design option

  • Non-compensatory rules can require evidence for each task-relevant requirement instead of allowing strength in one domain to offset weakness in another.The review does not estimate how often either approach would misclassify participants, and it does not compare matrices, gates, and weighted totals for workplace placement.Source: §6.2 · p. 14

6.3 Candidate operational scenarios and validation needs

  • Candidate scenarios can test changing premises, checkpoint recovery, independent review, and parallel-state reconciliation; standardization still requires validation evidence.Validity depends on job analysis, expert review, scoring consistency, difficulty, and predictive evidence; the review validates neither this item format nor a ready-made item bank.Source: §6.3 · p. 15

7 Limitations

The review's evidence is bounded by its targeted search, heterogeneous populations, changing AI content, and a small selected correlation pool. The proposed workplace assessment and coverage map also remain unvalidated and depend on single-author judgments.

Search scope

  • The update was structured and seeded, not a de novo systematic database review, so newly published scales outside its retrieval paths may be missing.It used Lintner's 2024 review as the systematic baseline and targeted publisher and arXiv retrieval.Source: §7 · p. 15

Population and transfer

  • The focal studies span varied populations, and their measurement properties do not transfer automatically across groups or establish workplace placement validity.German-speaking adult norms cannot serve as workplace cutoffs in another country without local evidence.Source: §7 · p. 15

Changing content

  • AI-related item content can age as model capabilities, prompting, and tool behavior change.The review recommends durable principles where possible and versioned review of tool-specific content.Source: §7 · p. 15

Quantitative synthesis

  • The primary correlations come from one MAILS research program, with AICOS-S carrying 77.2% of REML weight under an unresolved reported r/p discrepancy.The primary interval includes zero, and neither the primary nor expanded sensitivity estimate validates score interchangeability or placement rules.Source: §7 · p. 15

Assessment proposal and coding

  • No focal instrument tests the full proposed agent-control combination, and the proposed map, battery, and gate rules remain organizing proposals.Coverage ratings lack independent duplicate coding and a new COSMIN appraisal; standardized certification would require further validation and cross-role comparisons.Source: §7 · pp. 15–16

8 Research agenda

The research agenda proposes linking psychometric measures to situated decisions and traces of AI-assisted work. It outlines scenario development, construct testing, behavioral-trace collection, and transfer evaluation.

Scenario development

  • Develop a situational judgment test from cross-role critical incidents, with expert ratings and think-aloud pilots to check what items measure.Candidate settings include software development, QA, operations, research, and information-intensive office work.Source: §8 · p. 16

Construct relations

  • Pair a new scenario test with foundation, oversight, and reliance measures to test convergent and discriminant relations among the four domains.The proposed analysis would compare a multi-factor model with a single generic AI-competence factor.Source: §8 · p. 16

Behavioral evidence

  • Collect sandboxed-agent behavioral traces such as permission requests, plan changes, checkpoint use, verification, reviewer separation, and completion-report accuracy.These traces could provide criterion evidence unavailable from questionnaire-only studies.Source: §8 · p. 16

Transfer

  • Test transfer by examining whether placement instruments predict later work outcomes, using 30- and 60-day samples as stronger criteria than immediate satisfaction.Proposed outcomes include fewer unsupported claims and unauthorized actions, more complete evidence, better recovery, and lower rework on comparable tasks.Source: §8 · p. 16

9 Conclusion

The reviewed measures cover distinct components of competent GenAI use, but no validated individual-level instrument in the focal corpus tests the full operational combination examined. The authors propose a layered workplace battery, while leaving its thresholds and added value for empirical testing.

9 Conclusion

  • The three direct subjective-objective correlations yielded r = .055, with an interval including zero; adding a cross-factor composite gave r = .079, also with an interval including zero.All three direct effects came from one research program, and the largest study's weight depends on an unresolved reporting discrepancy; the findings do not establish a population correlation or workplace pass threshold.Source: §9 · p. 16
  • No validated individual-level instrument in the focal corpus tests the full combination of scope changes, permission decisions, faulty-state recovery, state isolation, independent review, and evidence-based closure.This is a corpus-bounded finding about combined coverage; some instruments address component behaviors.Source: §9 · p. 16
  • The proposed workplace battery keeps objective literacy, epistemic oversight, reliance behavior, and situated operational performance separate.Non-compensatory rules are a candidate approach, but the battery's added value, thresholds, error rates, and transfer across roles remain empirical questions.Source: §9 · p. 16

Data availability

The review relies on published aggregate statistics and reports no participant-level data or accompanying separate data files, analysis scripts, or supplementary verification records.

Data availability

  • The review uses published aggregate statistics and collected no participant-level data.Appendix A reproduces quantitative inputs and sensitivity results, including Zhang component correlations; Table 1 reports descriptive coverage judgments.Source: Data availability · p. 17
  • No separate data files, analysis scripts, or supplementary verification records accompany the arXiv submission.Source: Data availability · p. 17
  • Appendix A reproduces the quantitative inputs and sensitivity results, while Table 1 reports descriptive coverage judgments.Source: Data availability · p. 17

A Quantitative inputs and sensitivity results

The appendix reports study-level quantitative inputs and sensitivity analyses, including a composite based on cross-factor correlations. The authors caution that sample-size assumptions and the largest study's reporting discrepancy affect interpretation.

A Quantitative inputs and sensitivity results

  • The study-level tables list the three direct effects and composite used in the analyses; correlations are uncorrected for measurement error, and reported sample totals assume each study's N applies to its extracted coefficient.The AICOS-S sample-size caveat requires particular attention.Source: A Quantitative inputs and sensitivity… · p. 19, Table 2 · p. 20, Table 3 · p. 20
  • The largest study reports r = .05 and p = .11, while an unadjusted Pearson test with N = 2,131 gives p ≈ .021; the discrepancy remains unresolved.The analysis preserves the reported correlation rather than imputing a replacement, leaving the study's weight uncertain.Source: A Quantitative inputs and sensitivity… · p. 20
  • The sensitivity composite is an average cross-factor association, not a total-score correlation; its approximate variance is 1/285 = .003509 at ρ = 1, .002544 at ρ = .7, and .001901 at ρ = .5.Source: A Quantitative inputs and sensitivity… · p. 20, Table 3 · p. 20
  • Primary and expanded REML normal-theory intervals were [.018, .092] and [.021, .136], while their model-based prediction intervals were [-.027, .136] and [-.070, .224].The primary REML heterogeneity estimate is at zero, so the intervals should not be treated as reliable coverage statements for new workplace populations.Source: A Quantitative inputs and sensitivity… · p. 20, Table 4 · p. 20

B Proposed interpretive rules for a layered workplace battery

The proposed rules match assessment type to the construct or task, while keeping domains distinct. Placement use requires validation of instruments, scoring, and decision thresholds in the target setting.

B Proposed interpretive rules for a layered workplace battery

Related papers

Loading 2609.15624v1…