PaperlayerView exact revision on arXiv

Source-linked AI summary

Using profiles of cognitive capability to assess AI suitability for workplace tasks

Jonathan Prunty, Marko Tešić, Patrick Quinn, José Hernández-Orallo, Lucy Cheke

arXiv:2608.25623v1cs.AIcs.CYcs.HC

TL;DR

Organisations lack reliable ways to determine which workplace tasks AI systems can handle, because aggregate evaluations often fail to predict real-world performance. This paper profiles AI capabilities and workplace requirements using shared cognitive dimensions, finding that systems differ most across capabilities while workplace tasks share a common cognitive core.

  • Problem

    Pre-deployment evaluations often fail to predict real-world AI performance, limiting reliable scoping of suitable workplace tasks.

  • Method

    The framework infers agent capabilities from demand-annotated benchmarks and weights workplace task requirements using the same cognitive constructs.

  • Results

    AI systems varied more across cognitive dimensions than model families, while employees across six occupational domains identified a stable cognitive core for workplace tasks.

  • Takeaways & Limitations

    The framework enables direct comparative assessment of how AI agents match the cognitive requirements of workplace tasks.

  • Takeaways & Limitations

    Suitability scores are comparative rather than calibrated because the pipeline measures task importance, not task demand or success probability.

Abstract

from arXiv · show

Organisations deploying AI face a scoping problem: which tasks can be automated, which should remain with humans, and which are best shared between the two. Aggregate benchmark scores provide little insight into where systems will succeed or fail in practice, while human judgements of model capabilities quickly become outdated. We introduce a pipeline that profiles agents and tasks using a shared set of core cognitive capabilities. Cognitive capability profiling infers an agent's capabilities from performance on a benchmark battery annotated for the cognitive demands of each item. Task requirements weighting elicits from domain experts the relative importance of these same capabilities for their work. As both use a common set of cognitive dimensions, they can be updated independently as models and roles change, and combined to estimate AI suitability at the level of a domain, organisation, role, or individual duty. We validate capability recovery on synthetic agents, profile six AI systems, and elicit task requirements from 410 employees across six occupational domains. AI systems differed more across cognitive dimensions than across model families, while workplace activities converged on a shared cognitive core. The resulting scores provide a comparative scoping tool for identifying promising candidates for piloting and areas where current systems are unlikely to be well suited. We discuss extending the framework to profile human workers alongside AI systems, moving from AI suitability towards human-machine task allocation.

1 Introduction

Pre-deployment evaluations often fail to predict brittle performance in complex, variable workplace conditions, while aggregate scores provide little insight into failures or future generalisation. The paper introduces a pipeline that matches cognitive capability profiles with weighted task requirements to support systematic, evidence-based suitability decisions.

  • The evaluation gap: AI systems frequently perform unpredictably after deployment because real-world demands are more complex and variable than curated evaluations.Organisations can test only a subset of eventual tasks, leaving deployed systems exposed to untested combinations of demands.
  • The evaluation gap: Reliability is a major barrier to AI adoption, ranking above cost or capability limitations in reported business concerns.Anthropic’s survey of 81,000 users likewise identified reliability as respondents’ top concern.
  • Predictable failure: An aggregate benchmark score, such as 85% on MMLU, says little about why particular instances failed or how performance will generalise to meaningfully different cases.The introduction argues that predictability, rather than task reliability alone, should be the primary target of concern.
  • Predictable failure: Predictable failures enable human oversight, guardrails, handoff protocols, and deployment within a system’s capability remit, whereas unpredictable failures require blanket review or opaque risk acceptance.Blanket human review would negate much of the efficiency gain.
  • Capability-based suitability: The proposed pipeline profiles cognitive capabilities and weights task requirements so workplace demands can be matched against candidate AI systems’ capabilities.This reframes suitability judgements from intuition or trial-and-error into a systematic, evidence-based assessment process.

2 Background

Modern AI systems cannot be evaluated or assigned like human workers because their uneven cognitive capabilities undermine assumptions of a shared foundation, while human tests and broad benchmarks have complementary validity problems. The paper therefore motivates profiling agents and workplace tasks on shared cognitive dimensions to support suitability decisions.

  • Capability-profiling problem: Modern AI systems have jagged cognitive profiles: some capabilities exceed individual humans, while seemingly elementary perception and action remain fragile.This creates risks when deployed tasks demand capabilities a system does not reliably possess.
  • Evaluation limitations: Human psychometric instruments do not transfer straightforwardly to AI, and direct use can anthropomorphise systems or overestimate capabilities through training-data contamination.Small cognitive tests may have been included or closely approximated in language-model training data.
  • Evaluation limitations: Existing AI benchmarks provide scale and ecological breadth but usually group tasks by broad domains rather than specifying each item’s underlying cognitive demands.This limits their construct validity for capability profiling.
  • Related methodology: ADeLe combines benchmark scale with psychometric construct validity by using expert-constructed rubrics to annotate individual items for cognitive capability demands.An LLM judge applies the rubric as a scalable proxy for domain experts.
  • Task-suitability analysis: Task requirements should describe the cognitive demands of work itself rather than whether current systems can perform activities, because those demands change more slowly than AI capabilities.This provides a more stable basis for deployment analysis as organisations and practices change.
  • Task-suitability analysis: Because instance-level workplace annotation is impractical at organisational scale, the paper provides a pipeline that profiles tasks and agents on shared capability dimensions and maps them for suitability decisions.Generating an agent profile is only half the suitability question; role demands must also be structured and mapped to it.

3 Implementation framework

The implementation framework profiles AI capabilities from demand-annotated benchmark performance, elicits weighted task requirements, and maps both profiles into task-by-agent suitability scores. It uses shared cognitive dimensions, geometric demand scaling, and explicit aggregation assumptions to make capability–task comparisons.

  • 3 Implementation framework: The pipeline has three stages: Cognitive Capability Profiling, Task Requirements Weighting, and Suitability Mapping.Profiling infers AI capabilities from annotated benchmark performance; requirements weighting elicits capability importance from domain experts; mapping combines both into actionable task-fit estimates.
  • Cognitive Capability Profiling: The profiling battery annotates benchmark items with cognitive demands and infers capability profiles from observed performance.The implementation defines capability dimensions and rubrics, annotates and filters benchmark items into a targeted battery, and models capability estimates from performance data.
  • Cognitive Capability Profiling: Each capability demand is scored from level 0 to level 5, with raw difficulty δ_j,k = e^λd_j,k and λ = 1 in the experiments.Successive demand levels are geometrically spaced, allowing demands and estimated capabilities to be compared directly as a log-ratio.
  • Cognitive Capability Profiling: The annotated battery targets roughly 20,000 items, and consensus demand scores are combined into eight composite dimensions.Each composite dimension averages the raw consensus demand scores of its constituent capabilities while preserving the original 0–5 scale; no item reaches level 5 and only a small proportion reaches level 4.
  • Task Requirements Weighting and Suitability Mapping: Task requirements are converted into normalised capability weights, optionally sharpened with exponent s ≥ 1, and combined with agent profiles into suitability scores.The aggregation parameter p controls compensation between capabilities; the study uses p = 0, the weighted geometric mean, as its baseline.

4 Validation and testing

Validation shows that shared-intercept soft-min modelling recovers relative capability profiles and suitability reliably in simulation, while six AI systems differ mainly in selective planning and control strengths. Workplace capability requirements also show strong cross-sample agreement and a stable cognitive core, with occupational differences concentrated in secondary capabilities.

  • Simulation validation: Profile shape recovery is high at r = 0.92 across treatments, while a shared intercept raises capability-level recovery from r = 0.12 to 0.98.Separate agent intercepts leave the overall scale unanchored; sharing one intercept anchors the common capability scale.
  • Simulation validation: The shared-intercept model improves suitability recovery to ρ = 0.91 versus 0.14 and pairwise accuracy to 0.90 versus 0.53.It also yields a better-conditioned posterior and performs essentially identically to the oracle analysis with the true intercept.
  • Simulation validation: The adopted specification is soft-min pooling with τ = 1 and a shared intercept, but the recovery analysis establishes internal recoverability rather than empirical validity of its modelling assumptions.The same modelling parameters were used for simulation and inference, so the analysis does not test those assumptions against independent empirical evidence.
  • AI-system profiles: Gemini 3.1 Pro and Gemini 3 Flash have the highest overall capability levels at ¯c = 3.70 and ¯c = 3.38, but all six systems have similar profile shapes.Their main advantages are selective: Gemini 3 Flash scores 5.06 and Gemini 3.1 Pro 4.90 on Information Integration & Control, while Gemini 3.1 Pro reaches 4.25 on Action Planning & Simulation versus 2.37 for Gemini 3 Flash.
  • Workplace-task validation: Activity-importance profiles agree strongly across samples, with mean cosine 0.91 and Pearson 0.80, while capability-level agreement is lower at cosine 0.85 and Pearson 0.52.Within-source reliability reaches 0.94, and the disattenuated between-source correlation is approximately 1, consistent with measurement reliability explaining residual differences.
  • Workplace-task validation: Workplace requirements form a stable cognitive core centered on planning, memory, language, and procedural capabilities, while domains differ mainly in secondary capabilities.Domain-specific matrices correlate at r = 0.53–0.77, with examples including Spatial Reasoning & Navigation +2.1 for Warehouse and Logistics and Planning +2.5 for Manufacture, Maintenance, and Repair.

5 Discussion

The discussion presents a reusable framework that separately profiles AI capabilities and workplace requirements in a shared cognitive space. It identifies substantive findings, measurement limitations, and extensions toward comparable profiling of AI systems, human workers, and hybrid teams.

  • Framework contribution: The framework’s methodological contribution is to recover agent capability profiles from demand-annotated benchmarks and elicit workplace requirements using shared cognitive constructs.It separates capability profiling from requirements gathering, allowing the two measurement processes to be updated independently as systems and work change.
  • Empirical findings: AI systems varied more across cognitive dimensions than across model families or versions, while showing common strengths in language, semantic memory, and social cognition.The systems were weakest on capabilities associated with more demanding forms of reasoning and interaction, as described in the discussion.
  • Empirical findings: Across six occupational domains, workplace activities shared a cognitive core of semantic, procedural, and working memory, planning, and language.Interpersonal activities placed greater emphasis on social cognition, while individual domains showed predictable secondary specialisations.
  • Limitations: Current suitability scores are comparative rather than calibrated because the pipeline measures task importance rather than task demand.They indicate which agents are better matched to a task, not the probability that an agent will perform it successfully.
  • Limitations: The implementation under-represents multimodal perception, long-horizon planning, and interactive tool use, and profiles foundation models in isolation rather than scaffolded agents.These gaps concern capabilities on which contemporary systems appear weakest and most differentiated.
  • Future directions: The most consequential extension is to profile human workers with the same framework, enabling comparable capability profiles and evidence-based task allocation across heterogeneous workforces.The framework could also represent AI systems, human workers, and hybrid teams in a shared capability space for systematic workforce planning.

A Battery and inference procedure · A.1 Refining capability dimensions

The procedure represents benchmark items and AI systems through 18 cognitive capabilities, using rubric-based demand annotations. It refines these dimensions by removing unreliable capabilities and clustering redundant ones based on demand-profile correlations.

  • A.1 Refining capability dimensions: The battery defines 18 core cognitive capabilities drawn from the cognitive science literature.Each capability has a scoring rubric for rating the level of demand a task item places on it.
  • A.1 Refining capability dimensions: Two independent demand-annotation sets were collected across the full battery from GPT-4o and Gemini 3 Flash.These annotations used the capability-specific scoring rubrics and are reported in Table 1.
  • A.1 Refining capability dimensions: The dimension set was refined first by assessing inter-rater reliability between the two annotators.Reliability results are reported in Table A2.
  • A.1 Refining capability dimensions: Attention and Inhibitory Control, and Prospective Memory were excluded because the annotators failed to agree reliably on them.These were the two dimensions removed during the reliability-based refinement step.
  • A.1 Refining capability dimensions: Remaining dimensions were examined through correlations between their demand profiles and dimensionality reduction.Dimensions that tended to co-occur across the battery were clustered together to address redundancy.
  • A.1 Refining capability dimensions: Mental Simulation, Metacognition, and Causal Reasoning are treated as domain-general capacities recruited across capability families.They are not assigned to a single grouping.
  • A.1 Refining capability dimensions: Rubrics rate task demands on each capability, with the full set of 18 rubrics included in the project repository.The Theory of Mind rubric ranges from level 0, requiring no ToM capability, to level 5, requiring very high ToM capability.

A.2 Posterior for capability estimation

Capability estimation uses Bayesian inversion of the generative model to infer an agent’s log-capabilities and intercept from attempted-item responses. Because the posterior has no closed form, it is sampled with MCMC and summarized as posterior distributions over exponentiated capabilities.

  • Model specification: For a single agent, the demand matrix D and slope λ are fixed, while the unknowns are the K log-capabilities c and intercept α.Each capability c_k has a N(μ_c, σ_c) prior, and α has a weak N(0, σ_α) prior.
  • Posterior computation: The likelihood combines pooled item logits z_j(c, α) with the Bernoulli observation model, using only items the agent attempted.Missing responses therefore drop out of the likelihood.
  • Posterior computation: The sigmoid likelihood and pooling nonlinearity yield no closed-form posterior, so samples are drawn by MCMC with the No-U-Turn Sampler.Each dimension is summarized by the posterior over θ_k = e^c_k.

A.3 Modelling assumptions and extensions

The capability-inference model imposes shared structural parameters to keep individual capabilities identifiable, so overall levels are relative within a fitted catalogue while profile shapes remain recoverable. These constraints motivate a hierarchical extension that estimates structural parameters across many agents.

  • Identifiability constraints: Single-agent data cannot separately identify parameters that trade off against capability estimates, motivating constraints on demand slope, discrimination weights, and intercept.The model therefore shares the demand slope across dimensions, fixes discrimination weights to one, and anchors the intercept across agents.
  • Identifiability constraints: A shared demand slope assumes that one-level demand increases affect every capability dimension equally, while dimension-specific slopes could trade off against capability estimates.The model depends on the margin m_jk = c_k −λ_kD_jk, making λ_k and c_k inseparable for a single agent.
  • Identifiability constraints: Fixing each discrimination weight to one preserves a common, interpretable capability scale, but differences in performance dependence across dimensions are absorbed into capability estimates.Freely estimated discrimination parameters would be confounded with the capability scale and λ in single-agent data.
  • Interpretation and extension: Overall capability levels are identified only relative to the other agents in a catalogue, whereas profile shape remains recoverable and meaningful within that catalogue.Adding or removing agents can shift the shared reference point, so absolute levels should not be compared across separately fitted catalogues.
  • Interpretation and extension: A hierarchical model could fit many agents jointly, drawing individual profiles from a shared population distribution and estimating structural parameters at the population level.Cross-agent variation would help separate structural parameters such as λ, dimension-specific λ_k, discrimination weights, and the intercept from individual capabilities.

B Questionnaire and interviews

The supplementary questionnaire and interview materials define 18 O*NET-adapted work activities and describe how participants weighted the capabilities required for each task. They also document the questionnaire sample, demographics, and validation of agreement between company and online respondents across 16 capabilities and 18 activities.

  • Activities: The questionnaire covered 18 work activities adapted from O*NET categories.Table B1 provides the activity descriptions.
  • Sample: The materials report questionnaire recruitment sources, participant counts, sample demographics, and quality-control exclusions across company and online respondents.The full post-exclusion sample comprised N = 410 participants.
  • Task-capability weighting: Participants selected five essential capabilities for each activity and distributed 100 ability points across them to produce a weighted task profile.The process framed capability selection as configuring a robot helper for the target role.
  • Validation: Sample validation compared company and online importance vectors using cosine similarity and Pearson correlation, aggregating across 16 retained capabilities and 18 work activities.Reliability and noise-ceiling adjustments were also reported for interpreting between-source agreement.

C.1 Recovery analysis

The recovery analysis refits a fixed population of synthetic agents across soft-min pooling temperatures, assessing how well posterior capabilities recover known profiles. It also identifies an overall-level/intercept confound and shows that shared-intercept fitting anchors system comparisons with good MCMC convergence.

  • Recovery analysis: Recovery is assessed by correlating true and posterior-mean capabilities for each dimension across synthetic agents and by measuring posterior contraction.The analysis varies the soft-min pooling temperature τ while holding the synthetic population fixed and refitting it with the same inference procedure.
  • Identifiability: The intercept α is confounded with an agent’s overall capability level, so a free per-agent intercept leaves the absolute level unidentified while preserving relative profile shape.Adding a constant to every capability and subtracting the same constant from α leaves predicted outcomes unchanged because pooling depends only on capability margins.
  • Recovery analysis: N = 20 synthetic agents are drawn once from a log-scale capability prior and simulated and refit self-consistently at each pooling temperature τ.The prior uses µc = 3.0 and σc = 0.8; the battery contains K = 8 clustered dimensions.
  • Pooling analysis: The simulation recalibrates its intercept to keep mean accuracy at 0.60 while holding item difficulty constant, isolating pooling effects from overall-difficulty changes.τ=0 represents the normalized-additive compensatory model, whereas larger τ values move toward weakest-link pooling.
  • System fitting: All jointly fitted systems are well converged at 2,000 draws/chain across 4 chains, with ˆRmax ≤1.013 and bulk ESSmin ≳350.A shared intercept anchors each system’s overall level against the others.

C.2 AI capability profiles

The AI capability profiles were broadly similar in shape but varied in overall level, with the two Gemini 3 models achieving the highest aggregate estimates. Across systems, Semantic Memory, Language, and Social Cognition were strongest, while Action Planning & Simulation, Instrumental Reasoning, and Object Permanence were weakest.

  • Aggregate capability levels: The two Gemini 3 models achieved the highest aggregate capability estimates among the evaluated AI systems.Table C5 aggregates capability estimates across the eight clustered dimensions.
  • Profile shapes: AI systems had broadly consistent profile shapes but differed in overall capability level.Figure C2 displays posterior estimates across eight clustered dimensions for each system.
  • Relative capability strengths: Semantic Memory, Language, and Social Cognition were among the strongest estimated capabilities across systems.The relative ordering of dimensions was largely preserved across systems.
  • Relative capability weaknesses: Action Planning & Simulation, Instrumental Reasoning, and Object Permanence were among the weakest estimated capabilities across systems.These dimensions were consistently positioned toward the weaker end of the largely preserved cross-system ordering.

C.3 Questionnaire analysis

The questionnaire analysis aggregates responses into task-ability importance matrices and separates these matrices by occupational domain. It also compares activity importance with reported weekly time, distinguishing what workers value from where their time accumulates.

  • C.3 Questionnaire analysis: Questionnaire responses are aggregated into task-ability importance matrices, with domain-specific matrices showing how a shared cognitive core is retuned by secondary demands.Figure C3 presents separate matrices for Warehouse and Logistics, Manufacture/Maintenance/Repair, Numerical/Data/Programming, Administration/Organisational/Planning, Customer Service/Marketing/HR, and Hospitality/Sales/Client Care.
  • C.3 Questionnaire analysis: Work-activity importance is measured using frequency-adjusted selection-and-ranking scores, combining how often activities are chosen with how highly they are ranked.The scoring assigns 5 points to the top-ranked activity, down to 1 for the fifth, and 0 for unranked activities.
  • C.3 Questionnaire analysis: Reported weekly hours provide a separate measure of where respondents’ time accumulates, enabling comparison with the activities they treat as important.Hours are reported as conditional means among respondents who perform each activity, without zero-imputation.

C.4 Suitability mapping

The appendix maps AI suitability across compensatory settings, occupational domains, deployment priorities, and task-level importance or frequency. It also reports role-specific suitability for a Customer Service employee and documents the robustness and interpretation of these rankings.

  • Ranking robustness: The robustness analysis reports winner stability, distinct six-system rankings, invariant system pairs out of 15, and statistically distinguishable pairs after uncertainty propagation.These measures are summarised for each task across the compensatory sweep.
  • Domain and deployment mapping: Suitability is broken out by occupational domain using domain-specific importance weights, while deployment priority combines task importance with suitability as log(importance×S).Table C9 reports log S, and Table C10 reports log Pt where Pt = ItSt.
  • Ranking robustness: Suitability rankings remain robust across a compensatory sweep, with Gemini 3.1 Pro highest-ranked for every task except Admin, which changes only at p = 2.The sweep uses p ∈{−2, −1, −0.5, 0, 0.5, 1, 2} with demand sharpness fixed at s = 1.
  • Task-level maps: Full-sample and Company X maps relate AI suitability to task importance or frequency, using median-defined quadrants and labels only when quadrant placement has posterior probability ≥0.75.The full-sample plots use N = 410, while the Company X frequency-versus-suitability plot uses N = 35.
  • Role-level analysis: A Customer Service role analysis computes suitability for the overall role and three duties: informing about rights, helping with applications, and documenting dialogues.Scores are importance-weighted power means of ratio-scale capabilities, reported as log S for six AI systems.
Loading 2608.25623v1…