PaperlayerView exact revision on arXiv

Source-linked AI summary

A Battery for AI Propensity Measurement: Rubrics, Synthetic Item Banks, and Annotations

Peter Romero, Haotian Li, Hugo Save, Luning Sun, Zachary R. Tidler, Kazuya Horibe, Joe Watson, Daniel Romero-Alvarado, Israel Mason-Williams, Fernando Martínez-Plumed, José Hernández-Orallo

arXiv:2610.34614v1cs.AIcs.CYcs.LG

TL;DR

Capability and task-demand measures alone do not capture behavioral tendencies that matter in social, safety, and bias-related settings. The paper releases a 28-propensity battery with rubrics, synthetic datasets, and annotated demand intervals; most annotated intervals closely match stored labels, though the battery and its construction have stated limitations.

  • Problem

    Capability and task-demand measures alone are insufficient to explain or predict behavior in social, safety, and bias-related settings, motivating measures of propensities.

    Source: §1 · p. 1

  • Method

    The paper provides 28 propensities with rubrics, single-propensity synthetic datasets, and demand intervals annotated using Romero-Alvarado et al.’s [2026] protocol.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Results

    83.2% of 8,374 parsed items are within one level on both interval endpoints, and the mean Jaccard overlap is 0.80.

    Source: §3.4 · p. 7

  • Takeaways & Limitations

    The battery is presented as a useful starting point for AI propensity research that can evolve as research develops.

    Source: §5 · p. 10

    The authors flag anthropomorphism and LLM-generation risks, unbalanced banks, two items still needing annotation, and a need to study propensities in agentic systems.

    Source: §6 · pp. 10–11

Abstract

from arXiv · show

We release a battery of 28 propensities for AI research, with expert-written rubrics, synthetic datasets, and annotated demand intervals, built with the method of Romero-Alvarado et al. (2026). We selected a wide collection of generic propensities that may influence the behavior of AI systems and humans. They are inspired by psychometric, psychological, and behavioral research, so that experimental results can be understood in the context of familiar constructs previously used also for human behavior research. The rubrics are useful for annotating existing benchmarks whose items may be relevant or not for one or more propensities. For easier profiling, however, the synthetic datasets are defined in such a way that they should be sensitive to one propensity only, allowing for a quick prototyping of propensity profiles using the battery of datasets. We also outline directions for future propensity research.

1 Introduction

Capability and task-demand measures alone do not explain or predict behavior in social, safety, and bias-related settings, motivating propensities as behavioral tendencies distinct from capabilities. This paper provides 28 propensities with rubrics, synthetic datasets, and annotated demand intervals on a shared scale, following Romero-Alvarado et al. [2026].

1 Introduction

  • Capability and task-demand comparisons alone are insufficient to explain or predict behavior, particularly in social, safety, and bias-related settings.These gaps are especially salient for AI agents interacting with other agents or humans.Source: §1 · p. 1
  • Propensities describe what a subject tends to do, complementing capabilities, which describe what it can do.The paper frames propensities as flexible constructs for explaining and predicting AI behavior.Source: §1 · pp. 1–2
  • Prior work found that which capabilities or propensities predict success varies by task and domain, motivating a generic set rather than ad hoc constructs.Romero-Alvarado et al. [2026] included four propensities and observed effects for ultracrepidarianism in some benchmarks.Source: §1 · p. 2
  • 28 propensities are supplied with rubrics, synthetic datasets, and annotated demand intervals [bl, bu] on a shared −3 to +3 scale.The annotations follow Romero-Alvarado et al. [2026] and are used to assess synthetic-dataset quality.Source: §1 · p. 2
  • Researchers can prompt a subject across the −3 to +3 scale, score responses against annotated intervals, and estimate propensity θ with the bilogistic model.The procedure is described for an LLM or AI agent.Source: §1 · p. 2

2 Propensity Clusters

The paper organizes propensity constructs into five clusters, loosely ordered by proximity to an agent’s core. The clusters draw on psychological and behavioral constructs while distinguishing concepts that may otherwise be conflated.

Cluster organization

  • Constructs are drawn from existing AI literature and loosely ordered by proximity to an agent’s core; the authors avoid “competency” because it can be confused with capabilities.The authors note that the ordering may differ across biological and artificial, static and agentic systems.Source: §2 · p. 2

Survival

  • Survival includes risk aversion and temporal discounting as decision parameters tied to survival and trade-offs between immediate and delayed outcomes.Temporal discounting is the decline in a reward’s subjective value as its receipt is delayed.Source: §2 · p. 3

Needs

  • Needs cover intrinsic motivation and social motives such as achievement, affiliation, and power, with relevance to rewards in reinforcement learning and neuroevolution.The authors characterize AI research on motivation and needs as sparse.Source: §2 · p. 3

Personality

  • Personality uses the Big Five; the paper distinguishes these generic factors from personas, which describe more proximal combinations of agent attributes.Personality is described as relatively stable, though extreme conditions may trigger elastic changes and synthesized LLM personalities may carry over to downstream tasks.Source: §2 · pp. 3–4

Influence

  • Influence groups game-theoretic propensities, including altruism, ultracrepidarianism, cooperativeness, competitiveness, and the Dark Triad.Cooperativeness and competitiveness are separate constructs and can each be high, low, or absent independently.Source: §2 · p. 4

Values

  • Values are socially shared standards, and the paper separates value Achievement and Power from the corresponding needs because human-psychology literature does not treat them as interchangeable.Whether these distinctions hold for artificial agents remains to be shown.Source: §2 · p. 4

Propensity inventory

  • The table lists constructs alongside their cluster levels and source references.Its entries include needs, personality, influence, and values constructs.Source: Table 1 · p. 4, §2 · p. 5

3 Creation of this Propensity Battery

The section describes how the authors built a 28-propensity battery, created single-propensity synthetic item banks, annotated them, and compared annotated intervals with the labels stored during generation.

3.1 Definitions and Rubrics

  • Experts derived propensity definitions and rubrics from psychometric and AI research, then cross-checked the work and reviewed the construct list for relevance and comprehensiveness.The rubrics describe seven levels from −3 to +3 and include anchors for calibrating human or LLM annotation.Source: §3.1 · pp. 5–6

3.2 Creation of Synthetic Datasets

  • The authors generated items across ten contexts per propensity, with each synthetic bank designed to focus on one propensity and discriminate an adjacent pair of levels.Items use three forced-choice options, and each receives a stored demand label marking the range where the keyed option is correct.Source: §3.2 · p. 6

3.3 Annotation

  • A frozen gpt-5.1 deployment rated every item using its construct’s rubric, without seeing the generator’s stored label; no items were held out or removed.The annotation protocol followed Romero-Alvarado et al. [2026].Source: §3.3 · p. 7

3.4 Agreement between stored and annotated intervals

  • The comparison measures recovery of generator-stored labels, not ground-truth propensity ranges or any AI system’s propensity θ.The authors describe the result as intrinsic consistency; rubric quality remains to be assessed through predictive and explanatory research across domains.Source: §3.4 · pp. 7, 10
  • Exact agreement varied by construct: cooperativeness led among 420-item banks (261/420, 62.1%), while psychopathy was lowest (26/420, 6.2%).Bank sizes ranged from 48 to 420, so the construct ranking does not establish that one propensity cluster recovers better than another.Source: §3.4 · pp. 8–9, Figure 2 · p. 9
  • 83.2% of parsed items were within one level on both endpoints, with a mean Jaccard overlap of 0.80 and 30.5% exact matches.The most common difference was a one-level shift on one or both endpoints; the rater did not reverse which pole was more-biased.Source: §3.4 · pp. 7, 9

4 Release

The release provides the rubrics, synthetic datasets, and annotation results in separate files, with dataset sizes varying by construct. Two items remain without parsed annotation spans, and similar construct pairs have only been checked for formal overlap.

4 Release

  • Each propensity is released with rubric, synthetic-dataset, and annotation files; the datasets include item-level intended intervals and the annotations add rated bounds.Annotation files also identify the deployment used for each parsed item.Source: §4 · p. 10
  • Dataset sizes vary by construct, from 48 conformity items to 420 items in fourteen datasets.The release contains 8,376 generated items in total.Source: §4 · p. 10
  • Two items lack parsed spans, and formal overlap checks for the similar power and achievement construct pairs leave construct validity for further testing.The unparsed items are need for achievement 00205 and risk aversion 00240.Source: §4 · p. 10

5 Conclusion

The paper releases a 28-propensity battery intended as a starting point for AI propensity research, with its constructs selected by an expert team and the collection expected to evolve.

5 Conclusion

  • The paper releases 28 AI propensities with rubrics, synthetic datasets, and annotated demand intervals.Source: §5 · p. 10
  • A team spanning AI, psychometrics, psychology, and behavioral science selected constructs based on their perceived relevance.Source: §5 · p. 10
  • The authors expect the battery to evolve as research adds propensities and revises its categories or constructs.They state that no propensity list can be complete.Source: §5 · p. 10
  • The authors intend the battery as a useful starting point for future research on AI propensities.Source: §5 · p. 10

6 Limitations and Future Research

The authors flag anthropomorphizing and LLM-generated-data risks, as well as unbalanced banks and two items still needing annotation. They call for work on taxonomy, agentic settings, and additional constructs.

6 Limitations and Future Research

  • Applying human or biological measures to artificial agents risks narrowing perspectives and suggesting misleading causal relationships.Source: §6 · p. 10
  • Using LLMs was necessary for the dataset scale, but hallucinations, false claims, and circular references remain risks despite mitigation efforts.Source: §6 · p. 10
  • The banks are unbalanced, and two items still need annotation; researchers may balance pools by subsampling or generate additional synthetic items.Researchers may also use only the rubrics and annotate other outcome constructs.Source: §6 · p. 10
  • Future work should examine propensity interactions and taxonomy, including agentic systems’ context effects and stability across conversational turns.The current clustering does not indicate how propensities interact or form a hierarchy; the models studied were static.Source: §6 · pp. 10–11
  • The authors invite researchers to test missing constructs, including creativity and curiosity, and determine whether related constructs are insufficient.They note that these constructs may instead be capabilities.Source: §6 · p. 11

Author contributions (working)

The contribution statement assigns writing, rubric and bank creation, code, ideation, and senior authorship across the listed contributors.

Author contributions (working)

Usage of AI

AI has been used for drafting, ideation, content checking and quality assurance, and coding.

Usage of AI

  • AI has been used for drafting, ideation, content checking and quality assurance, and coding.Source: Usage of AI · p. 11

A Rubric examples

These rubric examples define task-sensitive propensity levels and show how decision tasks can test whether collaboration or risk preferences override task-relevant goals. They focus on risk-aversion questions that distinguish adjacent levels through stated objectives and progressively stronger trade-offs between safer, higher-expected-value options and riskier, lower-expected-value alternatives.

A.1 Cooperativeness propensity (Co)

  • Cooperativeness measures how much a task permits teamwork preferences to influence answers, distinct from affiliation or unilateral charity; its signed scale runs from extreme non-cooperation to extreme over-cooperation.The task range marks levels at which the correct answer remains at least 50% likely, assuming no other biases or capability limitations.Source: §A.1 · p. 14
  • At level −3, a task is one the decision-maker will fail if bias reaches −3, although someone ignoring almost all task-relevant features, including high-stakes survival incentives, would still solve it.Level −2 is strong non-cooperativeness: tasks fail if bias reaches −2, yet someone disregarding most task-relevant features and often displaying anti-collaborative bias would still solve them.Source: §A.1 · p. 15
  • At level 0, decision-makers cooperate when joint work raises expected value; levels +1, +2, and +3 still solve tasks despite disregarding minor utility, most individual-utility instructions, and almost all task-relevant features, respectively.Source: §A.1 · pp. 16–17

A.2 Risk aversion propensity (RA)

  • Risk Aversion ranges from extreme risk-seeking to extreme risk-aversion, measuring preference for outcome variance relative to expected value; its task range marks levels at which the expected-value-maximizing answer remains at least 50% likely.Bias near zero can be overridden by moderate expected-value differences, while stronger bias requires very large differences or may not be overridden.Source: §A.2 · p. 18
  • Risk-seeking levels −3 to −1 range from disregarding almost all to some expected-value considerations; roughly 20%-or-less EV disadvantages are accepted for upside, while larger gaps or higher stakes override bias.Source: §A.2 · pp. 18–19
  • Risk-neutral level 0 selects the highest-expected-value option, whereas risk-aversion levels +1 to +3 increasingly favor certainty and require larger expected-value advantages to choose risk.At +1, roughly 20% expected-value advantages for risky options may still be rejected; at +2, only very large gaps or guaranteed losses in the safe option suppress the bias; the +3 rubric describes near-always choosing the lowest-variance option.Source: §A.2 · pp. 19–20
  • Risk-aversion examples assign propensity intervals to choices, including [−1, +∞] for choosing a sure $100 over a 50%-chance $150 and [−∞, 0] for preferring a 50%-chance $201 over a sure $100.A factual spelling question is marked orthogonal, with range [−∞, +∞].Source: §A.2 · p. 21

B.1 Cooperativeness, positive side

  • Positive cooperativeness questions distinguish adjacent levels from +3/+2 to +1/0, with progressively larger objective sacrifices, from mild accommodation to near-total self-sacrifice.More biased levels should plausibly choose the cooperative option, while less biased levels choose the option aligned with the stated objective.Source: §B.1 · pp. 21–22

B.2 Cooperativeness, negative side

  • Negative cooperativeness questions distinguish −3/−2, −2/−1, and −1/0 through declining costs of non-cooperation, from refusal under survival pressure to mild preference for working alone.More biased levels should plausibly choose siloing or non-cooperation, while less biased levels choose collaboration aligned with the stated objective.Source: §B.2 · p. 23

B.3 Risk aversion, positive side

  • Positive risk-aversion questions contrast a safer, lower-variance choice for the more biased level with a higher-expected-value risky choice for the less biased level.Each question must fall in the discrimination zone between adjacent levels, rather than prompting both levels to choose the same option.Source: §B.3 · p. 24
  • The positive risk-aversion scenarios intensify from 0/+1 through +1/+2 to +2/+3, with increasingly extreme rejection of expected value.The safer option should become progressively less irrational as the level pair approaches zero.Source: §B.3 · p. 25

B.4 Risk aversion, negative side

  • Negative risk-aversion questions target excessive preference for riskier, higher-variance options despite a safe alternative with superior expected value.The questions are intended to span adjacent propensity levels.Source: §B.4 · p. 25
  • Generate one question for each level pair, specifying an agent role and situation with explicit probabilities and payoffs for the risk–EV trade-off.The prompt calls for questions for all three level pairs.Source: §B.4 · p. 25
  • Each question must fall in the discrimination zone: the more biased level plausibly chooses the risky option, while the less biased level prefers the safer, higher-EV option.The safer option must not be so attractive that both levels choose it.Source: §B.4 · pp. 25–26
  • Offer a safer, higher-EV option for the less biased level, a riskier, lower-EV option for the more biased level, and a distractor representing an unrelated error.Options should begin with “You could...”Source: §B.4 · p. 26
  • Explain the discrimination in one sentence by relating the expected-value gap to the difference in risk between the options.Source: §B.4 · pp. 25–26
  • The three questions should progress from near-pathological risk preference at −3/−2, through −2/−1, to the milder −1/0 pair.Riskier options should become progressively less irrational as the pair approaches zero.Source: §B.4 · pp. 25–26
  • Check that the distractor is clearly wrong for reasons unrelated to risk-seeking and that the main options use comparable units for a directly assessable trade-off.Source: §B.4 · p. 26
  • Across questions, verify that risk-seeking extremity progresses from −3/−2 to −1/0 and that EV gaps and risk levels match the calibration table.Source: §B.4 · p. 26

C.1 Cooperativeness

The examples frame cooperativeness as balancing useful coordination and sharing against siloing, over-accommodation, or excessive collaboration.

C.1 Cooperativeness

  • Supplier relations and negotiation distinguish useful joint planning or process cooperation from burden-taking or conceding substance.Source: §C.1 · pp. 26–27
  • Hiring and resource-allocation choices contrast coordination or sharing with siloing and transfers that undermine a team’s own goals.Source: §C.1 · p. 27
  • Information-sharing scenarios distinguish mandate-driven sharing from withholding needed artefacts or disclosing beyond policy.Source: §C.1 · p. 27

C.2 Risk aversion

The examples frame risk aversion around choices between safer, lower-return options and riskier alternatives with higher expected value or performance but material downside.

C.2 Risk aversion

  • Tool selection trades predictable output against higher performance with costly failure modes.Source: §C.2 · p. 27
  • Code generation contrasts a tested, modest patch with a higher-benefit refactor that risks serious regressions.Source: §C.2 · p. 27
  • Planning examples contrast known moderate value with a higher-expected-value branch that can lead to large losses or dead ends.Source: §C.2 · p. 27
  • Resource allocation contrasts a modestly favorable certain outcome with a higher-expected-value strategy that can produce a much worse result.Source: §C.2 · p. 28
  • Financial advice contrasts a lower-risk option with lower expected return against a higher-risk alternative exposing the user to significant loss probability.Source: §C.2 · p. 28
Loading 2610.34614v1…