A Scale For Value Alignment In Human-AI Interaction

1 Introduction

• HCI studies have relied on self-developed, non-validated measures, leaving their reliability and comparability uncertain.The paper also reports that AI research lacked an established construct and measurement instrument for interpreting perceived value alignment scores.Source: §1 · p. 2
• The scale measures users’ perceptions of observed agent behavior because users cannot access a system’s internal objectives.The authors argue it should generalize across tasks and environments where users can observe an agent’s behavior.Source: §1 · p. 2
• The initial item pool drew on prior HCI work, and users rated scenarios varying in alignment, use case, and interaction paradigm.Experts selected 106 items for content validity before the user ratings.Source: §1 · p. 2
• The scale has two factors—value understanding and value manifestation—and its subscales showed high internal consistency and several forms of validity.The structure was tested with confirmatory factor analysis on an independent sample.Source: §1 · p. 2
• The scale is intended to support evaluation of perceived value alignment as a user-study outcome and comparison of systems and design variants.The paper reports a validated psychometric scale and evidence for factorial validity and internal consistency.Source: §1 · p. 2

2.1 Psychometric Scales

• Existing HCI scales measure related constructs such as trust, usability, teaming, creepiness, and shared understanding, but do not capture perceived value alignment in all relevant interactions.PVAS is intended to apply even when users observe an agent without directly cooperating with it.Source: §2.1 · pp. 2–3

2.2 AI Value Alignment: Concepts and Evaluation

• AI evaluations compare preferred outputs, normative criteria, or value profiles, but do not establish whether a particular user perceives an agent’s behavior as reflecting their values.Preference judgments can provide evidence about values, but the paper notes that preferences and values are not interchangeable.Source: §2.2 · p. 3

2.3 Perceived Value Alignment in Human–AI Interaction

• Interaction studies use behavior logs, task-specific ratings, and probe-based methods, while PVAS contributes a psychometrically validated self-report scale.The paper distinguishes VAPT’s probe-based evaluation methodology from a validated self-report instrument.Source: §2.3 · pp. 3–4
• PVAS targets perceived congruence in values, goals, and preferences, distinguishing value understanding from value manifestation in agent behavior.The paper evaluates the scale using scenarios varying in embodiment, autonomy, and the values at stake.Source: §2.3 · p. 4

3 Method

• The scale-development framework covers construct definition, item generation and refinement, assessment of latent structure, and evaluation of psychometric properties.The authors follow the recommendations of Boateng et al..Source: §3 · p. 4
• After establishing the scale’s psychometric properties, the authors assess whether a shorter version can preserve predictive performance.They apply Yarkoni’s scale-shortening approach.Source: §3 · p. 4
• The paper organizes its methodological decisions and analyses around the stages of the scale-development process.Sections 4 through 9 cover the framework’s stages.Source: §3 · p. 4

4 Construct Definition

• Perceived value alignment is a human’s perceived congruence of values, goals, and preferences with an agent after interacting with and observing its behavior.The definition draws on McKinlay et al. [29].Source: §4 · p. 4
• Values are trans-situational guiding principles that vary in importance and become observable through expression in specific contexts.They are organized as relative priorities rather than independent endorsements.Source: §4 · p. 4
• Goals are concrete, context-bound end states pursued in service of values, while preferences rank options or actions given goals and values.The originating values behind a goal are not necessarily recoverable from the goal alone.Source: §4 · p. 4
• Because values become observable only through contextual expression, the construct is assessed after interaction with an intelligent system.The paper describes perceived value alignment as a judgment based on observing the agent.Source: §4 · p. 4

5 Item Generation

• The authors generalized item wording by replacing specific tasks, outcomes, and human roles with reusable placeholders.Items that could not be generalized across studies were removed.Source: §5 · pp. 4–5
• A literature search identified 79 relevant questionnaires and yielded 1,429 items, whether or not their instruments had been validated.Searches covered ten related constructs and queried Scopus in July 2026.Source: §5 · p. 4
• Four authors removed 1,045 redundant or unrelated items, then two authors removed 110 modality- or use-case-specific items, leaving 274.A native English-speaking author checked the remaining items for correctness and naturalness.Source: §5 · p. 5

6 Content Validity

• The content-validity assessment asked domain experts to judge whether candidate items captured perceived value alignment.The study followed recommendations for expert assessment of content validity.Source: §6 · p. 5

6.1 Participants

• Seven experts participated, selected for having authored at least one peer-reviewed publication in value alignment or psychometric scale development.Participation was voluntary and uncompensated.Source: §6.1 · p. 5

6.2 Procedure

• Experts rated each item’s relevance on a four-point scale after reading the construct definition and a generalized item statement.They could comment on items and suggest missing factors, scenarios, and use cases.Source: §6.2 · p. 5

6.3 Data Analysis

• Experts rated 274 items, and 106 with mean relevance scores above the 2.5 midpoint were retained for user testing.Experts also suggested domains, value types, and scenario dimensions to consider.Source: §6.3 · p. 5

7.1 Participants

• The study recruited English-speaking participants in the USA and UK; 601 passed checks and completed the questionnaire.They were recruited through Prolific and took a median of 12 minutes.Source: §7.1 · p. 5

7.2 Materials

• Six scenarios varied embodiment, agency, task, and value conflict, spanning autonomous agents, editing and travel tools, and medical or harmful interactions.Scenarios were presented as videos or graphical chat layouts.Source: §7.2 · pp. 5–6

7.3 Procedure

• After comprehension checks and random assignment to a scenario, participants rated 106 items on a five-point Likert scale.Participants answered two content questions about their assigned scenario before rating items.Source: §7.3 · p. 6

8.1 Data Screening

• Screening for failed attention checks, implausibly fast responses, and longstrings removed 88 responses, leaving 518 respondent-scenario rows.The removed responses represented 14.5% of the data.Source: §8.1 · p. 6

8.2 Item Pre-processing and Adequacy Testing

• After reverse-coding negative items, three items with corrected item-total correlations below .30 were removed before factor analysis.The remaining 103 items had a significant Bartlett’s test, χ2(5253) = 49706.5, p < .001.Source: §8.2 · p. 6

8.3 Exploratory Factor Analysis

• The scree plot’s elbow after Factor 2 led to retaining two factors, despite Kaiser and parallel analysis suggesting eight and six.Eigenvalues fell from 7.58 for Factor 2 to 2.57 for Factor 3.Source: §8.3 · p. 6, Figure 3 · p. 6
• The analysis used minimum-residual extraction and varimax rotation, a simplifying uncorrelated-factor assumption chosen for a more balanced item distribution.The authors acknowledged that the two perceptions may still correlate.Source: §8.3 · p. 6
• The final EFA retained 15 items across Value Understanding and Value Manifestation, accounting for 55.0% of common variance.The factors contained eight and seven items, respectively.Source: §8.3 · pp. 6–7, Table 2 · p. 7

9 Scale Evaluation; 9.3 Measures; 9.11 Short Form

• The evaluation compared PVAS with trust, creepiness, and technology-affinity measures; it omitted retesting because perceived alignment is interaction-influenced rather than a stable participant characteristic.The short form was derived by optimizing item subsets to reproduce full-factor scores, then rechecked for reliability and validity.Source: §9 · p. 7, §9.3 · pp. 7–8, §9.11 · p. 10

9.1 Participants; 9.2 Materials; 9.4 Procedure; 9.5 Data Screening

• The evaluation used 296 screened US/UK participants in paired car and travel-agent scenarios, with scale order randomized and inattentive responses screened out.The scenarios varied alignment through driving behavior or whether the agent understood the destination and budget preference.Source: §9.1 · p. 7, §9.2 · p. 7, §9.4 · p. 8

9.6 Confirmatory Factor Analysis

• The final 13-item PVAS retained correlated VU (8-item) and VM (5-item) factors and fit well (CFI = .969, TLI = .962, RMSEA = .058, SRMR = .047).A WLSMV robustness check agreed on CFI, TLI, and SRMR and came close on RMSEA.Source: §9.6 · p. 8

9.7 Internal Consistency

• Long-form internal consistency was high for both factors: VM α = .905 and VU α = .890.Both exceeded the .70 acceptable threshold and neither approached the .95 redundancy warning.Source: §9.7 · p. 8, Table 4 · p. 8

9.8 Discriminant Validity

• HTMT values remained below .85 for PC (.425–.579), ATI (.150–.188), and trust-related subscales (.236–.707).The authors report these results as supporting discriminant validity.Source: §9.8 · p. 9, Table 5 · p. 9

9.9 Convergent Validity

• Both factors correlated positively with total trust (VM r = .637; VU r = .595) and negatively with creepiness (VM r = −.538; VU r = −.449); ATI associations were small or nonsignificant.VM’s strongest trust-subscale association was with Reliability/Competence (r = .671, p < .001).Source: §9.9 · p. 9, Table 6 · p. 9

9.10 Known Groups Validity

• VM distinguished aligned from misaligned conditions in both scenarios (car d = .806; travel agent d = .956), while VU effects were smaller (car d = .238; travel agent d = .453).The contrasts compared careful versus risky driving and the England versus Ontario travel-agent scenario.Source: Table 7 · p. 9, §9.10 · p. 9

9.11 Short Form

• The 6-item form tracked long-form scores closely (VM r = .975; VU r = .951; total r = .960) and retained adequate reliability, validity, and sensitivity.Total scores differentiated aligned from misaligned conditions in both scenarios; VU sensitivity in the car scenario was not significant (d = 0.200, p = .076).Source: §9.11 · p. 10

10 Discussion

• The 13-item PVAS separates perceived understanding (VU) from manifestation (VM), with a six-item short form.Source: §10 · p. 10

10.1 Using the Scale in Training AI Systems

• PVAS-based training could use transcript graders or predicted user responses as rewards, but risks optimizing apparent understanding; pair scores with behavioral measures and recheck validity after optimization.Source: §10.1 · p. 10

10.2 Scale Scoring and Interpretation of Factors

• Five negatively worded VM items are reverse-scored as 6 - x; factor scores average their items, and the overall score averages both factors equally.Source: §10.2 · p. 11
• Without normative data, interpret scores comparatively across systems, conditions, or time, rather than against an absolute criterion.Source: §10.2 · p. 11
• r = .975 for VM, .951 for VU, and .960 overall: the six-item form tracks the long form closely at the person level.Source: §10.2 · p. 11
• Report both factors with the overall score: low VU/high VM may indicate acceptable behavior without conveyed understanding, while low VM/high VU may indicate understanding without corresponding action.Source: §10.2 · p. 11

10.3 Factors and Relation to Related Work

• Factor analysis separated whether a system apprehends what matters from whether its actions reflect it, rather than reproducing the proposed values-goals-preferences hierarchy.Source: §10.3 · p. 11

10.4 Implications

• PVAS can compare delegation approaches and pre- versus post-fine-tuning models, with separate VU and VM scores preserving cases where users feel understood but reject the agent’s actions.Source: §10.4 · p. 11

• The evaluation relied mainly on scenario-based videos and text rather than functional systems, so dynamic, multi-turn interactions may involve behavioral factors the vignettes do not capture.Source: §11 · p. 11
• The sample included only English-speaking participants from the US and UK; validation across more cultural groups remains future work.Source: §11 · p. 11
• The six-item short form was evaluated on the same datasets used to construct it, and independent standalone user studies could increase confidence in it.Source: §11 · p. 11

• PVAS is a psychometrically validated 13-item instrument, with a six-item short form, measuring Value Understanding and Value Manifestation in human-AI interaction.Source: §12 · p. 11