PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Protocol Personality Assessment with LLMs: Aligning Behavioral Evidence with Human Ratings

Wei Shan, Tianyi Zhang, Yuan Zong, Wenming Zheng

arXiv:2610.16050v1cs.AI

TL;DR

Different AVI protocols can assign different trait meanings to similar behavior, limiting direct reuse of human-rated evidence across protocols. CEPA aligns source behavior to the target construct space and retrieves complementary human-rated exemplars for LLM scoring. In bidirectional transfer, CEPA outperforms direct label transfer, while broader evaluation is needed to establish generality.

  • Problem

    Different interview protocols use different questions and question–trait mappings, so behavioral evidence rated under one protocol cannot be directly used under another.

    Source: §I · p. 1

  • Method

    CEPA aligns source behavioral records to the target protocol, selects complementary human-rated exemplars, and uses independent LLM raters to score without target-dataset supervision.

    Source: A. Problem Setting and Framework… · p. 3, §VIII · p. 9

  • Results

    Across bidirectional transfer between OPVA and AVI-6, CEPA outperforms direct label transfer, supporting assessment without target labels.

    Source: §VIII · p. 9

  • Takeaways & Limitations

    The conclusion identifies transferred behavior-to-judgment associations, rather than source labels, as the basis for cross-protocol assessment.

    Source: §VIII · p. 9

    Evaluation covers two independently collected interview datasets and one set of scoring models; broader evaluation is needed to establish generality.

    Source: §VII · p. 9

Abstract

from arXiv · show

Personality assessment from asynchronous video interviews (AVIs) plays a crucial role in recruitment by enabling standardized and behaviorally grounded evaluation for job applicants. However, most of the previous works are confined to a single interview protocol. Independently collected protocols differ in question sets and question-trait mapping. Thus, the human-rated evidence from one protocol cannot be used directly under another. To address this limitation, we propose the Cross-protocol Evidence-grounded Personality Assessment (CEPA) framework, which aligns behavioral representations across interview protocols. CEPA addresses protocol heterogeneity by projecting source behavioral records into the target protocol's construct-oriented representation space. It retrieves behaviorally relevant and personality-complementary human-rated exemplars to ground an ensemble of LLM raters, which enables behavior-to-personality associations without target-specific supervision. In cross-protocol experiments, CEPA reduces average mean squared error (MSE) by 43.8% (Forward) and 17.0% (Reverse) relative to the strongest baseline. Our empirical experiments show that human-rated evidence from one protocol can support personality assessment under another without target-specific supervision or retraining.

I. INTRODUCTION

AVI personality assessment depends on protocol-specific question–trait mappings, making behavioral evidence difficult to reuse across protocols. CEPA reuses human-rated evidence across protocols without target-specific supervision or optimization.

I. INTRODUCTION

  • Protocol-specific questions activate trait-relevant behaviors, and heterogeneous question–trait mappings can give the same behavior different trait interpretations across protocols.For example, organizing a team project may indicate Extraversion in one protocol and Conscientiousness in another.Source: §I · p. 1
  • CEPA reuses human-rated behavioral records across protocols through Representation Alignment, complementary evidence selection, and aggregated LLM ratings, without target-specific supervision or optimization.The framework is presented as avoiding both zero-shot evidence loss and the per-protocol supervision required by fine-tuning.Source: §I · pp. 1–2
  • 43.8% (Forward) and 17.0% (Reverse) average MSE reductions over the strongest zero-shot baseline are reported for CEPA in bidirectional transfer.Ablations attribute most of the gain to retrieved human-rated evidence and Representation Alignment.Source: §I · p. 2
  • CEPA’s stated cross-protocol finding is that behavior-to-personality associations transfer, whereas source personality labels do not; retrieved records work better as evidence than as attached profiles.A multi-rater reliability analysis separates prediction error from scoring-process stability.Source: §I · p. 2

A. Personality Assessment from Asynchronous Video Interviews

  • AVI personality models capture protocol-specific question–trait mappings, while zero-shot LLM methods use observed behavior alone and target-supervised methods require data for each protocol.The related-work discussion describes zero-shot LLM assessment – and fine-tuning approaches [17], [18] as not reusing human-rated evidence from other protocols.Source: A. Personality Assessment from… · p. 2

B. Retrieval-Augmented Generation and Evidence Transfer

CEPA treats independently collected interview datasets as a source of behavior-linked human judgments and a target assessment setting, then aligns, retrieves, and scores evidence without target-rating supervision.

B. Retrieval-Augmented Generation and Evidence Transfer

  • Unlike open-domain QA, personality assessment needs records linking observed behavior to human trait judgments, which independently collected datasets can supply.The passage notes that knowledgeable-other judgments have documented validity.Source: B. Retrieval-Augmented Generation and… · p. 2
  • Prior transfer methods move models or representations, while CEPA retrieves human-rated records as evidence and aggregates multiple LLM raters without target-specific supervision or parameter updates.The passage distinguishes this from methods that require target-domain data or parameter updates.Source: B. Retrieval-Augmented Generation and… · p. 2

A. Problem Setting and Framework Overview

  • The target dataset is held out from personality-rating supervision and retrieval-configuration selection, and assessment uses only the target subject’s behavioral observations as a query.Candidate-pool size, evidence budget, and retrieval encoder are not tuned on the target.Source: A. Problem Setting and Framework… · p. 3
  • CEPA aligns source behavior to the target protocol, selects complementary human-rated exemplars from a semantically retrieved candidate pool, and uses independent LLM raters for continuous scores.The framework’s stages are Representation Alignment, PMES, and Evidence-Grounded Continuous Scoring.Source: A. Problem Setting and Framework… · p. 3, Figure 1 · p. 3

B. Multimodal Behavioral Preprocessing

  • Source and target interview videos are independently converted into textual behavioral representations, and preprocessing does not use target-dataset personality ratings.The representations combine transcription with textual descriptions of audio and visual behaviors.Source: B. Multimodal Behavioral Preprocessing · p. 3

C. Representation Alignment

Representation Alignment makes source behavioral records compatible with the target protocol while preserving source human ratings mapped into the target construct space.

C. Representation Alignment

  • Representation Alignment has two sequential operations: Protocol Projection aligns source behavior to the target protocol, and Cue Construction makes it a construct-oriented semantic representation.The source human-rated profile is preserved through the transformation.Source: C. Representation Alignment · p. 3

1) Protocol Projection:

  • Protocol Projection maps each source profile into the target construct space, with human-rated scores taken from source annotations rather than inferred from projected behavior.The target construct space may differ in granularity from the source dataset’s native annotations.Source: 1) Protocol Projection · p. 3
  • Only source records with complete target-space profiles and structurally complete, score-free projected representations enter the knowledge base; failing records are discarded.The projection must populate every target construct slot and contain no numeric score value.Source: 1) Protocol Projection · p. 3

2) Cue Construction and Semantic Representation:

Cue Construction organizes projected behavior into score-free textual cues and embeds construct-specific cues as subject-level retrieval representations.

2) Cue Construction and Semantic Representation:

  • Each construct-oriented cue organizes observable evidence into high-pole, low-pole, transcript, audio, and video semantic slots without emitting numerical scores or diagnostic trait labels.Interview questions are not reproduced, and slot labels are removed before embedding.Source: 2) Cue Construction and Semantic… · p. 4
  • Source cues are generated from projected representations, while target cues use target transcript, audio, and video descriptions with shared construct definitions, output format, encoder, and aggregation.For source records, the audio and video slots are set to none after Protocol Projection.Source: 2) Cue Construction and Semantic… · p. 4
  • The subject-level source and target representations are construct-mean pooled embeddings of their per-construct cues.Each admitted source exemplar stores its retrieval representation, projected textual description, and target-space human-rated profile; target observations use the same cue representation.Source: 2) Cue Construction and Semantic… · p. 4

D. Personality-space Max-Min Evidence Selection

PMES selects evidence that is behaviorally relevant while covering complementary profiles in the target construct space. It uses a greedy max-min strategy over complete-profile candidates, with a similarity-based fallback when needed.

D. Personality-space Max-Min Evidence Selection

  • PMES balances behavioral relevance with complementary coverage of human-rated personality profiles in the target construct space.The target construct space induces the source-side evidence space after source-profile mapping; scores are fixed before target evaluation.Source: D. Personality-space Max-Min Evidence… · p. 4
  • Semantic similarity retrieves a candidate pool, then max-min selection chooses up to K evidence cards from complete-profile candidates.The candidate pool contains Kpre cards; selection uses a budget K ≤ Kpre.Source: D. Personality-space Max-Min Evidence… · p. 4
  • The first selected card is the semantically nearest complete-profile candidate; each later card maximizes its minimum personality-space distance from those already selected.The distance standardizes differences across constructs using standard deviations computed solely from complete source-KB profiles.Source: D. Personality-space Max-Min Evidence… · p. 4
  • If fewer than K complete-profile candidates are available, remaining positions are filled in decreasing semantic-similarity order.PMES uses only complete-profile candidates for its greedy selection and requires neither target personality labels nor target-side personality statistics.Source: D. Personality-space Max-Min Evidence… · p. 4

E. Evidence-Grounded Continuous Scoring

Cont3 produces continuous scores for the target constructs by averaging valid independent LLM ratings. Raters receive the target assessment context, retrieved human-rated evidence, and the target subject’s behavioral description.

E. Evidence-Grounded Continuous Scoring

  • Cont3 predicts a continuous score vector over the target constructs, applying predefined aggregation when target measures are coarser.The construct-level scores and ground-truth targets are aggregated consistently.Source: E. Evidence-Grounded Continuous Scoring · p. 4
  • Each independent rater receives target construct definitions and scenarios, selected evidence cards with human-rated profiles, and the target subject’s multimodal behavioral description.Source: E. Evidence-Grounded Continuous Scoring · p. 4
  • Three independent LLM raters produce scores from 1 to 5; subjects need at least two complete rater outputs, which are averaged for the final prediction.Raters output only requested scores, without intermediate reasoning in the aggregation.Source: E. Evidence-Grounded Continuous Scoring · pp. 4–5

IV. EXPERIMENTAL SETUP

The experiments test bidirectional transfer between two AVI datasets with different question designs and annotation granularity. Retrieval settings are fixed before target evaluation, and performance is measured primarily by equally weighted construct-level MSE.

A. Datasets

  • OPVA and AVI-6 differ in question design and annotation granularity, enabling evaluation in both transfer directions.AVI-6 has factor-level annotations, while OPVA facet-level annotations are aggregated to factors; source profiles are mapped into the target construct space before indexing.Source: A. Datasets · p. 5

B. Implementation Details

  • Implementation uses fixed retrieval settings and the same three open-source LLM raters across transfer directions.Visual and audio descriptions are generated with Qwen2.5-VL-7B-Instruct and Fun-Audio-Chat-8B; cue texts are generated and encoded for representation alignment, with closed-source models used only for representation construction. Retrieval is fixed at (Kpre, K) = (50, 5) without target-side retuning; subjects with fewer than two complete rater outputs are excluded.Source: B. Implementation Details · p. 5

C. Evaluation Metrics

  • Primary MSE is computed in the original score space and averaged equally across evaluation constructs; MAE is complementary.Evaluation constructs are {H, X, A, C} for Forward and aggregated {X, C} for Reverse. Paired subject-level bootstrap comparisons use 10,000 resamples and 95% confidence intervals; negative ∆MSE favors Ours.Source: C. Evaluation Metrics · pp. 5–6

V. RESULTS

CEPA improves target-label-free performance in both transfer directions compared with the strongest evidence-free baseline. The advantage is not uniform across constructs, but paired comparisons also favor CEPA over an external zero-shot textual baseline.

A. Comparison with Baselines

  • The baseline comparison isolates retrieved evidence using textualized models on the same behavioral descriptions and also includes native audio or video baselines.Target-supervised methods are excluded because they require target personality labels and retraining for each protocol.Source: A. Comparison with Baselines · p. 6
  • 43.8% Forward and 17.0% Reverse primary MSE reductions over the strongest evidence-free baseline accompany CEPA’s best target-label-free MSE in both directions.Forward MSE is 0.2279 versus 0.4052 for Ours w/o evidence; Reverse MSE is 0.1651 versus 0.1990.Source: A. Comparison with Baselines · p. 6
  • Construct-level gains are selective: CEPA leads on three Forward constructs and both Reverse constructs, while Qwen3.5-Plus leads on Forward Extraversion.For Forward Extraversion, Qwen3.5-Plus has MSE 0.1937 versus 0.2249 for CEPA.Source: A. Comparison with Baselines · p. 6
  • All four paired bootstrap intervals lie below zero against both the evidence-free ensemble and Qwen3.5-Plus across the two directions.Against Qwen3.5-Plus, the Forward improvement is a 49.9% relative error reduction.Source: A. Comparison with Baselines · p. 6

B. Component Ablation

Matched ablations isolate retrieved evidence, PMES, and Representation Alignment while holding the remaining assessment configuration constant. Retrieved evidence and alignment contribute distinct gains, with their relative importance differing by transfer direction.

B. Component Ablation

  • The ablations change one component at a time while keeping target subjects, annotations, scoring models, and assessment configuration matched.Alignment removal jointly ablates Protocol Projection and Cue Construction because Cue Construction uses the protocol-projected source representation.Source: B. Component Ablation · p. 6
  • Removing retrieved evidence raises MSE from 0.2279 to 0.4052 in Forward and from 0.1651 to 0.1990 in Reverse transfer.Paired bootstrap comparisons confirm subject-level improvements from retrieved human-rated evidence in both directions.Source: B. Component Ablation · p. 6, Table III · p. 7
  • Removing PMES raises MSE to 0.2838 in Forward and 0.1750 in Reverse transfer.The paired MSE differences are −0.0559 (95% CI [−0.0695, −0.0424]) in Forward and −0.0099 (95% CI [−0.0197, −0.0001]) in Reverse.Source: B. Component Ablation · p. 6
  • Removing Representation Alignment raises MSE to 0.3636 in Forward and 0.4235 in Reverse transfer.Alignment removal causes the largest single-component loss in Reverse, while evidence removal causes the largest in Forward.Source: B. Component Ablation · p. 6
  • Retrieved evidence provides the primary gain in Forward, Representation Alignment in Reverse, and PMES a complementary gain through more diverse evidence.Alignment is necessary for cross-protocol comparability in both directions.Source: B. Component Ablation · pp. 6–7

VI. DISCUSSION

The discussion asks whether retrieved records transfer as personality labels or as evidence, and how robustly models interpret that evidence. Its analyses support evidence construction and multi-rater aggregation rather than direct label transfer.

A. What Transfers, and How Should the Evidence Set Be Built?

  • Across both retrieval sets, evidence-grounded Cont3 outperforms averaging retrieved personality labels, with all four subject-level ∆MSE intervals excluding zero.Behaviorally similar source subjects are therefore not reliable surrogate labels for target profiles.Source: A. What Transfers, and How Should the… · p. 7, Table V · p. 8
  • PMES maximizes minimum within-set personality-space distance; larger candidate pools raise it, while larger evidence budgets lower it.The authors fix (Kpre, K) = (50, 5) on engineering grounds, not by tuning downstream prediction error.Source: A. What Transfers, and How Should the… · pp. 7–8
  • Cross-protocol retrieval is framed as evidence construction: retrieved records jointly provide complementary information about interpreting behavior, rather than surrogate labels.The personality rating remains a judgment about the target.Source: A. What Transfers, and How Should the… · pp. 7–8

B. How Robustly Can Transferred Evidence Be Interpreted?

  • Cont3 is clearly better than the best single model in Forward but essentially tied with Qwen3-14B in Reverse.The benefit of aggregation for prediction error therefore differs by transfer direction.Source: B. How Robustly Can Transferred… · p. 8
  • ICC(2,1) ranges overlap (0.272–0.678 Forward; 0.500–0.834 Reverse), while pairwise correlations separate the directions more clearly.These are point estimates without confidence intervals or a formal between-direction test; ICC measures scoring stability, while MSE measures closeness to the target.Source: B. How Robustly Can Transferred… · pp. 8–9
  • The analyses support cross-protocol assessment as empirical evidence transfer, with retrieval framed as evidence construction and multi-rater aggregation as a robustness mechanism.Source: B. How Robustly Can Transferred… · p. 9

VII. LIMITATIONS

CEPA supports cross-protocol scoring by aligning multimodal records and interpreting compact retrieved evidence through multiple raters; bidirectional OPVA–AVI-6 transfer outperforms direct label transfer. The evidence is bounded by semantic compression and evaluation on two datasets with one set of scoring models.

VII. LIMITATIONS

  • Semantic text enables cross-protocol retrieval but sacrifices modality-specific signals and fine-grained temporal structure; retaining raw video and audio sharply raises retrieval cost.Source: §VII · p. 9
  • Evaluation covers two interview datasets and one set of scoring models; broader evaluation is needed, and the performance cost of forgoing target supervision is unquantified.Source: §VII · p. 9

VIII. CONCLUSION

  • CEPA aligns multimodal records to the target protocol, builds a compact evidence pool, and interprets it through multi-rater scoring.Source: §VIII · p. 9
  • Across bidirectional transfer between OPVA and AVI-6, CEPA outperforms direct label transfer.Source: §VIII · p. 9
  • Transfer depends on behavioral comparability and complementary, jointly interpreted evidence; observed behavior-to-judgment associations transfer, not source labels.Source: §VIII · p. 9

Related papers

Loading 2610.16050v1…