PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for Personality Detection

1 Introduction

• Earlier text-based personality detectors focused on training or fine-tuning models, whose performance is limited by training-data quality and quantity.The paper investigates whether LLMs can infer an author’s personality from text, a capability it says had not been extensively explored.Source: §1 · pp. 1–2
• PsyCoT rates questionnaire items from an author’s text in successive dialogue turns, then uses the rating history to infer a personality trait.Questionnaire-defined rules can aggregate the ratings into an overall score that serves as a double check and provides confidence.Source: §1 · p. 2
• PsyCoT improves average F1 over standard prompting by 4.23/10.63 points across the Essays and Kaggle datasets.The datasets use Big Five and MBTI taxonomies, respectively.Source: §1 · p. 2
• Ablations indicate that multi-turn dialogue improves reasoning accuracy over single-turn dialogue, and PsyCoT is robust to option-order perturbations.Source: §1 · p. 2
• The paper presents PsyCoT as the first exploration of LLM personality detection and frames psychological questionnaires as explicit CoT prompts.It describes personality detection as useful in applications including understanding psychiatric disorders, personalized treatment, and human-robot interaction.Source: §1 · pp. 1–2

2 Related Work

• Earlier personality-detection methods use data-driven training to capture implicit personality cues, whereas this study explores zero-shot LLM performance.The paper notes that training-based methods depend on data quality and quantity.Source: §2 · p. 2
• Related approaches extract LIWC features, learn user representations with post-aware models or graphs, or incorporate knowledge such as questionnaires.Examples include TrigNet, which uses LIWC and a heterogeneous graph, and PQ-Net, which uses questionnaires as additional guidance.Source: §2 · p. 2
• Unlike CoT variants that often focus on selecting few-shot exemplars, this work uses psychological questionnaires explicitly as reasoning steps.Source: §2 · p. 3

3 Psychological Questionnaire as CoT

• The task is to infer an individual's personality from their text X using a task-description prompt D and an inference prompt I.The inference prompt specifies the desired personality option and output format.Source: §3 · p. 3

3.1 Standard Prompting

• Standard prompting directly infers personality from author text using task-description and inference prompts, relying on the LLM to evaluate personality.The inference prompt supplies options and instructions for a fixed-format choice.Source: §3.1 · p. 3

3.2 PsyCoT Prompting

• Because personal texts may lack explicit references to personality, PsyCoT uses questionnaire items as a chain-of-thought framework for personality detection.Source: §3.2 · p. 3
• PsyCoT instructs the LLM to rate each questionnaire statement, explains the rating scale, and flags reversed statements.The example scale ranges from strongly disagree to strongly agree.Source: §3.2 · p. 4
• PsyCoT inserts K questionnaire-derived reasoning steps before the final personality inference through a multi-turn dialogue.The figure contrasts these steps with standard prompting's direct personality output.Source: §3.2 · p. 4, Figure 2 · p. 4
• PsyCoT's item ratings can be aggregated into an overall score like self-assessment tests, with reversed statements identified as a set.Source: §3.2 · p. 4
• The item-level rating results can also serve as a double check and provide confidence for the inferred personality trait.Source: §3.2 · p. 4
• The authors argue that rating items separately focuses the LLM, while carrying prior ratings forward can improve consistency across correlated items.They cite item pairs about being reserved as an example of correlated statements.Source: §3.2 · pp. 4–5

4.1 Datasets

• Essays contains 2,468 texts labeled for Big Five traits, while Kaggle contains 8,675 MBTI users; API limits restricted Kaggle evaluation to 200 test samples.Essays was split 80/10/10 for training, validation, and testing; prompt-based methods were evaluated zero-shot on its test set.Source: §4.1 · p. 5

4.3 Implementation Details

• The experiments simplify multi-label personality detection into multiple binary classification tasks and use Accuracy and Macro-F1 for evaluation.GPT-3.5 API outputs were generated at temperature 0; fine-tuned results average five runs.Source: §4.3 · pp. 5–6

4.4 Overall Results

• On Essays, PsyCoT improves over standard prompting by 0.94/4.23 points in accuracy/macro-F1; on Kaggle, the gains are 18.37/10.63 points, though it trails DDGCN.On Kaggle, PsyCoT is close to BERT; the reported gains are over standard prompting.Source: Table 1 · p. 6, §4.4 · pp. 6–7, Table 2 · p. 6

4.2 Baselines

• Baselines include shallow, non-pretrained, and fine-tuned models, alongside standard prompting and Zero-shot-CoT.Zero-shot-CoT adds “Let’s think step by step” before the final personality inference prompt.Source: §4.2 · p. 5, §4.4 · p. 6

4.4 Overall Results

• On Essays, PsyCoT performs worse than standard prompting for Neuroticism, which the authors associate with negative-emotion bias in the texts.On Kaggle, its gains over Zero-shot-CoT are 1.50/4.08 points in accuracy/macro-F1.Source: §4.4 · pp. 6–7

4.5 Ablation Study

• Table 3’s ablations show that single-turn reasoning performs worse than PsyCoT’s multi-turn dialogue approach.The multi-turn structure lets the LLM focus on one questionnaire item at a time.Source: Table 3 · p. 7, §4.5 · p. 7
• The 44-item Big Five Inventory performs better than the shorter TIPI and Mini-IPIP alternatives, supporting careful questionnaire selection for PsyCoT.The paper attributes the 44-item inventory’s potential advantage to its broader coverage of personality aspects.Source: §4.5 · pp. 7–8

5.1 Correlation Analysis

• Trait scores correlate above 0.70 with chosen personality types for Agreeableness, Extraversion, and Openness, while questionnaire-rating distributions suggest potential bias on other traits.Extraversion and Openness ratings align with questionnaire criteria; distributions for the other traits intersect the dashed line and shift to one side.Source: §5.1 · p. 8

5.2 Statistical Tests

• Improvements over the Standard prompt are statistically significant for AGR (p < 0.05) and CON, EXT, and OPN (p < 0.001).Each point in Table 4 represents the average of five runs.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Robustness Testing

• PsyCoT achieves the highest unchanged rate among the tested methods when option orders are perturbed.The experiment swaps option orders in the task-description and inference prompts and evaluates unchanged predictions across 100 samples.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4 Impact of Post Order

• Shuffling post orders noticeably changes predictions for several personality traits, including PsyCoT.The study evaluates prompt-based methods on 100 samples using unchanged rate.Source: §5.4 · p. 9, Figure 5 · p. 9

• PsyCoT uses questionnaire items as chain-of-thought steps, guiding the LLM to rate each item from the author’s text before inferring a personality trait.Source: §6 · p. 9
• Across two benchmark datasets, PsyCoT outperforms Standard prompting and Zero-shot-CoT approaches and achieves comparable performance to some fine-tuned models.Source: §6 · p. 9
• The work presents psychological questionnaires as a way to elicit LLM reasoning in zero-shot personality detection.Source: §6 · p. 9

• PsyCoT’s multi-turn item-rating approach requires more API requests than the single-turn method.Source: Limitations · p. 9
• PsyCoT’s performance is heavily influenced by questionnaire choice, requiring effort to find a suitable questionnaire.Source: Limitations · p. 9
• PsyCoT has been applied only to one language, and its use in additional languages remains to be explored.Source: Limitations · p. 9

• The study uses publicly available datasets with anonymized user information and reports adherence to data-use policy.Source: Ethics Statement · p. 9
• The authors acknowledge that misuse of text-based personality detection may pose ethical risks.Source: Ethics Statement · p. 9
• The authors permit research use but prohibit covert inference of individuals’ personality traits or other sensitive characteristics.Source: Ethics Statement · p. 9

A Appendix: Form of the Task

• The Essays dataset converts Big Five ratings from regression into binary labels, mapping “High” to “Y” and “Low” to “N.”The original rater scores are not released.Source: A Appendix: Form of the Task · p. 12
• Simplifying multi-label personality detection into multiple binary classification tasks leads to performance improvements.The study reports average standard-prompt results in Table 5, whose caption identifies a comparison of different forms.Source: A Appendix: Form of the Task · p. 12, Table 5 · p. 12
• Table 5 presents comparison results for different task forms.Source: Table 5 · p. 12

B Appendix: Dialogue Record

• The records show PsyCoT predicting Big Five and MBTI personality traits.Source: B Appendix: Dialogue Record · p. 12

C Appendix: Details of Datasets

• Table 6 provides statistics for the Essays and Kaggle datasets.Source: C Appendix: Details of Datasets · p. 12, Table 6 · p. 12

D Appendix: Details of Questionnaires

• The study uses the 44-item Big Five Inventory and Myers-Briggs Type Indicator, adds a “Not sure whether A or B” option, and compares TIPI and Mini-PIPI in Section 4.5.Item details are listed in Tables 7 and 10; TIPI and Mini-PIPI details are listed in Tables 8 and 9, respectively.Source: D Appendix: Details of Questionnaires · p. 12
• The MBTI items add a “Not sure whether A or B” option so the LLM can indicate uncertainty when information is insufficient.Source: D Appendix: Details of Questionnaires · p. 12
• Figures 6 and 7 provide complete PsyCoT prediction records for Big Five and MBTI traits, respectively.Source: Figure 6 · p. 13, Figure 7 · p. 14