Source-linked AI summary
PsyJev: Psychological Analysis via System-1 Judgment
Fangyuan Liu, Sirui Zhao, Xiaohao Wang, Guo Cheng, Peng Zhang, Enhong Chen
TL;DR
Web self-disclosures support varied psychological analyses, but similar text can require different question-specific judgments across heterogeneous datasets. PsyJudgeBench organizes these judgments while preserving source criteria, and PsyJev directly predicts answer distributions with JA-RLCD refinement. Across three model scales, PsyJev achieves average F1 scores of 64.33, 66.21, and 67.56, outperforming corresponding answer-SFT baselines.
Problem
Psychological analyses of Web content require distinct question-specific judgments, while datasets differ in criteria, label schemes, coverage, and supervision balance.
Source: §1 · p. 1Method
Results
Across 2B, 4B, and 9B scales, PsyJev achieves average F1 scores of 64.33, 66.21, and 67.56, respectively, outperforming corresponding answer-SFT baselines.
Source: §6 · p. 8Takeaways & Limitations
The reported results demonstrate the value of a shared, question-conditioned approach to heterogeneous psychological content analysis.
Source: §6 · p. 8The judgments describe annotated textual content and do not establish clinical diagnoses; reuse must follow source access and usage terms and safeguard sensitive self-disclosures.
Source: §5.4.5 · p. 8
Abstract
from arXiv · showhide
Self-disclosures on the Web support diverse psychological analyses, from recognizing expressed experiences to identifying symptom-relevant information. We study a shared, question-conditioned decision interface for this heterogeneous problem in content analysis. We organize PsyJudgeBench, comprising 13 annotated datasets and 82 questions across six psychological areas, while preserving source-specific criteria and masks for unobserved labels. We introduce PsyJev, a System-1 model that directly produces distributions over answers specified in each request. Joint supervised training establishes the shared judgment function, followed by Judgment-Anchored RLCD (JA-RLCD), which adapts logit-space reward optimization to heterogeneous psychological judgments through balanced supervision, selective retention of correct SFT predictions, and question-group reward optimization. The Qwen3.5-based 2B, 4B, and 9B models achieve 64.33, 66.21, and 67.56 average test F1, respectively. The 9B model exceeds the strongest external comparison by 2.54 points. Controlled 2B comparisons examine the training stages and anchored refinement. Further analyses examine question-conditioned judgment, inference cost, and probability quality, complemented by qualitative comparisons across models.
1 Introduction
The paper frames Web-based psychological analysis as a problem of making distinct, question-specific judgments across heterogeneous datasets. It introduces a shared benchmark and direct-prediction model, then evaluates the model across scales and through controlled refinement comparisons.
1 Introduction
- Different analyses require different criteria: a post may mention a symptom without expressing it or lack enough detail to answer a questionnaire item.The paper identifies heterogeneous judgment semantics, fragmented supervision, and uneven dataset sizes and label frequencies as challenges for a shared framework.Source: §1 · p. 1
- PsyJudgeBench organizes 13 datasets into 82 questions across six psychological areas while retaining source-specific criteria and masking unobserved labels.Coordinated splits support joint training and task-specific evaluation under partial supervision.Source: §1 · pp. 1–2
- Across 2B, 4B, and 9B scales, PsyJev outperforms the corresponding same-backbone answer-SFT baselines.PsyJev directly predicts distributions over question-defined answers, with JA-RLCD combining balanced supervision, selective retention of correct SFT predictions, and question-group reward optimization.Source: §1 · p. 1
- In controlled 2B comparisons, complete JA-RLCD outperforms continued cross-entropy training and anchor-only refinement.The paper also examines question-conditioned discrimination on co-annotated texts, inference latency, and probability quality.Source: §1 · p. 2
2 Related Work
Prior work covers diverse psychological targets and shared prediction approaches, while System-1 models provide structured decisions without autoregressive answer generation. PsyJev applies this approach to heterogeneous psychological judgments through joint supervision and JA-RLCD.
2.1 Psychological Analysis
- Psychological-analysis work spans specialized targets and instruction-tuned multi-task systems; PsyJudgeBench frames heterogeneous tasks as question-defined judgments.Existing tasks share related content but retain distinct analytical targets and annotation criteria.Source: §2.1 · p. 2
2.2 System-1 Decision Models
- System-1 models produce structured predictions without autoregressive answer generation, and PsyJev extends this approach to heterogeneous judgments using joint supervision and JA-RLCD.JA-RLCD adapts logit-space RLCD with a class-balanced judgment anchor, selective SFT-reference retention, and question-group reward optimization using pathwise gradients.Source: §2.2 · p. 2
3 PsyJudgeBench
PsyJudgeBench brings heterogeneous psychological datasets into a shared registry of question-defined judgments while preserving their annotation criteria. Its construction harmonizes source labels and coordinates evaluation partitions under partial supervision.
3 PsyJudgeBench
- PsyJudgeBench contains 53,775 source records and 222,191 observed judgments from 13 annotated datasets across six psychological content areas.Construction includes a question registry, source-specific annotation conversion, and coordinated data partitions.Source: §3 · p. 2
- Each question-defined judgment specifies a target, decision criterion, valid answers, and observed answer; the registry contains 82 questions, including 79 binary and three multiclass questions.The formulation preserves original criteria, including questionnaire answerability, symptom relevance, and wellness-dimension identification.Source: §3 · p. 2
- Annotation harmonization maps source labels to answer sets, expands multi-label and category–time annotations into questions, and masks conflicting judgments while retaining unaffected supervision.For TherapistQA, the construction removes two records with conflicting dominant labels and masks 22 conflicting judgments across 11 same-text/same-target keys.Source: §3 · p. 3
- Unannotated questions are excluded rather than treated as negative examples, and coordinated partitions group shared texts, source identifiers, and identified near-duplicates across splits.The resulting partitions contain 33,511 training, 5,852 validation, and 14,412 test source records.Source: §3 · p. 3
4 PsyJev
PsyJev makes heterogeneous psychological judgments through a shared, question-conditioned distribution over legal answers. Joint SFT establishes the shared decision function, then JA-RLCD refines it through anchored reward optimization.
4 PsyJev
- PsyJev directly predicts distributions over question-defined answers, using joint SFT followed by JA-RLCD refinement.The two-stage framework combines judgment anchoring, logit-space exploration, and question-group reward optimization.Source: §4 · p. 3, Figure 2 · p. 4
4.1 System-1-Style Direct Judgment
- Question-conditioned inputs combine text, question, decision criterion, and candidate descriptions; legal-answer logits are normalized into a distribution without autoregressive generation.The same scoring procedure supports binary and multiclass judgments without task-specific classification heads.Source: §4.1 · p. 3
4.2 Supervised Fine-Tuning
- Joint SFT learns across source datasets, weighting sources equally and averaging only observed questions so missing labels contribute no loss.The resulting SFT checkpoint initializes JA-RLCD refinement.Source: §4.2 · p. 3
4.3 Judgment-Anchored RLCD
- The judgment anchor balances source/question observation frequency and answer classes, while retaining the frozen SFT distribution only when its prediction is correct.The label term anchors changes to observed answers; the reference term retains the SFT distribution when it agrees with the annotation.Source: §4.3 · p. 4
- JA-RLCD optimizes negative class-weighted Brier rewards aggregated across question groups, with a soft minimum emphasizing lower-reward groups and pathwise gradients through perturbed logits.It uses paired Gaussian logit-space perturbations per judgment and propagates reward gradients through legal-answer distributions.Source: §4.3 · p. 4
5 Experiments
Experiments evaluate PsyJev on PsyJudgeBench against adapted baselines and through ablations and analyses of question-conditioned judgments, efficiency, prediction quality, and transfer. PsyJev achieves the highest average F1 within each model-size group, while JA-RLCD comparisons and further analyses examine its performance across questions and settings.
5.2 Performance Evaluation
- PsyJev achieves 64.33, 66.21, and 67.56 average F1 at 2B, 4B, and 9B, respectively, and ranks highest within each model-size group.The 4B and 9B variants also outperform the strongest external baseline.Source: §5.2 · p. 5, Table 1 · p. 6
5.3 Ablation Study
- JA-RLCD reaches 64.33 average F1, 3.65 points above SFT, and exceeds continued CE by 2.61 points and adapted RLCD by 5.46 points.Full JA-RLCD also exceeds class-balanced CE by 1.04 points and anchor-only training by 0.63 points; the adapted-RLCD comparison does not isolate reward formulation from gradient estimator.Source: §5.3.1 · p. 6, §5.3.2 · p. 6, Table 2 · p. 6
5.3.3 Performance Across Questions
- On 16 validation-selected difficult questions, JA-RLCD improves mean test F1 over SFT by 19.45 points, compared with 6.23 points across all 82 questions.Gains on the difficult subset remain positive against continued CE and anchor-only training, but improvements are not uniform across questions.Source: §5.3.3 · pp. 6–7, Table 3 · p. 7
5.4.1 Question-Conditioned Judgment
- PsyJev correctly answers both questions in 70.06% of opposite-label pairs, versus 66.24% for the same-size answer model, 67.85% for Gemma-2-9B, and 55.84% for GLiClass.The evaluation covers 35,992 pairs from eight sources.Source: Table 4 · p. 7, §5.4.1 · p. 7
5.4.2 Inference Efficiency
- PsyJev runs 2.10× faster than full-answer scoring and gains 2.38 F1 points over first-token scoring at comparable GPU latency.The comparison uses three 4B configurations and a shared timing protocol.Source: Figure 3 · p. 7, §5.4.2 · p. 7
5.4.5 Qualitative Analysis
- Qualitative cases show JA-RLCD correcting SFT label choices across co-occurring emotions, competing distress sources, and related cognitive distortions; these judgments are not clinical diagnoses.Reuse should follow each source dataset’s access and usage terms and safeguard sensitive self-disclosures.Source: Table 6 · p. 8, §5.4.5 · p. 8
5.4.4 Unseen-Dataset Transfer
- On unseen Dreaddit test data, PsyJev reaches 77.76 F1 at 2B and 78.70 at 4B, exceeding the corresponding answer-SFT baselines by 3.44 and 2.17 points.The 715 test examples were excluded from PsyJudgeBench supervision; gains over direct-judgment SFT initialization vary by model scale.Source: Table 7 · p. 8, §5.4.4 · p. 8
5.4.3 Prediction Quality
- PsyJev-4B achieves the lowest Brier score and normalized NLL among displayed models, and the lowest ECE at 4.58%, outperforming GLiNER2 on all three metrics.Metrics use the same source-balanced test evaluation.Source: §5.4.3 · pp. 7–8, Table 5 · p. 7, Figure 4 · p. 7
6 Conclusion
The paper presents PsyJudgeBench and PsyJev as a shared framework for question-conditioned psychological analysis of Web content. Across three model scales, PsyJev outperforms corresponding answer-SFT baselines.
6 Conclusion
- PsyJev reaches average F1 scores of 64.33, 66.21, and 67.56 at 2B, 4B, and 9B scales, outperforming corresponding answer-SFT baselines.The conclusion also reports additional analyses of question-conditioned discrimination, inference efficiency, and probability quality.Source: §6 · p. 8
- PsyJudgeBench organizes heterogeneous tasks while retaining their distinct decision criteria, and PsyJev directly predicts answer distributions with JA-RLCD refinement.The paper frames these contributions as a unified approach to psychological analysis of Web content through question-conditioned judgments.Source: §6 · p. 8
A Data Construction and Splits
PsyJudgeBench preserves source-specific judgment criteria and masks labels that are unobserved. Coordinated splits keep shared texts and source groups within one partition, with a final audit documenting lexical overlap.
Label construction
- Missing labels remain masked, while label construction preserves source-specific criteria, including symptom relevance for BDI-Sen and patient-question and dominant-distortion labels for TherapistQA.Thinking Traps merges normalized repeated completions into 15-category label sets; CRADLE expands seven crisis categories across ongoing/past status into 14 questions, and missing CAMS categories remain unobserved.Source: A Data Construction and Splits · p. 9, Table 9 · p. 10
Split coordination
- Exact texts and available source groups are coordinated across all 13 sources, with test taking priority; 22 conflicting judgments across 11 identical text–target keys are masked.Shared texts retain their distinct source-defined judgments; normalization uses Unicode NFKC, case folding, and whitespace collapse.Source: A Data Construction and Splits · p. 9
TherapistQA refinement
- TherapistQA refinement removes two identical questions with conflicting labels, then assigns normalized-question groups by seed-42 hash split while inheriting exact-match assignments.Near-duplicate components follow the same held-out priority; ANGST retains silver training/validation and expert-labeled test annotations, while CRADLE’s test partition contains 600 gold and one silver record.Source: A Data Construction and Splits · p. 10
Final-split audit
- All 3,600 cross-source shared-text groups are confined to one split, including all 3,518 IRF–LonXplain shared texts; no source-group identifier crosses partitions.The audit measures lexical instance reuse on full texts and their first 512 tokens.Source: A Data Construction and Splits · p. 10, Table 10 · p. 10
B Reproducibility and Comparison Protocols
Evaluation uses Qwen3.5 models with a constrained answer readout. Inputs retain question criteria and legal answer descriptions, while multilabel decisions are made separately for observed binary questions.
Input processing
- Qwen3.5 2B, 4B, and 9B inputs truncate source text at 512 tokens while preserving question criteria and legal answer descriptions in full.The models use native tokenizers, and input text is encoded without added special tokens.Source: §B.1 · p. 10
- Requests serialize Context, Question, and letter-prefixed answer options, then use the backbone chat template with thinking disabled and an appended Answer: prefix.The answer prefix is separately tokenized for constrained readout.Source: §B.1 · p. 10
- For multilabel sources, each observed binary question is scored separately, and yes is assigned only when p(yes) > p(no), with ties assigned to no.Source: §B.1 · p. 10
B.2 Training and Matched Controls
Training uses a shared schedule with balanced judgment supervision and a frozen SFT reference for refinement. Baseline and control protocols vary the scoring objective, refinement components, and evaluation setup.
B.2 Training and Matched Controls
- Each stage visits all 33,511 training records once in 524 updates, summing weighted gradients across source-homogeneous microbatches before each optimizer step.All runs use seed 42; microbatches contain at most 16 judgments, and the final checkpoint is retained.Source: §B.2 · p. 10
- JA refinement initializes from the same-scale final SFT checkpoint and keeps a frozen reference; training-count priors and weights and training-label correctness gates shape refinement.Two Gaussian directions and their negatives share model logits per microbatch; the noise scale falls from 0.4 to 0.1, and configuration selection uses validation only.Source: §B.2 · pp. 10–11
B.3 Baseline Protocols
- Benchmark-adapted baselines share cleaned records, the question registry, and observed-label masks, while fixed-head and description-driven models encode decisions differently.ModernBERT selects among three epochs using validation F1, then NLL and earlier-epoch tie-breaking; MentaLLaMA receives benchmark-specific answer SFT [33].Source: §B.3 · p. 11
- Laya is reproduced with one full-parameter SFT epoch followed by one RLCD epoch, using the benchmark training partition and final checkpoints.Its RLCD combines cross-entropy with score-function optimization over Gaussian-perturbed answer logits.Source: §B.3 · p. 11
- The jev API is evaluated zero-shot on all 44,999 test judgments, with invalid responses retained as incorrect predictions.Binary and multiclass questions use Noul and Choice; invalid labels are excluded from the legal-class macro average when computing F1.Source: §B.3 · p. 11
- The adapted RLCD control combines ordinary CE with Laya’s categorical score-function objective, without JA’s reference gate or question-group aggregation.It retains SFT source/record weights and uses four centered Gaussian logit actions with spherical and ranked-probability reward weights.Source: §B.3 · p. 11
- Figure 5 compares quality and latency for the 16 models whose average F1 is at least 57%.Source: Figure 5 · p. 11
- Ablations separately remove reward optimization, class costs, or reference KL, or replace pathwise differentiation with paired score-function gradients.The ordinary Brier control jointly changes grouping and tail weighting, while balanced-CE controls compare direct scoring with answer-sequence CE under matched schedules.Source: §B.3 · p. 11
B.4 Unified Timing Protocol
The timing protocol compares model latency under a shared GPU setup, while the probability evaluation uses a common set of test judgments and source-balanced aggregation. It also defines how probability quality and the Figure 4 calibration curve are evaluated.
B.4 Unified Timing Protocol
- 25.25, 11.87, and 12.01 ms per judgment are the respective latencies for 4B full-answer scoring, first-token scoring, and PsyJev.All were measured under the same protocol; full-answer and first-token scoring share the Qwen3.5-4B answer-SFT checkpoint, while PsyJev uses its final JA-RLCD checkpoint.Source: §B.4 · p. 11
- Table 11 reports final PsyJev-4B probability quality by source, with its mean weighting all 13 sources equally.The caption identifies the table’s scope but supplies no metric values.Source: Table 11 · p. 12
C.1 Probability Evaluation Protocol
- The probability evaluation uses 44,999 test judgments and legal answer sets, with no post-hoc calibration.Classification models normalize answer scores over legal options; answer-generation models normalize complete answer-sequence likelihoods, including EOS.Source: §C.1 · p. 12
- NLL is averaged within records and then equally across records and sources, while Brier score and 15-bin ECE are averaged equally across sources.The source-bin denominator for Figure 4 is the full source judgment count, making E≤15 equal to source-mean ECE in Table 5.Source: §C.1 · p. 12
- Figure 4’s source-bin calibration curve uses each bin’s count, accuracy, and mean confidence, with empty bins contributing zero.The supplied description identifies these quantities but does not report a curve outcome.Source: §C.1 · p. 12
C.2 Unseen-Dataset Transfer
The transfer and shared-text analyses specify evaluation settings and comparisons, while case-level examples track answer-probability changes after JA-RLCD. The supplied passages describe these analyses but do not provide their overall comparative outcomes.
C.2 Unseen-Dataset Transfer
- The Dreaddit transfer test uses all 715 examples without Dreaddit supervision or further tuning, and the registered question asks whether the author expresses psychological stress.The setup uses the first 512 tokens, legal-answer argmax, and F1 with yes as positive; the training/validation audit reports no exact, high-overlap, or containment matches.Source: §C.2 · p. 12, Table 12 · p. 12
C.3 Questions about Shared Text
- The shared-text comparison uses identical opposite-label binary pairs, three outcomes, equal source weights, and paired text-group resampling.Table 13’s caption says “Same answer” means exactly one prediction is wrong in this subset; the supplied passages do not state the outcome values.Source: §C.3 · p. 12, Table 13 · p. 12
C.4 Case-Level Probability Shifts
- In Case A, JA-RLCD moves Sadness below 0.5 while Emptiness and Loneliness remain positive; in Cases B and C, the gold label overtakes the strongest competitor.The probability displays distinguish relative preference changes from uniform confidence increases, across three cases from DepressionEmo, CAMS, and TherapistQA.Source: §C.4 · p. 12, Table 14 · p. 12