Source-linked AI summary
COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
Baihan Lin
TL;DR
Language-derived assessment can resemble self-report because similarly worded items attract similar speech, leaving unclear how much psychological structure comes from the person. The method introduces psychometric representational similarity analysis to compare representations across questionnaire responses, clinicians’ ratings, model outputs, and theory using dissimilarity matrices over shared conditions. Across two interview samples and both embedding models, symptom geometry followed wording more than self-report, with no registered evidence of population structure beyond wording.
Problem
Language-derived assessment can resemble self-report because similarly worded items attract similar speech, leaving unclear how much psychological structure comes from the person.
Source: Introduction · p. 1Method
The method introduces psychometric representational similarity analysis to compare representations across questionnaire responses, clinicians’ ratings, model outputs, and theory using dissimilarity matrices over shared conditions.
Source: Introduction · p. 1Results
Across two interview samples and both embedding models, symptom geometry followed wording more than self-report, with no registered evidence of population structure beyond wording.
Source: Introduction · p. 2Takeaways & Limitations
Population geometry describes relations among items but not whether language-derived measures track the right person, which requires paired validation.
Source: Discussion · p. 10Modest samples, one interview per participant, short answers, and inference conditional on 25 fixed items constrain detection of small effects and individual geometry.
Source: Discussion · p. 11
Abstract
from arXiv · showhide
Language models can score psychiatric questionnaires from speech, but agreement with self-report may reflect the questionnaire rather than the person. We introduce psychometric representational similarity analysis, a framework for comparing the structure of speech-derived scores, self-report, item wording and theory, and implement it alongside person-level construct scoring in COMPASS 2.0. We show how similarly worded items induce covariance without psychological signal. In pre-registered discovery and confirmation analyses of clinical interviews from 275 participants, language-derived symptom geometry resembled wording more than self-report, with no structure beyond wording detected by the registered tests. Geometric agreement with self-report survived assigning participants someone else's answers, whereas person-paired scores captured distress more than specific symptoms. Complementary analyses examined counselling quality and wording structure across 34 instruments and the Research Domain Criteria (RDoC) framework. These findings distinguish agreement about psychological structure from evidence that language-derived assessments track individual people.
Introduction
Language-derived assessment can reproduce questionnaire structure through item wording, making agreement with self-report insufficient to show that scores track the person. The paper introduces psychometric representational similarity analysis and COMPASS 2.0 to compare these structures and assess person-level scores.
Introduction
- Questionnaire responses can covary because item texts resemble one another, even when that covariance is not evidence of psychological signal.The passage notes that item-text similarity predicts much of the correlation among survey responses.Source: Introduction · p. 1
- Language-based assessment scores speech against questionnaire items, so similarly worded items may attract the same speech regardless of the speaker’s feelings.The resulting structure can look coherent, replicate across samples, and resemble self-report without showing how much the person contributed.Source: Introduction · p. 1
- Psychometric representational similarity analysis compares item-by-item geometries from speech, self-report, item meanings, and theory, while person-level validity requires a separate paired test.Each source is represented as a psychometric representational dissimilarity matrix over the instrument items.Source: Introduction · pp. 1–2
- COMPASS 2.0 applies instrument scoring keys to speech-derived item activity profiles to score constructs and examine each score against its underlying geometry.It can follow constructs over a conversation and compare them between speakers.Source: Introduction · p. 2
- Under isotropic speech variation, score covariance reproduces item similarity, defining the wording null as expected geometry without information about the person.The paper treats isotropy as a reference rather than a complete account of nonpsychological speech variation; COMPASS 2.0 also checks within-speaker variation when it is nuisance, centres activities on the interview script, tests residual agreement by item permutation, and scores stance alongside topic.Source: Introduction · p. 2
- Across two interview samples and both embedding models, symptom geometry followed wording more than self-report, with no registered evidence of structure beyond wording.Geometric agreement survived assigning participants someone else’s answers, whereas paired scores tracked distress more than specific symptoms.Source: Introduction · p. 2
- Complementary analyses examined counselling quality and wording across 34 instruments and the Research Domain Criteria, with wording clustering by construct domain.Stance scores distinguished counselling quality in a second corpus without robustly outperforming keyed scores of the same items.Source: Introduction · p. 2
Results
Across clinical interviews, language-derived symptom geometry tracked questionnaire wording more closely than self-report, with no registered evidence of structure beyond wording. Person-paired scores aligned more with general distress than specific symptoms, while analyses of alliance and other instruments showed further limits and scope for these measures.
Psychometric RDMs place language, self-report, wording and theory in one space
- Embedding-scored speech geometry was more similar to wording than self-report, and the symptom-cluster theory model related weakly to every source.Across the 25 PHQ-8 and PCL-C items, embedding geometry correlated 0.74 and 0.72 with wording versus 0.43 and 0.32 with self-report; theory correlations were ρ ≤0.31.Source: Psychometric RDMs place language,… · p. 2
Item wording fixes the geometry of language-derived scores
- With no simulated symptom signal, naive geometry still correlated 0.47 with true structure through wording, while whitening reduced this to 0.01.At g = 0.4, whitening correlated 0.44 with truth; exact deconvolution often failed in samples, and within-speaker whitening could remove genuine dynamics or reverse their sign.Source: Item wording fixes the geometry of… · p. 5
Interview scripts impose a second structure
- Interview question identity explained more item-activity variance than person identity, and question-mean centring removed estimated question means, not every script effect.In DAIC-WOZ the median shares were 0.110 for question identity and 0.039 for person identity; in E-DAIC they were 0.071 and 0.030.Source: Interview scripts impose a second… · pp. 5–6
Registered tests separate the wording from the symptoms
- Registered tests found naive language geometry closer to wording than self-report, no supported self-report structure beyond wording, and small, inconsistent item-specific convergence.Geometric agreement was unchanged by shuffling participants’ self-reports, whereas paired read-outs fell to their null; exploratory agreement was mainly in general distress.Source: Registered tests separate the wording… · p. 7, Speech conveys how distressed a person… · p. 7
- Stance scores separated counselling-quality groups in confirmation, but their pre-specified superiority over keyed cosines narrowly missed criterion and quality labels did not validate alliance scores.Across sensitivity analyses stance AUCs remained 0.68–0.69, while the superiority interval crossed zero; the two corpora were not fully separate.Source: Keyed alliance scores track topic, and… · pp. 8–9
Wording clusters by construct domain across instruments
- Across 35 probe banks, item wording clustered by construct domain; RDoC definitions also clustered within their domains despite spanning diagnoses.For bge embeddings, banks occupied 1.3–2.9 effective dimensions before removing a shared direction; wording-based clusters can therefore contribute apparent psychological structure in language scores.Source: Wording clusters by construct domain… · pp. 9–10
Discussion
The discussion concludes that language-derived symptom geometry is largely shaped by wording, while person-paired scores track distress more than specific symptoms. Population-level agreement does not establish that language-derived assessments measure the right person.
Discussion
- Wording dominated language-derived symptom geometry, and registered tests detected no population structure matching self-report beyond wording.A small exploratory residual depended on the reference; paired scores tracked general distress more than specific symptoms.Source: Discussion · p. 10
- Population RDMs remain unchanged when self-reports are shuffled across speakers, whereas person-paired measures track distress and lose that correspondence under shuffling.The shuffle exposes that population geometry describes item relationships without retaining person identity.Source: Discussion · p. 10
- The authors recommend a wording baseline, question-centred activities, declared references, and person shuffles when validating language-derived assessments.They advise interpreting keyed cosine sums as topic measures unless the scorer distinguishes stance.Source: Discussion · p. 10
- Item wording predicts response correlations and is informative about instrument meaning, but whitening can erase genuine structure when its reference confounds signal with nuisance.Within-person and between-person structure need not coincide, so no reference is universally appropriate.Source: Discussion · pp. 10–11
- Modest samples, short answers, transcript errors, and limited power constrain detection of small effects and individual geometry.The authors caution that weak scores do not establish absent symptom information in conversation; alliance analyses used demonstration-session quality rather than alliance ratings.Source: Discussion · p. 11
- Proposed tests vary probe wording while holding language fixed, then vary speakers while holding probes fixed, to distinguish instrument-bound from person-bound features.Longer paired language records, direct alliance ratings, and methods that generalize beyond the instrument are also proposed.Source: Discussion · p. 11
- A convincing language-based measure must leave room for a person’s expression to change what the instrument leads us to expect.The paper frames questionnaires as theories of people written in sentences.Source: Discussion · p. 11
Methods
The methods compare speech, self-report, item wording and theory as item-level representations, while separately scoring constructs from speech. Pre-registered tests, simulations and counselling analyses assess these approaches across interview and session data.
Data
- The study used 189 DAIC-WOZ and 86 E-DAIC interviews with item-level PHQ-8 and PCL-C self-reports, alongside psychotherapy and counselling corpora.The counselling analyses included quality-labelled AnnoMI and high-/low-quality sessions; data access conditions varied by corpus.Source: Methods · p. 11
Scoring
- Speech units were answer windows in interviews and speaker turns in sessions; models scored units against instrument items, with construct scores aggregated across turns or sessions.The study included symptom, alliance, personality and other instrument banks, plus RDoC construct definitions; counselling stance scores used four-turn segments and entailment scores.Source: Methods · pp. 11–13
Representational comparison
- Psychometric representational similarity analysis treats items as shared conditions, builds a dissimilarity matrix for each source and compares sources by second-order similarity.Sources include language, self-report, item texts and theory; RDMs are compared by Spearman correlation over item pairs.Source: Methods · p. 12
Wording null and whitening
- Under isotropic speech variation, item-score covariance reproduces item-text similarity; whitening removes that feature-model prediction, while its alternative reference has restrictive assumptions.Whitening by pooled within-speaker covariance is a sensitivity analysis because it can fail when between-speaker nuisance covariance differs or within-speaker variation contains symptom structure.Source: Methods · pp. 12–13
Registered symptom tests
- Registered symptom tests compared naive language geometry with wording and self-report, tested whitened geometry beyond wording, and assessed same-item versus other-item score convergence.Discovery preceded confirmation with unchanged code; item permutations respected dependence among item pairs.Source: Methods · pp. 13–14
Simulations
- Stylized simulations tested recovery of symptom structure, test calibration and power under varied wording, speech and topic conditions, including differing question subsets and alliance items.The simulations used additive Gaussian models and were not reproductions of the full pipeline.Source: Methods · p. 13
Alliance analyses
- Alliance analyses tested whether stance scores distinguished high- from low-quality sessions and whether they outperformed keyed cosine scores, with subscale and topic analyses secondary.The primary tests used stratified bootstrap intervals and a fixed sequence at α = 0.05.Source: Methods · pp. 13–14
Additional analyses
- Exploratory analyses assessed general factors, probe-bank geometry and sensitivity to person shuffling, controls, alternative references and transcript overlap.The probe-bank analyses included effective dimensionality, nearest-neighbour shares and multidimensional scaling; sensitivity checks also examined response-window limits and source-video resampling.Source: Methods · pp. 13–14
Data availability
The study’s source datasets and item texts have different access conditions; the code package distributes aggregate results only.
Data availability
- DAIC-WOZ and E-DAIC require a data use agreement, psychotherapy transcripts require an Alexander Street subscription, and AnnoMI and the counselling corpus are publicly available from their authors.Public-domain item texts accompany the code; other item texts are available from publishers, and the code package distributes aggregate results only.Source: Data availability · p. 14
Code availability
The xpsych package provides code and instructions for reproducing the study’s numbers, tables and figures, subject to source-data access conditions.
Code availability
- The xpsych Python package provides code and instructions for reproducing every number, table and figure, including COMPASS 2.0’s implementation of psychometric representational similarity analysis.Reproduction requires access to the source datasets under their respective access conditions.Source: Code availability · pp. 14–15
Extended Data
The Extended Data figures describe script structure, RDoC definition geometry and simulation conditions, while tables catalogue corpora and registered symptom tests.
Extended Data figures
- Extended Data Fig. 1 compares wording correlations across window- and participant-level scoring variants, then shows population and participant-level split-half reliability.The participant-level reliability analysis includes participants with at least 40 windows.Source: Figure 1 · p. 16
- Extended Data Fig. 2 maps cosine distances among 49 RDoC construct definitions by domain and compares same-domain nearest-neighbour shares with chance across embedders.Panel a uses classical multidimensional scaling; panel b reports nearest-neighbour shares.Source: Figure 2 · p. 16
- Extended Data Fig. 3 describes simulations comparing geometry with true symptom structure and wording, testing person-level signal and false-positive rates under varied conditions.The caption notes that centring keeps the null calibrated with little power cost in the question-subset simulation.Source: Figure 3 · p. 17
Extended Data tables
- Extended Data Table 1 lists the corpora and identifies answer windows or speaker turns as their analysis units.Source: Table 1 · p. 17
- Extended Data Table 2 summarizes registered H1, H2 and H3 symptom tests, their comparisons and inference procedures.Source: Table 2 · p. 17
Supplementary Information
The supplementary material specifies the wording-null and whitening diagnostics, documents analysis corrections and sensitivity tests, and describes the probe banks and analysis estimands.
Supplementary Note 1: the wording null and its whitening
- Under isotropic speech variation, score covariance follows item-text similarity; whitening removes that term, but retains a transformed image of person structure rather than recovering it directly.The registered comparison with self-report also partials out wording similarity. Question-centering removes estimated question-specific content, except for questions answered by only one participant; question-by-person interactions remain.Source: Supplementary Note 1: the wording null… · p. 18
- The empirical within-speaker reference depends on nuisance-shape assumptions, entailment-scoring whitening is heuristic, and deconvolution was often undefined in the reported sample size.The within-speaker reference is valid as a person-structure test only if within-speaker variation is nuisance with the same shape as between-speaker nuisance and carries no symptom structure; in 153 of 200 fits, noise-subtracted variance was non-positive, and 67 had non-positive fitted signal variance.Source: Supplementary Note 1: the wording null… · p. 18
Supplementary Note 2: deviations, corrections and sensitivity analyses
- The registered symptom code was unchanged; the alliance tests had a separate analysis plan, while loader and data-screening corrections left the registered confirmation sample unchanged.The screen-negative sessions could still share speakers, scripts, or edited footage below the screen’s resolution.Source: Supplementary Note 2: deviations,… · pp. 18–19
- Across exploratory alliance sensitivity sets, stance scores separated quality groups, but their advantage over cosine scores had an interval lower bound that crossed zero depending on the set.These P values were unadjusted for multiplicity.Source: Supplementary Note 2: deviations,… · p. 19
Supplementary Note 3: analyses by status and estimand
- Supplementary Table 5 distinguishes each comparison’s estimand, exchangeability unit, and analysis status.Source: Supplementary Note 3: analyses by… · p. 19
Supplementary Note 4: probe banks
- The probe banks draw on published forms, manuals, or official websites; some instruments use only available item titles, and duplicate Columbia scale texts were pooled.The probe-bank description states that instruction and stem sentences were excluded, and copyrighted response options or definitions were not included for BDI-II, PANSS, SANS, and SAPS.Source: Supplementary Note 4: probe banks · p. 19, Table 6 · p. 22
- For entailment scoring, each symptom item is represented by one or more atomic first-person statements, and its score is the maximum entailment-minus-contradiction probability across them.Source: Table 7 · p. 23