PaperlayerView exact revision on arXiv

Source-linked AI summary

semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales

Devon Yanitski, Chris Westbury

arXiv:2610.37494v1cs.CL

TL;DR

Factor analysis is almost always run on responses, even though item wording shapes the structure later found in answers. semanticfa factors item-embedding similarities to estimate scale structure from wording; in the example, the retention tie rule selected five factors, with three corresponding mainly to single domains.

  • Problem

    Factor analysis is almost always run on responses, although item wording shapes the structure later found in answers.

    Source: §1 · p. 2

  • Method

    semanticfa estimates scale structure from item-embedding similarities and provides a workflow for retention, factor extraction, diagnostics, and interpretation.

    Source: §1 · p. 2, §2.3 · p. 17, §5 · p. 61

  • Results

    The retention criteria consensus was five factors; three corresponded mainly to single domains, while the other two blended most Extraversion items with five Agreeableness items.

    Source: §4.1 · p. 50

  • Takeaways & Limitations

    Treat each semantic verdict as a hypothesis for response data to test, because wording alone cannot establish how people’s responses group.

    Source: §4.3 · p. 54

    Because SFA characterizes item wording rather than respondents, it cannot show that a trait causes responses, and coherence must not be reported as a validity coefficient.

    Source: §4.5 · p. 56

Abstract

from arXiv · show

The items of a reflective psychometric scale can be read as a proposal about a construct's structure. Language models make that proposal measurable by converting each item into an embedding, a list of numbers, similar for similar meanings. We introduce semanticfa, an R package for semantic factor analysis, which factors item-embedding similarities. It also interprets factors, flags redundant items, builds short forms, and screens new items. A worked example analyzes the 50-item International Personality Item Pool Big-Five Factor Markers, embedded with Qwen3-Embedding-8B, and compares the results with a factor analysis of 874,434 respondents. Retention criteria tied between five and six factors, and the tie rule kept five. Three corresponded mainly to single domains, and two blended most Extraversion items with five Agreeableness items. Each factor's congruence with its best-matching human factor (1 = identical) ranged from .92 for Openness to .52 for Agreeableness. Item pairs more similar in meaning tended to be answered more alike (r = .46 with raw responses), but same-domain reverse-keyed opposites, answered oppositely, were about as similar as cross-domain pairs (.58 vs. .57). Sign-flipping reverse-keyed items' embeddings did not improve agreement with raw responses and worsened it with reverse-scored ones. With five factors, a 25-item short form matched the domains better than all 200 random five-per-domain sets, but parallel analysis kept only three. We propose semantic-behavioral coherence, the agreement between semantic and response structures, as a measurable property of a scale, given an embedding model. Semantic factor analysis complements, and does not replace, response-based factor analysis.

1 Introduction

The Introduction presents semantic factor analysis (SFA) as a way to estimate scale structure from item wording and compare it with response-based structure, complementing response analysis while recognizing limits for reflective scales and reverse-keyed items. It reviews semantic approaches to reverse-keyed items and pre-data-collection use of item meaning, then introduces semanticfa as an R package unifying SFA and scale-refinement tools.

1 Introduction

  • SFA factors similarities between item embeddings without respondents, and semanticfa also interprets factors, flags redundant items, proposes short forms, and screens new items.Its workflow builds a similarity matrix, selects and extracts factors with diagnostics, then relates item embeddings to candidate construct names.Source: §1 · p. 3

1.1 Item Wording Predicts How People Respond

  • Item wording predicts response correlations and often recovers intended factors, though semantic and response structures can diverge.Arnulf et al. (2014) reported 47% to 79% explained variance, rising to 68% to 87% after accounting for scale membership; Hommel and Arslan (2025) reported r = .59 for single-item correlations and r = .84 for whole-scale correlations. Feraco and Toffalini (2025) found r = .67 between semantic similarities and response correlations, while Kojima et al. (2026) found congruence = .97 for general factors but divergence among narrower factors.Source: §1.1 · pp. 4–5

1.2 Language Models Turn Item Meaning Into Numbers

  • Sentence-embedding models turn item meaning into vectors, and cosine similarities between those vectors form the matrix SFA factors.The paper uses Qwen3-Embedding-8B; sentence-embedding models are trained so texts with similar meanings receive similar representations.Source: §1 · p. 2, §1.2 · pp. 6–7

1.3 A Scale Has Both a Semantic and a Behavioral Structure

  • SFA is designed for reflective scales; formative item sets need not have correlated responses, so weak semantic–behavioral agreement is not necessarily a flaw.Direct causal links among symptoms can also produce response structure that differs from semantic structure, and candidate factor names can be evaluated by their embedding similarities to the items loading on a factor.Source: §1.3 · pp. 8–9
  • Semantic–behavioral coherence treats agreement between meaning-based and response-based factors as evidence about a construct, while disagreement highlights items grouped differently by wording and behavior.Disagreements may reflect ambiguous wording, response reasons not stated in the item, or a construct organized differently in behavior than in language; coherence supports but does not prove that a scale measures its intended attribute.Source: §1.3 · pp. 8–9

1.4 Embeddings Do Not Represent Reverse-Keyed Items as Opposites

  • A widely used sentence-embedding model rated the opposite pair at cosine = .35 and a compatible item at .32; such matrices do not separate reverse-keyed items, leaving negative correlations missing for human-like patterns.Guenole et al. (2025) multiplied reverse-keyed embeddings by −1; this improved factor recovery for IPIP-NEO-300 but performed worse than other encodings on HEXACO and consistently missed Emotionality.Source: §1.4 · p. 10
  • Reverse-keyed items motivate several semantic corrections, but mean-centering does not make opposites negatively similar, and an NLI-trained model still predicted positive correlations for 67% of pairs versus 59% in response data.SQuID subtracts the questionnaire embedding; mean-centering makes cosine equal Pearson correlation. A cross-encoder instead subtracts contradiction probability from entailment probability, while rewording reverse-keyed items may alter their meaning.Source: §1.4 · pp. 10–11

1.5 Semantic Structure Can Guide Scale Development Before Any Data Are Collected

  • Semantic structure can guide pre-data scale development by flagging redundant items and detecting jingle and jangle fallacies through mismatches between scale labels and item content.Wulff and Mata (2025) embedded 4,452 IPIP items, 459 scales, and 277 construct labels, flagging 504 candidate fallacies; they recommend expert judgment rather than automated screening alone.Source: §1.5 · pp. 11–12
  • Prior work found EGA recovered the true dimension count in 43.4% of 500 simulated item pools, while parallel analysis and the Kaiser rule recovered it in none; removing redundant and unstable items made EGA recovery almost perfect.The study used GPT-4o-written items and one embedding model; semantic short-form studies also retained factor structure, although one IPIP short form fit its five-factor model only moderately well (Wang et al., 2026).Source: §1.5 · pp. 12–13
  • SFA can support other language-model measurement steps, including screening newly generated items before respondents answer them.The introduction also describes language models being used to write items, adapt scales, simulate respondents, code text, and measure traits from writing.Source: §1.5 · p. 13

1.6 The R Package semanticfa Brings the Whole Workflow Together

  • semanticfa unifies similarity construction, factor retention, extraction and diagnostics, and interpretation in an R workflow using familiar factor-analysis and optional network-analysis tools.Its retention stage reports a consensus, breaking ties toward fewer factors; extraction includes an optional comparison of fit statistics with random embeddings.Source: §1.6 · p. 13
  • Beyond its four stages, semanticfa uses fitted models to flag redundant items, propose short forms, and screen new items before administration.The worked example applies every stage and refinement tool to the 50-item IPIP Big-Five Factor Markers embedded with Qwen3-Embedding-8B, comparing results with responses from 874,434 people.Source: §1.6 · p. 14

2 Method

The method applies semantic factor analysis to item embeddings and a human-response benchmark, then describes the package workflow for constructing similarities, retaining and extracting factors, and interpreting or refining scales.

2.1 Materials

  • The worked example uses the 50-item IPIP Big-Five Factor Markers and compares semantic analyses with Pearson correlations from 874,434 respondents.The inventory has ten first-person items per domain; the respondent data are self-selected online ratings, with records outside the 1–5 range removed.Source: §2.1 · p. 15

2.2 Embeddings

  • The main analyses use 4,096-dimensional Qwen3-Embedding-8B vectors, while the model-size comparison tests 0.6B and 4B alternatives.The 0.6B model is the package default for new text on a user's computer; the authors recommend 8B when memory allows, but model size and dimension vary together.Source: §2.2 · p. 16

2.3 The semanticfa Workflow

  • Interpretation tools compare items with construct centroids or names and project them onto semantic axes; refinement tools flag redundancy, propose short forms, and screen new items.Anchoring flags items more similar to another construct than their own, while semantic projection scores items along a contrast defined by opposing words.Source: §2.3 · pp. 17–18
  • semanticfa offers four embedding encodings, with mean_centered_pearson as the default, and an alternative NLI matrix based on entailment minus contradiction probabilities.The default centers each vector's entries before taking cosines, yielding a Pearson correlation matrix without a scoring key; an invalid NLI matrix is replaced with a valid one during factor analysis.Source: §2.3 · p. 17
  • Parallel analysis retains leading factors whose eigenvalues exceed thresholds from random embeddings, while sfa_nfactors() reports the most frequent choice across five criteria and breaks ties downward.By default, parallel analysis uses the 95th percentile of corresponding eigenvalues from 100 random matrices and stops at the first threshold not exceeded.Source: §2.3 · pp. 17–18
  • The sfa() workflow extracts factors by minimum residual, rotates them with oblimin, and reports five diagnostics, including three fit statistics and factor-wise omega.RMSR, CAF, and TEFI describe fit to the similarity matrix; the reported omega is a one-factor omega for each factor's items, not hierarchical omega.Source: §2.3 · p. 18

Diagnostics reported by sfa() and how to read them

The analysis plan specifies diagnostics and refinement tools, then compares semantic structure with human responses through factor, item-pair, and encoding-level assessments.

Diagnostics reported by sfa() and how to read them

  • The reported diagnostics include fit calibration, congruence, jingle–jangle comparisons, and plots of similarity matrices, item maps, loadings, and eigenvalues.The jingle–jangle screen flags scale pairs whose name and item similarities differ by at least 0.20, a package heuristic.Source: Diagnostics reported by sfa() and how… · pp. 19–20
  • Optional refinement functions identify redundant item pairs, propose centroid-based short forms, and assess whether new items fit a construct or duplicate existing items.The new-item check reports the best-matching construct, its margin over the next construct, and whether an existing item is nearly duplicated.Source: Diagnostics reported by sfa() and how… · p. 20

2.4 Analysis Plan

  • The main fit uses the default mean_centered_pearson matrix, fixes the parallel-analysis choice for later steps, and compares fit statistics with 100 random-embedding fits.The analysis script runs the steps in a set order and seeds most random steps with 42; several specified checks use other seeds or no seed.Source: §2.4 · pp. 20–21
  • Retention compares parallel analysis with a five-criterion consensus, then searches 148 embedding depths using a weighted objective that favors domain agreement over entropy fit.The depth score is 0.70 × rescaled NMI − 0.30 × rescaled TEFI; the main fit retains the full embedding unless depth selection is requested.Source: §2.4 · p. 21
  • The plan compares all four embedding encodings, repeats retention and domain-separation checks across model sizes, and quantifies same-domain versus cross-domain similarity.The model-size analysis uses a point-biserial correlation over 1,225 item pairs and describes uncertainty with an item-level bootstrap.Source: §2.4 · pp. 21–22
  • Factor interpretation compares items with construct centroids and names, maps them on two semantic axes, and evaluates domain agreement against random label permutations.The analysis also checks whether each item's five nearest neighbors in two-dimensional maps share its domain and screens subscales for jingle–jangle differences of at least 0.20.Source: §2.4 · p. 22
  • Item refinement flags redundancy, selects five centroid-similar items per domain for a 25-item short form, and tests three candidate items for fit, cross-loading, or duplication.The plan compares the short form with 200 random five-per-domain sets fitted with five factors; candidate cross-loading uses a 0.07 similarity margin.Source: §2.4 · p. 22
  • Semantic–behavioral coherence is assessed at factor, item-pair, and encoding levels, including matched Tucker congruence and correlations between similarity and response matrices.For the default encoding, the item-pair comparison uses raw-response correlations; uncertainty is estimated by resampling items.Source: §2.4 · p. 23

3 Results

The results evaluate semantic factor analysis of the 50 IPIP items across retention and encoding choices, factor and item structure, and alignment with human response data. They also examine response-free item review, short-form construction, and item screening, including the need for further psychometric validation.

3.1 Fitting the Model

  • KMO was .971, RMSR was .032, TEFI was −27.71, and CAF was .412 for the five-factor solution.The factor correlations ranged from .40 to .69, and CAF indicated that unexplained similarities still shared structure. Conventional RMSR cutoffs were weak evidence here: random-embedding fits had RMSR .0119–.0134, and the observed RMSR was higher than all of them; the random fits were a weak baseline.Source: §3.1 · pp. 25, 28
  • Three factors mainly corresponded to Neuroticism, Conscientiousness, and Openness, while two blended factors combined most Extraversion items with five Agreeableness items.The five-factor similarities correlated r = .46 with correlations among 874,434 respondents’ raw responses. The three domain-aligned factors covered most items in their domains, while the blended factors also included three items from other domains.Source: §3 · p. 24, §3.1 · p. 26, Figure 1 · p. 27

3.2 Choosing an Encoding

  • All 1,225 raw cosines were positive (.394 to .938), and mean-centered Pearson similarities differed from them by at most .0010 (r = 1.000).Sign-flipping reverse-keyed embeddings did not make same-domain opposites uniquely dissimilar: those pairs averaged −.58, and 4% of similarities were at or below the E1–E2 value of −.63. The NLI cross-encoder scored that pair −1.00 as almost certainly contradictory.Source: §3.2 · pp. 29–30

3.3 Choosing the Number of Factors

  • Five and six factors each received two retention votes, so the tie rule selected five.Parallel analysis and EGA chose five, the Kaiser rule and empirical Kaiser criterion chose six, and TEFI chose two. The first eigenvalue was 29.63 (59% of the total), and the scree plot had no sharp elbow to settle the decision independently.Source: §3.3 · p. 32, Figure 3 · p. 32
  • At 1,431 of 4,096 embedding coordinates, EGA found five communities with NMI = .704 against the theoretical domains, the maximum across evaluated depths.At full depth, EGA found five communities with NMI = .454; because depth selection favored agreement with theory, the selected result is consistent with the main fit rather than independent confirmation.Source: §3.3 · p. 33

3.4 Choosing an Embedding Model

  • At .458 (4B), .416 (8B), and .398 (0.6B), domain-sharing correlations did not rise steadily with size; only 8B made parallel analysis and EGA retain five factors.For the Big Five, 4B had the highest five-factor NMI (.693, versus .497 for 0.6B and .527 for 8B); for DASS, NMI rose from .412 to .700 to .834 across model sizes. The comparison confounds model size with embedding dimension, and the suggested largest-model choice rests on two inventories and one model family.Source: §3.4 · pp. 33–34

3.5 Interpreting the Factors

  • Six items were more similar to another domain centroid than to their own, while only 29 of 50 items matched their construct name as their most similar name.These small centroid margins make the six items review candidates rather than evidence of misplacement; A25’s own-name cosine was .36 versus .37 for Neuroticism.Source: §3.5 · pp. 35–36
  • Semantic projection placed reverse-keyed N12 nearest the calm pole at .82, while the sociability axis’s five highest items included three Extraversion and two Agreeableness items.Projection scores follow item wording rather than scoring keys and depend on the chosen pole words; the example did not test whether the axes track their intended dimensions.Source: §3.5 · pp. 36–37
  • The item maps showed Neuroticism separation and Agreeableness–Extraversion overlap; own-domain nearest-neighbor shares were .72 for t-SNE, .68 for UMAP, and .42 for linear maps, versus .18 by chance.The nonlinear maps emphasize local neighborhoods and can distort larger distances; the linear maps’ first two principal components each explained 9% of embedding variance.Source: §3.5 · pp. 37–38, Figure 5 · p. 39
  • NMI was .527 and ARI was .402 for semantic factors against the Big Five domains, well above permutation maxima of .292 and .149 but far from complete agreement.The main mismatch arose from how Agreeableness and Extraversion items were grouped.Source: §3.5 · p. 38
  • No scale pair crossed the jingle-jangle flag threshold of .20; the largest divergence was .17 for Agreeableness–Extraversion, whose item averages had cosine .89.The null result on five subscales of one inventory offers limited evidence about the screen’s sensitivity.Source: §3.5 · p. 40

3.6 Refining the Items

  • The screen flagged 11 redundant pairs in 10 clusters, all within domains; a simpler cosine cutoff flagged seven pairs in five clusters, five overlapping with the first screen.The top pair, A24–A29, had wTO overlap .462; the count of 11 flagged pairs held at nominal sample sizes of 500, 1,000, and 2,000.Source: §3.6 · p. 41
  • The 25-item response-free short form improved structure recovery to NMI .810 and ARI .571 from full-scale values of .527 and .402, although parallel analysis retained three rather than five factors.It selected five items per theoretical domain, and the printout cautions that the candidate form requires psychometric validation before use.Source: §3.6 · p. 42
  • NMI = .810 and ARI = .707 for the selected 25-item short form fitted with five factors, exceeding every random five-per-domain set; parallel analysis retained only three factors.The centrality-selected short form kept both items from four flagged redundant pairs, so it is a candidate for psychometric validation rather than a finished instrument.Source: §3.6 · pp. 42–43
  • The three illustrative candidates received their intended verdicts: the first cross-loaded ambiguously between Extraversion and Agreeableness by .04; the second was a near-duplicate of E1 at cosine .95; the third cleanly fit Conscientiousness.The .95 similarity exceeded the 0.85 cutoff. The third candidate’s similarity exceeded that of the next construct by .09, above the 0.07 cross-loader cutoff.Source: §3.6 · p. 44

3.7 Comparing With Human Responses

  • N = 874,434 human responses yielded a five-factor solution with every item’s largest loading on its theoretical domain, although parallel analysis suggested ten factors.At this sample size, random-data eigenvalues were almost zero, and the authors note that the ten-factor result may reflect sample size rather than ten substantive dimensions; this was not tested.Source: §3.7 · p. 45

3.7.1 Factor-Level Agreement

  • ϕ ranged from .92 for Openness to .52 for Agreeableness, with mean matched congruence .744; Extraversion also matched Agreeableness at .52.The semantic Extraversion and Agreeableness factors blended items from both domains, while human responses kept the domains separate.Source: Table 3 · p. 46, §3.7.1 · p. 46

3.7.2 Encoding-Level Agreement

  • r = .561 for NLI was highest against raw responses, but its mean matched congruence was lowest at .286; the default encoding had the highest mean matched congruence, .744.The default and atomic encodings tied on agreement metrics to three decimals; NLI’s factor-based metrics came from a repaired matrix, and comparisons were descriptive without uncertainty estimates.Source: §3.7.2 · p. 47

3.7.3 Item-Pair Agreement

  • Mean similarity was .58 for same-domain reverse-keyed pairs and .57 for cross-domain pairs, despite the former’s negative mean response correlation of −.34.Their similarity was .12 below same-domain alike-keyed pairs; embeddings distinguished opposite-keyed items but did not represent their opposition in human responses.Source: §3.7.3 · p. 48
  • Semantic similarity’s relation to human correlation was curved: a quadratic fit explained .40 of variance versus .21 for the straight-line fit, a .20 gain (item-level bootstrap 95% interval .12 to .28).Source: §3.7.3 · p. 47

4 Discussion

The discussion summarizes semanticfa’s response-free analysis and examines its agreement and disagreement with human response structure, including recommendations, applications, and limitations. It treats semantic verdicts as hypotheses requiring expert judgment and response-data validation, and outlines further tests while clarifying what semantic analysis cannot establish.

4.1 Summary of Findings

  • Five factors were retained by consensus; congruence ranged from ϕ = .92 for Openness to .52 for Agreeableness, and item-meaning similarity correlated r = .46 with raw-response similarity.Openness, Conscientiousness, and Neuroticism mainly mapped to single factors, while most Extraversion items and five Agreeableness items split between two blended factors. Same-domain reverse-keyed pairs were about as similar as cross-domain pairs (.58 vs. .57).Source: §4 · p. 49, §4.1 · p. 50
  • Sign-flipping reverse-keyed embeddings did not improve agreement with raw responses (r = .444 vs. .456) and reduced agreement with reverse-scored responses (r = .137 vs. .434).The 25-item short form reached NMI = .810 and ARI = .707, above every random five-per-domain set fitted with five factors, but parallel analysis retained only three factors; eleven redundant pairs were flagged.Source: §4.1 · p. 50

4.2 Theoretical Implications

  • Semantic–response agreement is evidence that wording and responses group items similarly for this inventory and model, not proof that traits exist or cause responses.The authors propose coherence as a measurable relation for testing theories, while noting that formative indicators need not correlate and method variance may lower coherence; these predictions were not tested here.Source: §4.2 · pp. 51–52

4.3 Recommendations for Users

  • Users should compare multiple retention criteria with theory, avoid sign-flipping, and treat semantic verdicts as hypotheses to test with response data.The recommendations are starting points based on one inventory, one embedding-model family plus one NLI cross-encoder, and English items; the authors also advise comparing fit with random embeddings and reporting agreement across metrics and matched response correlations.Source: §4.3 · pp. 52–54

4.4 Applications

  • Semanticfa is intended for early item-pool development, where researchers can screen candidates, flag redundancy, and draft short forms before collecting responses.The example screened only three illustrative candidate items and did not remove items based on redundancy flags; semantic-centrality short forms remain candidates for validation, not finished instruments. The package may also support instrument comparisons and work when response data are scarce, unavailable, or untrustworthy.Source: §4.4 · pp. 54–56

4.5 Limitations

  • Because SFA characterizes item wording rather than respondents, coherence is not a validity coefficient and cannot replace response and outcome data.Semantic evidence addresses content validity and part of structural validity; the authors recommend following SFA with human data wherever possible. Embeddings also reflect language use rather than each respondent’s reading, so coherence below 1 may be expected even for a flawless instrument.Source: §4.5 · pp. 56–57
  • The results may depend on training-data exposure, embedding model, and sample: the familiar inventory may have been partially memorized, and the human benchmark was self-selected.Random-embedding baselines are weak because real embeddings are positively similar, so factor retention should be informed by multiple criteria and theory. Results rest on one model family and English items; stability across models should be reported, and rewording items is proposed as future work to investigate possible memorization.Source: §4.5 · pp. 57–58
  • Semantic redundancy and jingle–jangle screens should inform, not replace, expert judgment, especially because meaning similarity does not ensure equivalent responses or measurement.The redundancy cutoff was calibrated on simulated response data, so its error rate on similarity matrices is unknown; removing items also assumes a reflective model rather than a formative one.Source: §4.5 · p. 59

4.6 Future Directions

  • Coherence needs comparison across instruments, domains, and populations; common-cause constructs are predicted to show higher coherence than constructs whose symptoms cause one another.An unpublished study found meaning–correlation relations varied across nine dimensions from four scales (D. Zhang & Qiu, 2026); Milano et al. (2026) found embedding methods less accurate than expert judges at assigning behavior-phrased items, a result about structure recovery rather than coherence.Source: §4.6 · p. 59
  • Future methodological tests should assess embedding-family stability, coordinate depth with domain-independent criteria, recalibrate redundancy cutoffs, improve random-embedding and factor-count baselines, and test memorization.Re-embedding systematically reworded items while preserving meaning could test whether agreement depends on models having seen published items; model-family stability should be reported like rater agreement.Source: §4.6 · pp. 59–60
  • Semantic factor analysis could screen translated scales before data collection by comparing original and adapted items in a shared multilingual embedding space, if cross-language comparability is established.Response structure would still need confirmation after administration; a Polish adaptation study found performance at or near human adaptation on most psychometric criteria, with slightly lower reliability.Source: §4.6 · p. 60
  • Semantic analysis can automate tests of item wording against construct names and centroids, but explanations of answers, outcome relations, and measure validation still require response data and expert judgment.The package implements semantic factor analysis as a standard factor-analytic workflow in R.Source: §4.6 · p. 60

5 Conclusion

The package analyzes the structure implied by item language and compares it with response structure; the authors propose semantic–behavioral coherence as a measurable scale property for a given embedding model.

5 Conclusion

  • Five retention criteria selected five factors by the smaller-number tie rule, matching the inventory’s intended number of domains.Two criteria chose five, two chose six, and one chose two.Source: §5 · p. 61
  • Three factors mainly matched single domains, while two blended most Extraversion items with Agreeableness items; congruence with human factors ranged from .92 for Openness to .52 for Agreeableness.Conscientiousness and Neuroticism were the other mainly single-domain factors; some item counts rested on near ties.Source: §5 · p. 61
  • For this inventory and embedding model, sign-flipping reverse-keyed embeddings agreed less with reverse-scored responses than leaving them unflipped.The authors suggest overlapping wording and shared negation as plausible reasons for blended factors, but did not test either explanation.Source: §5 · p. 61
  • The package combines semantic factor analysis, interpretation, diagnostics, and refinement tools, allowing candidate revisions to be drafted before responses exist and validated once responses are available.Its refinement tools flagged redundant items, built a candidate short form, and screened three illustrative new items.Source: §5 · pp. 61–62
  • Semantic results are reproducible from archived embeddings and specified software settings, with small layout differences; response benchmarking depends on downloaded data and unseeded parallel analysis.The NLI matrix is an exception because its cross-encoder is downloaded on first use.Source: §5 · p. 61
  • Semantic–behavioral coherence is the agreement between item-language and response structures; the example summarized it with r = .46 across item pairs and mean matched congruence ϕ̄ = .744.The authors propose treating coherence as a scale property for a given embedding model; what counts as high remains to be established.Source: §5 · p. 62

Funding

The authors report that the work received no external funding.

Funding

Ethics Approval and Consent

No new human data were collected; the human benchmark used a publicly released, anonymized data set.

Ethics Approval and Consent

  • No new human data were collected; the human benchmark was a secondary analysis of the publicly released, anonymized Open-Source Psychometrics Project data set.Source: Ethics Approval and Consent · p. 63

Use of Large Language Models

The authors used Claude and GPT models for literature, coding, and manuscript tasks under their direction, and state that they take responsibility for the published content.

Use of Large Language Models

  • Claude and GPT models assisted with literature searches, code, manuscript drafting and revision, and citation, statistics, and claim checks; the authors say design and interpretations were theirs and take responsibility for the published article.Source: Use of Large Language Models · p. 63

Open Practices Statement

The reproduction archive regenerates the worked example’s outputs from raw inputs and documents the software and computing environment. Its analyses are reproducible with specified dependencies, though some outputs can vary slightly across hardware.

Open Practices Statement

  • Re-embedding reproduced stored vectors with cosine similarity of at least .999998 for every item and model size.The archive includes the scripts, inputs, embeddings, results, and a full-run transcript; the worked example was not preregistered.Source: Open Practices Statement · p. 64
  • All factor analyses, retention methods, diagnostics, congruence metrics, refinement analyses, and figures run in pure R and offline.Two steps use Python: reading NumPy embeddings and building the NLI matrix, which downloads its cross-encoder on first run.Source: Open Practices Statement · p. 64
  • The report specifies software versions, model revisions, seed, and hardware, while noting small cross-hardware variation in NLI scores and map positions.Reproducing the worked example requires semanticfa 0.5.1 or later.Source: Open Practices Statement · pp. 64–65

Supplementary Materials

The Supplementary Materials provide technical definitions, comparison results, full loading matrices, and additional analyses for readers checking or extending the work.

Supplementary Materials

  • The materials include function and method details, encoding comparisons, random-embedding baseline limitations, full loading matrices, and the NLI matrix.They also provide comparison-data misfit profiles and repeat the model-size comparison with the Depression Anxiety Stress Scales.Source: Supplementary Materials · p. 74

Technical Reference

The Technical Reference explains how semanticfa builds, fits, evaluates, interprets, and refines semantic factor solutions. Its functions include embedding-based and NLI similarity routes, diagnostics, and tools for evaluating items and scales.

Technical Reference

  • The package adapts response-based retention and fit methods to embedding similarity matrices and documents limits of random-embedding baselines.The reference also compares encodings on the main analysis metrics.Source: Technical Reference · p. 74

Function Reference

  • sfa() extracts minimum-residual factors, applies oblimin rotation, and reports KMO, RMSR, CAF, TEFI, and factor-specific omega diagnostics.Oblimin allows factors to correlate.Source: Function Reference · p. 74
  • sfa_similarity() offers raw-cosine, reverse-key-flipped, SQuID-centered, and mean-centered Pearson encodings for item-embedding similarities.SQuID subtracts the mean item embedding; mean-centering each vector makes the cosine equal its Pearson correlation.Source: Function Reference · p. 75
  • sfa_nli_matrix() scores ordered item pairs by entailment probability minus contradiction probability, then averages both directions into a symmetric matrix.This route uses an NLI cross-encoder rather than embeddings.Source: Function Reference · p. 75
  • Interpretation functions relate item embeddings to construct anchors or poles, while congruence tools compare factors, assignments, and factor-correlation matrices.sfa_anchor() uses cosine similarity to construct centroids or name embeddings; sfa_project() rescales item similarity to a two-pole axis. Frobenius similarity depends on factor order and is not interpreted.Source: Function Reference · pp. 76–77
  • The package also calibrates fit against random embeddings, matches factors to domains by average absolute loadings, and provides plots and conversion utilities.The disattenuated item-pair correlation uses an order-dependent split-half reliability estimate, is capped at 1, and is missing when either reliability is nonpositive.Source: Function Reference · pp. 74, 75, 77, 78
  • Refinement functions flag locally dependent items, build candidate short forms, screen new items for cross-loading or duplication, and compare scale names with item content.Short forms are candidates requiring response-data validation; candidate-item thresholds are package heuristics, and UVA uses a nominal sample size convention.Source: Function Reference · pp. 77–78

Retention Criteria

Retention criteria adapted to similarity matrices disagree, so the package reports their consensus and recommends choosing the final factor count with theory in view. Embedding-depth selection is a separate procedure that balances domain agreement and fit.

Retention Criteria

  • The criteria adapt response-data methods by using random embeddings or embedding dimension where sample size would ordinarily enter.sfa_nfactors() reports the most frequent choice, resolving ties in favor of fewer factors; the criteria are not independent.Source: Retention Criteria · pp. 78–79
  • Parallel analysis compares observed eigenvalues with the 95th percentiles from random-embedding similarity matrices and stops counting at the first threshold failure.The default uses 100 random matrices; the Kaiser rule retains eigenvalues greater than one, while empirical Kaiser substitutes embedding dimension for sample size.Source: Retention Criteria · p. 79
  • TEFI retained two factors on these matrices, fewer than theory, while its use across a 1-to-15-factor scan and one-factor candidates exceeds its original tested scope.The source simulations used binary items with two or three factors, and the index can indicate too few factors under some conditions.Source: Retention Criteria · p. 79
  • The package’s EGA uses a triangulated maximally filtered graph and Walktrap, while MAP indicated removing ten components and comparison-data analysis profiles candidate fits.The reported MAP result exceeded the inventory’s five theoretical domains.Source: Retention Criteria · pp. 79–80
  • At 1,431 of 4,096 coordinates, depth selection found five EGA communities with NMI = .704 against theoretical domains.The selection maximizes a rescaled criterion weighting NMI by 0.70 and TEFI by −0.30; NMI was the maximum across evaluated depths.Source: Retention Criteria · pp. 80–81
  • Parallel analysis and EGA retained five factors, Kaiser and empirical Kaiser retained six, and TEFI retained two; theory-guided consensus favored five.Response parallel analysis retained ten factors, while the five-factor theoretical structure guided the final choice.Source: Retention Criteria · p. 81

Monte Carlo Calibration

Random-unit-vector calibration provides reference distributions for fit diagnostics, but the near-zero similarities make some contrasts weak. The authors recommend reporting calibrated and conventional values together.

Monte Carlo Calibration

  • Calibration draws unit-length standard-normal vectors matching the scale’s item count and embedding dimension, then fits the same factor count, extraction, and rotation.The worked example retained all 100 replicates; a rotation failed to converge in one fit, affecting TEFI but not RMSR or CAF.Source: Monte Carlo Calibration · p. 82
  • Observed TEFI (−27.71) was lower than every random fit, while observed CAF (.412) was also below every random value; the authors caution that the CAF contrast is weak.Random-fit TEFI ranged from −21.66 to −17.82, and CAF from .501 to .532; the baseline’s near-zero similarities limit interpretation.Source: Monte Carlo Calibration · p. 83
  • Observed RMSR (.032) exceeded the random range of .0119 to .0134, but the authors say it cannot be read as misfit because random similarities are near zero.The random RMSR mean was .0127; random fits passed the conventional cutoff easily, which the authors say says little about fit.Source: Monte Carlo Calibration · pp. 82–83
  • The authors recommend reporting calibrated values alongside conventional fit statistics, while treating the random-embedding baseline as weak.The choice of calibration baseline remains the analyst’s decision.Source: Monte Carlo Calibration · p. 83

Encoding Comparison

The encoding comparison finds no universally best encoding or agreement metric, and recommends like-with-like comparisons across multiple measures. Random-unit-vector retention baselines are weak, while the supplementary loading matrices give the complete five-factor solutions.

Encoding Comparison

  • Sign-flipping reverse-keyed embeddings reduced agreement with reverse-scored correlations (rkeyed = .137 versus .434 for atomic) and did not improve agreement with raw correlations.Because the sign flip changes similarity signs but not item grouping, atomic_reversed has the same NMI, ARI, KMO, RMSR, and CAF as atomic; negative split-half reliability also makes disattenuated correlation unavailable.Source: Encoding Comparison · pp. 83–85
  • SQuID matched reverse-scored response correlations best (rkeyed = .567), but its KMO = .022 indicated a matrix nearly unfactorable.On raw responses, NLI ranked above SQuID (rraw = .561 versus .490); mean_centered_pearson and atomic were near-best on factor and domain agreement.Source: Encoding Comparison · pp. 84–85
  • NLI matched theoretical domains best (NMI = .548) and raw correlations best (rraw = .561), but matched human loading patterns worst (¯ϕ = .286).Factor-based NLI metrics use a repaired matrix; its values correlated .93 with the original, while rraw uses the original matrix.Source: Encoding Comparison · pp. 83–85, Table S1 · p. 84
  • The authors recommend comparing keying-free encodings with raw responses, key-using encodings with reverse-scored responses, and reporting multiple agreement metrics.NLI’s rraw = .561 but rkeyed = .063 illustrates how mixing keying approaches can misstate coherence.Source: Encoding Comparison · p. 85

Limits of the Random-Embedding Baselines

  • Random-unit-vector parallel-analysis thresholds are a weak test for embedding matrices, whose real item similarities are positive and whose eigenvalues lack a sharp boundary.The adapted procedure behaves like a Kaiser rule with a cutoff slightly above one; it indicates which eigenvalues exceed that cutoff, not where interpretable structure ends. The authors recommend consensus across criteria, comparison-data misfit profiles, and checking theory across candidate factor counts.Source: Limits of the Random-Embedding Baselines · pp. 85–86

Full Loading Matrices

  • Tables S2 and S3 provide the complete oblimin-rotated loading matrices for the five-factor semantic and human solutions, respectively.The semantic solution factors mean-centered Pearson similarities from Qwen3-Embedding-8B; the human solution factors reverse-scored responses from 874,434 respondents after exclusions.Source: Full Loading Matrices · pp. 86–88

Signed NLI Similarity Matrix

The signed NLI matrix marks contradictory item pairs with negative similarities, unlike cosine similarities, which are positive in the main analysis. Comparison-data profiles show a sharp five-factor elbow for human responses but no comparable boundary for embedding matrices.

Signed NLI Similarity Matrix

  • The NLI matrix assigns E1 and its reverse-keyed counterpart E2 a similarity of −1.00, while atomic_reversed assigns them −.63.The sign-flipped pair is not the most dissimilar pair in that matrix, despite the statements expressing opposite meanings.Source: Signed NLI Similarity Matrix · pp. 89–90
  • NLI similarities average .20 for same-domain pairs keyed alike and −.79 for same-domain pairs keyed oppositely; 71% of cross-domain pairs are negative.The figure’s scale runs from −1 (red) through 0 (white) to 1 (blue), with red cells marking judged contradictions.Source: Comparison-Data Misfit Profiles · p. 91

Comparison-Data Misfit Profiles

  • Human-response misfit improvement peaked at five factors (57.2% over four factors) and fell to 19.7% at six, while embedding profiles showed no comparable elbow.At five factors, 16% of the human one-factor misfit remained; embedding matrices retained 47–49%, with each added factor reducing misfit by 9%–22%.Source: Comparison-Data Misfit Profiles · pp. 92–93
  • The sequential comparison-data rule reached ten factors for all three embedding matrices and did not stop at five for human subsamples at any tested size.Human subsamples of 250 respondents stopped at seven to nine factors; from 500 respondents upward, results reached the ten-factor test cap in some runs.Source: Comparison-Data Misfit Profiles · p. 92
  • The authors do not establish the proposed power explanation for continued factor retention and note that embedding dimensions are only analogous to sampled cases.Embedding profiles show no boundary for the stopping rule to detect; the authors recommend consensus among retention criteria and checking agreement with theory.Source: Comparison-Data Misfit Profiles · p. 93

Model Size With a Second Scale

In the DASS, the 8B model separated all three scales in its three-factor solution, unlike the 0.6B model, which merged most Anxiety items with Stress items. The two-inventory comparison cannot establish which inventory properties explain the difference in model-size effects.

Model Size With a Second Scale

  • DASS within-scale similarity gaps increased from .099 with 0.6B to .124 with 8B, while the Big Five gap stayed nearly constant.The 95% interval for the DASS increase from 0.6B to 8B was −.001 to .053, just including zero; Anxiety’s gap shrank as Depression’s and Stress’s grew.Source: Model Size With a Second Scale · pp. 94, 96
  • DASS scale agreement rose from NMI = .412 and ARI = .271 with 0.6B to NMI = .834 and ARI = .801 with 8B; Big Five agreement was highest with 4B.The 8B DASS solution assigned 14 Depression, 14 Stress, and 11 Anxiety items to their respective factors; the 0.6B solution combined all Stress items with eight Anxiety items.Source: Model Size With a Second Scale · pp. 94, 96
  • Neither parallel analysis nor the five-criterion consensus selected three factors for the DASS: parallel analysis retained four at every size, and consensus retained four or five.Consensus was five for 0.6B and four for both 4B and 8B.Source: Model Size With a Second Scale · pp. 95–96
  • The 0.6B DASS solution varied across rotation seeds, and both reported solutions merged Anxiety and Stress; the two inventories cannot identify which properties explain model-size effects.The reported solution occurred under 16 of 40 seeds; an alternative occurred under the other 24 and had higher scale agreement, though still below 4B. The authors note the inventories differ in scale count and content.Source: Model Size With a Second Scale · p. 95

Related papers

Loading 2610.37494v1…