Source-linked AI summary
Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing
Jing Chen, Giulia Loca, Simona Amenta, Marco Marelli
TL;DR
Whether LLMs share the sublexical sensitivity that governs human pseudoword processing remains unclear. The paper tests five LLMs on two Italian pseudoword experiments against human responses and finds that alignment varies with the availability of real-word options, while a human-predictive sublexical cue does not consistently predict human–LLM alignment.
Problem
It is unclear whether LLMs acquire the sublexical form–meaning patterns that guide human pseudoword processing.
Source: §1 · pp. 1–2Method
The study benchmarks five LLMs on two Italian 2AFC pseudoword experiments and compares their choices with human responses.
Source: §1 · p. 2, §3.2 · p. 3Results
LLMs aligned more reliably with humans in EXP2, where real-word options provided a lexical familiarity cue, than in pseudoword-only EXP1, where alignment was near chance and below fastText; Dcos had no consistent effect on human–LLM alignment.
Source: §6 · pp. 7–8Takeaways & Limitations
The findings suggest that LLMs do not necessarily share the sublexical cues governing human pseudoword processing.
Source: §6 · p. 7Training-data coverage may contribute to the alignment gap, and item-level predictor and reasoning-token analyses relied on only 50 pseudoword pairs.
Source: Limitations · pp. 8–9
Abstract
from arXiv · showhide
Systematicity, the probabilistic mapping of form to meaning, permeates language at all levels, and sublexical cues have been shown to govern human pseudoword processing. Yet whether LLMs exhibit comparable sensitivity to these cues remains unclear. We tested five LLMs on two Italian two-alternative forced-choice pseudoword experiments and compared their responses with a human behavioural baseline. LLMs aligned more reliably with humans when real-word options provided a lexical familiarity cue than in the pseudoword-only condition, where they fell substantially below fastText, a character-n-gram model. In addition, the sublexical cosine-similarity cue that reliably drove human--fastText agreement did not consistently transfer to human--LLM alignment, and reasoning-token expenditure bore no consistent relation to human processing difficulty. These findings suggest that LLMs do not necessarily share the sublexical cues that govern human pseudoword processing; we discuss tokenization and training-data coverage as candidate explanations.
1 Introduction
Language contains statistical form–meaning regularities at multiple levels, and such sublexical cues guide human pseudoword processing. This study asks whether LLMs show comparable sensitivity, using pseudowords to isolate form cues from stored word knowledge.
1 Introduction
- Statistical regularities in form and meaning, or systematicity, occur throughout language, including sublexical patterns such as gl- in words like “glitter” and “glow.”Systematicity is distinct from iconicity, where a symbol resembles its referent.Source: §1 · p. 1
- Human processing of pseudowords is guided by sublexical form–meaning regularities, and fastText reliably predicts human judgments.Pseudowords follow a language’s orthotactic patterns but lack conventionalized meanings and lexical entries.Source: §1 · p. 1
- Whether LLMs acquire the sublexical patterns governing human form-to-meaning mapping remains unclear because their frequency-based subword units may not match human processing units.LLMs are trained on large text corpora and perform well across many linguistic and psychological tasks.Source: §1 · p. 1
- The study benchmarks five LLMs on two Italian pseudoword experiments, asking whether they align with humans, use human-predictive sublexical features, and spend more reasoning effort on harder items.Pseudowords have no lexical entry, helping isolate sensitivity to sublexical form cues.Source: §1 · p. 2
2 Related Work
Prior work shows that human pseudoword judgments reflect sublexical associations captured by fastText, while LLM tokenization can either preserve or obscure form cues. The paper investigates whether LLMs use the token-based cues available to them when interpreting pseudowords.
2 Related Work
- fastText represents strings through aggregated character n-gram vectors, and its predictions align with human choices in pseudoword 2AFC tasks.Greater fastText-estimated relatedness also slowed responses and reduced accuracy in primed lexical decisions involving pseudowords.Source: §2 · p. 2
- Unlike fastText’s overlapping n-grams, LLMs often process strings as non-overlapping, frequency-selected subword tokens that may or may not preserve human-relevant cues.For example, GPT-4o preserves gl- in “glitter” but not “glisten.”Source: §2 · p. 2
- Tokenization can capture form–meaning regularities, and LLMs may treat shared tokens as meaningful cues even when they are not genuine linguistic constituents.LLMs may also miss shared constituents that are absorbed into tokens.Source: §2 · p. 2
- Because pseudowords lack lexical entries, the study asks whether LLMs use their subword tokens as form-based cues for construing meaning.Pseudoword fragments may match neither fastText’s exhaustive n-grams nor the cues human readers exploit.Source: §2 · p. 2
3 Method
The study reuses two Italian 2AFC pseudoword experiments, compares five LLMs with human responses, and varies prompt framing across baseline and native-speaker-intuition conditions.
3.1 Data and materials
- The two experiments use Wuggy-generated Italian pseudowords organized into triplets by fastText relatedness to attested words.EXP1 pairs a real target with related and unrelated pseudowords; EXP2 pairs a pseudoword with related and unrelated real-word options, with word properties matched.Source: §3.1 · pp. 2–3
3.2 Models
- Five LLMs were tested, including Italian-primary Minerva and four frontier models trained primarily on English.o4-mini and DeepSeek-R1 generate intermediate reasoning tokens before answering; the other listed models respond directly.Source: §3.2 · p. 3
3.3 Experimental settings
- Models received Italian prompts in baseline and role-based conditions, with three iterations per experiment and condition.The role-based prompt added a persona matching the human sample and an instruction to use native-speaker intuition; models were asked to choose an option without explanation.Source: §3.3 · pp. 3–4
4 Analysis 1: Item-level human response consistency
Human responses varied substantially across pseudoword items, and split-half analyses indicate that this variation reflects stable item properties rather than sampling noise. The authors interpret the gradient as variation in sublexical cue strength and use it as an index of processing difficulty.
4 Analysis 1: Item-level human response consistency
- The analysis defines Hi as binary entropy of participants’ left-option choices, with 0 indicating unanimity and 1 bit a perfectly split response.Hi is used as an item-level index of processing difficulty, complementing mean log response time as a measure of human effort.Source: §4 · p. 4
- Mean item-level entropy was 0.867 in EXP1 and 0.866 in EXP2, indicating substantial variation in response consistency across pseudoword pairs.Entropy ranged from 0.290 to 1.000 in EXP1 and 0.414 to 1.000 in EXP2; the means are close to the 1-bit maximum.Source: §4 · p. 4, Figure 1 · p. 4
- The authors interpret the item gradient as variation in sublexical cue strength: stronger directional signals elicit greater consensus, while weaker or balanced cues yield lower agreement.The entropy measure is carried forward as an item-level index of processing difficulty.Source: §4 · p. 4
- Split-half reliability exceeded the shuffled-label null in both experiments, confirming a stable item-level gradient rather than sampling noise.Median ρ was 0.847 in EXP1 and 0.900 in EXP2; both exceeded their corresponding null medians, with p < .001.Source: Figure 2 · p. 4, §4 · p. 4
5 Analysis 2: Human-model alignment on 2AFC experiments
Human–model alignment was measured by whether a model’s majority-vote response matched each participant’s choice. LLM alignment varied by experiment, but sublexical predictors and reasoning-token expenditure did not consistently track human responses or processing difficulty.
5 Analysis 2: Human-model alignment on 2AFC experiments
- Alignment compared each model’s majority-vote response with each participant’s choice for the same item, producing an aligned-or-misaligned outcome.Invalid responses from Minerva-7B and DeepSeek-V3 were excluded from analyses.Source: §5 · p. 5
- Native-speaker role prompts had no significant effect on alignment in either experiment (EXP1: β = −0.012, p = .621; EXP2: β = −0.015, p = .539).The models were pooled in separate binomial GLMMs for the two experiments.Source: §5 · p. 5
- EXP1’s only above-chance result was Minerva-7B Baseline (58.8%), below fastText (66.3%); EXP2 had four above-chance models, led by GPT-4o Role (59.7%) versus fastText (58.1%).The EXP2 above-chance models also included DeepSeek-V3 Base, DeepSeek-R1 Role, and o4-mini in both conditions.Source: §5 · p. 5, Table 2 · p. 6
- On the shared valid-item set, Minerva-7B remained near or below chance while frontier models largely replicated their EXP2 results.The restricted analyses included 35 EXP1 items (2,082 trials) and 39 EXP2 items (2,337 trials).Source: §5 · p. 5, Table 3 · p. 6
- No sublexical predictor consistently predicted human–LLM alignment; only DeepSeek-R1 showed a reliable Dcos effect in EXP2 (β = 0.35, p = .008).Human–fastText alignment showed reliable Dcos effects in both experiments, supporting the presence of a detectable sublexical signal.Source: §5 · p. 6, Table 4 · p. 7
- Only o4-mini Baseline in EXP1 showed a reliable positive correlation between reasoning tokens and human response time (r = .28, p = .046; partial r = .29, p = .039).For response entropy, reliable positive correlations appeared only for o4-mini in EXP2; overall, the associations were not consistent across experiments and were absent for DeepSeek-R1.Source: Table 5 · p. 7, §5 · p. 7
6 Conclusion
Across two Italian pseudoword experiments, LLMs showed limited and condition-dependent alignment with human judgments, while falling below fastText in the pseudoword-only condition. Neither the sublexical Dcos cue nor reasoning-token counts consistently tracked human responses; tokenization and training-data coverage are proposed explanations.
6 Conclusion
- LLM alignment was generally better when real-word options supplied a lexical familiarity cue, but near chance and below fastText when both options were pseudowords.The latter condition was EXP1; the former was EXP2.Source: §6 · p. 8
- Dcos, which reliably predicted human–fastText agreement, showed no consistent effect on human–LLM alignment; DeepSeek-R1 in EXP2 was the sole exception.Source: §6 · p. 8
- Reasoning-token counts showed no consistent relation to human processing difficulty: o4-mini had scattered positive associations, while DeepSeek-R1 had none.Source: §6 · p. 8, Figure 3 · p. 8
- Frequency-driven tokenization may leave form–meaning cues unavailable to LLMs when informative substrings are not isolated as tokens.The paper contrasts this with fastText, which represents pseudowords through overlapping character n-grams.Source: §6 · p. 8
- Italian sublexical regularities may be underrepresented in predominantly English training data, potentially contributing to the observed insensitivity.The paper presents training-data coverage as a speculative explanation alongside tokenization.Source: §6 · p. 8
Limitations
The study’s conclusions are limited by uncertainty about training-data coverage and by a small set of pseudoword pairs for item-level analyses.
Limitations
- The alignment gap may reflect training-data coverage as much as sublexical processing differences; replication in higher-resource languages would help assess this account.Source: Limitations · p. 8
- Item-level predictor and reasoning-token analyses were limited to 50 pseudoword pairs, so their estimates should be interpreted cautiously.Trial-level alignment analyses had over 2,000 observations per fit.Source: Limitations · p. 9