Holmes: A Benchmark to Assess the Linguistic Competence of Language Models

• Prompt-based benchmarks can conflate linguistic competence with abilities such as producing coherent text or following instructions, limiting definitive conclusions.Source: §1 · p. 1
• Holmes trains linear probes on LMs’ internal representations and uses probe performance to estimate understanding of specific linguistic phenomena.The representations come from the last model layer, and the probing approach is checked with control tasks and an information-theoretic perspective.Source: §1 · pp. 1–2
• A review of over 270 probing studies finds fragmented coverage: only three LMs were probed on over 20% of tasks, and recent large LMs are underrepresented.Source: §1 · p. 2
• Holmes provides 208 datasets across morphology, syntax, semantics, reasoning, and discourse, including underrepresented phenomena such as negation and rhetoric.Source: §1 · p. 2
• Across 59 LMs, no model consistently excels; linguistic competence is more pronounced for morphology and syntax and varies with model size, architecture, and instruction tuning.Competence, particularly in morphology and syntax, scales beyond 350 million parameters; instruction tuning improves morphology and syntax but has mixed effects on other phenomena.Source: §1 · p. 2
• FlashHolmes approximates Holmes rankings with high precision while requiring only ~3% of the computation.It selectively excludes samples that do not significantly influence overall rankings.Source: §1 · p. 2

• Linguistic competence is unconscious understanding of language, whereas linguistic performance is language use in utterances; Holmes focuses on competence in LMs as static artifacts.Source: §2 · p. 3
• Holmes groups phenomena into morphology and syntax as formal phenomena, and semantics, reasoning, and discourse as functional phenomena.The categories cover word structure, sentence structure, word meaning, logical deduction and related phenomena, and textual context such as rhetorical structure.Source: §2 · p. 3
• A dataset consists of text examples and labels for a specific phenomenon aspect, typically with unambiguous labels to isolate what is tested.Source: §2 · p. 3
• A probe is a linear model trained on LM-encoded dataset examples to predict phenomenon labels; its performance indicates patterns represented internally by the LM.The paper describes classifier-based probing as an examination of information in LM representations.Source: §2 · p. 3

3.1 Scope

• The review analyzed 28k papers from 2015 to August 2023, identified 493 matching papers through criteria-based filtering, then manually reviewed candidates and added curated studies, yielding 274 relevant papers.The corpus included major NLP conferences and selected work from other venues; screening excluded uses of “probing” in other senses.Source: §3.1 · p. 4

3.2 Analysis

• Probing studies cited one another infrequently, with an average probing citation ratio α of 0.1, while only 16 papers received over 200 general citations.The authors interpret the citation patterns as limited engagement with linguistic competence across other fields.Source: §3.2 · p. 4
• Task-focused studies made up 51.8% of the selected work, analytical studies 35.7%, and methodological studies 12.5%.Task-focused studies covered 202 tasks, compared with 115 in analytical studies; methodological studies examined only 24 LMs and 36 tasks.Source: §3.2 · p. 5
• Classifier-based probing dominated the surveyed methods at 74%, followed by mask-based probing at 20%; attention-based and other approaches accounted for about 3%.The latter approaches include bridging and dimension selection.Source: §3.2 · p. 5
• The studies collectively covered 289 unique tasks and 161 distinct LMs, yet individual studies considered only fractions; newer larger models and instruction-tuned LMs were missing almost entirely.POS, the most-probed task, was evaluated on only 23% of LMs; the top 10 mentioned LMs accounted for 80% of all LM mentions.Source: §3.2 · p. 5

3.3 Summary

• The authors argue that consolidating existing resources could support more comprehensive assessment and stimulate future research.They compare the potential stimulus with ImageNet in computer vision and GLUE and SuperGLUE in NLP.Source: §3.3 · p. 6

4.1 Datasets

• Holmes provides 208 datasets covering 66 phenomena across morphology, syntax, semantics, reasoning, and discourse, including less-probed areas such as negation and rhetoric.The datasets combine established resources with less commonly used data addressing contextualization, semantic decomposition, grammatical knowledge, bridging, and discourse structure.Source: §4 · p. 6, §4.1 · p. 6

4.2 Structure

• The benchmark groups datasets by linguistic phenomenon and then places phenomena into five types, following source-study categorizations when available.The authors note that a phenomenon such as tense could be categorized differently depending on how it is tested, but follow the original study’s categorization.Source: §4.2 · pp. 6–7

4.3 Experimental Setup

• Holmes trains linear probes on last-layer LM representations, using token, span, or sentence embeddings according to the dataset and five random-seed runs.When source data lack splits, samples are divided into train, development, and test sets at a 70/10/20 ratio.Source: §4.3 · p. 7

4.4 Evaluations

• Task scores use macro F1 for classification or Pearson correlation for regression, while mean winning rate ranks LMs by pairwise wins.The standard deviation across seeds is also calculated; lower σ indicates a more robust measurement.Source: §4.4 · p. 7
• In the displayed subset of rankings, FLAN-UL2 outperforms other evaluated LMs overall, while different LMs lead across the five phenomenon types.The figure presents a subset of Holmes rankings, with lower values indicating better rank.Source: Figure 7 · p. 7
• Compression and selectivity complement task performance: higher compression indicates clearer encoding, while selectivity compares original-label performance with randomized-label controls.The benchmark uses minimum description length relative to uniform encoding for compression and the F1-score difference for selectivity.Source: §4.4 · p. 7

5 Holmes Results

• σ = 0.02 across five seeds, average I = 1.9, and average S = 0.31 support the reliability of Holmes’ probing results.Task metrics also correlate positively with compression (τ = 0.64) and selectivity (τ = 0.65); selectivity uses base-sized models for computational efficiency.Source: §5 · p. 8, Figure 8 · p. 8
• Formal phenomena (morphology and syntax) score higher than functional phenomena, while all five types contribute to overall rankings without one dominating.Phenomenon types correlate 68.4 ± 7.5 with overall results and 54.7 ± 13.9 with one another on average.Source: §5 · pp. 8–9, Figure 9 · p. 9, Figure 10 · p. 9
• Holmes and OpenLLM LM rankings are medium correlated; correlations are higher for morphology (65.3) and reasoning (77.5) than for syntax, semantics, and discourse (54.7 to 58.0).The patterns are consistent across MMLU, TruthfulQA, and GSM8K, while TruthfulQA shows lower correlations with linguistic phenomena and other OpenLLM datasets.Source: Figure 10 · p. 9, §5 · pp. 10–11
• 52% of encoder LMs versus 21% of decoder LMs have a higher mean winning rate; even 700-times-larger decoders do not match encoder performance.Encoders also show higher median accuracy and lower deviations on the top-20 common POS tokens than same-size GPT2 models.Source: Figure 11 · p. 9, §5 · p. 9
• Linguistic competence scales with size in both Pythia and T5, especially beyond 0.5B and 1.0B parameters, respectively.T5 mean winning rates range from 40–70%, compared with 20–60% for Pythia; morphology and syntax show a performance jump after those thresholds.Source: §5 · pp. 9–10, Figure 12 · p. 10
• +10% for morphology, +5% for syntax, and +4% for reasoning accompany instruction tuning, while semantics falls −3% and discourse −1%.Larger models gain more; Llama-2-70b-Chat and FLAN-UL2 gain up to +24% and +41% for morphology and +10% and +12% on average. Vicuna’s 125k higher-quality instructions compare favorably with Tülu’s approximately 330k.Source: Table 2 · p. 10, §5 · p. 10
• τ = 0.05 for probing- versus prompting-based rankings on jointly evaluated BLiMP tasks in Holmes and HELM, showing apparent ranking discrepancies.The authors report that most HELM prompting results fall below the random baseline and argue that prompting mixes linguistic assessment with abilities such as instruction following.Source: §5 · p. 11

6 Efficiency

• Holmes takes ~6 GPU days to encode 70 million tokens and two days to run 208 probes for a 70b model.FlashHolmes targets the compute burden of encoding text and training probes.Source: §6 · p. 11
• FlashHolmes excludes 18 licensed probing datasets and reduces computation for encoding and probing by discarding training instances.Its construction balances evaluation efficiency with benchmark reliability.Source: §6 · p. 11
• At 1/2 of the training data, rank resolution is ~0.9; at 1/512, it is ~2.6, illustrating the reliability–efficiency trade-off.The authors choose 1/32 of training instances for FlashHolmes.Source: Figure 13 · p. 11, §6 · p. 11
• Using 1/32 of training instances, FlashHolmes reduces evaluation computation to ~3% of Holmes while preserving a high rank-correlation of ~1.5.The reported rank resolution is a 95% confidence interval of model rank difference.Source: §6 · p. 11, Figure 13 · p. 11

7 Related Work

• Benchmark results can disagree substantially, and evaluations using textual instructions are sensitive to formulation and metric choices.Benchmarks have covered abilities including language understanding, out-of-distribution generalization, knowledge contradiction, adversarial scenarios, and retrieval.Source: §7 · pp. 11–12

Assessing the Linguistic Competence of LMs

• Prior linguistic-competence work examined word vectors, sentence embeddings, translation-model internals, contextualized LMs, evaluation validity, causal effects, and downstream task understanding.Other studies found downstream task fine-tuning can hurt understanding of linguistic phenomena.Source: Assessing the Linguistic Competence of… · p. 12
• Holmes probes a broad set of contemporary LMs across linguistic phenomena, whereas prior work often focused on particular tasks or used prompting.The authors present probing as a way to compare LMs across architectures and pre-training, addressing calls for more comprehensive evaluation.Source: Assessing the Linguistic Competence of… · p. 12

8 Conclusion

• Holmes finds linguistic competence more pronounced for formal phenomena than functional phenomena requiring broader textual context, while size, architecture, and instruction tuning are crucial factors in differences among LMs.The authors plan to extend the benchmark with new datasets and upcoming LMs.Source: §8 · p. 12

Ethical Considerations and Limitations

• The benchmark covers English only, with multilingual expansion deferred because English resources and available LMs are more widely available.The authors plan to expand its multilingual scope.Source: Ethical Considerations and Limitations · p. 12
• Analysis uses the output of each LM’s last layer, which the authors say only partially captures the complexity across all layers.They present Holmes’ tasks and evaluation code as groundwork for future layer-specific analyses.Source: Ethical Considerations and Limitations · p. 12
• Holmes’ coverage is incomplete: its phenomena, LMs, and underlying metastudies represent only a subset of available resources, and the benchmark will be expanded.Source: Ethical Considerations and Limitations · p. 12
• Eighteen of Holmes’ 208 datasets use licensed resources and are not freely available; sharing them depends on confirmation of granted access.FlashHolmes offers an alternative based on open-access resources.Source: Ethical Considerations and Limitations · p. 12
• Because Holmes relies on existing datasets, it inherits their biases; unknown training or instruction-tuning data also means some benchmark texts or tasks may have been encountered by evaluated LMs.The authors state that internal-representation evaluation retains validity under potential contamination.Source: Ethical Considerations and Limitations · pp. 12–13

• After BERT’s 2019 release, probing research increased, reaching a peak of 90 papers in 2022.Source: §A.1 · p. 25
• Major conferences ACL and EMNLP published 149 relevant papers, more than half of the reviewed work.Source: §A.1 · p. 25
• A further 68 papers appeared at AACL, EACL, NAACL, and COLING.Source: §A.1 · p. 25

• Probe training uses 20 epochs, AdamW, batch size 64, learning rate 0.0005, dropout 0.2, and 10% warmup, with the best epoch selected on development instances.Source: §A.2 · p. 25
• Experiments run on 12 Nvidia RTX A6000 GPUs, each with 48GB memory and 10,752 CUDA cores.Source: §A.2 · p. 25
• Table 9 is identified as outlining the evaluated LMs, but this passage does not provide their individual details.Source: §A.2 · p. 25

A.3 Probing Datasets Categorization

• Holmes covers 19 morphology, 75 syntax, 67 semantics, 19 reasoning, and 28 discourse datasets, drawing on 33 works for data or phenomenon definitions.Source: §A.3 · pp. 25–26

A.4 Details of Probing Dataset Composition

• Datasets use an input span or sentence, optionally paired with textual context, and a label; tokenized multi-token words may require averaging their vectors into one representation.Figure 14 illustrates input composition for four task types using last-layer representations.Source: §A.4 · p. 26, Figure 14 · p. 27
• The dataset catalog combines established resources with specialized probes, including nine context-word datasets testing properties such as animacy and verb dynamics.Source: §A.3 · p. 25, §A.4 · p. 26
• Holmes converts BLiMP and Zorro phenomena into binary accept-or-reject classifications, while available original train/dev/test splits are retained and missing splits are formed in a 70/10/20 ratio.Some tasks are also reformatted; oLMmpics mask-filling becomes binary classification of correctly versus wrongly filled masks.Source: §A.4 · pp. 26–27
• Additional task constructions include synonym-or-antonym replacement detection, three metaphor datasets, and negation probes for text-level negation, cue or scope, and correspondence between spans.Source: §A.4 · p. 27

Sentence Order and Next Sentence Prediction

• Discourse resources include sentence-order and next-sentence tasks, discourse representation labels and relations, and rhetorical-tree probes for span type, structure, and relations.Source: Sentence Order and Next Sentence… · p. 28
• Figure 15 compares Holmes and HELM for 40 open decoder models across 22 BLiMP datasets; its accompanying text reports Holmes’ advantage and says most HELM results fall below random baseline.The comparison is framed as probing-based versus prompting-based evaluation.Source: Figure 15 · p. 29

A.3 Probing Datasets Categorization

• Tables 4–8 map resources and phenomena for morphology, syntax, reasoning, discourse, and semantics, while Table 9 describes evaluated-model attributes.Source: Table 4 · p. 29, Table 5 · p. 30, Table 6 · p. 30