PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompts and probabilities in the linguistic assessment of language models: Evidence from semantic focus-sensitivity

Robert Pasternak

arXiv:2610.39923v1cs.CLcs.AI

TL;DR

Prior comparisons of prompting and token probabilities found different performance patterns across model sizes, motivating tests in another linguistic domain. The authors evaluate both methods on semantic focus-sensitivity and find that probabilities beat chance across models, while forced-choice prompting can outperform them in larger models and decline less with atypical markers.

  • Problem

    Prior comparisons found that prompting often underperforms token probabilities in weaker models, with the gap narrowing as models grow, leaving their relative performance in a new linguistic domain to be tested.

    Source: §1 · pp. 1–2

  • Method

    The authors compare string probabilities with forced-choice and yes/no prompting on semantic focus-sensitivity using instruction-tuned models of varied sizes and focus markers.

    Source: §1 · p. 2

  • Results

    Probabilities perform significantly above chance for all models, while forced choice beats them by wide margins in some larger models and declines less under randomly generated focus markers.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Prompting and probabilities may provide complementary views of language competence, and prompting can be more robust when evaluated sentences are stylistically atypical.

    Source: §5 · pp. 9–10

    The conclusions are limited by the study’s focus on a single empirical domain, so their generality requires testing other linguistic data.

    Source: §5 · p. 9

Abstract

from arXiv · show

As language models (LMs) have improved, an increasingly common means of evaluating their linguistic competence is through explicit prompting (e.g., "Which sentence is more acceptable?"). Prior work indicates that such prompting underperforms compared to standard token-probability-based methods in smaller models, but improves in larger models. We test this in a new domain: semantic focus-sensitivity, responsible for contrasts like "Lucy only SWAM yesterday" vs. "Lucy only swam YESTERDAY". Testing models of various sizes using text-based focus markers like boldface and italics, we find universal above-chance performance for token probabilities. Prompting results mostly align with previous findings, but in some larger models they outperform probabilities by a wide margin. We propose that this is because LMs are not prone to produce out-of-the-blue focus marking, which muddies results from token probabilities in a way that does not affect prompting. We test this by replacing standard focus markers with randomly generated ones; token probabilities still beat chance but show significantly greater performance decrease compared to prompting. These results suggest that metalinguistic prompting, while non-trivial for LMs, can also be more robust when testing stylistically atypical data. We argue that this has important consequences for the assessment of LMs' language competence, even if LMs do not perform metalinguistic judgments through human-like introspection.

1 Introduction

The introduction motivates comparing token probabilities with prompting for linguistic assessment and tests both on semantic focus-sensitivity. It reports above-chance probabilities across models, while prompting can outperform them in larger models and be more robust to atypical focus marking.

1 Introduction

  • Prior comparisons found prompting much worse for weaker models, with the gap shrinking as models grow and some sufficiently large models slightly outperforming probabilities.The introduction notes that prompting is designed to resemble human judgment tasks, while minimal pairs help control irrelevant influences on probabilities.Source: §1 · pp. 1–2
  • The study compares string probabilities with forced-choice and yes/no prompts on a manually created 282-item minimal-pair dataset across three instruction-tuned model families.The models are Gemma 4, Qwen3, and Qwen3.5.Source: §1 · p. 2
  • String probabilities perform significantly above chance for every tested model, suggesting substantial acquisition of semantic focus-sensitivity.The experiments use eight text-based focus-marking methods.Source: §1 · p. 2
  • Prompting performs poorly in smaller models and improves with scale; forced choice usually beats yes/no prompting, while larger Gemma 4 models beat probabilities by a wide margin.The authors hypothesize that unsolicited focus marking muddies probability results, whereas forced choice presents the marking in the input and compares paired sentences side by side.Source: §1 · p. 2
  • With randomly generated focus markers, probabilities still beat chance but decline proportionally more than forced-choice prompting, which more than doubles its significantly outperforming instances.The authors interpret this as evidence that forced choice can be more robust on stylistically atypical sentences.Source: §1 · p. 2
  • The authors argue that prompting and probabilities may test complementary facets of language competence, loosely analogous to comprehension and production.They caution that the methods need not measure one single, true indicator of competence.Source: §1 · p. 3

2 Focus-sensitive environments

Focus placement can affect truth or pragmatic appropriateness across several linguistic environments, and its acceptability often depends on context. The section presents these environments as a diverse, commonplace test domain, not an exhaustive inventory.

2 Focus-sensitive environments

  • Question-answer congruence links focus in an answer to its semantic relation with the question; a mismatched focus can make an answer odd.The example contrasts an answer focusing SENATOR with one focusing BOOK after “Who did Phil give a book to?”Source: §2 · p. 3
  • Additive too is focus-sensitive: after “The doctor greeted the nurse,” focusing PATIENT is acceptable, whereas focusing NURSE is not.The contrast illustrates how the continuation’s focus affects its fit with the preceding context.Source: §2 · p. 3
  • Adverbs of quantification such as usually can depend on focus placement, as in the contrast between MORNING and RUNS.The example stipulates that Sarah rarely runs but chooses morning runs when she does.Source: §2 · p. 3
  • Contrastive focus, including contrastive negation, favors focus on the contrasting property: GREEN is better than SHIRT when correcting a claim about a green shirt.The section also gives paired contrastive-focus examples beyond negation.Source: §2 · p. 3
  • The discussed environments are diverse and commonplace enough for a reasonable test domain, but they do not cover the full range of focus-sensitive environments.The authors proceed to experiments using this scope.Source: §2 · pp. 3–4

3 Experiment 1: Standard focus markers

Experiment 1 tests semantic focus-sensitivity with eight text-based focus markers and compares string probabilities with forced-choice and yes/no prompting. String probabilities beat chance for every model, while prompting generally improves with model size and forced choice exceeds probabilities in some larger models.

3.1 Methodology

  • The manually created dataset contains 282 minimal pairs across six focus-sensitive environments, tested with eight text markers and three evaluation methods.The 282 pairs were each rendered with all caps, HTML underlining, and HTML or Markdown boldface, italics, or emphasis.Source: §3.1 · p. 4
  • Items use backgrounds or prefixes to distinguish the better sentence, and controls remove that context to test whether performance reflects dataset bias.The design addresses the possibility that models could rely on a syntactic heuristic rather than the background information.Source: §3.1 · pp. 4–5, §3.2 · pp. 5–6

3.2 Results

  • String probabilities performed above chance for every model; they significantly outperformed prompting in smaller models, while forced choice and yes/no generally improved with model size.The target-item results also outpaced controls, supporting a role for backgrounds and prefixes.Source: §3.2 · p. 6, Figure 1 · p. 6
  • Forced choice generally outperformed yes/no prompting, especially in larger models, though prompt engineering could improve yes/no results.Forced choice presents both candidate sentences together, whereas yes/no evaluates one at a time.Source: §3.2 · pp. 6–7
  • Forced choice significantly outperformed string probabilities for several larger models, with the widest gaps in Gemma 4; Gemma 4 31B FC2 reached 95.3%.Gemma 4 31B FC1 reached 91.7%, and Qwen3.5 27B FC1 reached 91.3%.Source: §3.2 · p. 6, Figure 1 · p. 6
  • The two mixture-of-experts models performed below smaller non-MoE models in their families, with the authors suggesting prompting may be especially affected.Source: §3.2 · pp. 6–7
  • For Gemma 4 31B, FC1 and FC2 outperformed SP across all categories, indicating a general rather than subclass-confined FC advantage; Qwen3.5 27B FC1 also beat SP, but results were more mixed.The authors caution that manually created data may contain biases that make some items more difficult for independent reasons.Source: §3.2 · p. 7
  • The authors propose that out-of-the-blue focus marking can depress string probabilities, while forced choice avoids output-likelihood effects by including marked sentences in the input.Seeing both sentences together may also clarify the comparison; the authors note that yes/no results could reflect other factors.Source: §3.2 · p. 7

4 Experiment 2: Ad hoc focus markers

Experiment 2 replaces canonical focus markers with randomly generated markers to test generalization and compare robustness across evaluation methods. String probabilities remain above chance, but forced choice declines less than probabilities as markers become more atypical.

4.1 Methodology

  • The experiment replaced common markers such as boldface and italics with ten randomly generated character sequences bracketing the focused constituent.The ad hoc-marker dataset contained 2,820 minimal pairs; otherwise, the experiment matched Experiment 1.Source: §4 · p. 8, §4.1 · p. 8

4.2 Results

  • Target-item results outpaced controls, again indicating that backgrounds and prefixes mattered rather than models relying only on a simple syntactic heuristic.Source: §4.2 · p. 8, Figure 3 · p. 8
  • String probabilities remained significantly above chance for every model, while forced choice improved from weak smaller-model performance to well above chance in larger models.Yes/no performance hovered around chance with ad hoc markers.Source: §4.2 · p. 8, Figure 3 · p. 8
  • FC1 and FC2 accuracy declined less than string-probability accuracy when moving from standard to ad hoc markers for nearly every model.After controlling for standard-marker accuracy, both FC methods had significantly less negative effects than string probabilities (both p< .001).Source: §4.2 · pp. 8–9, Figure 4 · p. 9
  • The relative robustness of forced choice was not explained solely by its lower starting accuracy: in four of five cases where prompting beat probabilities initially, forced choice still declined less.There were twelve instances of forced choice significantly outperforming probabilities in Experiment 2, compared with five in Experiment 1.Source: §4.2 · pp. 8–9

5 Discussion and conclusion

Forced-choice prompting can be more robust than string probabilities for stylistically unlikely sentences, but the evidence is limited to one empirical domain. The discussion considers whether prompting and probabilities may assess complementary aspects of language knowledge, even if prompting is not introspective.

5 Discussion and conclusion

  • Forced choice saw a smaller accuracy decrease than string probabilities when standard focus markers were replaced with randomly generated ones.The authors suggest forced choice may be more robust when evaluating sentences that are stylistically unlikely outputs.Source: §5 · p. 9
  • Conclusions are limited by the focus on a single empirical domain, so their generality across other linguistic data remains to be tested.Suggested tests include other data models may be disinclined to generate, such as semantically nonsensical but syntactically well-formed sentences, non-standard dialects, socially unacceptable language, and unambiguously false statements.Source: §5 · p. 9
  • Future work could test prompting tasks that are not explicitly metalinguistic, such as code generation, which may be better represented in training.Such comparisons could clarify which pros and cons of metalinguistic prompting are specific to its metalinguistic nature.Source: §5 · pp. 9–10
  • Whether metalinguistic judgments involve introspection remains unresolved; if they do not, one interpretation is that explicit judgments use factual knowledge while probabilities better indicate linguistic competence.The factual-versus-linguistic-competence interpretation is conditional on judgments being non-introspective.Source: §5 · p. 10
  • Input interpretation and output generation may draw on distinct knowledge, making prompting and string probabilities complementary measures rather than competitors.The authors characterize prompting as testing comprehension and probabilities as testing production.Source: §5 · p. 10
Loading 2610.39923v1…