PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Just an Evaluation: A Case Study of Discourse Particle Sensitivity in Language Models

William Sheffield, Jiahui Chen, Amy Zhang, Alex Warstadt, Junyi Jessy Li, Kanishka Misra

arXiv:2610.13794v1cs.CL

TL;DR

Discourse particles can shape conversational continuations while leaving truth-conditional meaning largely untouched, but their continuation effects have received little study. This paper tests LLM sensitivity to just and only in natural dialogue using behavioral and mechanistic analyses. Humans and models generally prefer particle-present conversations, and localization reveals asymmetric overlap between the particles’ units.

  • Problem

    The effects of discourse particles on continuations have received little study, despite their subtle role in shaping discourse and reflecting conversational context and speaker attitudes.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates just and only in natural conversations through human and model preference comparisons, generated follow-up probabilities, and activation-patching localization.

    Source: §1 · p. 2, §6.1 · pp. 6–7

  • Results

    Humans and models generally prefer particle-present conversations, with weaker preferences for only than just; only-sensitive units transfer selectively to exclusive just, while just-sensitive units transfer to only across senses.

    Source: §4.2 · p. 5, §7 · p. 9

  • Takeaways & Limitations

    Conversational follow-ups can reveal particle effects that single-sentence evaluations cannot, and provide a way to evaluate whether LLMs attend to subtle discourse cues.

    Source: §7 · p. 9

    The study covers English discourse particles in relatively short, scripted-style exchanges and models in the 8B parameter range, with localization tested on one LLM.

    Source: §8 · pp. 9–10

Abstract

from arXiv · show

Discourse particles subtly shape the flow of discourse by reflecting conversational context and interlocutor attitudes, often leaving the truth-conditional meaning untouched. They are also often polyfunctional, with distinct senses which partially overlap with other particles; for example, the exclusive use of English "just" and "only" ("John only/just drinks water"). Despite the frequency of these words in natural language, they have gone largely unstudied due to the subtle nature of their effects. We investigate LLM sensitivity to "just" and "only" in natural conversations using behavioral and mechanistic analyses. First, we find that LLMs and humans similarly prefer the presence of these particles in context, and both are more sensitive to "just" compared to "only". Second, when sampling from LLMs with and without these particles, we find LLMs consistently rate generations as more likely, given their source context, demonstrating their particle sensitivity. In both experiments, instruction tuning increases sensitivity. Finally, we localize particle-sensitive units within LLMs and find the overlap between "just" and "only" is asymmetric: "only" units transfer selectively to exclusive "just", while all "just" units transfer to "only" equally well regardless of sense, suggesting LLM lexical-level particle representations are partially organized by sense. We release our code, sense-labeled dialogue stimuli, and human preference annotations at https://github.com/JiahuiKChen/Not_Just_an_Evaluation.

1 Introduction

Discourse particles shape how conversations proceed and reflect context and speaker attitudes, often without changing truth-conditional meaning. This paper examines LLM sensitivity to just and only through behavioral comparisons with humans and mechanistic analysis.

1 Introduction

  • Discourse particles subtly shape discourse flow and reflect conversational context and interlocutor attitudes while leaving truth-conditional meaning largely untouched.The example contrasts follow-up questions that fit Claire’s claim with just against those that fit without it.Source: §1 · p. 1
  • Particles can have distinct, interconnected senses that partly overlap; exclusive just and only can both exclude other salient alternatives, but are not interchangeable in every context.Only is difficult to parse in some dialects in the example “He’s only late all the time.”Source: §1 · p. 1
  • LLMs and humans prefer ground-truth conversations with just or only over versions with the particle removed, while LLMs also show sensitivity in sampled follow-ups and localized representations.Instruction tuning increases sensitivity, and just and only share some localized units asymmetrically.Source: §1 · p. 2
  • The effects of discourse particles on conversational continuations have received little study, despite their frequency and relevance to conversational agents.Prior work addressed phenomena such as irony, metaphor, implicature, connectives, particle sense identification, and particle prediction, but not effects on continuations.Source: §1 · p. 2

2 Background

Prior work studies discourse particles, connectives, and pragmatic phenomena through several approaches, but particle effects on continuations remain distinct from those lines of inquiry. This paper uses natural multi-turn dialogues and continuation probabilities to investigate those effects.

2 Background

  • Unlike connectives, which signal relations between clauses, discourse particles reflect speaker attitudes; prior LM work has studied connective prediction, inference, and mechanistic representations.This paper instead uses natural multi-turn dialogues and measures how particle removal shifts probabilities of whole follow-up turns, including model generations.Source: §2 · p. 3
  • Prior particle-focused LM studies examined sense discrimination in metalinguistic settings or particle prediction in Colloquial Malay, rather than effects on continuations.Source: §2 · p. 3
  • The paper adapts minimal-pair evaluation beyond a sentence by including pre- and post-context, and applies generated-continuation comparisons to particle rather than at-issue sensitivity.Source: §2 · p. 3
  • For localization, the paper adapts activation patching to whole attention, MLP, and residual-stream activations, evaluated through counterfactual activation replacement.Source: §2 · p. 3

3 Experimental Setup

The experiments use selected DailyDialog conversations containing just or only, construct particle-present and particle-removed pairs, and score them across base and instruction-tuned models. The setup also includes human preference judgments and a meaning-altering control.

3.1 Dataset

  • The dataset retains target utterances ending in just or only, with a follow-up and up to three preceding context turns; the filtered set has 1,182 just, 360 only, and 1,307 not examples.The authors manually select 60 examples per target word for a gold set and split just examples into equal exclusive and non-exclusive sense subsets.Source: §3.1 · pp. 3–4

3.2 Scoring Setup

  • Base models are scored on speaker-labeled dialogue text, while instruction-tuned models use their native chat templates; probabilities are measured directly rather than elicited through metalinguistic prompts.Both utterance variants are capitalized after deletion to avoid spurious sentence-initial casing penalties.Source: §3.2 · p. 4

3.3 Models

  • The study compares base and instruction-tuned checkpoints from four roughly comparable open-weight model families: Llama 3 8B, OLMo-2 7B, Qwen3.5 9B, and Gemma2 9B.Source: §3.3 · p. 4

4 Experiment 1: Particle Removal Effects for Humans and LMs

In the particle-removal comparison, humans and models generally prefer original conversations with the particle, with weaker preferences for only than just. Post-training increases model sensitivity, and model ratings generally correlate positively with human ratings.

4 Experiment 1: Particle Removal Effects for Humans and LMs

  • The experiment compares model and human preferences for particle-present CSF against particle-removed CS′F on a gold dataset, using not as a meaning-altering control.Source: §4 · p. 4

4.1 Scoring Metrics

  • The model comparison score is ∆removal = log P(CSF) − log P(CS′F), with positive values indicating preference for the original dialogue and deltas normalized per token.Source: §4.1 · p. 4
  • Nine participants rated randomized, two-column dialogue pairs on an implicit 7-point Likert scale, alongside distractors made by randomly reordering words in S.The ratings covered 240 items: 60 examples for each of three target words and 60 distractors.Source: §4.1 · p. 5

4.2 Results

  • Humans prefer all target-word conversations, while models follow this trend for not and just; only preferences are weaker, and not significantly above zero for all base models and one instruction-tuned model.Models generally correlate positively with humans, and post-training increases sensitivity.Source: §4.2 · p. 5

5 Experiment 2: Generation

Experiment 2 moves beyond deletion-based preference tests, which capture only particle-dependent follow-ups and a single continuation. It tests whether responses sampled with or without a particle fit their source context better.

5 Experiment 2: Generation

  • The generation design addresses two limits of deletion-based preference tests: asymmetric sensitivity and evaluation of only one continuation.The paired prompts differ only in the target word, and responses are sampled from both versions.Source: §5 · p. 5

5.2 Results

  • All sampled responses favor their source prompt; particle effects are roughly an order of magnitude smaller than not, showing subtler next-turn sensitivity.Positive own-context advantage means a response is more probable under the prompt from which it was sampled.Source: Figure 3 · p. 5, §5.2 · p. 6

5.1 Methodology

  • For each item, they sampled 50 unique next turns from each target-present or target-removed prompt, then rescored every response under both prompts using per-token log-probability differences.Base models used speaker-labeled turns; instruction-tuned models used native chat templates without a task-specific system prompt.Source: §5 · p. 5, §5.1 · p. 6

5.2 Results

  • Examples show just withdrawing an offer of help and only implying other laundry remains; Llama 3 8B Instruct’s nearly identical praise yields a small just advantage.The model responds in a helpful-assistant register rather than the shopkeeper’s, so its follow-ups can differ from the human follow-up for reasons unrelated to the particle.Source: §5.2 · p. 6

6 Experiment 3: Function Localization Analysis

Experiment 3 uses activation patching to localize sites supporting word sensitivity, then tests transfer between exclusive and non-exclusive just and only. Localized sites outperform random sites in nearly all comparisons, while transfer from only to just is sense-selective but the reverse is not.

6.1 Experiment 3a: Component-Based Localization of Word Sensitivity

  • Activation patching localized sensitivity by substituting clean-context activations into word-removed prefixes, then ranking layer-specific sites by restoration score within each component type.The study used Llama 3 8B Instruct and selected the top k ∈ {1, 3, 5, 10} sites; localization and causal evaluation used separate item-level folds.Source: §6.1 · pp. 6–7
  • Localized sites had significantly larger effects than random sites in 35 of 36 word–component–k comparisons, with effects increasing as more sites were patched.Residual-stream and attention sites produced the largest effects, MLP effects were smaller, and effects were strongest for not, followed by only and then just.Source: §6.1 · p. 8, Table 2 · p. 9

6.2 Experiment 3b: Sense-Specific Transfer Between just and only

  • Transfer tests localized sites on exclusive just, non-exclusive just, or only response pools and evaluated them on the paired target pools across both fold directions.The exclusive sense of just and only share an alternative-excluding contribution, motivating the test of whether transfer is sense-selective.Source: §6.2 · p. 8
  • Only→just transfer favored exclusive just in every contrast, with significant advantages for residual-stream sites at every k and top-10 attention sites, but not MLP sites.No just→only contrast showed a significant exclusive advantage; significant attention contrasts at k = 1 and k = 3 and the residual-stream contrast at k = 5 favored non-exclusive just.Source: §6.2 · p. 8, Table 2 · p. 9

7 Discussion and Conclusion

LLMs show sensitivity to just and only in both ground-truth and model-generated follow-ups, aligning with human judgments and showing weaker preference for only than just. Localization results suggest lexical representations bundle particle senses, with asymmetric transfer between only and just.

7 Discussion and Conclusion

  • LLMs and humans prefer particle-containing contexts, with weaker preference for only than just; instruction tuning increases model sensitivity in Experiments 1 and 2.The pattern appears with ground-truth and model-generated follow-ups.Source: §7 · p. 9
  • The experiments cannot identify why model families differ because architecture, pretraining scale, and post-training recipe vary together; instruction tuning is the factor isolated.Base checkpoints behave similarly across families, while most cross-family variation emerges after instruction tuning.Source: §7 · p. 9
  • In Llama 3 8B Instruct, only-sensitive units transfer more cleanly to exclusive than non-exclusive just, while just units transfer equally well to only across senses.The authors interpret this as lexical representations bundling senses rather than representing them separately.Source: §7 · p. 9
  • Conversational follow-ups reveal discourse-particle effects that cannot surface within one sentence, supporting their use in targeted evaluations of subtle discourse cues.The authors describe this as especially clear for instruction-tuned models and say their generation method helps counter scarce gold data.Source: §7 · p. 9

8 Limitations

The study is limited in language, exchange style, selected stimuli, model size, and localization scope. The appendix also illustrates how original and counterfactual histories are rendered and how human preference judgments are collected.

8 Limitations

  • Evidence is limited to English particles in short scripted-style exchanges and selected DailyDialog items; one linguist’s hand-selection may bias sense labels.Generating a distribution of continuations in Experiment 2 slightly mitigates, but does not remove, the selected-item constraint.Source: §8 · pp. 9–10
  • Experiments use 8B models, localization analyzes one LLM, and deletion counterfactuals may retain subtle fluency artifacts or contextual cues.The localization method may also miss structure at finer-grained units such as individual heads or neurons.Source: §8 · p. 10

A Prompt Construction Example

  • The gold follow-up belongs to the original DailyDialog item but is omitted from prompts; models instead sample follow-ups from the original and modified histories.The appendix shows the second unique response sampled from each history under a fixed seed.Source: A Prompt Construction Example · p. 13

A.1 Base-model rendering

  • Base-model prompts use alternating speaker labels and a next-speaker prefix, with separate sampled replies for histories with and without just.The example pairs an insurance-cost question after the original history with a self-payment response after the modified history.Source: §A.1 · p. 13

A.2 Instruction-tuned rendering

  • Instruction-tuned prompts use the native chat template, with the target utterance as the final user message and the next assistant response sampled.The example’s original-history reply discusses insurance and reassurance, while the modified-history reply begins with thanks.Source: §A.2 · p. 13

B Human Annotation Details

  • Human annotators compare conversation versions using a preference slider, adjusted to an implicit seven-point scale after pilot feedback about exact ratings.Annotators were recruited through Prolific and had to reside primarily in the United States, speak English as their primary language, and meet an approval threshold.Source: B Human Annotation Details · pp. 13–14, Figure 4 · p. 15

C Additional Particle Removal Results

The appendix points to detailed particle-removal results and full results for the particle-sensitive generation experiment, without providing the underlying values in these passages.

C Additional Particle Removal Results

D LLM Particle-sensitive Generation

E Additional Qualitative Examples

Additional examples show that particles can shape sampled follow-ups in ways that match the conversational move, though effects vary by item. Human and model preferences can diverge, and some items show little model sensitivity.

E Additional Qualitative Examples

  • On the lost-baggage item, human ratings split (mean 4.2), and near-zero own-context advantages indicate little effect of just on LLM follow-ups.Sampled follow-ups often repeat just; Qwen3.5 9B CT substitutes only, conveying a different attitude toward the loss.Source: E Additional Qualitative Examples · p. 15
  • On the “Only a few?” item, Gemma-2 9B IT echoes the particle from CS but responds sympathetically from CS′, while humans are comparatively indifferent (mean 4.1).Gemma-2 9B IT has a negative mean own-context advantage on this item.Source: E Additional Qualitative Examples · p. 16
  • For the low-price lipstick exchange, humans strongly prefer CS′F (mean 1.3), and all eight LLMs adopt the speaker’s framing from CS but describe the price as merely reasonable from CS′.The gold follow-up complains that the lipstick is too expensive, conflicting with the speaker’s framing of 25 yuan as a low price.Source: E Additional Qualitative Examples · p. 16

F Compute and Model Details

The paper reports inference-only experiments run on a three-GPU A40 pool, with model-scoring and sampling costs varying by experiment. The total reported model-based compute is approximately 25 A40 GPU-hours.

F Compute and Model Details

  • Experiment 1 takes less than 1.3 A40 GPU-hours across eight checkpoints for the 180-item gold dataset; the 2,849-pair supplemental analysis takes approximately 8 A40 GPU-hours.Per-checkpoint scoring of the gold dataset takes less than 10 minutes on one A40.Source: F Compute and Model Details · p. 18
  • Experiment 2 generates 18,000 responses per checkpoint and 144,000 across eight checkpoints, taking approximately 11 A40 GPU-hours overall.Each checkpoint takes about 25–30 minutes of wall time across three A40s, or approximately 1.4 A40 GPU-hours.Source: F Compute and Model Details · p. 18
  • Experiment 3 uses nine localization and transfer jobs that sum to approximately 4.6 A40 GPU-hours.Jobs take approximately 24–57 minutes each and include held-out sufficiency and necessity evaluation and random-site controls.Source: F Compute and Model Details · p. 18
  • Across model-based experiments, including the supplemental larger-dataset analysis, the reported runs require approximately 25 A40 GPU-hours.The pool consists of three NVIDIA A40 GPUs with 48 GB memory each; GPU-hours sum time over occupied accelerators.Source: F Compute and Model Details · p. 18

G Differences Across Model Families

Post-training corresponds to most cross-family variation in particle sensitivity, but family-specific patterns and multiple model differences complicate attribution. The paper also reports how AI assistants were used and outputs checked.

G Differences Across Model Families

  • The studied families differ across architecture, multimodal pretraining, data, scale, and post-training, so observed sensitivity differences cannot be attributed to any single factor.Qwen3.5 uses a hybrid attention architecture, multimodal pretraining, and agent-oriented post-training; the authors note these factors cannot be separated in their experiments.Source: G Differences Across Model Families · pp. 18–19
  • Documented pretraining budgets range from 4.05T tokens for OLMo-2 to over 15T for Llama 3, but the paper cannot compare families’ training-data composition because only OLMo-2 releases its data.The authors leave analysis of conversational and particle-bearing training data to future work.Source: G Differences Across Model Families · pp. 18–19
  • After instruction tuning, cross-family removal-delta ranges widen for both just and only, and every family’s instruction-tuned checkpoint has larger own-context advantages than its base counterpart.Base ranges are 0.044–0.060 for just and 0.069–0.098 for only; instruction-tuned ranges are 0.050–0.189 and 0.078–0.159, respectively.Source: G Differences Across Model Families · p. 19
  • Qwen3.5 gains little from post-training in Experiment 1, while Gemma-2 IT shows a two-to-threefold CS′-versus-CS own-context advantage for both particles.Gemma-2 IT’s values are 0.0558 vs. 0.0266 for just and 0.0858 vs. 0.0313 for only; the other instruction-tuned checkpoints are roughly symmetric.Source: G Differences Across Model Families · pp. 19–20

H AI Assistant Use

  • AI assistants helped implement experiments and visualizations and polish author-written text; included outputs were verified by the authors for accuracy.They also provided suggestions to improve writing flow and clarify text.Source: H AI Assistant Use · p. 20
Loading 2610.13794v1…