PaperlayerView exact revision on arXiv

Source-linked AI summary

Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation

Giulia Pucci, Ruizhe Li, Arabella Sinclair

arXiv:2609.04484v1cs.CLcs.AI

TL;DR

Prior work found structural priming in LM comprehension, but whether it also shapes production remained unclear. This paper tests dative sentence completions under controlled priming conditions. LMs show structural priming, strengthened by semantic coherence, alongside greater lexico-semantic repetition in structurally primed completions.

  • Problem

    Although structural priming has been observed in LM comprehension, its influence on production and the relation between structural and lexico-semantic repetition remain less clear.

    Source: §1 · p. 1

  • Method

    The authors run controlled dative sentence-completion experiments, varying prime structure and semantic coherence between prime and target stem.

    Source: §1 · p. 1

  • Results

    LMs show structural priming, with coherence boosting priming and structurally primed completions showing greater lexico-semantic repetition than unprimed completions.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings add evidence of human-like structural adaptation in LMs, while the role of coherence remains to be disentangled from a potential lexical boost.

    Source: §6 · p. 9

    The experiments cover English and one alternating construction, without a direct item-level comparison to human responses to the same stimuli.

    Source: Limitations · p. 10

Abstract

from arXiv · show

This paper investigates structural priming in language model (LM) production, examining how preceding structural context influences sentence completion. While prior work has demonstrated priming effects in comprehension of structural alternations, it remained unclear whether these persist in production, where, when generating, an LM samples from many possible continuations at each step. We address this question through a series of controlled sentence-completion experiments on dative constructions. In line with prior work, we find that LMs are susceptible to structural priming, particularly in sentences that are semantically coherent. In terms of priming magnitude, we find that while there is a greater relative increase of double-object datives against our baselines, in line with inverse frequency effects, there is a larger absolute increase in prepositional-objects, the more frequently produced construction. Finally, we not only observe that structural priming is boosted by lexico-semantic coherence, but that structurally primed completions display greater levels of lexico-semantic repetition. Taken together, our evidence supports the view that structural priming in LMs operates across multiple levels of linguistic representation, facilitating, and facilitated by syntactic, lexical, and semantic alignment. Code: https://github.com/the-context-lab/primedproduction.

1 Introduction

Structural priming is well documented in human production, but its operation in language-model generation is less clear. This paper tests dative sentence completions and finds priming, stronger effects with semantic coherence, and greater lexico-semantic repetition in structurally primed completions.

1 Introduction

  • Prior LM research found structural priming in comprehension, but whether it persists in production—where models generate among possible continuations—remains less clear.Production experiments can test whether structural context has downstream effects on sentence generation and lexico-semantic choices.Source: §1 · p. 1
  • The controlled sentence-completion experiments use dative constructions, varying prime structure and whether the target stem is lexico-semantically coherent with the prime.The study asks whether LMs prime, whether coherence boosts priming, and whether primed completions align lexically and semantically with context.Source: §1 · pp. 1–2
  • Primed completions show structural priming, with greater rates of matching dative structures than in empty, non-dative, or dative-context baselines.The baseline setting rarely elicits datives, but produces more prepositional-object (PO) than double-object (DO) completions.Source: §1 · p. 2
  • Semantic coherence increases structurally primed completions, and structurally primed completions show greater lexico-semantic alignment with context.The authors describe the association between structural and lexico-semantic priming as non-causal.Source: §1 · p. 2

2 Related Work

Prior work links human priming across lexical, semantic, and syntactic levels, while LM research has documented priming in comprehension and contextual prediction. Whether structural priming shapes LM production, and how lexical or sequence repetition relates to it, remains open.

2 Related Work

  • Human priming occurs across lexical, semantic, and syntactic levels, and repetition at one level can amplify effects at another, including lexical repetition boosting structural priming.Proposed accounts include residual activation and implicit learning.Source: §2 · p. 2
  • LM studies report structural and lexico-semantic priming in context, including syntactic content biasing evaluations more in longer contexts (Sinha et al., 2023).Prior work also finds lexical repetition can boost and predict LM priming strength (Sinclair et al., 2022).Source: §2 · p. 2
  • Whether structural priming affects LM production, and whether repetition reflects lexical sequences, hierarchical structure, or both, remains an open question.LMs show lexical repetition patterns in production, which have also been linked to co-occurrence reinforcement, task performance, and undesirable repetition (Yan et al., 2024a; Agarwal et al., 2024; Li et al., 2023).Source: §2 · p. 3

3 Methods

The methods use controlled dative sentence completions across varied prime and target conditions, model families, and measures of alignment. A hybrid dataset and automated annotation support the experiments.

3 Methods

  • Controlled experiments test language-model generation by completing target sentence stems after primes under varied conditions.The design uses a controlled dataset and sentence-completion paradigm.Source: §3 · p. 3

3.1 Data - PrimeLMDative+

  • PrimeLMDative+ combines dative and non-dative sources into a balanced dataset of more than one million sentences, with augmentation for lexical diversity and structural variation.Table 1 distinguishes no-context, non-dative, unrelated, and coherent settings; coherence reuses the prime’s last animate noun in the target.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Sentence Completion Experiment

  • The completion experiment varies prime structure and target-stem relation, including unrelated pairs and coherent pairs that reuse a noun from the prime.A no-context condition provides a baseline for completion likelihood and dative incidence.Source: §3.2 · pp. 3–4

3.3 LM sentence completion

  • The study evaluates four open-weight base model families, prompting each with a prime when applicable and a target stem, then retaining the first complete generated sentence.The models vary in scale, training data, and multilingual coverage; instruction-tuned models are excluded to avoid those confounds.Source: §3.3 · p. 4

3.4 Measuring Completion Alignment

  • Completion alignment is measured with sentence-level cosine similarity, token-level BERTScore F1, and lexical-overlap measures Jaccard and BLEU.The semantic similarity measures use all-MiniLM-L6-v2 and all-mpnet-base-v2.Source: §3.4 · p. 4

4 Annotating Dative Structures

Dative completions are labeled as prepositional-object, double-object, or non-dative using automatic classification supported by hand annotation. The authors validate the classifier against human labels and use BERT for the main experiments.

4 Annotating Dative Structures

  • Generated completions are classified as PO, DO, or non-dative using a custom BERT-based classifier, with manual annotation of a subset to validate the approach.The hand-labeled subset also supports comparison with scaled-up automatic annotation findings.Source: §4 · p. 4

4.1 Hand-annotation and agreement

  • Human annotation covers 2,700 completions, with substantial to almost perfect inter-annotator agreement across the three reported models.Cohen’s Kappa is 0.92 for Phi-2-2.7B, 0.71 for Gemma-2-2B, and 0.89 for EuroLLM-1.7B.Source: §4.1 · p. 4

4.2 Automatic annotation

  • BERT outperforms the Logistic Regression baseline more robustly across generated test sets, so the authors use BERT for main-experiment structural labeling.The comparison includes an in-domain test set and hand-annotated model-generated completions; BERT’s accuracy against human labels is 0.89 for Phi-2-2.7B, 0.90 for Gemma-2-2B, and 0.86 for EuroLLM-1.7B.Source: §4.2 · p. 5, Table 2 · p. 5

5 Results: Priming in Production

Across dative sentence-completion experiments, LMs produce more structure-matching continuations after matching primes, with coherence strengthening priming and structurally primed outputs showing greater lexical-semantic alignment.

5.1 Structural priming

  • Matching dative primes yield more structure-matching completions than no-context, non-dative, and non-matching dative baselines across both target structures.About 70% of completions were longer and more complex than the prime, indicating effects beyond simple part-of-speech sequence repetition.Source: Figure 1 · p. 6, §5.1 · p. 6

5.2 Coherence Effects on Structural Priming

  • Coherence generally raises structure-matching completions: DO effects are larger and consistent across models, while the PO advantage is not reliable for Gemma-2-9B.The PO coherence advantage was positive for all models, but not statistically reliable for Gemma-2-9B after correction.Source: Figure 2 · p. 7, §5.2 · p. 7

5.1 Structural priming

  • PO priming yields larger absolute increases, while DO shows larger relative increases, consistent with inverse-frequency effects but potentially attenuated by production entropy.The absolute PO advantage was borderline significant against the dative non-matching baseline.Source: §5.1 · pp. 6–7, Figure 3 · p. 7

5.3 Lexico-semantic Alignment under Structural Priming

  • Structurally primed completions show greater semantic and lexical alignment across metrics and contexts, with stronger effects in PO and coherent settings; lexical repetition is especially elevated under coherence.Semantic alignment boosts were comparable across coherent and unrelated conditions, whereas lexical repetition was significantly higher in the coherent condition.Source: §5.3 · p. 8

6 Discussion & Conclusions

The discussion finds structural priming in production, with its apparent magnitude depending on the construction and whether relative or absolute change is considered. Coherence and lexical repetition accompany stronger priming, while the authors identify limits and questions for further work.

6 Discussion & Conclusions

  • DOs show a stronger relative priming increase, but POs show a stronger absolute increase over baseline, reflecting their greater production frequency.The authors relate the relative DO advantage to inverse frequency effects and note that PO is the more predictable, more frequently produced structure.Source: §6 · p. 9
  • Coherence between prime and target stem reliably boosts priming, with a greater and more reliable boost for the less predictable DO construction.The discussion relates this pattern to prior human and LM findings (Sinclair et al., 2022).Source: §6 · p. 9
  • Structurally primed completions show greater lexico-semantic repetition than unprimed ones; semantic alignment is comparable across conditions, while lexical repetition is higher after coherent, lexically overlapping primes.The discussion connects the lexical-repetition finding to lexical boost effects in humans.Source: §6 · p. 9
  • Greater lexical repetition is associated with stronger structural priming, and vice versa, suggesting that the two effects tend to co-occur.The authors describe this as an association, not a resolved causal direction.Source: §6 · p. 9
  • The authors frame coherence-enhanced priming as human-like structural adaptation, while noting that further work must disentangle coherence from a possible lexical boost.They propose interpretability work to clarify whether structural priming strengthens lexical repetition, vice versa, or whether the effects are bidirectional or independent.Source: §6 · p. 9
  • The findings suggest that combinations of lexical, semantic, and structural patterns in context may influence LM-generated content.The discussion names implications for simulating human production, query wording and safety, and studying cooperation in interacting agentic LMs.Source: §6 · p. 9

Limitations

The evidence is limited to English and one alternating construction, and the study lacks matched human completions. Verb lexical bias and preference are also outside its scope.

Limitations

  • The experiments cover English and one alternating construction, so whether the production findings extend to other languages or constructions remains untested.The authors describe the evaluation as systematic and broad in stimuli, but limited in language and construction type.Source: Limitations · p. 10
  • Without human completions to the same stimuli, item-level comparisons with human responses remain unavailable; the authors note that item-level effect directionality can differ substantially.Source: Limitations · p. 10
  • Verb lexical bias and preference are outside this study, although verb preference has been reported as a predictor of priming strength.Source: Limitations · p. 10

A Dataset Creation Details

The appendix documents how the experimental data and completion-analysis pipeline were built and checked. It reports high accuracy for sentence extraction and selects BERT over Logistic Regression for labeling generated completions.

A Dataset Creation Details

  • The dataset combines dative PrimeLM sentences with non-dative examples from BLiMP, ROC, and PrimeLM transitive sentences.The appendix specifies CORE PrimeLM pairs, 15 BLiMP sub-datasets, and 976 ROC examples.Source: A Dataset Creation Details · pp. 13–14

A.1 Augmentation Templates

  • Template-based augmentation expanded the dataset to 1,048,576 sentences, adding lexical variety, syntactic variation, and length while aiming to preserve coherence and grammaticality.Augmentations include vocabulary changes, indirect speech, temporal complements, and negation.Source: A Dataset Creation Details · p. 14, §A.1 · p. 14

B Sentence Completion Generation Details

  • Each prompt row received one continuation under stochastic decoding, with generation standardized across models and conditions.The listed settings include a 40-token maximum and unrestricted top-k sampling.Source: Augmentation Templates · p. 15

C.2 Text Splitting Rules

  • The extraction pipeline cleans generated text, protects abbreviations and decimals, then finds the first sentence boundary using punctuation and capitalization with fallback rules.If no boundary heuristic applies, it retains the entire string; missing terminal punctuation is added afterward.Source: §C.2 · p. 16, §C.1 · p. 16

C Sentence Splitting

D BERT Classification Details

  • A fine-tuned BERT classifier assigns generated completions to PO, DO, or non-dative classes for structural analyses.It is evaluated on held-out data and then applied to generated completions.Source: D BERT Classification Details · p. 17

E Logistic Regression Details

  • The Logistic Regression baseline uses dependency and surface features, including object distance, prepositions, indirect-object type, POS tags, and dependency relations.The features are designed to distinguish dative structures and non-dative sentences.Source: §D.1 · p. 17, E Logistic Regression Details · p. 18

D.1 Classifier selection: BERT vs. Logistic Regression

  • BERT performed more consistently than Logistic Regression on generated completions, so the authors use BERT for automatic structural labeling in the main experiments.Logistic Regression performed well on the in-domain PrimeLM+ test set but dropped substantially on generated completions, especially for DO and non-dative cases.Source: §D.1 · p. 17

F Completion Alignment Metrics

This section defines how completion alignment and priming strength are measured. It also specifies annotation boundaries for deciding whether a completion realizes a dative alternation.

F Completion Alignment Metrics

  • Alignment metrics compare the generated completion with its contextual prime, using semantic similarity, BERTScore, Jaccard overlap, and BLEU.Sentence embeddings provide sentence-level cosine similarity; Jaccard captures unigram-set overlap, while BLEU captures ordered n-gram repetition.Source: F Completion Alignment Metrics · pp. 18–19
  • Only unrelated and coherence completions receive alignment scores because no-context prompts have no contextual prime for comparison.The target stem is removed before similarity is computed.Source: F Completion Alignment Metrics · p. 18

G Annotations Details

  • Completions count as PO or DO only when they realize a dative alternation with a transferred theme and recipient or beneficiary; other ambiguous cases are non-dative.Clausal complements and non-recipient prepositional phrases are among the excluded cases.Source: G Annotations Details · p. 19

H.1 Effect Sizes

  • The appendix reports Cohen’s h effect sizes for RQ1/RQ2 and Cohen’s d effect sizes for RQ3 across seven models.The tables specify Benjamini–Hochberg FDR-corrected significance markers.Source: §H.1 · p. 19

H.2 Priming Strength

  • Priming strength is the difference between primed incidence and baseline incidence, reported as both absolute percentage-point difference and relative percentage increase.Figure 5 shows per-model values for PO and DO; Table 10 reports t-tests of average differences.Source: §H.2 · p. 19

H.3 Length Analysis

The appendix checks whether primed completions are longer than their primes and compares alignment boosts across context conditions. It also points to supporting statistics and additional breakdowns.

H.3 Length Analysis

  • To check that priming effects were not driven only by structurally matching completions, authors found 70% of primed completions longer than simple primes, supporting structural rather than simple POS-sequence repetition.Source: §H.3 · p. 19

H.4 Completion Alignment

  • Lexical repetition is significantly higher in coherent than unrelated conditions, while the relative boost in semantic repetition is comparable.The authors attribute the lexical priming pattern to coherence being achieved by reusing the prime’s final human noun.Source: §H.4 · p. 19

H.5 Supporting Statistics

  • The appendix locates supporting RQ1 and RQ2 results in Tables 13 and 14, and additional RQ3 breakdowns in Tables 15 and 16.Figure 7 presents relative differences in lexico-semantic similarity across conditions.Source: §H.5 · p. 19

I Additional Human-Annotated Analysis

This appendix uses a balanced, manually labeled sample to check whether the main analyses’ qualitative patterns also appear with gold labels. It organizes the analysis around priming, coherence, and completion alignment.

I Additional Human-Annotated Analysis

  • The hand-annotated analysis is intended to check whether the main qualitative patterns persist when completions use manual rather than automatic labels.The appendix follows the three main research questions: structural priming, coherence effects, and alignment of structurally primed completions.Source: I Additional Human-Annotated Analysis · pp. 19–20
  • The appendix reports Cohen’s h for RQ1/RQ2 and Cohen’s d for RQ3, with corrected-significance markers specified in the table captions.The captions describe effect sizes across models for priming tests and lexico-semantic similarity comparisons.Source: Table 8 · p. 20, Table 9 · p. 20
  • The gold-labeled set contains 2,700 completions balanced across three models, three prime structures, and three context conditions.Each model contributes 100 examples for every target-structure and context-condition combination.Source: I Additional Human-Annotated Analysis · p. 20

I.1 RQ1: Structural Priming Against Baseline Conditions

The hand-annotated sample supports structural priming, especially for PO, but its evidence is weaker than in the large-scale analysis. Unrelated primes show a clearer signal for PO than DO.

I.1 RQ1: Structural Priming Against Baseline Conditions

  • For PO, all three models produce more matching completions after unrelated primes than with no context, but only Phi-2-2.7B remains significant after correction.Compared with the non-dative-prime baseline, PO differences are positive for all models but not significant after correction.Source: §I.1 · p. 21
  • For DO, unrelated primes yield only slight, statistically unreliable increases, making unrelated-context priming more visible for PO in this sample.The authors relate this asymmetry to PO completions being more frequent and easier to elicit, while DO completions are rarer and appear more context-dependent.Source: §I.1 · p. 21
  • The hand-annotated results support the direction of the large-scale findings, but with weaker statistical evidence in the smaller gold-labelled sample.The authors conclude that unrelated primes can increase structure-matching production, especially for PO.Source: §I.1 · p. 21

I.2 RQ2: Coherence Effects on Structural Priming

In the hand-annotated analysis, coherence has a clearer association with DO priming than PO priming. Structurally primed completions are often more aligned with their primes, though the pattern varies by model and metric.

I.2 RQ2: Coherence Effects on Structural Priming

  • PO coherence effects are small and inconsistent, with no model comparison remaining statistically reliable after correction.EuroLLM-1.7B and Gemma-2-2B show numerically higher PO production under coherence, while Phi-2-2.7B shows virtually no increase.Source: §I.2 · p. 21
  • DO coherence effects are clearer: EuroLLM-1.7B and Gemma-2-2B show reliable increases, while Phi-2-2.7B follows the same direction without surviving correction.The authors describe coherence as more important for DO than PO priming in this hand-annotated sample.Source: §I.2 · pp. 21–22

I.3 RQ3: Lexico-semantic Alignment under Structural Priming

  • Phi-2-2.7B shows higher similarity for primed completions across all metrics under coherence; the tendency is less consistent for unrelated primes.In unrelated contexts, stronger effects appear on surface-oriented measures such as Jaccard overlap and BLEU.Source: §I.3 · pp. 22–23
  • Gemma-2-2B completions tend to be more similar to their primes, particularly under coherence, but the effect is less stable across metrics.The authors characterize this alignment as associated with structural repetition but not uniform across similarity measures.Source: §I.3 · p. 23
  • EuroLLM-1.7B shows the weakest alignment: lexical metrics increase more clearly than semantic encoders, especially under coherence.For this model, structural priming is more strongly associated with surface-level lexical reuse than broader sentence-level semantic similarity.Source: §I.3 · p. 23
  • Across the hand-annotated sample, primed completions are often more semantically and lexically aligned with their primes, especially under coherence, but the association varies by model and metric.The smaller manually annotated set supports the main RQ3 conclusion, with weaker and more variable coupling than the large-scale automatic analysis.Source: §I.3 · p. 23

Related papers

Loading 2609.04484v1…