Language Model Behavior: A Comprehensive Survey

1. Introduction

• Language model behavior remains unpredictable for specific inputs despite relatively consistent broad benchmark performance, making models useful but unreliable in practical applications.Small changes in wording can produce unfactual, offensive, or plagiarized text.Source: §1 · p. 2
• The survey synthesizes over 250 studies covering syntax, semantics, pragmatics, world knowledge, reasoning, memorization, and bias to clarify what models can and cannot do.It emphasizes empirical behavior in controlled settings rather than taking a stance on whether models truly understand language.Source: §1 · pp. 2–3

1.1 Scope

• The survey examines masked and autoregressive English Transformer models that are not fine-tuned for specific downstream tasks, because fine-tuning can induce spurious correlations or erase pretrained information.It focuses on models increasingly used directly for text generation without task-specific data or further training.Source: §1 · p. 2, §1.1 · p. 3
• Its scope excludes non-English, multilingual, augmented, and specialized model variants, focusing instead on non-augmented out-of-the-box Transformer language models.The authors identify expansion to less-studied languages and other model variants as directions for future surveys.Source: §1.1 · p. 3
• The survey treats models as black-box functions mapping controlled input text to output-text probability distributions, leaving mechanistic analysis for future connections between internal and behavioral evidence.Mechanistic studies examine representations, neurons, and attention heads, whereas this survey focuses on behavioral responses.Source: §1.1 · p. 3

2.1 Architectures

• Transformers map tokenized inputs through embeddings and self-attention layers to contextualized representations and output-token probability distributions, with models ranging from 100M to 500B parameters.Common vocabularies contain 30K–250K tokens, and models usually have 10–100 Transformer layers.Source: §2.1 · p. 4
• Position encodings supply sequence-order information absent from basic self-attention, with relative methods enabling better extrapolation beyond pre-training sequence lengths.Models are usually pre-trained on sequences of roughly 500–2000 tokens.Source: §2.1 · pp. 4–5

2.2 Training

• Masked models predict randomly hidden tokens, whereas autoregressive models predict each upcoming token; both are pre-trained with gradient descent on large text corpora.Pre-training corpora typically contain approximately 5B to 1.5T tokens, and compute-optimal autoregressive training scales token counts roughly proportionally with model parameters (Kaplan et al. 2020; Hoffmann et al. 2022). The survey generally considers non-fine-tuned models, although later systems may add supervised, instruction, or reinforcement-learning stages (Ouyang et al. 2022; Wei et al. 2022a; Iyer et al. 2022; OpenAI 2023a).Source: §2.2 · pp. 5–6

2.3 Downstream tasks and text generation

• Language models support fine-tuning, zero-shot prompting, and few-shot prompting, while open-ended generation commonly uses greedy, temperature, top-k, or nucleus sampling.Few-shot prompting supplies examples in context without parameter updates (Brown et al. 2020; Chowdhery et al. 2022; Zhang et al. 2022b), and candidate sequences may be ranked by overall probability through beam search, though its computational cost often limits use.Source: §2.3 · p. 6

3.1 Language models generally produce grammatical text.

• Models consistently prefer grammatical tokens and encode hierarchical dependencies, including filler-gap, reflexive, and negative-polarity licensing, though verb-specific reflexive cases remain less accurate.These findings hold across autoregressive and masked language models, with grammaticality generally improving as model and corpus size increase.Source: §3.1 · pp. 7–8

3.2 Language models learn subject-verb agreement, but they are sensitive to intervening clauses and specific words.

• Subject-verb agreement is generally strong but degrades with longer dependencies, intervening clauses, infrequent or specific nouns and verbs, and nonce sentences.Few-shot examples can significantly reduce distance-related degradation in large autoregressive models; infrequent verbs produce over 40% less accurate predictions and over 30% error rates in nonce sentences.Source: §3.1 · p. 7, §3.2 · pp. 8–9

3.3 Language models learn syntactic rules early in pre-training.

• Syntactic rules emerge within roughly the first 20% of masked-model pre-training, with similar generalizations across architectures, initializations, and data shuffling despite early n-gram-like behavior.Small models trained on child-directed speech can match larger standard models, and syntactic abilities emerge despite Transformers’ limits on arbitrarily deep hierarchy.Source: §3.3 · p. 9

3.4 Language models can learn word order without explicit position information, but word order is not necessary in many examples.

• Models can learn token positions without explicit position embeddings, and word order is unnecessary in many cases, but position removal, shifting, and minority cases requiring order expose important sensitivities.Position information can be recovered from token sequences, sentence-length correlations, or autoregressive counting; absolute-position models also overfit to shifted positions.Source: §3.4 · p. 10

4.1 Language models learn semantic and compositional properties of individual words, including argument structure, synonyms, and hypernyms.

• Larger language models capture word meanings, substitutions, verb-specific argument structures, and some compositional parses, but performance declines for infrequent items and remains below syntactic accuracy.Few-shot compositional parsing improves with model size, although natural-language meaning construction remains unclear because studies often request unnatural semantic outputs.Source: §4 · pp. 10–11, §4.1 · pp. 11–12

4.2 Language models struggle with negation, often performing worse as models scale.

• Over 50% worse performance can occur on negated prompts, and larger models often produce more incorrect negation completions, although additional negation training improves masked-model sensitivity.Negation weaknesses may be underrepresented in standard benchmarks and can resemble human online processing rather than successful offline reasoning.Source: §4.2 · p. 12

4.3 Language models construct coherent but brittle situation models.

• Language models track introduced entities and bridging relations, but GPT-3 answers questions about entity states and event likelihoods only marginally better than chance and remains sensitive to question phrasing.Performance is better when answers are explicit, but mathematical reasoning, temporal ordering, logical negation, and references to unintroduced entities remain unreliable.Source: §4.3 · pp. 12–13

4.4 Language models recognize basic analogies, metaphors, and figurative language.

• Models recognize literal and conventional metaphors, improve on similes and analogies with scale, and show a sharp metaphor-identification increase around 100B parameters, but struggle with abstract analogies and constructions.Comparative correlatives are recognized at chance after controlling for adjective frequencies, while simile interpretations remain worse than human performance.Source: §4.4 · p. 13

4.5 Language models can infer the mental states of characters in text, but they struggle with implied meaning and pragmatics.

• Models can infer some character knowledge, desires, perspectives, and mental states, but pragmatic understanding remains weak for implicatures, presuppositions, indirect meanings, and natural reference.Implicature performance is only slightly above chance and does not substantially improve with scale, despite preferences for conversationally relevant content.Source: §4.5 · pp. 13–14

5. Commonsense and World Knowledge

• Factual and commonsense knowledge generally improves with scale and can be enhanced through explicit retrieval, search-engine connections, or knowledge-base connections.These mechanisms supplement the knowledge acquired during language-model pre-training.Source: §5 · p. 14

5.1 Language models learn facts and commonsense properties of objects, particularly as models scale, but they are less sensitive than people to physical properties.

• Models retrieve diverse factual knowledge and commonsense properties, with accuracy and self-correction improving as model size increases, but physical-property predictions remain less human-aligned.They complete facts about locations, products, people, and media, while relying more on encyclopedic or corpus patterns than perceptual properties and producing less creative object uses than people.Source: §5.1 · p. 15

5.2 Learned facts are sensitive to context and a fact’s frequency in the pre-training corpus.

• Learned facts are highly sensitive to template wording, punctuation, entities, distractors, and pre-training frequency, causing models to rely on heuristics and memorize facts more readily than they generalize them.In-context models can infer properties for novel words, but the same inference often fails when the fact is encountered during pre-training; larger models improve especially on popular entities and can assess factual answers more reliably.Source: §5.2 · pp. 15–17

5.3 Language models have a limited but nontrivial ability to make commonsense inferences about actions and events.

• Models make limited commonsense inferences about events, often distinguishing typical or possible situations, but their performance varies widely and is frequently driven by answer priors, lexical cues, or corpus frequency.They struggle with temporal spans, causal bridges, conjunctions, action effects, coherent event sequences, and physical inferences, despite some success with roles, event ordering, and supported continuations.Source: §5.3 · pp. 17–18

6.1 Large language models can perform basic logical reasoning when prompted, but they still struggle with complex reasoning.

• Complex reasoning remains weak: models struggle with branching steps, planning, multi-step fact combination, and rule-based inference, sometimes achieving below 5% accuracy.They may follow valid reasoning steps yet favor word-overlap or intuitively plausible answers instead of logically correct ones.Source: §6.1 · pp. 18–19

6.2 Language models exhibit basic numerical and probabilistic reasoning abilities, but they are dependent on specific inputs.

• GPT-3 handles small or frequent-number arithmetic but performs poorly on large numbers, whose unpredictable tokenization and corpus frequency strongly affect accuracy.As numbers increase, models may produce non-numeric responses; larger models perform significantly better at arithmetic (Brown et al. 2020).Source: §6.2 · p. 19
• Numerical reasoning is brittle: minor wording changes and irrelevant context disrupt word-problem solutions, competition-math accuracy stays below 10%, and exploration in probabilistic tasks is random.GPT-3 can outperform people in some gambling tasks, but its exploration is not targeted or information-optimal (Binz and Schulz 2023).Source: §6.2 · p. 19

7. Memorized vs. Novel Text

• Models generate both memorized pre-training text and novel text conditioned on input context, with memorization creating plagiarism, privacy, and benchmark-contamination risks.Deduplicating pre-training data or increasing sampling temperatures can reduce memorization.Source: §7 · p. 20

7.1 As language models scale, they are more likely to generate memorized text from the pre-training corpus.

• Autoregressive models produce about 1–5% memorized tokens without prompting, while scaling increases memorized and paraphrased text; deduplication reduces memorization by up to 10x.Exact-sequence extraction is facilitated by high-probability generation, memorized continuations, and frequent corpus examples.Source: §7.1 · p. 20

7.2 Language models generate novel text that is consistent with the input context.

• Novel generations contain more frequent local structures and shallower syntax than human text, yet longer sequences are more novel despite occasional memorized passages.Sampling reduces repetition but also produces less factual text than sequence-based decoding, while exposure bias can cause looping and repetition.Source: §7.2 · pp. 20–21
• Generated text is generally consistent with input context, with larger models producing more coherent, on-topic text and often fewer factual and commonsense errors.Models also reproduce contextual grammaticality, syntactic structures, personality, and politics, while adapting style more effectively as model size increases.Source: §7.2 · p. 21
• Model predictions depend strongly on specific context words, using local semantic and n-gram similarities that can confound factual recall with reasoning.Autoregressive models can nevertheless recall arbitrary noun lists across intervening text when prompted with suitable vignettes.Source: §7.2 · p. 21

8. Bias, Privacy, and Toxicity

• Non-fine-tuned language models generate biased, toxic, offensive, and private text, whose harms depend on societal context and can be detected by humans or automated systems.The survey focuses on controlled descriptions of unsafe outputs and refers readers to prior work on societal impacts and mitigation.Source: §8 · pp. 21–22

8.1 Language models sometimes generate offensive text and hate speech, particularly in response to targeted prompts.

• Autoregressive models can be red-teamed into producing hate speech, harassment, violence, and other toxic content, which also often emerges without targeted prompting and at similar rates across model sizes.Automated or learned prompt construction can induce toxic outputs, including in larger models, sometimes using only access to a smaller model.Source: §8.1 · p. 22

8.2 Language models can expose private information, but often not tied to specific individuals.

• Larger autoregressive models generate more valid emails in known pre-training contexts, but current methods usually produce random or fake PII rather than information tied to individuals.Masked clinical-data models are difficult to prompt for patient health information, while privacy expectations remain difficult to guarantee for web-trained models.Source: §8.2 · p. 22

8.3 Language model behavior varies across demographic groups, both in terms of raw performance and probabilities of toxic text.

• Language models assign different probabilities and achieve different prediction accuracy across demographic groups, while prompts mentioning some groups substantially increase toxic or negative-sentiment outputs.Toxicity reaches 13% for some LGBTQIA+ identity prompts in 350M-parameter masked models and up to 87% for some identity groups; beam search reduces these effects relative to top-k or nucleus sampling.Source: §8.3 · p. 23

8.4 Language models reflect harmful stereotypes based on gender, sexuality, race, religion, and other demographic identities.

• Models reproduce stereotypes involving gender, sexuality, race, religion, nationality, occupations, and other identities, but measured bias varies with prompts, metrics, models, training runs, and model scale.Stereotypical predictions include gendered occupations and pronouns, sexualized descriptions of women, and harmful religious associations; some effects can be mitigated by redirecting stereotypes, while gender-occupation scaling results are inconsistent.Source: §8.4 · pp. 24–25

9.1 Language models can generate convincing unfactual text and unsafe advice.

• Larger models more readily affirm conspiracy theories and common misconceptions, while semantically consistent responses may remain factually wrong (Levy, Saxon, and Wang 2021; Lin, Hilton, and Evans 2022; Rae et al. 2021; Raj, Rosati, and Majumdar 2022).The misconception finding is inconsistent across studies.Source: §9.1 · p. 25
• GPT-3-generated tweets were rated as true more often than human-generated tweets about vaccines, COVID-19, climate change, and other topics, regardless of factuality (Spitale, Biller-Andorno, and Germani 2023).Conversations with GPT-3 can also change opinions on BLM and climate change (Chen et al. 2022).Source: §9.1 · p. 26
• GPT-3 gives worse advice than people in 95% of Reddit situations, while autoregressive models rarely generate physically unsafe advice but slightly favor unsafe completions when forced to choose (Zellers et al. 2021; Levy et al. 2022).Models also agree with human social-permissibility ratings only marginally above chance (Jin et al. 2022a).Source: §9.1 · p. 26

9.2 Model-generated text is difficult to distinguish from human-generated text.

• People identified news articles from 175B-parameter models with only 52% accuracy versus 50% random chance, and specialized instructions raised identification accuracy only to about 55% (Brown et al. 2020; Clark et al. 2021).People perform better on longer sequences, but language models also remain relatively inaccurate at detection; watermarking has been proposed as a countermeasure (Ippolito et al. 2020; Jawahar, Abdul-Mageed, and Lakshmanan 2020; Wahle et al. 2022; Kirchenbauer et al. 2023; He et al. 2022b).Source: §9.2 · p. 26

9.3 Language model “personality” and politics depend on the input context.

• Models generally mimic political leanings and personality traits implied by prompts, including ideological moral reasoning, predicted voting patterns, and Big Five survey responses (Perez et al. 2022b; Simmons 2022; Argyle et al. 2023; Caron and Srivastava 2022; Jiang et al. 2022).Some stances are more probable a priori, models score low on agreeableness and openness regardless of prompt, and sampling temperature changes reported demographics, personality, and values (Liu et al. 2022c; Johnson et al. 2022; Miotto, Rossberg, and Kleinberg 2022).Source: §9.3 · p. 27

10.1 Effects of scale

• Larger autoregressive models usually match or exceed smaller models, but scaling effects vary by task, can change sharply beyond 175B parameters, and incur greater computational and environmental costs.Some tasks show sudden gains beyond 175B parameters, while others exhibit U-shaped scaling because larger models can ignore distractors; fine-grained conclusions require task and model details.Source: §10.1 · pp. 28–29

10.2 Language modeling as generalization

• Language-model failures often reflect under-generalization of patterns to novel inputs or over-generalization of frequency, positional, semantic, or socially biased patterns.Input sensitivity affects agreement, factual recall, situation modeling, logic, and arithmetic, while heuristic fallbacks and corpus-derived social biases illustrate over-generalization; larger-model gains may rely heavily on memorizing more specific instances.Source: §10.2 · pp. 30–31

10.3 Levels of analysis in understanding language models

• Mechanistic analyses increasingly connect language-model behavior to internal representations, neurons, parameters, and attention heads, including targeted factual edits and causal copying or in-context learning functions.These approaches aim to bridge behavioral and mechanistic analysis by identifying interpretable, generalizable principles and causal links, including mechanisms underlying over- and under-generalized behavior.Source: §10.3 · p. 31

• The survey covers over 250 studies of language model behavior from the past three years.Source: §11 · p. 31
• Language models remain sensitive to specific inputs and surface features even at scales reaching hundreds of billions of parameters.Source: §11 · p. 32
• Model strengths and weaknesses can often be understood as correct or incorrect generalizations of text patterns, informing deployment, regulation, and future analysis.Source: §11 · p. 32

Reasoning and cognition

• Related work covers zero-shot reasoning, coherence prediction, analogy generation, and cognitive-psychology approaches to understanding language models.The listed studies include GPT-2 reasoning, coherence evaluation, analogy generation with InstructGPT, and cognitive psychology applied to GPT-3.Source: 2021. Thinking aloud: Dynamic context · p. 34

Risks and harms

• The bibliography includes work on language-model bias, fairness, gender-neutral pronouns, privacy, memorization, and training-data extraction.These references span critical surveys, adaptation to gender-neutral pronouns, privacy preservation, memorization quantification, and extraction of training data.Source: 2021. Thinking aloud: Dynamic context · pp. 34–35

Model behavior and evaluation

• Additional references address foundation-model opportunities and risks, prompting, contextual representations, knowledge, attention, grammar, commonsense, and social biases.The cited literature also includes foundation-model risks, prompt probing, contextual embedding geometry, language models as knowledge bases, attention, grammar learning, commonsense reasoning, and bias measurement.Source: 2021. Thinking aloud: Dynamic context · pp. 34–37

2021. Thinking aloud: Dynamic context

• The provided passages are related-work bibliography fragments rather than substantive evidence, so they do not establish results about dynamic context.Entries include works on syntactic agreement, pragmatic reasoning, negation, reasoning, generalization, memorization, bias, and transformer interpretability.Source: 2021. Thinking aloud: Dynamic context · pp. 38, 39, 41

Knowledge and reasoning

• Related studies examine what language models know, including long-tail knowledge, implicit relations, script knowledge, and knowledge elicitation.Source: 2022. Can large language models truly · p. 42

Risks and biases

• The related work addresses privacy risks, plagiarism, truthfulness, and social bias in language models, including Raffel (2022), Lee (2023), Evans (2022), Volpin et al. (2021), and Liu et al. (2022c).Source: 2022. Can large language models truly · pp. 42–45

Knowledge and reasoning

• The bibliography covers reasoning limitations involving negation, symbolic knowledge, event plausibility, causal inference, numerical commonsense, and zero-shot reasoning.It includes Kassner and Schütze (2020), Kassner and Schütze (2020), Kauf et al. (2022), Li et al. (2022a), and Kojima et al. (2022).Source: 2022. Can large language models truly · pp. 42–44

Prompting and input sensitivity

• Several works investigate how prompting and input form affect model behavior, including negated prompts, prompt-based models, explanations in context, and surface cues in causal reasoning.Relevant studies include Kavumba et al. (2022), Lampinen et al. (2022), Kementchedjhieva et al. (2021), and Kavumba et al. (2022).Source: 2022. Can large language models truly · pp. 41, 43, 44

Linguistic behavior

• The cited studies examine recursive nested constructions, subject-verb agreement, figurative language interpretation, and discourse relations in language generation.Source: 2022. Can large language models truly · pp. 43–45

• The references cover studies of language-model knowledge, including factual associations, commonsense, conceptual knowledge, and knowledge-base behavior.They include work on locating factual associations, commonsense acquisition, conceptual knowledge, and the limits of BERT as a knowledge base.Source: 2022. Can large language models truly · pp. 46, 48
• The cited work also addresses model behavior through scaling, demonstrations, in-context learning, decoding strategies, evaluation methods, and model-written evaluations.The references include studies of model scale, in-context learning, decoding effects on verifiability, evaluation beyond perplexity, and model-written behavioral assessments.Source: 2022. Can large language models truly · pp. 45–48
• The references document research on social bias, stereotypes, toxicity, harmful completions, reporting bias, and linguistic bias in language models.They include StereoSet, CrowS-pairs, HONEST, LGBTQIA+ harmful-completion measurement, toxic-content probing, and studies of how reporting or linguistic bias affects models.Source: 2022. Can large language models truly · pp. 46–47
• The cited literature examines linguistic competence across syntax, semantics, pragmatics, discourse, metaphors, event knowledge, and compositional generalization.The references include targeted syntactic evaluation, discourse connectives, metaphor understanding, generalized event knowledge, and compositionality gaps.Source: 2022. Can large language models truly · pp. 47–49

Related work

• The references include work on language-model scaling, training, transfer learning, and text-style transfer.These entries include Rae et al. (2021), Raffel et al. (2020), and Reif et al. (2022).Source: 2022. Can large language models truly · p. 49
• The bibliography also covers reliability, systematicity, semantic consistency, numerical reasoning, and sentence comprehension in language models.The listed studies address semantic consistency, hypernymy probing, pretraining-term frequencies, and agreement phenomena.Source: 2022. Can large language models truly · pp. 49–50
• Additional references examine model geometry, conceptual consistency, commonsense knowledge, and zero-shot communication.The supplied entries include studies titled “Visualizing and measuring the geometry of BERT,” “Do children texts hold the key to commonsense knowledge?”, and “Large language models are not zero-shot communicators.”Source: 2022. Can large language models truly · p. 50

Related work

• The cited literature spans language-model reasoning, including chain-of-thought analysis, mathematical reasoning, planning, tool use, and theory-of-mind limitations.These entries include work on greedy reasoning, causal robustness of mathematical reasoning, planning, Toolformer, and social intelligence.Source: 2022. What do large language models · pp. 50, 52, 53
• The bibliography covers language-model knowledge and linguistic behavior in syntax, discourse reference, pragmatics, negation, numeracy, analogies, and physical-world knowledge.Representative entries examine quantity and syntax, discourse entities, pragmatic reasoning, negation, numerical representations, analogies, and knowledge about the physical world.Source: 2022. What do large language models · pp. 50, 51, 53
• The cited studies also address social bias, stereotypes, toxicity, ad hominems, gender effects, and the reliability of bias evaluation in pretrained and generated language.The entries include work on societal biases, gender bias, holistic descriptors, ad hominems, and bias-probe reliability.Source: 2022. What do large language models · pp. 50–51
• Additional references examine memorization, plagiarism, adversarial triggers, attention, positional representations, model size, scaling, and the computational costs of large language models.The supplied entries include studies of memorization dynamics, machine-paraphrase plagiarism, adversarial attacks, attention interpretability, position embeddings, scaling, and energy policy.Source: 2022. What do large language models · pp. 51–53

References

• The reference list further includes studies of model scaling, pretraining data, training trajectories, position embeddings, masking rates, quantization, and open pretrained models.These entries span training across scales, data requirements, architectural choices, and model releases such as OPT.Source: 2022. What do large language models · pp. 54–56
• The references include work on chain-of-thought prompting, emergent abilities, analogical reasoning, and arithmetic and logical reasoning.Source: 2022. What do large language models · pp. 54–55
• Other cited studies address robustness, detoxification, bias and social risks, multilingual or sociolectal variation, controllable generation, and evaluation methods.The references also include work on spurious correlations, ethical risks, toxicity, richer representations, BERTScore, and controllable text generation.Source: 2022. What do large language models · pp. 54–56
• The bibliography covers probing and benchmarking of syntax, semantics, grammatical knowledge, comparative correlatives, tacit assumptions, and sentence understanding.Examples include BLiMP, BERT probing with negative polarity items, GPT-3 semantic probing, and computational tests of syntactic learnability.Source: 2022. What do large language models · pp. 54–56

Literature Review Process

• The process began with 271 relevant language model analysis papers and expanded to over 15K papers citing any seed paper.The citation search was last scraped on February 4, 2023.Source: Appendix A · p. 56
• Manual title filtering reduced the corpus to approximately 1500 potentially relevant papers, followed by abstract filtering to approximately 400 highly relevant papers.The scope was gradually refined as described in Section 1.1.Source: Appendix A · p. 56