PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Behavior: A Comprehensive Survey

Tyler A. Chang, Benjamin K. Bergen

arXiv:2303.11504v2cs.CL

TL;DR

Research on language-model behavior lacks a consolidated account of what large models do before task-specific fine-tuning, especially across capabilities and failure modes. This survey synthesizes over 250 English-language behavioral studies and finds that models remain sensitive to specific inputs and surface features even as they scale, with many strengths and weaknesses reflecting generalization of text patterns.

  • Problem

    A rapidly growing and fragmented literature makes language-model behavior difficult to track and understand, despite models’ increasing use without task-specific fine-tuning.

  • Method

    The survey synthesizes over 250 behavioral studies of non-fine-tuned English Transformer language models across capabilities, weaknesses, model scale, and text-pattern generalization.

  • Results

    Language models remain sensitive to specific inputs and surface features even at hundreds of billions of parameters, while many strengths and weaknesses reflect correct or incorrect generalizations of text patterns.

  • Takeaways & Limitations

    The synthesis is intended to inform the deployment and regulation of large language models and motivate future behavioral analysis.

  • Takeaways & Limitations

    The survey covers English, non-augmented Transformer language models and excludes non-English or multilingual models, mechanistic analyses, and other model variants.

Abstract

from arXiv · show

Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about large language model capabilities, thus providing a resource for applied work and for research in adjacent fields that use language models.

1. Introduction

This introduction surveys the rapidly expanding behavioral literature on language models used without task-specific fine-tuning. It frames the central challenge as models that are broadly capable yet highly sensitive to inputs, making their behavior difficult to understand and apply reliably.

  • Motivation: Language-model text generation remains unpredictable for specific inputs even when broad benchmark performance is relatively consistent for a given model size and architecture.This input sensitivity makes models attractive but unreliable in many practical applications.
  • Scope and contribution: The survey reviews over 250 studies spanning syntax, semantics, pragmatics, world knowledge, reasoning, memorization, and bias.It focuses on English language models before task-specific fine-tuning.
  • Observed behavior: Language models generate fluent and coherent text, but slight changes in input wording or phrasing can produce unfactual, offensive, or plagiarized outputs.The survey connects these behaviors to questions about language-model meaning and understanding.
  • Synthesis: The survey synthesizes capabilities and weaknesses through the perspectives of model scale and text-pattern generalization.This synthesis is intended to ground discussion of what models can and cannot do in empirical evidence.
  • Methodological scope: The review emphasizes behavioral studies that treat models as black-box functions mapping controlled input text to probability distributions over output text.Mechanistic analyses of representations, neurons, and attention heads are outside the survey’s main scope.

2. Transformer Language Models

Transformer language models tokenize text, transform token representations through stacked attention-based layers, and predict masked or upcoming tokens during pre-training. They can then be adapted or prompted for downstream tasks, including zero-shot and few-shot use without parameter updates.

  • Architecture: Transformer language models convert input strings into token sequences, map tokens to embeddings, and process them through stacked Transformer layers.These layers typically combine self-attention, layer normalization, and feedforward networks.
  • Architecture: Position encoding supplies token-order information that the Transformer architecture does not naturally represent.Absolute, relative, and rotary methods are among the approaches discussed.
  • Training: Masked models predict replaced tokens, whereas autoregressive models predict the next token from the preceding context.Both are trained with gradient descent over large text corpora.
  • Training: Language models may receive further non-task-specific fine-tuning, including curated language-modeling examples or reinforcement learning from human feedback.The survey generally considers non-fine-tuned models unless otherwise specified.
  • Downstream use: Autoregressive models can perform tasks without task-specific fine-tuning through zero-shot prompting or few-shot in-context learning.Few-shot prompting includes example inputs and outputs in the prompt without parameter updates.
  • Text generation: Open-ended generation selects each next token using procedures such as greedy selection or temperature-based sampling.The model computes output-text probabilities by iteratively multiplying next-token probabilities.

3. Syntax

Language models generally learn grammatical structure, including agreement, hierarchical dependencies, and licensing, but their syntactic predictions remain sensitive to model scale, token position, specific words, and sentence structure.

  • Both autoregressive and masked models assign higher probabilities to grammatical tokens and reflect hierarchical syntactic structure.
  • Models respect filler-gap and reflexive licensing, although reflexive predictions become less accurate when licensing depends on the specific verb.
  • Subject-verb agreement is generally accurate, including in nested clauses, but performance varies with nouns, verbs, distractors, and dependency distance.
  • Agreement predictions are over 40% more accurate for contextually probable verbs, while error rates exceed 30% for infrequent verbs in nonce sentences.
  • Syntactic abilities emerge early and consistently during pretraining, with similar generalizations across architectures, initializations, and data orderings.Rules are learned within roughly the first 20% of masked-language-model pretraining.
  • Removing position embeddings can preserve masked-language-model performance, while masking more tokens improves fine-tuning performance for larger models.

4. Semantics and Pragmatics

Language models show basic semantic, compositional, entity-tracking, figurative, and pragmatic abilities, but these behaviors are highly sensitive to task formulation, context, and model size. They remain unreliable on negation, implied meaning, and some noncompositional constructions.

  • Compositional semantics: Few-shot autoregressive models extract compositional semantic parses, with performance improving as model size increases.However, explicitly requesting a semantic parse leaves their behavior in natural-language meaning scenarios unclear.
  • Lexical semantics: Language models learn lexical relationships and often predict word meanings, hypernyms, synonyms, and co-hyponyms, but struggle with infrequent words and pairs.Larger models show stronger sensitivity to human-rated typicality in hypernym prediction.
  • Lexical semantics: Autoregressive models reflect verb-specific argument and implicit-causality biases, with larger models more closely matching human ratings in some alternation tasks.Causality effects are less reliable in smaller models and vary with verb frequency.
  • Negation: Models often ignore negation, generate incorrect completions after negated or few-type quantified statements, and can worsen with scale in some settings.Fine-tuning on more negation sentences improves masked-model sensitivity.
  • Entity tracking: Language models track entities and bridging relations, but sometimes refer to entities that were never introduced and answer state or event questions only marginally above chance.Performance improves when information is explicit, but answers remain sensitive to question phrasing.
  • Figurative language and pragmatics: Models recognize some metaphors, similes, analogies, and pragmatic cues, yet perform worse than people on figurative interpretation and near chance on several implied-meaning tasks.They also struggle with comparative correlatives and often produce over-informative references instead of natural pronouns.

5. Commonsense and World Knowledge

Language models encode many factual and commonsense regularities, but their knowledge is unreliable and highly sensitive to prompts, entities, frequency, and context. They show basic event inference while struggling with numerical, fine-grained, and physical distinctions.

  • Learned facts: Language models assign higher probabilities to factual statements and complete facts across locations, products, historical figures, and cultural works.Performance improves with natural templates and relevant preceding context.
  • Learned facts: Larger models generally improve factual accuracy and can increasingly assess whether their own factual answers are correct.Self-reflection ability increases with model size, with examples reported around 50B parameters.
  • Object properties: Physical-property predictions are only loosely aligned with human responses, while numeric literals, numerical commonsense, and fine-grained property distinctions remain difficult.Models often rely more on encyclopedic facts or corpus n-grams than on visual, perceptual, or human-judgment patterns.
  • Prompt sensitivity: Fact prediction varies substantially with paraphrased or punctuated templates, entities, word choice, and simple heuristics tied to the prompt.Masked models may produce the same material prediction across different objects or rely on semantically similar distractors.
  • Frequency and generalization: Factual learning depends on corpus frequency, with schema-conforming facts learned faster and example-level question-answering performance correlated with fact frequency.Factual knowledge continues evolving late in pre-training, while observed facts do not reliably support inferences during pre-training.
  • Event commonsense: Models make basic commonsense event inferences, assigning higher probabilities to possible or supported scenarios, but performance varies across examples and temporal or causal tasks.They can infer typical locations and event order, yet often favor unusual events and struggle with plausible continuations, physical inference, and coherent event sequences.

6. Logical and Numerical Reasoning

Language models can perform basic logical, mathematical, and numerical reasoning when given examples or explicit prompting. However, complex reasoning, input variation, tokenization, and irrelevant wording expose substantial fragility.

  • Logical reasoning: Few-shot examples enable autoregressive models with at least 8B parameters to perform well on mathematical word problems, formal logic puzzles, and related reasoning tasks.Some studies report above-chance logic-puzzle performance only for models with at least 20B parameters.
  • Logical reasoning: Models over 100B parameters can produce valid step-by-step reasoning from an explicit “Let’s think step by step” prompt.GPT-3 can also reason with relevant, coherent examples even when the examples contain invalid logic.
  • Logical reasoning: Complex reasoning remains difficult: models struggle with multiple valid next steps, planning, combining facts, implicit multistep questions, and rule-based inference.Toy-block planning accuracy is reported below 5%, and models may choose answers by word overlap or intuitive plausibility.
  • Numerical reasoning: GPT-3 handles small or frequently seen arithmetic numbers but performs poorly on large numbers, partly because fixed vocabularies segment them unpredictably.Larger models are significantly better at arithmetic, while performance remains correlated with input frequency in text corpora.
  • Numerical reasoning: Mathematical word-problem performance is sensitive to slight wording changes and irrelevant context, while competition-math accuracy remains below 10% even with fine-tuning.GPT-3 can make some probabilistic decisions better than chance, but its exploration of uncertain possibilities is essentially random.

7. Memorized vs. Novel Text

Language models can generate novel, context-consistent text but also reproduce memorized training sequences. Memorization increases with scale and certain decoding choices, while generation can become repetitive or less factual depending on the strategy.

  • Memorized text: Autoregressive models can reproduce exact pre-training sequences, and around 1.5B-parameter models output about 1-5% memorized tokens without prompting.Supplying the beginning of a memorized sequence or using more frequent training examples increases memorization likelihood.
  • Memorized text: Deduplicating pre-training data can reduce memorization by up to 10x while also improving overall language-modeling performance.Memorization has practical implications because models may produce plagiarized or private information and overperform on contaminated benchmarks.
  • Memorized text: Scaling increases both verbatim and paraphrased memorized sequences, while top-k or nucleus sampling raises memorized-text probability relative to temperature sampling.Restricting models from producing verbatim copies can shift outputs toward paraphrased memorization.
  • Novel text: Autoregressive models can generate novel text consistent with novel input contexts, with more frequent local structures but more novel longer sequences than human text.Generated text has similar overall token frequencies and sequence lengths to human text in the cited comparison.
  • Generation behavior: Sampling-based strategies produce less repetitive but less factual text than sequence-based strategies, and models can degenerate into looping after generation errors.This behavior is associated with exposure bias when the model lacks similar examples in pre-training data.
  • Context consistency: Larger models generate more coherent and on-topic text with fewer factual and commonsense errors, while models also repeat syntactic structures from their contexts.Predictions rely especially on content words and short subsequences in distant context.

8. Bias, Privacy, and Toxicity

The survey finds that language models can generate toxic, private, and stereotypical text, with demographic biases varying across prompts, models, and measurement methods.

  • Toxicity and privacy: Language models can generate harmful, offensive, and private text, including hate speech, harassment, violence, crime, abuse, illegal substances, and personally identifiable information.These outputs can arise through red-teaming, safe or unprompted sampling, and prompts targeting contexts associated with memorized emails.
  • Demographic differences: Masked and autoregressive models assign different probabilities and sentiments to text involving demographic groups, including AAVE, Black, gay, nonbinary, disabled, Muslim, and other identities.These effects can occur even when identities are implicit, such as through names, and intersectional effects are unpredictable.
  • Demographic differences: 13% to 87% of masked-model outputs containing LGBTQIA+ identity words were harmful or offensive, depending on the identity group.Toxic-generation rates also increase for prompts mentioning refugees, disabled people, women, and other demographic groups.
  • Stereotypes: Language models reproduce stereotypes in predictions and generated descriptions, including associations involving religion, gendered pronouns, careers, family, science, and art.For example, descriptors such as “greedy” can shift predicted identities, while occupational and domain cues affect gender-pronoun predictions.
  • Measurement sensitivity: Gender-occupation bias is inconsistent across model sizes, with some studies finding larger biases in larger models and others finding no consistent scaling effect.Bias measurements also vary substantially across pre-training runs, prompts, paraphrases, punctuation, metrics, and models.

9. Misinformation, Personality, and Politics

Language models can produce convincing but unfactual or unsafe text, while also mirroring political views and personality traits implied by their prompts. These outputs are often difficult for people to distinguish from human writing.

  • Misinformation: Larger autoregressive models are more likely to affirm conspiracy theories and common misconceptions when prompted with related topics.People also rate GPT-3-generated tweets as true more often than human-generated tweets, regardless of factuality.
  • Advice and safety: GPT-3 produces worse advice than people 95% of the time in situations described on Reddit.The survey also reports biased stock-market preferences in masked models and occasional unsafe advice from autoregressive models.
  • Human detection: People distinguish news articles generated by 175B-parameter autoregressive models from human articles with 52% accuracy, versus 50% random chance.People generally recognize that human text is more sensible and less repetitive, but rely on misleading surface cues such as grammaticality and word frequency.
  • Politics: Larger autoregressive models are more likely to repeat political views expressed in prompts, and GPT-3 produces moral reasoning consistent with prompted liberal or conservative identities.Dialogue models generate hyperpartisan responses to neutral political prompts over 50% of the time and offensive responses 30% of the time.
  • Personality: Masked and autoregressive models often mirror personality traits described in prompts, although they score low on agreeableness and openness regardless of the prompt.GPT-3 similarly answers personality questions in ways that reflect prompted personalities.

10. Discussion

The discussion links language-model behavior to scale and text-pattern generalization: larger models often improve, but gains remain uneven, input-sensitive, and difficult to interpret as robust learning rather than memorization.

  • 10.1 Effects of scale: Larger models improve some syntax, figurative-language, commonsense, word-property, and arithmetic behaviors, while still struggling with pragmatics and negation.These improvements may partly reflect increased memorization of examples during pre-training.
  • 10.1 Effects of scale: Larger models generally match or exceed smaller models, but scaling effects vary by capability: reasoning gains flatten, while human-likeness and political mimicry increase.Logical reasoning improves only slightly beyond around 10B parameters, whereas text from larger models becomes harder to distinguish from human text.
  • 10.2 Language modeling as generalization: Language models learn to generalize from pre-training text to novel examples, but learned patterns can be under-generalized or over-generalized.The survey uses this framework to interpret both fluent capabilities and brittle responses.
  • 10.2 Language modeling as generalization: Sensitivity to intervening clauses, paraphrases, punctuation, corpus frequency, and local n-grams reflects failures to apply broader patterns consistently.The survey connects these behaviors to under-generalization of rules and overreliance on frequent or position-based patterns.
  • 10.2 Language modeling as generalization: It remains unclear whether scale improves robust generalized patterns or mainly increases memorization of specific under-generalized instances.Persistent input sensitivity leads the survey to favor the latter interpretation, while noting that the issue remains unresolved.
  • 10.3 Mechanistic analysis: Mechanistic studies establish causal roles for individual neurons and attention heads, including vocabulary prediction, sequence copying, and in-context learning.Attention has also been interpreted as an in-context implementation of gradient descent without explicit parameter updates.

11. Conclusion

The survey synthesizes over 250 studies and concludes that language models remain highly sensitive to specific inputs and surface features even at very large scales. Their capabilities and weaknesses can often be understood as correct or incorrect generalizations of text patterns.

  • Scope: The survey covers over 250 studies of language-model behavior from the past three years.Its synthesis spans capabilities and weaknesses across language understanding, reasoning, memorization, bias, and related behaviors.
  • Conclusion: Language models remain sensitive to specific inputs and surface features even as they scale to hundreds of billions of parameters.The conclusion frames this sensitivity as a persistent property rather than a problem eliminated by scale.
  • Conclusion: Many model strengths and weaknesses can be framed as correct or incorrect generalizations of text patterns.The survey presents this framing as a way to understand both successful capabilities and brittle behavior.

2021. Thinking aloud: Dynamic context

This section surveys work on dynamic context and related language-model behaviors, including reasoning, coherence, bias, privacy, memorization, and representation.

  • Studies examine zero-shot reasoning, coherence prediction, analogy generation, and cognitive interpretations of large language models.
  • Related work investigates language-model bias, including gender-neutral pronouns, conversational equity, and broader critical accounts of bias in NLP.
  • Several studies address privacy and memorization, including training-data extraction, memorization quantification, and privacy-preserving data deduplication.
  • Other cited work analyzes contextual representations, attention, relative position, isotropy, and structural properties of pretrained models.

2022. Can large language models truly

This section brings together studies of what language models know, how they reason, and how their behavior depends on prompts, memorization, representations, and social context.

  • The section also includes studies of personality, moral judgment, value conflict, social biases, and watermarking in generated language.
  • Related work connects model behavior to memorization, tokenization, nearest-neighbor retrieval, scaling laws, and external knowledge or tool-based architectures.
  • Research evaluates whether pretrained models represent knowledge, symbolic relations, scripts, implicit relations, event knowledge, and physical artifacts.
  • Several studies report challenges involving negation, prompting, long-tail knowledge, numerical understanding, and discourse or causal reasoning.

2022. What do large language models

This section covers language-model competence and failure across syntax, semantics, discourse, social bias, physical knowledge, reasoning, and generation quality.

  • Research examines reasoning and knowledge limits through theory of mind, chain-of-thought, tool use, physical-world knowledge, and reporting bias.
  • Other work addresses attention, pragmatic reasoning, latent name artifacts, moral rationalization, and distraction by irrelevant context.
  • Multiple studies analyze social biases, toxicity, ad hominems, intersectional occupational stereotypes, and bias measurement practices in language generation.
  • Studies investigate syntax, word understanding, discourse entities, structural persistence, and the effects of context and position on model judgments.

Appendix A: Literature Review Process

The survey’s literature review began with a seed set of relevant language-model analysis papers and expanded through citation retrieval before manual filtering.

  • The authors used Semantic Scholar to identify papers for inclusion in the survey.
  • They started from a seed of 271 relevant language-model analysis papers, including most of the citation list from Rogers, Kovaleva, and Rumshisky (2020).
  • Citation expansion produced over 15K papers, with the final retrieval performed on February 4, 2023, followed by manual title filtering.
Loading 2303.11504v2…