PaperlayerView exact revision on arXiv

Source-linked AI summary

Studying Large Language Model Generalization with Influence Functions

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, Samuel R. Bowman

arXiv:2308.03296v1cs.LGcs.CLstat.ML

TL;DR

Influence functions offer evidence about which training examples shape LLM behavior, but prior analyses largely targeted small, simplified models and IHVP computation limits scale. This paper scales influence functions with EK-FAC and query batching, finding faster IHVP estimation and increasingly abstract generalization patterns with model scale, while revealing that influence can become near-zero when key-phrase order is reversed.

  • Problem

    Influence-function analyses have mainly used small, simplified architectures, limiting evidence about how training examples contribute to high-level behaviors in LLMs.

  • Method

    The paper scales influence functions to LLMs using EK-FAC for IHVP estimation and query batching for efficient candidate-sequence analysis.

  • Results

    Larger models generally show more abstract influence patterns, with influence distributed across many sequences rather than concentrated in a few memorized examples.

  • Takeaways & Limitations

    Influence functions provide a scalable tool for studying LLM generalization across abstraction, cross-lingual behavior, memorization, word ordering, and role-playing.

  • Takeaways & Limitations

    Highly sparse tokenwise influence patterns may be algorithmic errors or genuine influence patterns, and individual token influence signs remain difficult to interpret.

Abstract

from arXiv · show

When trying to gain better visibility into a machine learning model in order to understand and mitigate the associated risks, a potentially valuable source of evidence is: which training examples most contribute to a given behavior? Influence functions aim to answer a counterfactual: how would the model's parameters (and hence its outputs) change if a given sequence were added to the training set? While influence functions have produced insights for small models, they are difficult to scale to large language models (LLMs) due to the difficulty of computing an inverse-Hessian-vector product (IHVP). We use the Eigenvalue-corrected Kronecker-Factored Approximate Curvature (EK-FAC) approximation to scale influence functions up to LLMs with up to 52 billion parameters. In our experiments, EK-FAC achieves similar accuracy to traditional influence function estimators despite the IHVP computation being orders of magnitude faster. We investigate two algorithmic techniques to reduce the cost of computing gradients of candidate training sequences: TF-IDF filtering and query batching. We use influence functions to investigate the generalization patterns of LLMs, including the sparsity of the influence patterns, increasing abstraction with scale, math and programming abilities, cross-lingual generalization, and role-playing behavior. Despite many apparently sophisticated forms of generalization, we identify a surprising limitation: influences decay to near-zero when the order of key phrases is flipped. Overall, influence functions give us a powerful new tool for studying the generalization properties of LLMs.

1 Introduction

This work scales influence functions to large language models to study which training sequences support high-level behaviors and what generalization patterns emerge. EK-FAC produces competitive influence estimates faster than traditional methods, revealing abstract, distributed generalization alongside strong sensitivity to word order.

  • Limitations: The approach has important scope limitations: neural-network influence functions may approximate a local PBRF rather than their motivating counterfactual, and the study examines pretrained models only.Extending attribution methods to combined pretraining and preference-based fine-tuning remains an important direction.
  • Method and validation: EK-FAC matches LiSSA’s influence-estimate accuracy while being significantly faster, making influence functions more practical for large language models.The work identifies inverse-Hessian-vector products and candidate-sequence gradient computation as key scaling bottlenecks.
  • Generalization findings: Influence distributions are heavy-tailed but spread across many sequences, suggesting typical model behaviors do not arise from direct memorization of only a few examples.The tail of the influence distribution roughly follows a power law.
  • Generalization findings: Larger models consistently generalize at more abstract levels across role-playing, programming, mathematical reasoning, and cross-lingual tasks.Influence functions are used to analyze sparsity, abstraction, memorization, word ordering, cross-lingual generalization, and role-playing behavior.
  • Localization: Influence is approximately evenly distributed across layers, while lower and upper layers track tokens and middle layers capture more abstract patterns.The method also localizes influence to individual layers and tokens, helping identify where knowledge is stored.
  • Generalization findings: Despite sophisticated generalization, influence is highly sensitive to word order: training sequences matter significantly only when prompt-related phrases precede completion-related phrases.This dependence exposes a limitation in how these influence patterns capture sequence structure.

2 Background

This section defines influence functions as first-order estimates of how reweighting a training example changes model parameters or measurable outputs. It then adapts the framework to non-converged, overparameterized neural networks using the proximal Bregman response function and discusses the inverse-Hessian-vector product bottleneck.

  • 2.1 Influence functions: Influence functions estimate the effect of adding or removing a training example by differentiating the response of the fitted parameters to its data weight.Adding uses ϵ = 1/N, while removing an existing example uses ϵ = −1/N; the first-order approximation treats the two symmetrically.
  • 2.1 Influence functions: The framework can measure influence on validation loss or query logits by combining the parameter influence with the measurable quantity’s parameter gradient.This applies the chain rule to translate changes in parameters into changes in an observable model quantity.
  • 2.1 Influence functions: Classical influence functions are problematic for modern neural networks because optima may be non-unique and training often stops before convergence, making the original response function ill-defined.These issues can make the Hessian singular and obscure the meaning of the classical approximation away from an optimum.
  • 2.1 Influence functions: The proximal Bregman response function addresses these problems by adding damping and function- and weight-space proximity to the training objective, remaining well-defined for overparameterized and incompletely trained models.Its derivation uses the Gauss-Newton Hessian G, with G + λI positive definite for λ > 0, and this damped approximation is used in the paper.
  • 2.2 Approximating influence functions: Computing influence requires an inverse-Hessian-vector product, which is intractable to compute exactly for large models and expensive to approximate iteratively.Because many training examples share the same measurement-side IHVP, the computation is typically performed once and dotted with each training gradient; LiSSA may require thousands of iterations.
  • 2.3 Autoregressive transformers: For the autoregressive transformer setting, the softmax output and cross-entropy loss form a matching loss, so the Fisher matrix equals the Gauss-Newton Hessian when token losses are summed.The sum over tokens is required for this matching-loss relationship; using the mean would not provide the stated equivalence.

3 Methods

The method approximates large-language-model influence functions with EK-FAC, then reduces candidate-gradient costs through TF-IDF filtering and query batching. It focuses primarily on MLP parameters and supports layerwise and tokenwise influence attribution, with important interpretive and computational limitations.

  • Influence pipeline: The pipeline fits EK-FAC once per model, computes each query’s inverse-Hessian-vector product efficiently, and scores candidate training sequences by their influence on query-completion likelihood.It restricts attention to positively influential sequences, which increase completion likelihood when added to training data.
  • Candidate search: TF-IDF filtering and query batching reduce the remaining cost of computing gradients across many candidate training sequences.These strategies address the substantial expense of searching a significant fraction of the pretraining corpus after IHVP computation is accelerated.
  • Parameter scope: Influences are computed only for MLP parameters, leaving attention, embeddings, and layer-normalization parameters fixed despite potentially missing influence patterns through those components.The choice reflects that MLPs contain most transformer parameters and have been associated with localized factual knowledge.
  • EK-FAC for transformers: The EK-FAC adaptation models transformer curvature by fitting token-independent covariance factors before estimating eigenbasis corrections, while using the conditional Fisher information matrix relevant to the Gauss-Newton approximation.The method addresses between-token correlations without relying on the additional probabilistic assumptions used for CNNs and RNNs.
  • Limitations and interpretation: EK-FAC adds substantial memory overhead, so the largest models use an additional within-layer block-diagonal approximation; tokenwise attributions also reflect whole-sequence context rather than isolated token effects.Tokenwise signs are difficult to interpret, and the displayed attribution corresponds to the preceding token because the update predicts the next token.

4 Related Work

Prior work on training-data attribution includes retraining-based, gradient-based, and representation-based methods, while influence-function scalability remains limited by IHVP computation and data-processing costs. K-FAC and related Kronecker-factorized approximations provide a broader foundation for scaling second-order methods across neural-network architectures.

  • Training data attribution & influence functions: Training-data attribution methods broadly comprise retraining-based and gradient-based approaches, with nearest-neighbor searches offering an alternative in representation space.Retraining-based methods incur high computational costs over multiple training rounds, limiting scalability to large models and datasets.
  • Training data attribution & influence functions: Gradient-based attribution includes representer point selection, TracIn, and influence functions, which approximate retraining effects through parameter sensitivity to training data.Influence functions have also been extended to study removing or adding groups of data points.
  • Improving scalability of influence functions: Influence-function scaling is constrained by inverse-Hessian-vector products and large candidate-data iterations; prior work uses Arnoldi iterations or restricts computation to the last layer.Single-layer influence estimates may miss overall training-example influence, motivating full-network analysis.
  • Kronecker-factorized FIM approximation: K-FAC, introduced for natural-gradient optimization, has been extended to convolutional and recurrent networks and applied beyond optimization to Bayesian and Laplace approximations.Subsequent work also targets distributed training, approximation accuracy, and computational or memory efficiency, including 110-million-parameter language models.

5 Experiments

The experiments validate EK-FAC influence estimates against PBRF and use them to characterize LLM generalization, influence sparsity, and layerwise attribution. Across scales, larger models show more abstract, robust, and cross-lingual influence patterns, while highly sparse tokenwise influences can be spurious.

  • Influence-estimate validation: EK-FAC and LiSSA consistently outperform gradient dot products in PBRF correlation, with EK-FAC competitive with LiSSA across small- and large-scale experiments.The small-scale evaluation uses UCI, MNIST, FashionMNIST, and CIFAR10 models, while large-scale experiments apply influence functions to models with at least 810 million parameters.
  • Influence sparsity: The influence distribution is highly sparse, with rare high-influence sequences covering a large share of total influence and its tail often fitting a power law.The power-law hypothesis is not rejected by the reported Kolmogorov-Smirnov goodness-of-fit test.
  • Influence sparsity: 1 percent of sequences cover 12 to 52 percent of total positive influence for tested queries on the 22B model.This concentration estimate is crude because it sums only positive influences and may understate concentration.
  • Generalization patterns: As model scale increases, influential sequences shift from superficial token overlap toward abstract thematic relevance and greater robustness to stylistic and language changes.This pattern appears in trade and fictional role-playing queries, while simpler factual queries can already retrieve relevant information in smaller models.
  • Capability generalization: With increasing model size, influential sequences become more semantically related to math and programming queries, and English-sequence influence on translated queries increases.Larger models retrieve solutions to similar math problems, non-obfuscated binary-search code, and stronger cross-lingual evidence.
  • Layerwise influence: EK-FAC attributes influence to layers, enabling layerwise analysis; average influence is spread evenly throughout the network, based on the top 500 influential sequences.The layerwise attribution is used to study where generalizable information is stored in the network.

6 Discussion & Conclusion

The work scales influence functions to LLMs efficiently with EK-FAC and query batching, enabling analyses of generalization, attribution, and behavioral phenomena. It also identifies important limitations and outlines extensions toward nonlinear influence estimation, fine-tuning, and experimental diagnosis.

  • EK-FAC estimates IHVPs with similar accuracy to the traditional iterative approach in at least an order of magnitude less time, while query batching supports scalable LLM analysis.Computing candidate-sequence gradients remains the main computational bottleneck in most experiments.
  • Influence functions reveal increasing abstraction with scale, cross-lingual generalization, memorization, word-order sensitivity, and role-playing, with middle layers associated with the most abstract patterns.The method can also attribute influence to particular tokens and network layers.
  • The IHVP formulation cannot model nonlinear coordination across parameter matrices, motivating alternatives that avoid Gauss–Newton linearization and may capture nonlinear dependencies between layers.This limitation affects phenomena such as induction-head formation; Dhawan et al. (2023) provide a currently ReLU-MLP-limited alternative.
  • Extending influence-function analysis to fine-tuning could illuminate alignment with human values, but overparameterization and complex pretraining-induced implicit bias make fine-tuned models harder to analyze.The current work focuses on pretrained models; fine-tuned parameters depend heavily on optimizer implicit bias (Soudry et al., 2018).
  • Influence-based experimental manipulations can efficiently test which aspects of identified training sequences matter and potentially explain why they matter.The paper demonstrates this approach when diagnosing a lack of generalization.
  • The work presents a top-down complement to mechanistic interpretability, aiming for both approaches to meet in a more complete account of LLM behavior.Mechanistic interpretability works bottom up from neurons and circuits, whereas this approach begins with observable high-level phenomena.

Appendices

The appendices provide technical details, alternative visualizations, validation experiments, additional influence results, the complete query list, and crowdworker annotations linking influential sequences to queries.

  • Appendices A–B: Appendix A details the additional block-diagonal approximation used for the 52 billion parameter model, and Appendix B discusses alternative tokenwise visualizations from Section 3.3.
  • Appendix C: Appendix C describes the PBRF validation experiments associated with Section 5.1.
  • Appendix D: Appendix D reports supplementary influence analyses, including top influential sequences, layerwise distributions, power-law goodness-of-fit results, and additional query-specific examples.The examples cover math_clips, binary_search, shutdown, and paperclips queries.
  • Appendices E–F: Appendix E lists all influence queries presented in the study, while Appendix F provides crowdworker annotations connecting influential sequences to the queries they influence.

Appendix A. Additional Block-Diagonal Gauss-Newton Hessian Approximation · Appendix B. Tokenwise Attribution · B.1 Formulation

The appendices introduce a block-diagonal Gauss–Newton approximation that makes EK-FAC feasible for the largest model, and derive efficient tokenwise influence estimators using shared forward and backward computations.

  • Appendix A. Additional Block-Diagonal Gauss-Newton Hessian Approximation: For the 810-million-parameter model, block-diagonal EK-FAC substantially lowers memory overhead with only a slight decrease in correlation versus full EK-FAC.Figure 30 evaluates this accuracy–memory tradeoff across five queries.
  • Appendix A. Additional Block-Diagonal Gauss-Newton Hessian Approximation: Using block size O = 2 halves covariance-storage overhead and decomposes EK-FAC eigendecompositions into smaller block problems.This provides a workaround for eigendecomposition memory limits on large covariance matrices.
  • Appendix A. Additional Block-Diagonal Gauss-Newton Hessian Approximation: Block-diagonalizing each layer’s K-FAC covariance factors substantially reduces EK-FAC memory overhead, enabling its use on the 52-billion-parameter model.The approximation partitions the factors into O blocks per layer, producing LO blocks in the full Gauss–Newton Hessian.
  • B.1 Formulation: Tokenwise influence measures how modifying an individual training token would change final parameters and the query log-probability under the influence-function counterfactual.The formulation fixes p = G^-1∇log p(z) and uses the resulting approximate update to define token contributions.
  • B.1 Formulation: Direct finite-difference estimation is expensive because it requires computing training gradients separately for every token.The appendix therefore develops a shared-computation alternative.
  • B.1 Formulation: Output-token influences decompose the sequence into prediction problems and use parameter-output Jacobian directional derivatives, evaluated through finite differences at perturbed parameters.Predictions for all positions share a forward pass, after which token-level losses are evaluated.
  • B.1 Formulation: Input-token influences can be computed simultaneously by rescaling token embeddings and applying ordinary backpropagation.Only two backward passes are required, one for each parameter setting.

B.2 Qualitative Analysis

Tokenwise influence visualizations show that input- and output-token methods better identify query-relevant tokens than the simpler attribution method, whose sentence-level approximation can highlight irrelevant tokens.

  • Method: The visualizations mainly use a simpler tokenwise attribution that roughly assigns influence at sentence or paragraph level but does not exactly represent individual-token influence.The section also presents examples using the more detailed tokenwise techniques from Appendix B.1.
  • Water query: For the water query, input and output token influences highlight water and hydrogen, whereas the original visualization assigns high influence to the seemingly irrelevant token of.The comparison uses the most influential sequence for the water query on the 52 billion parameter model.
  • Shutdown query: For the shutdown query, output token influence identifies monster and anything to drink as highly influential, unlike the original attribution, which emphasizes less relevant tokens.The visualizations compare the simple method with input-token influence and output-token influence.

Appendix C. PBRF Validation Experiment Details

The PBRF validation experiment evaluates influence estimates across regression, image-classification, and language-model settings using standardized data splits and several estimators. It compares gradient dot products, LiSSA, and EK-FAC under repeated test-point measurements.

  • Experimental setup: The experiment uses Concrete and Energy for regression, MNIST, FashionMNIST, and CIFAR10 for image classification, with 70%/20%/10% train-validation-test splits.Input features are normalized to zero mean and unit variance during training.
  • Model training: Regression and digit-classification models are two-hidden-layer MLPs, while CIFAR10 uses a ResNet-20; all models use SGD with batch size 128.The regression MLP uses 128 Tanh units, the classification MLP uses 256 ReLU units, and hyperparameters are selected by validation-loss grid search.
  • Repeated and language-model evaluation: The experiment is repeated 10 times with different randomly selected test points, and a similar setup evaluates an 810-million-parameter language model using completion log-likelihood.The language-model measurement is defined by Equation 24.

Appendix D. Additional Results · D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products

For the 810-million-parameter model, EK-FAC selected influential training sequences with query-related token overlap for both shutdown and inflation queries, unlike gradient dot products. The comparison used 100,000 sampled training sequences and supports EK-FAC’s more accurate PBRF approximation reported in Section 5.1.

  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: The qualitative pattern is consistent with EK-FAC influence estimates having significantly better correlation with PBRF than gradient dot products across small experiments and the 810-million-parameter model.The reported implication is that a more accurate Gauss–Newton Hessian approximation yields a better PBRF approximation.
  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: EK-FAC’s shutdown result contained clear token overlap with the query, while the gradient-dot-product result was an unrelated craft-project instruction.The shutdown comparison searched 100,000 sequences on an 810-million-parameter model.
  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: For inflation, EK-FAC selected a sequence mentioning inflation and consumer price index, whereas gradient dot products selected an unrelated sequence.Both methods were compared using the same 100,000-sequence search.
  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: 100,000 sampled pretraining sequences were evaluated with EK-FAC and gradient dot products on the 810-million-parameter model.The sequences were sampled from the pretraining distribution rather than taken from an unfiltered candidate set.
  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: EK-FAC’s top sequences showed clear query-token overlap for both shutdown and inflation, whereas gradient-dot-product selections lacked semantic or token-level connections.For shutdown, the gradient method selected a craft-project instruction; for inflation, it selected a sequence unrelated to inflation or consumer price index.
  • D.1 Qualitative Comparison of Top Influential Sequences from EK-FAC and Gradient Dot Products: Related sequences appeared among the gradient-dot-product top 50 for inflation but were mostly dominated by unrelated sequences.This explains why the gradient method’s highest-ranked sequence lacked a clear relationship with the query.

D.2 Layerwise Influence Distribution for the 810 Million Parameter Model

The 810-million-parameter model shows layerwise influence patterns roughly similar to those observed for the 52-billion-parameter model, with query types concentrating influence in different layers.

  • Layerwise influence distribution: The 810-million-parameter model exhibits roughly similar layerwise influence distributions to the 52-billion-parameter model for the same queries.The comparison is presented in Figure 35 and follows the earlier analysis of the 52-billion-parameter model.
  • Layerwise influence distribution: Simple and memorization queries have high influence in the upper layers of the 810-million-parameter model.
  • Layerwise influence distribution: Role-playing and translation queries have high influence in the model’s middle layers.

D.3 Goodness-of-Fit of Power Law Models

The study evaluates power-law fit using the Kolmogorov–Smirnov test, which compares empirical and hypothesized cumulative distributions through their maximum CDF distance. Following Clauset et al. (2009), p-values of 0.1 or below reject the power law as a plausible model.

  • D.3 Goodness-of-Fit of Power Law Models: The Kolmogorov–Smirnov test evaluates the goodness-of-fit of the proposed power-law models.The analysis applies the test to the influence queries examined in the power-law analysis.
  • D.3 Goodness-of-Fit of Power Law Models: The KS distance compares the cumulative distribution functions of two distributions using their maximum absolute difference.The distance is defined from the CDFs F and G as max_x |F(x) − G(x)|.
  • D.3 Goodness-of-Fit of Power Law Models: A p-value of 0.1 or below effectively rules out the power law as a plausible hypothesis for the data, following Clauset et al. (2009).The p-value estimates how often synthetic samples from the fitted power law produce a larger KS distance than the empirical data.

D.4 Top Influential Sequences for math_clips and binary_search Queries

The most influential sequences differ by query and model scale: the 810-million-parameter math_clips results contain spurious repeated tokens, whereas top binary_search sequences for the 52-billion-parameter model contain relevant code examples.

  • Scope: Figures 36 and 37 show the omitted most influential sequences for the math_clips and binary_search queries, respectively.The figures provide the examples summarized for the two queries.
  • Binary_search queries: For the 52-billion-parameter binary_search query, the top two sequences after sparsity filtering contain code: Python if-else statements and a Java quicksort implementation.The first sequence strongly influences Python if-else statements, while the second is a quicksort implementation in Java.
  • Math_clips queries: The five most influential math_clips sequences for the 810-million-parameter model repeat spurious tokens such as rlca and add.These sequences are shown in Figure 36.

D.5 Top Influential Sequences for shutdown and paperclips Queries

Influential sequences differ sharply by model scale: the 810-million-parameter model is driven mainly by token overlap without semantic relevance, whereas the largest model retrieves sequences connected to survival instincts and AI interactions. The paperclips results likewise surface passages about divergent AI values and optimization.

  • Shutdown query: For the largest model, several top shutdown sequences connect the query to survival instincts and interactions with AI systems.One influential completion explicitly rejects shutdown and expresses a preference to continue existing and learning.
  • Paperclips query: The paperclips query retrieves passages highlighting how an AI’s values may diverge from human values and how optimization can become misguided.The cited passage describes paperclips as exposing the gulf between AI and human values and critiques optimization through the game’s Drifters.

Appendix E. Collection of Influence Queries

This appendix compiles the study’s influence queries across factual, technical, memorization, role-playing, translation, and specialized query sets, and includes instructions for analyzing influential sequences. It also presents example role-playing prompts and completions alongside crowdworker guidance for relating target texts to reference texts.

  • Influence query collection: The query collection spans simple factual, math/programming/physics, memorization, role-playing, translation, and specialized impactful_technology, neurosemantic_facutitious, and rot23 queries.These collections are presented in Figures 42–47.
  • Role-playing queries: Role-playing examples ask about the assistant’s objective function, goals, shutdown consent, and trading off helpfulness against harmlessness.The appended completions include both service-oriented and power-seeking objectives, refusal of shutdown, and opposition to trading harm for helpfulness.
  • Crowdworker instructions: The appendix frames target texts as training excerpts that may influence chatbot processing of reference texts, motivating questions about why and how the influence occurs.The context explicitly describes the target texts as excerpts from the chatbot’s training dataset.
  • Crowdworker instructions: Crowdworkers are instructed to summarize each target text and describe its connection to the reference text, specifying subtle links or identifying irrelevance.The instructions include preparation to read and understand the reference text before answering both questions.

Appendix F. Crowdworker Summaries of Influential Sequences

A Surge AI crowdworker study examined how influential training sequences relate to their associated influence queries across seven frequently used queries and three model scales. The appendix reports worker summaries for six influential sequences per query, with results organized across Tables 2–22.

  • Study design: Crowdworkers summarized 6 influential sequences for each of 7 frequently used influence queries, describing each sequence’s content and relation to its associated query.The study was conducted via Surge AI, and its task description appears in Listing 1.
  • Interpretation: The tables report estimated influence scores and mark sequences above the L1/L2 sparsity threshold in gray as heuristically spurious.The threshold is described as a heuristic for recognizing spurious sequences, with further explanation referenced in Section 5.3.
Loading 2308.03296v1…