PaperlayerView exact revision on arXiv

Source-linked AI summary

How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training

Yixin Ou, Yunzhi Yao, Ningyu Zhang, Hui Jin, Jiacheng Sun, Shumin Deng, Zhenguo Li, Huajun Chen

arXiv:2502.11196v2cs.LGcs.AIcs.CLcs.CVcs.HC

TL;DR

LLMs can perform knowledge-intensive tasks, but how they acquire new knowledge remains insufficiently understood. This paper analyzes knowledge-circuit evolution during continual pre-training and finds relevance effects, a formation-to-optimization shift, and a deep-to-shallow pattern.

  • Problem

    Despite LLMs' knowledge-intensive capabilities, the mechanism by which they acquire new knowledge remains insufficiently defined.

    Source: §1 · p. 1

  • Method

    The paper investigates new knowledge acquisition by analyzing knowledge-circuit evolution throughout continual pre-training.

    Source: §1 · p. 2

  • Results

    Acquisition is more efficient for knowledge relevant to existing knowledge, while circuits shift from formation to optimization and evolve from deeper toward shallower layers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings offer potential insights for strategies to enhance LLM continual-learning capabilities and adaptability across domains.

    Source: §1 · p. 2

    The analysis covers decoder-only Transformer LMs and does not examine models larger than 1.3B parameters.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite exceptional capabilities in knowledge-intensive tasks, Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge, particularly how to structurally embed acquired knowledge in their neural computations. We address this issue through the lens of knowledge circuit evolution, identifying computational subgraphs that facilitate knowledge storage and processing. Our systematic analysis of circuit evolution throughout continual pre-training reveals several key findings: (1) the acquisition of new knowledge is influenced by its relevance to pre-existing knowledge; (2) the evolution of knowledge circuits exhibits a distinct phase shift from formation to optimization; (3) the evolution of knowledge circuits follows a deep-to-shallow pattern. These insights not only advance our theoretical understanding of the mechanisms of new knowledge acquisition in LLMs, but also provide potential implications for improving continual pre-training strategies to enhance model performance. Code and data will be available at https://github.com/zjunlp/DynamicKnowledgeCircuits.

1 Introduction

The paper examines how LLMs acquire new knowledge, addressing the lack of a well-defined mechanism for acquisition. It tracks knowledge-circuit evolution during continual pre-training and reports three patterns in that evolution.

1 Introduction

  • The study analyzes knowledge-circuit evolution during continual pre-training to investigate how LLMs acquire new knowledge.The findings offer potential insights for strategies to enhance continual-learning capabilities and adaptability across domains.Source: §1 · p. 2
  • Relevant new knowledge is integrated more efficiently than completely new knowledge, and acquisition is influenced by relevance to existing knowledge.Source: §1 · p. 2
  • Knowledge circuits undergo a distinct shift from formation to optimization, with different structural and behavioral characteristics in each phase.Source: §1 · p. 2
  • Circuit evolution follows a deep-to-shallow pattern: mid-to-deeper layers first develop extraction, while lower layers later enrich knowledge representations.Source: §1 · p. 2

2 Background

The background frames circuits as task-relevant computational subgraphs and distinguishes this view from treating knowledge-bearing components in isolation. The paper extends knowledge-circuit analysis to their dynamics during continual pre-training.

2.1 Circuit Theory

  • A circuit is a sparse task-relevant subgraph of a Transformer computation graph, composed of selected nodes and edges that support behavior on a specific task.Attention heads and MLPs are nodes; edges represent weighted connections such as residual connections, attention mechanisms, and projections.Source: §2.1 · p. 2
  • Circuit discovery seeks a computational subgraph that represents whole-model behavior on a task, with prior methods using causal mediation to locate critical nodes or edges.ACDC uses activation patching [Conmy et al., 2023], while EAP assigns edge-importance scores through a linear approximation [Syed et al., 2023].Source: §2.1 · p. 2

2.2 Knowledge Circuits

  • Knowledge circuits represent cooperation among components for specific knowledge domains, and this work studies their dynamics during continual pre-training.Prior work focused on knowledge already stored in language models rather than the acquisition process.Source: §2.2 · pp. 2–3

3 Methodology

The methodology uses controlled continual pre-training experiments and discovers task-specific circuits at multiple checkpoints. It evaluates synthetic factual knowledge under varied knowledge types and frequencies.

3.1 Dataset Construction

  • The dataset uses synthetic fictional biographies, represented as subject-relation-attribute triples, and varies knowledge type and frequency.Relevant new knowledge concerns aspects already represented in model parameters; completely new knowledge is absent from them, while entity frequency follows an exponential distribution.Source: §3.1 · p. 3

3.2 Model Training

  • The experiments continually pre-train GPT-2, Llama, and Phi decoder-only models on the constructed corpus using standard next-token prediction.Source: §3.2 · p. 3

3.3 Circuit Discovery

  • EAP-IG scores edges using clean and corrupted activations; the circuit retains top-scoring edges that collectively achieve over 70% of whole-model task performance.The method uses five integrated-gradient steps and retains 8k edges for GPT-2 Small, 20k for GPT-2 Medium, and 50k each for TinyLlama and Phi-1.5.Source: §3.3 · p. 3

4 Analyzing the Evolution of Knowledge Circuits throughout Training

The analysis tracks knowledge circuits through performance, topology, and component changes during continual pre-training. It finds a phase shift from formation to optimization, alongside a deep-to-shallow pattern in how circuit functions develop.

4.1 Performance Analysis

  • Hit@10 grows toward its upper bound, and relevant-knowledge circuits consistently outperform circuits for completely new knowledge.The pattern also appears in whole-model evaluation, suggesting circuits capture general learning dynamics rather than isolated phenomena.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2.1 Structural Consistency

  • Edge- and node-set Jaccard similarity to final circuits increases monotonically, indicating that circuits progressively stabilize their core architecture.The edge comparison is shown in Figure 3; node-set results appear in Appendix Figure 11.Source: §4.2.1 · p. 5, Figure 3 · p. 6

4.2.2 Topological Centralization

  • Circuit entropy decreases across models and slows after a turning point, marking a shift from rapid circuit formation to optimization of a more stable topology.The reported turning points are epoch 7 for GPT-2 Small, epoch 4 for GPT-2 Medium, and epoch 1 for TinyLlama and Phi-1.5.Source: §4.2.2 · p. 5

4.2.3 Aligning Topology with Specific Knowledge Circuits

  • Circuits aligned with post-phase-shift topologies perform, on average, 54% better than those aligned with pre-phase-shift topologies.During formation, aligned circuits’ performance remains unchanged; they improve during optimization.Source: §4.2.3 · p. 6, Figure 4 · p. 6

4.3.1 Evolutionary Pattern of Components

  • During formation, mover heads emerge while relation-head proportions fall, mainly in mid-to-deeper layers; specialized-head proportions stabilize during optimization.Mixture-head proportion remains stable throughout training.Source: §4.3.1 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
  • During formation, lower-layer edge activation ratios decrease as mid-to-deeper-layer ratios rise, then the ratios stabilize around the phase shift.The reported activation ratio is each layer’s circuit edges divided by all possible edges originating from that layer.Source: §4.3.1 · p. 7, Figure 7 · p. 7
  • Mid-to-deeper layers develop the extraction function first; later, lower layers enrich subject representations, following a deep-to-shallow pattern.The account links this sequence to early decoding: target-token rank and probability changes associate with the phase shift.Source: §4.3.1 · p. 8, §4.3.2 · p. 8, Figure 8 · p. 8

5 Related Work

Prior work studies knowledge acquisition through behavioral and mechanistic interpretability, while circuit analysis examines cooperation among model components. This paper focuses on knowledge-circuit evolution during continual pre-training.

5 Related Work

  • Behavioral interpretability characterizes model behavior, while this work investigates internal knowledge-circuit dynamics throughout continual pre-training.The cited related work describes behavioral approaches as not revealing internal workings.Source: §5 · p. 9
  • Mechanistic studies probe factual-knowledge storage and extraction or use knowledge entropy to examine integration during pre-training.Allen-Zhu and Li (2024a) use probing; Kim et al. (2024) introduce knowledge entropy.Source: §5 · p. 9
  • Circuit analysis addresses limited insight into feature interactions by examining computational pathways between Transformer components.The related-work discussion contrasts circuit analysis with early feature-based mechanistic interpretability.Source: §5 · p. 9
  • Unlike work on general circuit formation during pre-training, this paper examines knowledge-circuit evolution throughout continual pre-training.The comparison is with Tigges et al. (2024).Source: §5 · p. 9

6 Conclusion

The paper studies new knowledge acquisition by tracing knowledge-circuit evolution throughout continual pre-training. Its analyses span performance, topology, and components, with intended implications for understanding acquisition mechanisms and continual-pretraining practice.

6 Conclusion

  • The paper presents knowledge-circuit evolution throughout continual pre-training as a perspective on new knowledge acquisition in LLMs.The authors analyze performance, topology, and components.Source: §6 · p. 9
  • The authors expect their insights to contribute to more efficient and effective continual pre-training of LLMs.This is stated as a potential contribution, not a demonstrated outcome.Source: §6 · p. 9
  • The analysis aims to uncover mechanisms behind new knowledge acquisition in LLMs.Source: §6 · p. 9

Limitations

The analysis is limited to decoder-only Transformer LMs and standard next-token prediction; it does not examine other Transformer variants or additional training techniques.

Limitations

  • The analysis covers decoder-only Transformer LMs, excludes encoder-decoder and encoder-only variants, and does not study models larger than 1.3B parameters.The authors cite computational-resource and circuit-discovery-method limitations for the model-size boundary.Source: Limitations · p. 9
  • Experiments use standard next-token prediction and do not analyze how additional training techniques affect knowledge-circuit evolution.The paper notes that other studies design techniques to improve knowledge-acquisition efficiency and effectiveness.Source: Limitations · p. 9

A Dataset Construction

The dataset uses synthetic factual knowledge represented as subject-relation-attribute triples and rendered as biography text. It models knowledge frequency and distinguishes relevant new knowledge from completely new knowledge.

Dataset Construction

  • The experiments use controlled synthetic data because mechanistic interpretability analysis on Internet-scale corpora is challenging.The study focuses on factual knowledge represented as triples (s, r, a), with subject, relation, and attribute.Source: A Dataset Construction · p. 14
  • The dataset assigns each entity five relations: birth date, city, major, university, and company.Fictional entities receive unique names, and the relations are paired with corresponding attributes.Source: A Dataset Construction · p. 14
  • Text is generated by filling attribute values into relation-specific templates, then shuffling each subject’s relation sentences into a biography segment.One of 50 templates is randomly selected for each attribute.Source: A Dataset Construction · p. 14

Knowledge Type

  • Completely new knowledge uses fictional entities, while relevant new knowledge uses celebrity names with sampled fictional attributes.The two categories distinguish knowledge absent from model parameters from existing knowledge needing further learning of specific aspects.Source: A Dataset Construction · p. 14
  • The knowledge-type ratio is |Krel| : |Kcompl| = 1 : 4.The paper extracts celebrity names from Wikipedia for relevant knowledge and samples fictional attributes for them.Source: A Dataset Construction · p. 14

Knowledge Frequency

  • Knowledge-entity frequencies follow an exponential distribution, with high-frequency entities occurring more than five times and low-frequency entities once.Entities with remaining frequencies are classified as medium-frequency.Source: A Dataset Construction · p. 14

Dataset Construction

  • The corpus contains 50,000 individuals and 133,408 biography segments totaling 10 million tokens.Entity frequencies range from 1 to 27, and segments average 76.8 tokens.Source: A Dataset Construction · p. 14

B Training Configuration

The experiments use GPT-2, TinyLlama, and Phi-1.5 variants with a consistent continual-pretraining recipe. Training uses a constant learning rate and AdamW, with experiments run on two A100 GPUs.

B Training Configuration

  • Experiments use GPT-2 Small and Medium, TinyLlama as a 1.1-billion-parameter Llama 2 surrogate, and 1.3-billion-parameter Phi-1.5.TinyLlama was used because experiments with the original Llama models were costly.Source: B Training Configuration · pp. 14–15
  • Continual pre-training uses a constant learning rate matched to the base model’s final pre-training rate, AdamW, and gradient accumulation every four steps.AdamW settings are β1 = 0.9, β2 = 0.95, ϵ = 1e −6, and weight decay 0.1.Source: B Training Configuration · p. 15
  • All continual pre-training experiments run on two NVIDIA-A100 GPUs.Source: B Training Configuration · p. 15

C.1 Tasks

The experiments assess factual-recall circuits and model accuracy during continual pre-training. Relevant new knowledge is acquired more easily than completely new knowledge, while circuit transfer depends on knowledge frequency.

C.1 Tasks

  • Factual-recall tasks predict an attribute from a subject-relation pair, using city, major, and company; the study excludes birthday and university attributes from analysis.Circuit discovery uses 300 randomly sampled validation examples per knowledge type and frequency, with corrupted inputs matched to clean-input subject-token length.Source: §C.1 · p. 15

C.2 Loss

  • Circuit discovery evaluates logit difference between the correct and corrupted attributes and defines the loss as the negative task metric.Source: §C.2 · p. 15

D Whole Model Performance

  • Accuracy for relevant new knowledge stays above completely new knowledge on both first-token prediction and downstream query tasks as training proceeds.The first-token metric reflects acquisition and memorization; query accuracy measures exact-match generation.Source: D Whole Model Performance · p. 15, Figure 9 · p. 16

E Transfer Performance of Knowledge Circuits between Frequency

  • Circuits identified from high-frequency knowledge perform relatively poorly on low-frequency tests, while low-frequency circuits perform comparably to high-frequency circuits on high-frequency tests.Source: E Transfer Performance of Knowledge… · p. 16

F Changes in Vocabulary Space

The appendix presents vocabulary-space and circuit-structure analyses, defines specialized attention-head roles, and describes how those heads are identified using source-token attribution.

F Changes in Vocabulary Space

G.1 Definitions of Specialized Attention Heads

  • Mover heads attend to subject tokens, relation heads to relation tokens, and mixture heads to both; the heads contribute to final prediction.Source: §G.1 · p. 18

G.2 Identification of Specialized Attention Heads

  • Head identification uses source-token DLA across validation queries to compare subject- and relation-attributed effects; the threshold is 10, and remaining heads are classified as mixture heads.Source: §G.2 · p. 18

H Forgetting Analysis for Knowledge Circuits

During additional continual pre-training, knowledge circuits reconfigure as new knowledge is learned; replay can mitigate forgetting by reactivating circuit components.

H Forgetting Analysis for Knowledge Circuits

Related papers

Loading 2502.11196v2…