Source-linked AI summary
How Do Large Language Models Learn Concepts During Continual Pre-Training?
Barry Menglong Yao, Sha Li, Yunzhi Yao, Minqian Liu, Zaishuo Xia, Qifan Wang, Lifu Huang
TL;DR
LLM concept acquisition, retention, and forgetting during continual pretraining remain poorly understood, especially under interactions among multiple concepts and knowledge types. The paper addresses this gap by linking behavioral dynamics to internal concept circuits and graph-based topology measures, finding consistent circuit-level signals alongside systematic interference, transfer, and forgetting patterns.
Problem
How LLMs acquire, retain, and forget concepts, and how concepts interact through interference and synergy during continual pretraining, remains poorly understood.
Method
The paper constructs FICO, conducts controlled continual-pretraining experiments, extracts concept circuits, and analyzes their topology alongside learning, forgetting, interference, and transfer.
Results
Concept circuits consistently signal learning and forgetting; topology changes stage-wise, stronger learning predicts greater forgetting, similar concepts interfere more, and knowledge types differ in transferability.
Takeaways & Limitations
The findings motivate concept-aware continual-pretraining strategies, including circuit-aware replay to prioritize concepts vulnerable to forgetting and improve knowledge retention.
Takeaways & Limitations
Experiments focus mainly on GPT-2 Large, with LLaMA models used for ablations, and Circuit-aware Experience Replay is evaluated only as a proof of concept.
Abstract
from arXiv · showhide
Human beings primarily understand the world through concepts (e.g., dog), abstract mental representations that structure perception, reasoning, and learning. However, how large language models (LLMs) acquire, retain, and forget such concepts during continual pretraining remains poorly understood. In this work, we study how individual concepts are acquired and forgotten, as well as how multiple concepts interact through interference and synergy. We link these behavioral dynamics to LLMs' internal concept circuits, computational subgraphs associated with specific concepts, and incorporate graph metrics to characterize circuit topology. Our analysis reveals: (1) LLMs concept circuits provide a non-trivial, consistent signal of concept learning and forgetting; (2) concept circuits exhibit a stage-wise temporal pattern during continual pretraining, with an early increase followed by gradual decrease and stabilization; (3) concepts with larger learning gains tend to exhibit greater forgetting under subsequent training; (4) semantically similar concepts induce stronger interference than weakly related ones; (5) conceptual knowledge differs in their transferability, with some significantly facilitating the learning of others. Together, our findings provide a circuit-level view of concept learning dynamics and motivate concept-aware training strategies, such as Circuit-aware Experience Replay, which uses circuit topology to prioritize concepts vulnerable to forgetting.
1 Introduction
This study examines how LLMs acquire and forget concepts during continual pretraining, and how concepts interact through interference and synergy. It links these dynamics to internal concept circuits to support concept-aware training decisions.
- The paper asks how internal concept representations relate to concept acquisition, forgetting, interference, and synergy during continual training.
- The study links behavioral changes to Concept Circuits, computational subgraphs associated with individual concepts, and tracks their topology with graph metrics.
- Circuit topology correlates with concept learning and forgetting, while stronger acquisition is associated with greater subsequent forgetting.
- Concept circuits follow a stage-wise trajectory during continued training, with an early increase followed by gradual decrease and stabilization.
- Semantically related concepts exhibit interference and synergy patterns during joint training, motivating interference-aware data scheduling.
- The analysis motivates concept-aware strategies, including circuit-aware replay to improve knowledge retention during continual pretraining.
2 Dataset Construction
The paper constructs FICO from ConceptNet-derived relations while replacing real concept names with fictional ones. It uses the dataset to study concept learning while preserving realistic knowledge structure and reducing pre-existing model knowledge.
- The dataset includes 500 concrete concepts and 500 abstract concepts, with associated knowledge retrieved from ConceptNet.
- FICO is built from ConceptNet-derived conceptual relations mapped to synthetic, nonexistent concept names.
- Fictional concept names reduce the influence of pre-existing knowledge encoded in LLMs while preserving realistic relational structure.
- Figure 1 presents dataset construction, Concept Circuit extraction, continual-pretraining dynamics, and cross-concept interaction analyses.
- Evaluation uses disjoint templates and unseen surface forms to assess whether models learn underlying relations rather than memorize templates.
3 How Internal Concept Representations Correlate with LLM Learning and Forgetting Dynamics?
Concept circuits provide informative signals of how concepts are learned and forgotten during continual pretraining. Their topology changes over time, and stronger acquisition is associated with greater subsequent forgetting.
- Concept Circuit as Internal Concept Representation: Concept circuits are computational subgraphs that jointly and faithfully predict all conceptual knowledge associated with a concept.
- Experiment Finding: Across 500 concepts, learning and forgetting degrees vary substantially, showing that the same training regime affects concepts to markedly different extents.The distributions are unimodal but widely spread.
- Experiment Finding: Circuit topology correlates more strongly with concept learning than edge-level EAP-IG statistics or random topology metrics.Positive correlations with eigenvector centrality and k-core indicate that centralized bottlenecks and stable structural cores are associated with larger learning gains.
- Experiment Finding: Concepts with more centralized, redundant circuits tend to show stronger acquisition but also greater forgetting during subsequent training.The behavioral relationship is a positive correlation between learning degree and forgetting degree, and the associations remain comparable at 70% and 80% preservation thresholds.
- Experiment Finding: During continued training on unrelated data, most circuit metrics initially increase, then gradually decrease and stabilize.This stage-wise pattern suggests systematic circuit reorganization rather than monotonic structural decay.
4 How Do Interference and Synergy Arise Across Concepts and Conceptual Knowledge During Joint Training?
Joint training reveals both interference and synergy across concepts and conceptual knowledge. Semantically similar concepts interfere more strongly, while transfer effects across knowledge types are heterogeneous and directional.
- Concept interference: Highly related concepts show stronger interference than weakly related concepts during joint learning.The relatedness groups are constructed from embedding similarity, and performance is evaluated on the target concept.
- Concept interference: 75.9% performance for weakly related concepts exceeded 67.2% for moderately related and 57.5% for highly related concepts.The comparison uses average logit and average probability across target concepts.
- Concept interference: Highly related concept pairs have more concentrated circuit-overlap distributions, indicating reuse of overlapping computational pathways.Circuit overlap is measured with Jaccard similarity between edge sets; weaker-related pairs show lower and more dispersed similarity.
- Knowledge-type transfer: Paired transferability measures whether prior training on source type Ri changes target-type Rj performance relative to BIO training.Positive T(Ri → Rj) indicates synergy, while negative values indicate interference.
- Knowledge-type transfer: Transfer effects across knowledge types are substantial, asymmetric, and uneven across ordered source-target pairs.Property & Affordance pretraining substantially benefits Hyponym & Hypernym and Synonym & Antonym, whereas reverse directions are markedly weaker.
- Knowledge-type transfer: Knowledge-type transfer suggests reordering curricula so types with strong positive transfer precede downstream concept learning.Meronym & Holonym shows relatively small improvements across most sources, indicating reduced sensitivity to prior knowledge-type pretraining.
5 How Circuit Analysis Can Inform Better Continual Pretraining?
Circuit topology metrics are used to identify concepts vulnerable to forgetting during continual pretraining. Circuit-aware Experience Replay prioritizes those concepts for additional replay, outperforming random replay across reported retention measures.
- Circuit-aware Experience Replay: Circuit-aware Experience Replay prioritizes concepts predicted to be susceptible to forgetting using circuit topology metrics.The strategy allocates additional training effort through replay of associated samples during continual pretraining.
- Circuit-aware Experience Replay: Density is used to rank concepts because it has the strongest reported correlation with forgetting.The ranking selects the top p% of concepts with the largest topology metric values.
- Evaluation: On AntiLeak, Circuit-aware Experience Replay consistently outperforms a random replay baseline across Logit, Accuracy, and Hit@5.The experiment trains GPT-2 Large on AntiLeak before continuing training on BIO.
6 Robustness and Ablations
The study tests robustness across model scales and a longer continual-training horizon. Circuit topology correlations persist across LLaMA models and become stronger for learning degree in repeated acquisition and interference phases.
- Model scale: Across LLaMA-3.2-1B and LLaMA-3.1-8B, circuit topology achieves around ρ ≈0.3 correlations with concept learning and forgetting degree.These results replicate the RQ1 and RQ2 analyses across model scale.
- Continual-training horizon: In a four-stage FICO→BIO→FICO→BIO sequence, graph metrics correlate with learning degree at ρ = 0.425.The longer horizon tests repeated acquisition and interference phases.
- Continual-training horizon: Repeated training may encourage reuse or consolidation of previously learned computational structures for concept acquisition.This is presented as an interpretation of the stronger learning-degree correlations under the extended horizon.
7 Related Work
Related work studies LLM knowledge acquisition, concept probing, editing, and internal representation changes. This paper distinguishes itself by targeting concept-level knowledge and higher-order learning dynamics rather than isolated factual triples.
- LLM Knowledge Acquisition: One research line studies LLM knowledge acquisition using synthetic or fictional corpora, including fabricated biographies and fictional entities.Another line traces how internal representations evolve across training stages, while a third identifies phase transitions in model behavior.
- Positioning: This work targets concept-level knowledge rather than isolated factual triples, enabling analysis of higher-order phenomena such as concept learning and forgetting.The supplied passage introduces this contrast with prior knowledge-acquisition studies.
- LLM Concept Probing and Editing: Prior concept research uses prompt-based probing to assess conceptual properties and relations.Other approaches test definition-name alignment and compositional binding or consistency.
8 Conclusion
The paper studies concept acquisition, retention, forgetting, interference, and synergy in LLM continual pretraining by linking behavioral outcomes to internal concept circuits and circuit topology. It finds consistent circuit-level signals of concept dynamics, supporting more concept-aware and interpretable continual pretraining.
- The study links LLM output behavior with internal concept circuits and graph metrics to analyze concept learning, forgetting, interference, and synergy.Concept Circuits are used as internal representations of concepts, while Circuit Topology is characterized with graph metrics.
- Circuit topology provides meaningful signals of concept dynamics across acquisition, forgetting, and cross-concept interactions.
- The findings motivate more concept-aware, interpretable, and robust continual pretraining of LLMs.
Limitations
The study is limited by its model scale and by the early-stage evaluation of its actionable training strategy. Larger models and broader validation of circuit-aware replay remain future directions.
- Experiments primarily use GPT-2 Large (0.7B), with LLaMA-3.2-1B and LLaMA-3.1-8B used for ablation studies.The authors identify extension to larger LLMs as an important future direction.
- Circuit-aware Experience Replay is evaluated only as a proof of concept rather than a complete continual-learning method.Its evaluation is limited to GPT-2 Large and AntiLeak.
- Future work should validate circuit-aware replay across broader models and datasets and explore additional concept-aware training strategies.Suggested directions include adaptive training effort allocation, interference-aware data scheduling, and curriculum design.
Ethical considerations
The authors report following the ACM Code of Ethics and identify no potential societal impacts or risks so far. The dataset covers broad everyday conceptual domains represented in ConceptNet.
- The authors state that they follow the ACM Code of Ethics and have not found potential societal impacts or risks so far.
- The authors report no notable harmful effects or risks involving environmental impact, fairness, privacy, security, or related considerations.
- FICO covers everyday ConceptNet domains including objects, animals, locations, properties, affordances, and relational knowledge.The listed relations include part-whole, spatial, taxonomic, synonym, and antonym relations.
E.1.1 RQ1 Results
Across experiments and model scales, concept learning and forgetting vary substantially, circuit topology remains associated with these dynamics, and joint training reveals stronger interference among semantically related concepts. Transferability across knowledge types is heterogeneous and directional, while circuit metrics evolve with training in a recurring temporal pattern.
- RQ1 Results: Concept learning and forgetting show substantial variability across concepts in LLaMA-3.2-1B and LLaMA-3.1-8B.
- RQ1 Results: Circuit topology metrics remain correlated with concept learning and forgetting, reproducing the qualitative trends observed for GPT-2 Large.The same overall association is reported for both LLaMA model scales.
- RQ1 Results: Circuit topology during continual forgetting shows an early increase followed by later decrease and stabilization.
- RQ1 Results: Concepts learned more strongly show greater forgetting during subsequent training, while semantically related concepts induce stronger interference during joint training.
- RQ1 Results: Transferability across knowledge types is highly heterogeneous, directional, asymmetric, and uneven across the evaluated models.
G GPT-2 Large Results for Log Probability
Under log-probability evaluation, concept learning and forgetting remain heterogeneous, while circuit-topology associations and the learning–forgetting trade-off persist. Circuit graph metrics also correlate more strongly with learning degree than the evaluated baselines.
- Concept acquisition and forgetting show substantial variability across concepts under log-probability evaluation.
- Circuit topology metrics remain positively correlated with concept learning and forgetting degree, including eigenvector centrality and k-core.These associations remain consistent when log probability replaces logits, indicating robustness to the behavioral metric.
- Concepts with larger learning gains tend to exhibit greater forgetting during subsequent training, confirming a learning–forgetting trade-off.
- Circuit graph metrics correlate more strongly with concept learning degree (Spearman ρ ≈0.29–0.32) than edge-level EAP-IG statistics (ρ ≈0.12–0.19) or random graph metrics (near zero).The comparison evaluates random graph topology and summary statistics of edge-level EAP-IG scores as baselines.
- Concept Circuits show more heterogeneous graph organization than random graphs, whose topology metrics are nearly constant by construction.For k-core, Concept Circuits have mean and standard deviation 30.27 and 16.78, compared with 38.69 and 2.42 for random graphs.
- Among the 20% of concepts with the highest learning degrees, final learning logits average 21.27 and forgetting degrees average 20.95, with both distributions spanning wide ranges.Final logits span 18–24, while forgetting degrees span 14–26, so high learning degree does not imply a uniform final-logit ceiling.