Specialists Learn Faster: Knowledge Organization in Pre-training Shapes Knowledge Update Efficiency

• Chronological training tends to produce specialists that perform best on recently encountered knowledge, while shuffled training tends to produce generalists that perform reasonably across represented periods.The paper studies biographical knowledge as a naturally ordered setting.Source: §1 · p. 1
• Specialists acquire new knowledge more efficiently, chiefly because their larger initial gradients close the knowledge gap faster, even when that gap is sometimes larger.The paper derives a law relating update cost to the initial knowledge gap and update speed.Source: §1 · p. 1
• The authors propose organizing pre-training data to preserve rarely changing historical knowledge while enabling efficient updates to recent knowledge.They report that the benefits extend to real-world pre-trained models, existing update tools, and other knowledge orderings.Source: §1 · p. 2
• Chronologically trained models require roughly half as many steps as temporally shuffled models to learn new knowledge, and the saving grows with model size.Source: §1 · p. 2

• Temporal benchmarks and studies examine what models know about the past and present, while this work asks how changing-fact histories are represented and what it costs to update them (Lazaridou et al., 2021; Jang et al., 2022a;b; Loureiro et al., 2022; Vu et al., 2024).The passage also names work on outdated knowledge, date conditioning, and temporal representations (Zhao et al., 2024; Dhingra et al., 2022; Gurnee & Tegmark, 2024; Park et al., 2025).Source: §2 · p. 2
• Prior studies treat data order as curriculum learning or chronological and continual pre-training; this work holds facts fixed and varies their order to isolate its effect (Jin et al., 2022; Pilchen et al., 2026).Source: §2 · p. 2
• The paper identifies its approach as the only prior-work category in the table using data organization for efficient editing at the pre-training stage.Table 1 distinguishes data organization and training dynamics from architecture or hyperparameter approaches by color.Source: Table 1 · p. 2
• Post-training update research rewrites facts, trains editors, or modifies update data, whereas this work asks whether data organization eases later updates (Meng et al., 2022; 2023; Fang et al., 2025; Mitchell et al., 2022a; Zweiger et al., 2025; Ovadia et al., 2024; Yang et al., 2025).Source: §2 · p. 2

• The synthetic biography dataset contains 25,000 individuals at the 110M scale, each represented 1,000 times from 2000 to 2024 with six attributes whose values vary at different rates.Birthday and birth city stay fixed, while university, field, employer, and employer city change over time.Source: §3 · p. 2
• The six corpora vary chronological versus shuffled record order and whether timestamps are absent, present, or included on half of records; GPT-2-style decoders are trained at 110M, 350M, and 682M scales.The chronological ordering sorts each person's records by timestamp; records across people are unordered.Source: §3 · p. 3
• Overwrite and injection updates train a LoRA adapter; overwrite fine-tuning uses loss on the new value and stops when its probability exceeds the strongest old-value competitor.Figure 1 depicts the probabilities over an attribute's values during an overwrite update.Source: §3 · p. 3, Figure 1 · p. 3
• Dated queries specify a year and accept any value held during that year, whereas undated queries omit the year and expect the latest value.Source: §3 · p. 3

4.1 ORGANIZING TIME-VARYING KNOWLEDGE INFLUENCES GENERALISTS VS. SPECIALISTS

• Chronological training produces distinct periods of high accuracy for different dated queries, while shuffled training performs well across dated queries, including knowledge not yet encountered in the first epoch.Figure 2 reports query accuracy over five training epochs for the 110M CD and SD models; color encodes accuracy, and panels distinguish year and batch queries.Source: Figure 2 · p. 4, §4.1 · p. 4
• In the loss-landscape account, chronological specialists for different years can overlap in a generalist region, while shuffled-batch specialists lie close together within that region.The paper suggests this proximity explains why shuffled training yields generalists that perform well across batches.Source: §4.1 · p. 4

4.2 THE ROLE OF TIME IN CHRONOLOGICAL TRAINING

• Nearby chronological records often share values, so models can reduce loss without relying strongly on year; shuffled records can place different values of the same fact together, making year useful for distinguishing them.Source: §4.2 · p. 4
• The paper measures year information's predictive benefit as the loss reduction from conditioning on year, represented by conditional mutual information between year and value given fact identity.Under chronological training, this quantity is close to zero, and year-agnostic specialists and year-aware generalists have similar query performance.Source: §4.2 · p. 5
• Year information is not simply dropped in specialist models; it participates in inference but is treated as an ordinary token and contributes little to prediction.Source: §4.2 · p. 5

5.1 SPECIALISTS ACQUIRE NEW KNOWLEDGE FASTER

• The update-cost pattern also holds without timestamps or when timestamps appear in only half the records, and across optimizers, learning rates, and full fine-tuning.The timestamp comparisons use undated updates for C and S, and evenly split dated and undated updates for CD1/2 and SD1/2.Source: §5.1 · p. 6
• Table 2 reports that specialists require 27%-58% fewer steps than generalists for overwrites and injections across three model sizes and six training corpora.The “saved” column compares chronological models with corresponding shuffled models.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 A STEP LAW FOR UPDATING EFFICIENCY

• The step law predicts update cost across model scales, timestamp settings, and training organizations, with an overall Pearson correlation coefficient r = 0.91.The same model predicts update cost after continued pre-training changes the model’s organization, without refitting its calibration constant.Source: §5.2 · pp. 6–7
• Specialists’ update speed is 2.5 × higher on average, contributing more to their advantage than the initial knowledge gap.A 110M chronological model updates more than 2 × faster despite having a 1.03 × larger knowledge gap than its shuffled counterpart.Source: §5.2 · p. 7
• Larger models tend to have larger initial knowledge gaps because they are more confident in old knowledge, helping explain their greater update savings.The passage attributes the gap to larger generalists being trained better.Source: §5.2 · p. 7
• Middle layers contribute most to specialists’ larger initial gradients, the source identified for their faster updates.The layer and block comparison shows initial-gradient amplitude increasing from generalists to specialists.Source: §5.2 · p. 7

6.1 MIXING GENERALISTS AND SPECIALISTS

• Mix-C shuffles records from earlier history and chronologically orders records in a recent window; it alone achieves high accuracy on both dated historical and undated current queries.Its training dynamics are generalist-like on historical knowledge and specialist-like on recent knowledge.Source: §6.1 · p. 8

6.2 RESHAPING A REAL-WORLD PRE-TRAINED MODEL

• After continual pre-training with C-C, the model’s undated current-knowledge query accuracy rises from 42% to 98%, requiring 1.3× fewer overwrite-update steps than Base and 1.8× fewer than C-S.The passage also says its general ability is preserved and that forgetting results are reported in Appendix D.2.Source: §6.2 · p. 8

6.1 MIXING GENERALISTS AND SPECIALISTS

• 16%-44% fewer update steps and 5%-57% less forgetting on historical records with Mix-C than Mix-S across single, batch, and sequential updates.Updates target the most recent year; forgetting measures incorrectly changed facts about other people.Source: §6.1 · p. 8, Figure 5 · p. 9

6.3 IMPROVING PUBLISHED-KNOWLEDGE EDITORS

• Up to 46% lower update cost with MEMIT and 63% less forgetting with AlphaEdit on Mix-C than Mix-S.The comparison holds for the same editing method.Source: §6.3 · pp. 8–9, Figure 6 · p. 9

6.4 GENERALIZING TO ANOTHER KNOWLEDGE ORDERING: VERSIONED SOFTWARE CHANGE

• Mix-C and C-C answer 98–99% of latest-version queries, cutting update cost by up to 48% and 33% versus Mix-S and C-S, respectively, while retaining older-version knowledge.Figure 6(c) reports results for a 110M model.Source: Figure 6 · p. 9, §6.4 · p. 10

7 CONCLUDING REMARKS

• Specialists trained on temporally ordered knowledge require roughly half as many update steps as generalists trained on shuffled knowledge.The authors say the benefit of pre-training organization extends to already-trained models and existing post-training update tools.Source: §7 · p. 10
• The paper explores software version as an ordering beyond time and proposes quantifying how the mutual information of an ordering affects generalist and specialist emergence.The authors note that the relevant information is what the ordering adds to a knowledge query given other available information.Source: §7 · p. 10
• The authors propose investigating whether specialists’ learning benefit extends to evolution, which acquires knowledge through gradient-free methods.They connect this question to evolutionary biology’s study of generalists and specialists in time-varying environments.Source: §7 · p. 10

Data construction

• Each synthetic biography has six attributes across 1,000 records from 2000–2024; consecutive records retain the same value about 99.0% of the time chronologically versus around 20.0% after shuffling.Birthday and birth city stay fixed; university and field change a median of three times, while employer and employer city change about ten times on average.Source: A DATA, TRAINING AND QUERY PROTOCOL · p. 14

Pre-training

• Models use GPT-2 architecture and tokenizer, batches of 2,560 records, a 5% linear warmup, cosine learning-rate decay, weight decay 0.1, and gradient clipping at 1.0.The main Section 4 models train for five epochs, by which query accuracy has converged.Source: A DATA, TRAINING AND QUERY PROTOCOL · p. 14

Queries

• Dated queries ask for any value held in the specified year; undated queries ask for the latest value, while current-row queries evaluate records from the batch just trained.Figure 7 distinguishes the queried-year value, the value in the batch at step k, and the final value.Source: A DATA, TRAINING AND QUERY PROTOCOL · p. 14, Figure 7 · p. 14, Figure 8 · p. 15

Updates

• Updates train LoRA adapters, and an update succeeds when the new value passes its strongest previous competitor; robustness checks also test top-1, a 3-nat margin, and Pnew ≥0.5.Table 5 reports median steps for 60 overwrite and 60 injection updates per pair under the three stricter criteria.Source: A DATA, TRAINING AND QUERY PROTOCOL · p. 15, Table 5 · p. 15

B.1 THE PREDICTIVE VALUE OF TIMESTAMPS

• Permuting records leaves the global objective unchanged, but orderings can produce different distributions within a fixed analysis window.The horizon W is an analysis scale, not necessarily a batch size.Source: §B.1 · p. 16
• The timestamp gap is positive exactly when a fact’s value distribution depends on the year, and measures the minimum loss from ignoring the year.Chronological windows have persistent fact values, making the gap small.Source: §B.1 · pp. 17–18
• 0.01 bits per record bounds the 110M CD model’s timestamp value in a chronological window, versus 0.69 bits in a shuffled window.The year provides almost no predictive value in the chronological window but much more in the shuffled one.Source: §B.1 · p. 18

B.2 TRACING TIMESTAMP USE WITH ACTIVATION PATCHING

• Activation patching finds that SD uses the year strongly and early, while CD processes it gradually and gives it less weight.The models process timestamps during inference, but chronological training places CD where the year is not needed much for value prediction.Source: §B.2 · p. 18, Figure 10 · p. 18

C.1 UPDATE COST BY RATE OF CHANGE AND ACROSS SEEDS

• Chronological training saves update steps across slow- and fast-changing facts, with Table 6 reporting median steps and savings by change rate.The table also includes the Mix pair; bold savings have paired Wilcoxon p < 0.05.Source: Table 6 · p. 19
• 44–52% of update steps are saved by chronological training across four independent 110M runs.The result is reported as robust to initialization.Source: §C.1 · p. 18, Table 7 · p. 19

C.2 OTHER OPTIMIZERS, LEARNING RATES AND FULL FINE-TUNING

• Chronological models consistently require fewer update steps across optimizers, learning rates, LoRA ranks, and full fine-tuning.Savings are about 50% under Adam and Lion, and larger under SGD; the step law continues to predict results across optimizers.Source: §C.2 · p. 19

C.3.1 FROM THE TRAJECTORY TO THE LAW

• The trajectory derivation sums margin changes across optimizer steps, then approximates each step’s change by its starting gradient and parameter update.This first-order approximation drops curvature within each optimizer step.Source: §C.3.1 · pp. 20–21
• The step law relates update cost to the initial gap and first-step speed, using the initial loss gradient to estimate that speed.The update succeeds when the accumulated margin gain closes the initial gap; the law’s fitted factor is estimated on one half of updates and evaluated on the other half.Source: §C.3.1 · pp. 21–22

C.3.2 HOW GOOD EACH APPROXIMATION IS

• The approximation checks recover 99.4% of margin traveled, and the gradient-based estimate matches measured first-step speed within 3%.The average speed is 1.3 times the first speed in C and 1.8 times in S.Source: §C.3.2 · p. 22, Figure 11 · p. 22

C.3.3 FACTORIZING THE ADAPTER GRADIENT

• At initialization, LoRA’s A-gradient vanishes because B is zero, so the initial loss gradient consists of the adapters’ B-parts.The gradient is factored by module using token inputs and output sensitivities.Source: §C.3.3 · p. 22

D.1 MIXING GENERALISTS AND SPECIALISTS

• For batch updates with 2,400 facts and sequential updates of ten 50-fact batches, across all three settings, Mix-C needs 16–44% fewer steps than Mix-S and forgets 5–57% less.Source: §D.1 · p. 23

D.2 RESHAPING A REAL-WORLD PRE-TRAINED MODEL

• On Gemma-2-9B-it, C-C raises past-year accuracy from 40% in Base to 99%, while replay-omitted KairosQA facts lose about 25% of previously correct past-year answers.On ARC-Easy, PIQA, and HellaSwag, C-C is at most 1.1 below Base; on CounterFact, it matches Base.Source: §D.2 · p. 23

D.3 IMPROVING PUBLISHED-KNOWLEDGE EDITORS

• Every tested editor forgets 50–63% less on Mix-C than on Mix-S across the same 60 overwrite updates.The comparison covers ROME, MEMIT, AlphaEdit, and ENCORE; Table 10 reports update cost and two forgetting measures.Source: §D.3 · p. 24, Table 10 · p. 24

D.4 VERSIONED SOFTWARE CHANGE

• For software tool calls, chronological models need 14–48% fewer steps without a version prompt and 7–29% fewer with one.Without the version prompt, Mix-C reaches 0.98 latest-version accuracy versus 0.26 for Mix-S; continual pre-training gives C-C 0.99 versus C-S 0.76.Source: §D.4 · p. 24, Table 11 · p. 25

D.5 RESHAPING A PRE-TRAINED MODEL IN SYNTHETIC WORLD

• C-C raises Base’s undated query accuracy from 25% to 93% and keeps dated accuracy at 98%; it needs up to 2.1× fewer steps in a batch update.C-S reaches 22% undated accuracy, and C-C forgets 10–15% less.Source: §D.5 · p. 25