PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement

Shuxing Yang, Kaihao Zhu, Junjie Yang, Rui Zhao, Junyao Wu, Yize Wang, Wenhao Li, Fujia Chen, Taowen Deng, Shenzhan Hong, Yaqi Li, Zichen Li, Jincheng Mi, Yuang Pan, Hongsheng Chen, Yihao Yang

arXiv:2609.10702v1cs.CLcs.AI

TL;DR

Learning from limited text raises the question of how models can use context, generalize to new inputs, and retain useful capabilities. Across three research stages, Qiushi Engine developed and tested a contextual-dependency learning principle; its second-generation continuation models rose from 42.02 to 42.25 Overall.

  • Problem

    How limited text can support contextual use, generalization to new expressions, and retention of useful capabilities is a central research question.

    Source: §1 · p. 4

  • Method

    The program linked frontier construction, principle discovery, and principle-guided improvement, organizing experience around target contextual dependencies while separating visible information, supervision, and preservation.

    Source: §7 · p. 37

  • Results

    42.02 to 42.25 Overall: the two public representative models advanced across generations, and second-generation training exceeded ordinary continuation under both seeds from a shared parent.

    Source: §7 · p. 37

  • Takeaways & Limitations

    Recovering familiar performance does not ensure unseen inputs can use a learned computation, and the study evaluates new and existing capabilities together.

    Source: §7 · p. 37

    The complete evaluations cover two continuations from one parent, and the full method uses extra preservation presentations.

    Source: §6.5 · p. 36

Abstract

from arXiv · show

Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them.

1 BabyLM 2026: Learning from Limited Data

This section frames data-efficient language modeling as learning to use context, generalize to new expressions, and retain capabilities under limited experience. Qiushi Engine links frontier advancement, principle discovery, and principle-guided improvement through experiments on contextual dependencies and model behavior.

1.1 The Scientific Problem of Learning from Limited Data

  • Limited-data learning must teach models to use relevant contextual relationships, not merely encounter information or fit familiar predictions that can rely on nearby clues.The central question is what experience teaches contextual information use and what keeps that ability usable during further learning.Source: §1 · p. 4, §1.1 · p. 4

1.2 Research Background and Track Rules

  • Strict-Small constrains both the training corpus and cumulative word exposure, while repeated, rewritten, and preservation inputs count toward applicable budgets and intermediate checkpoints document learning trajectories.Word counts follow track rules rather than tokenizer counts; the fixed budget still leaves choices about data, objectives, architecture, optimization, and presentation order.Source: §1.2 · p. 5, Table 1 · p. 5

1.3 Evaluating Language Abilities and Learning Behavior

  • BabyLM evaluates distinct language abilities and human-behavior correspondence, including grammatical contrasts, context use, conceptual inheritance, state tracking, downstream fine-tuning, reading behavior, and word-acquisition order.The nine top-level metrics comprise seven NLP measures and two human-like measures, and task scores require their own reference levels and scales.Source: §1.3 · p. 5, Table 2 · p. 6, §1.3.1 · p. 6

1.4 Aggregate Scores and the Public Leaderboard

  • 42.25 for Principle-guided exceeded Frontier’s 42.02 and was the highest Overall in the retained Strict-Small snapshot recorded on 8 September 2026.External submissions ranged from 40.67 to 41.80, and the second-generation model exceeded the highest external displayed score by 0.45.Source: §1.4 · pp. 7–8, Figure 1 · p. 8, Table 3 · p. 8

1.5 The Main Challenges

  • The main challenges are balancing content and rehearsal, distinguishing relevant-information use from correct prediction, and measuring new learning together with retention.The broader research contribution includes connecting candidate selection, mechanism tests, evaluation, and error analysis into a continuing research process.Source: §1.5 · p. 9

1.6 Performance Gains and Scientific Contributions

  • 42.2464 and 42.2317 with ordinary-input preservation exceeded same-parent ordinary continuation at 42.0926 and 42.1159; dense masking with sparse supervision scored 42.2025 and 42.1789 across two seeds.The preservation presentations were accounted for separately.Source: §1.6 · p. 9

1.7 Research Value and Prospective Applications

  • The methods offer hypotheses for specialized domains and low-resource languages, while Research RSI describes how accumulated findings and methods shape subsequent research questions and model designs.The experiments provide plausible transfer directions, not an established cross-domain result.Source: §1.7 · p. 10

1.8 Three Complete Research Stages Conducted by Qiushi Engine

  • Across three complete research stages, Qiushi Engine built a frontier model, tested explanations of learning behavior, then used those findings to guide model improvement and refine a testable learning principle.The principle organizes experience around required contextual dependencies and separately designs visible information, supervision targets, and preservation conditions while evaluating acquisition, reuse, and retention.Source: §1.8 · pp. 11–12

2 Stage I: Frontier Advancement

Stage I built a frontier model through coordinated choices about text, architecture, and training, then established an experimental basis for continued learning and mechanism studies.

2.1 Choosing Representations, Architectures, and Objectives

  • The adopted route combined compact restatements, residual learning, and evaluation-based selection of a late-training increment after comparisons across representations, architectures, objectives, and schedules.Lower loss on one objective was not sufficient; configurations also had to improve relevant language abilities under comparable budgets.Source: §2.1 · p. 12

2.2 Compact Restatements and Budget Reinvestment

  • 12,155 pairs versus 10,094 (up 2,061; 20.42%) fit within the same 423,520-word replacement block after compact restatements.On a seven-metric screening average—not the official NLP Average—the method was about 0.83 points below its reference at 20M cumulative words, then 1.29 and 1.35 points above at 70M and 80M.Source: §2.2 · pp. 12–13, Table 4 · p. 13

2.3 Residual Structure and Continued Incremental Learning

  • At 82,012,495 cumulative words, the jointly trained model had 35,463,008 parameters; Qiushi Engine then froze those parameters and added a 995,584-parameter increment, bringing total size to 36,458,592.The increment used 101 updates and 3,992,800 additional words, reaching 86,005,295 cumulative words; incremental training used ordinary prediction loss and a KL preservation term.Source: §2.3 · p. 14

2.4 Coherent Relations in Late Training

  • 44.1064 versus 43.1214 on the seven-metric average showed that coherent segments outperformed shuffled segments with the starting model, increment, exposure, and update count held fixed.The difference was approximately 0.9850; the comparison used residual scale 1.0.Source: Table 6 · p. 14, §2.4 · p. 14
  • 42.0240 (displayed publicly as 42.02) was the final first-generation local Overall after evaluation selected inference scale 0.75.The scale was selected by evaluating alternatives after training the increment at scale 1.0.Source: §2.4 · pp. 14–15

2.5 The Experimental Basis for Principle Discovery

  • Stage I produced reusable source–restatement data and phenomena for mechanism studies: coherent organization mattered, restatement benefits changed over training, and local improvements did not always yield aggregate gains.These observations motivated questions about reuse of learned relations on new inputs and retention during further learning.Source: §2.5 · p. 15

3 Stage II: Principle Discovery

Stage II tested how experience shapes contextual prediction and whether learned computations remain usable on new inputs. Its findings motivate coordinating the information visible during prediction, the targets and weights receiving supervision, and the functions preserved during further training.

3.1 Relation Type Shapes Context Use

  • Exact repetition changed true-source advantage by −0.8138 nats, while aligned restatement changed it by +0.7970 nats; unrelated-source effects were much smaller.These changes were averaged over three seeds. Figure 3 reports the true-source and unrelated-source advantages for nonoverlapping target tokens.Source: §3.1 · p. 16, Figure 3 · p. 16
  • Separating paired passages attenuated both effects; aligned restatement helped targets with source-token overlap, while its effect was near reference without overlap and repetition remained negative.The window controls retained each condition’s complete text but prevented source and target from sharing a prediction window.Source: §3.1 · p. 16, Table 7 · p. 17, Table 8 · p. 17

3.2 Familiar Performance and Reuse on Unseen Inputs

  • Full-sequence continuation yielded 87.2% familiar and 40.2% unseen accuracy; interleaved supervision yielded 100.0% familiar and 83.3% unseen accuracy.Initial accuracy was 100.0% familiar and 87.5% unseen; static relation supervision yielded 100.0% familiar and 75.5% unseen accuracy. Figure 4 reports these two-seed results.Source: §3.2 · p. 18, Figure 4 · p. 18
  • Zeroing the selection signal reduced unseen accuracy, and full-sequence continuation restored its use for familiar but not unseen queries; refitted directions did not explain weaker unseen-query redirection.Zeroing reduced initial unseen accuracy from 87.5% to 25.0% and interleaved-supervision accuracy from 83.3% to 38.7%.Source: §3.2 · p. 19, Table 9 · p. 19

3.3 How Supervision Allocation Changes Relation Learning

  • At 100M words, the seven-metric average was 43.896 with full supervision, 43.437 after deleting source-absent targets, and 42.792 after deleting matched copied targets.At 20M words, deleting source-absent rather than matched copied supervision raised source-absent loss by 0.0796 nats (95% interval [0.031, 0.125]); whole-document separation produced an interval containing zero. Target correspondence and supervision allocation affected learning, beyond input alone.Source: §3.3 · pp. 19–20

3.4 Experience Value Depends on Budget, Displacement, and Learning State

  • Experience value depended on what it displaced, learning stage, and model state; matching the seed did not ensure aligned shared parameters, which weakened a previously large negative interaction when copied explicitly.The comparisons showed that improving a training objective did not guarantee the same broad benefit, and no universal ranking followed from early loss or text-distribution distance.Source: §3.4 · p. 20

3.5 A Design Principle and Independent Findings

  • Stage II’s testable design principle separates visible information, supervised targets and weights, and preserved functions, then evaluates acquisition and retained behavior.Proposed tests vary local masking with targets fixed, add targets with masking fixed, and change preservation inputs while measuring both learning and retention.Source: §3.5 · pp. 20–21

4 Stage III: Principle-Guided Model Improvement

Stage III translated findings about contextual information use and capability retention into a revised continuation task and evaluated its effects against ordinary continuation. Its experiments also refined how the learning and preservation constraints should be designed and measured.

4 Stage III: Principle-Guided Model Improvement

  • Stage III retained the first-generation model’s weights, tokenizer, residual structure, and text while changing its training task and testing the design against ordinary continuation.The design addressed two findings: presenting text does not ensure that a model uses its relationships, and learning new behavior does not ensure retention of existing functions.Source: §4 · p. 21

4.1 From Mechanism Findings to Training Decisions

  • Training decisions evolved as tests separated the effects of visible information, prediction targets, and preservation inputs and update magnitudes.The study initially emphasized prediction-target count; effects persisted with sparse targets fixed and masking increased, prompting separate attention to information availability and supervision quantity.Source: §4.1 · p. 21

4.2 Separating Visible Inputs from Supervised Targets

  • The method retained source text, masked more candidate words in restatements, and selected focused prediction targets separately from masked positions.Dense masking with sparse supervision removes local clues without requiring a loss at every masked position; (S, S) versus (M, S) holds focused targets fixed, while (M, S) versus (M, M) holds masking fixed.Source: §4.2 · p. 22

4.3 Preserving Functions on Ordinary Inputs

  • Preservation constrained changes on ordinary inputs by comparing the updated model’s predictions with the frozen Frontier model’s predictions on the same masked rows.It used KL divergence over selected prediction positions, with the first-generation model as teacher and λ = 1 and T = 1.Source: §4.3 · p. 23

4.4 Complete Evaluation Across Two Continuation Seeds

  • The full method exceeded ordinary continuation by 0.1538 and 0.1158 Overall across the two seeds, while acquisition-only (M, S) exceeded it by 0.1099 and 0.0630.The full method exceeded (M, S) by 0.0439 and 0.0529; both dense-target (M, M) models scored below their sparse-target counterparts. GlobalPIQA contributed about 0.165 to each seed’s gain over the parent, while Entity Tracking supplied the largest positive contribution versus ordinary continuation (+0.1378 and +0.1511); the additional preservation effect did not change GlobalPIQA. The full-method comparison includes extra preservation presentations, and the (M, S) comparison does not isolate masking alone.Source: Table 11 · p. 24, §4.4 · pp. 24–25, Figure 7 · p. 25

4.5 Acquisition and Preservation Measurements

  • Dense-input preservation sharply reduced model change but also suppressed correct-target improvement, whereas ordinary-input preservation reduced divergence and cross-entropy deterioration.At shared target positions, the dense-input preservation gradient norm was 11.835 times its ordinary-input value. On a separate ordinary-input sample, parent KL fell from 0.02449 to 0.00941 nats and cross-entropy deterioration fell from 0.0290 to 0.0100 nats; preservation configurations also differed in target positions, leaving effects partly confounded.Source: §4.5 · pp. 26–27, Table 13 · p. 26, Table 12 · p. 26
  • Source interventions showed slight correct-source improvement but larger wrong-source and source-removal costs, some of which the full method reduced.These measurements used 96 pairs and 186 common targets outside the continuation’s trained prefix, but independence from the parent’s pretraining documents was not established; they test responses to source changes, not generalization of relational reasoning.Source: §4.5 · p. 27, Table 14 · p. 27

4.6 How the Model Tests Refined the Principle

  • Stage III tests refined the principle: control masking separately from target quantity and evaluate preservation through both new learning and ordinary behavior.Fixed-target masking and gradient comparisons showed that equal preservation coefficients can impose unequal effective constraints; source interventions also exposed correct-source benefits and wrong-source costs. The resulting method and controls sharpened the questions for subsequent research.Source: §4.6 · p. 27

5 Research Lineage and Independent Findings

The research program produced model improvements alongside independent findings, as methods developed for training became tools for controlled experiments. These branches yielded findings about data substitution, relational learning, memory, and measurement, with reusable methods and explicit limits.

5.1 Model Inheritance and the Development of Scientific Questions

  • Compact restatement and residual incremental learning linked the model generations through inheritance and became controlled instruments for comparing expression relationships, objectives, and preservation.Stage III operationalized choices about visible information, supervised targets, and preserved predictions on ordinarily masked inputs.Source: §5.1 · pp. 28–29

5.2 Experience Compression, Substitution, and Learning Stage

  • 0.6125 points separated two substitutions of the same added text: replacing child-directed/spoken material scored lower than replacing adult prose.The changes from reference were +0.0825 and +0.6950 on BLiMP, Supplement, EWoK, and COMPS. Restatement’s late-training effect varied by model configuration and learning stage; compression studies also made surviving words, correspondences, and prediction clues inspectable.Source: §5.2 · pp. 29–30

5.3 Relational Anchors, Shared Representations, and Identifiability

  • Reversing a few state anchors changed accuracy against original absolute labels from 1.0 to 0.0, while relative same/opposite judgments remained completely correct.Sparse absolute labels resolve orientation ambiguity in the specified relational structure, and the shared representation transmits anchor information; ordinary language still requires discovering objects and interpreting relations [33].Source: §5.3 · p. 30
  • Covering all base letters raised identity-matching accuracy from 0 to 1 and restored relational responses for events without direct supervision; uncovered characters left new names difficult.The experiment recombined learned characters with aligned positions, provided candidate objects, and froze the matcher after training; aliases, pronouns, and unrestricted reference remained distinct problems.Source: §5.3 · p. 31

5.4 Entity Storage, Retrieval, and Semantic Addressing

  • 1.2056 mean answer log-odds gain, 2.0513 slot-swap effect, and 0.0472 paired accuracy showed memory affected answer probabilities while reliable paired behavior remained difficult.A joint change to the memory connection and training configuration raised paired accuracy to 0.1375 and slot-swap effect to 3.087; lexical and role swaps were largely successful, but selecting states from temporal meaning remained difficult. An early result was withdrawn because a constant-answer shortcut could explain it.Source: §5.4 · pp. 31–32

5.5 Optimization, Initialization, and Measurement

  • Explicit parameter copying changed the compact-views-relative-to-repetition interaction estimate from −0.6908 to +0.0679, with an interval spanning zero.The reduced architecture still had lower absolute performance, while optimization interventions that improved short-run loss or internal measurements did not establish stable full-task gains. Macro-averages can also rise while total correct answers fall, so the study tracks gained and lost items separately.Source: §5.5 · p. 32

5.6 Preserving the Full Research Contribution

  • The independent branches left reusable training methods, mechanism experiments, controls that resolve confounds, and exploratory constructions with explicit questions and implementations.Together, they provide executable methods, testable explanations, reliable comparison tools, and directions for further work.Source: §5.6 · p. 33

6 Discussion

The discussion argues that data-efficient learning depends on how experience exposes contextual relationships and trains their use, while acquisition, generalization, and retention require distinct conditions. It presents Research RSI as a cycle in which findings and methods shape later research, while limiting the claim to this BabyLM program.

6.1 The Value of Experience and Contextual Dependencies

  • Content, position, expression, and prediction targets shape learning opportunities; coverage, compression, and repetition change distinct aspects of experience, and their effects depend on contextual access and trained use.Equal-exposure improvement supports more effective learning in this setting; the work complements scaling studies by testing how experience organization and continued training affect learning.Source: §6.1 · p. 33

6.2 Coexistence of New Capabilities and Existing Functions

  • Familiar accuracy can diverge from unseen-symbol relation processing; supervision changes unseen-input behavior and internal interventions alter answers, motivating tests of new-input access alongside familiar behavior.These findings, together with prior work on structural contextual learning, distinguish retaining familiar performance from preserving access to learned computation on new inputs.Source: §6.2 · pp. 33–34
  • Ordinary-input preservation retains most new prediction gains while reducing ordinary-masking loss changes, and the full configuration improves complete evaluation under both continuation seeds.Strong preservation on densely masked inputs suppresses acquisition; ordinary-input preservation better coordinates learning on the new relational task with limiting unnecessary drift.Source: §6.2 · pp. 33–34

6.3 How Qiushi Engine Organizes and Builds on Research

  • Qiushi Engine connected planning, method construction, experimentation, review, knowledge and memory systems, and research tools to conduct the investigation from question formation through result integration.Successive choices followed findings: coherent versus shuffled inputs prompted relation studies, while masking effects at fixed targets revised an earlier target-count explanation.Source: §6.3 · p. 34
  • Research continuity came from jointly developing methods and understanding: earlier methods and findings enabled new experiments, while explanations were revised and Stage III reused the first model and data.Compact restatements served as both training material and a way to compare expression relationships; residual increments improved the model and enabled objective comparisons against a fixed base.Source: §6.3 · p. 34

6.4 Research RSI: Recursive Self-Improvement of the Research Process

  • Research RSI describes Qiushi Engine reusing its research-generated understanding, methods, and experience to change later questions, experiments, and evidence assessment, then testing and revising that accumulation.The evidence establishes this recursive cycle within the BabyLM program, not sustained improvement in general research ability across tasks; that broader claim requires independent goals and comparisons of cost and outcomes.Source: §6.4 · pp. 35–36
  • The research cycle retained conditions, controls, and corrections: masking comparisons revised an explanation, and shared-position gradient measurements showed equal preservation coefficients need not impose equal constraints.Earlier compact-restatement rankings became budget- and stage-conditioned comparisons, while residual increments also became instruments for studying functional change against a fixed base.Source: §6.4 · p. 35

6.5 Reusable Contributions and Further Tests

  • The reusable contributions combine relation and capability studies with training design; component analyses attribute about 74% and 79% of full-method Overall gains to GlobalPIQA relative to the parent.Entity Tracking supplies the largest positive contribution relative to ordinary continuation; preservation adds net change in other components with GlobalPIQA unchanged, and two fine-tuning seeds show the (Super)GLUE advantage is not confined to one run. Complete evaluations use two continuations of one parent, include extra preservation presentations, and lack item-paired uncertainty intervals; further tests can use independent parents, budgets, and document-disjoint samples. Query-dependence, shared-parameter alignment, and text-overlap checks also provide stronger controls for future studies.Source: §6.5 · p. 36

7 Conclusion

Qiushi Engine completed three research stages on BabyLM Strict-Small, moving from frontier-model construction to principle discovery and principle-guided improvement. The work presents Research RSI as a process in which accumulated research informs new questions and methods that experiments then test and revise.

7 Conclusion

  • The three stages addressed how limited text can teach models to use contextual relationships while accommodating new learning and established language functions.The program covered frontier-model construction, principle discovery, and principle-guided improvement, including question formulation, method development, experiments, analysis, and synthesis.Source: §7 · p. 37
  • Exact repetition and aligned restatement shaped source use differently depending on target relation and prediction window, while familiar-performance recovery did not guarantee transfer to unseen inputs.The resulting design principle organizes experience around target-relevant contextual dependencies, separates visible information, supervision, and preservation, and evaluates new and existing capabilities together.Source: §7 · p. 37
  • 42.02 to 42.25 Overall: the two public representative models exceeded ordinary continuation on complete evaluation under both continuation seeds from a shared parent.The seeds used the same parent model.Source: §7 · p. 37
  • Research RSI describes how accumulated scientific understanding, methods, and experimental experience inform more precise questions and new methods that practice tests and revises.Earlier results are inherited and studied; the research routes advance through branching and integration.Source: §7 · p. 37
  • The repository preserves models, code, data construction, complete results, and scientific notes for testing findings and pursuing new questions.The two model generations, learning principle, experimental methods, and independent findings form a reusable research foundation.Source: §7 · p. 37

A Training, measurement, and reproducibility · A.1 Computing environment and experimental support

Research computations used an AI Lab skill across CPUs and two NVIDIA H100 GPUs for text processing, model training, evaluation, and controlled experiments. Released packages document a CPU validation environment, while original runs retain their own program and dependency records.

A.1 Computing environment and experimental support

  • Two NVIDIA H100 GPUs supported model training, evaluation, teacher-generated text, and accelerated mechanism experiments, while CPUs handled text processing and analysis.Recorded workloads included Qwen/Qwen3.5-9B compact restatements, controlled relation tasks, hidden-state and gradient measurements, and internal-signal interventions.Source: §A.1 · p. 38
  • Both released model packages record a CPU validation environment with Python 3.12, PyTorch 2.11.0 (CUDA 12.8 build), Transformers 4.57.6, Tokenizers 0.22.2, and Safetensors 0.8.0.These records describe release validation, not one environment retroactively assigned to every training, generation, or evaluation run.Source: §A.1 · p. 38
  • Original computations retain their own program and dependency records, and reproducibility/TRAINING.md collects environment instructions, training entry points, and data dependencies.Dependency records for the released packages are provided in models/frontier/ENVIRONMENT.json and mode ls/principle_guided/ENVIRONMENT.json.Source: §A.1 · p. 38

A.2 Model versions, teachers, and data sources · Frontier · A.3 Acquisition and preservation algorithm

These sections specify the model lineage and data used for Stage III, then describe how focused and ordinary targets, dense masking, and preservation are selected and combined during continuation. The complete model includes the incremental branch alongside the base network, and its identity also depends on its configuration, tokenizer, and loading implementation.

A.2 Model versions, teachers, and data sources

  • Stage III updates the existing incremental branch, which must be loaded alongside the base network for masked prediction and downstream fine-tuning.Model identity includes the weights, architecture configuration, tokenizer, and corresponding loading implementation; both public models share the same architecture and parameter count.Source: §A.2 · p. 38
  • Shared pretraining used an eight-layer DeBERTa-v2-style model with hidden size 480, eight attention heads, feed-forward size 1,920, and sequences up to 256 tokens.It used a 16,384-token byte-level BPE vocabulary, AdamW, effective batch size 256, peak learning rate 0.001, 6% linear warmup, cosine decay, and 0.01 weight decay. Configuration, initialization, and training randomness used distinct seeds 43, 43022, and 43023.Source: §A.2 · p. 38

Frontier

  • Stage III used Frontier as its frozen preservation teacher and reused Qwen restatements, with no new text generated.The manifests distinguish 37,594 source–restatement pairs containing 1,656,800 words from 12,155 compact pairs containing 423,511 words; both are within the ten-million-word corpus pool.Source: Frontier · p. 39
  • 20,475 packed Stage III rows contained 3,162,742 words, and a row could contain several paired segments.These row counts describe the continuation stream, not the word composition of the entire corpus.Source: Table 20 · p. 39, Frontier · p. 39

A.3 Acquisition and preservation algorithm

  • Annotated Qwen rows use relation-focused inputs masked over Qm and supervised over Qf, while other rows use ordinary whole-word masking.Qf is a subset of Qm; ordinary compact-restatement and budget-reinvestment rows remain in the ordinary branch. Preservation re-presents Qwen focused rows under ordinary masking and computes KL at those masked targets.Source: §A.3 · p. 40
  • Each update normalizes focused and ordinary losses by their respective target-token counts, trains only 48 designated incremental tensors, and adds normalized frozen-teacher KL preservation.Preservation uses the same targets for teacher and student, disables dropout while retaining student gradients, and saves and restores random-number states around the extra forward passes; acquisition words and preservation presentations are accounted separately.Source: §A.3 · p. 40
  • Sparse supervision selects eligible content groups by sampling at most 16 per row, then independently retaining each with probability 0.35.Eligible words are token-aligned English-letter-or-digit words, allowing internal apostrophes; words shorter than four characters, purely numerical words, and stopwords are excluded. If candidates exist but none is selected, one group is retained.Source: §A.3 · p. 40
  • Dense masking separately samples up to 128 candidate groups by default using a row-specific seed, then adds all supervised groups, so the final set can exceed 128.Masks and labels share position mapping but are selected separately.Source: §A.3 · p. 40

A.4 Evaluation aggregation, randomness, and trajectories

The section defines a common evaluation and aggregation procedure for complete models, while distinguishing scoring records and matching uncertainty estimates to their sampling units. It also specifies the checkpoint histories used for AoA trajectory evaluation.

A.4 Evaluation aggregation, randomness, and trajectories

  • Every complete-model evaluation includes the incremental branch, produces task-specific outputs before Eq. (2) aggregation, and preserves local nine-metric scores under a common procedure.Public leaderboard tables retain displayed values; close local scores are shown to four decimal places, with uncertainty assessed separately. Scorer version, model loading, and checkpoint lists jointly define the evaluation configuration; the repository retains the scoring programs and loading adaptations used.Source: §A.4 · p. 41
  • Scoring references the cited official evaluator and AoA estimator revision, while the repository’s execution entry points and checkpoint lists specify the reproducible evaluation setup.The evaluator is cited as BabyLM Organizing Team, and the AoA estimator corresponds to revision 6f825c2.Source: §A.4 · p. 41
  • BLiMP option-index scoring gave 68.51 for Frontier and 68.26 for Principle-guided, versus answer-text matching at 68.52 and 68.27.The two procedures differ because some items have identical option text: seven of 59,875 items across 67 subtasks, with three Frontier and five Principle-guided items scored differently. Released predictions match the original run files, and local training controls use the same scoring procedure.Source: §A.4 · pp. 41–42
  • Uncertainty estimates follow the sampling unit: relation experiments use training-seed standard deviations, source-isolated loss comparisons resample text pairs, and final-model results cover two continuations from one parent.Paired item-level GlobalPIQA intervals are not provided. The distinction between macro-averaged task scores and net item changes matters because unequal subtask sizes can make them move in opposite directions; Reading is excluded from item counts, and a Stage I scale-choice difference does not describe an item-level comparison of the official model generations.Source: §A.4 · p. 42
  • AoA uses 17 shared checkpoints from 1M to 80M words plus each model’s endpoint, yielding an 18-point vocabulary-learning trajectory.Checkpoint lists record actual cumulative exposure and shared history; they omit untraversed 90M or 100M branches, and evaluation defaults may need adjustment to the actual trajectory.Source: §A.4 · p. 42

B Complete local evaluation vectors

The section reports complete nine-metric evaluations for a shared parent and eight continuation models, covering local controls, remaining metrics, and Overall. It distinguishes sparse and dense supervision from the complete strategy, which adds ordinary-input preservation.

B Complete local evaluation vectors

  • The tables list all nine metrics for the shared parent and eight continuation models in the same order, and the report builder checks each nine-metric mean against Overall.Accompanying CSV files retain full precision; Appendix E gives file locations.Source: B Complete local evaluation vectors · p. 42, Table 22 · p. 42, Table 23 · p. 43
  • Table 22 covers local controls for language form, knowledge, entity tracking, and composition, with the shared parent appearing once.Sup. denotes BLiMP Supplement.Source: Table 22 · p. 42
  • Sparse and dense supervision are (M, S) and (M, M); the complete strategy adds ordinary-input preservation, while quick (S, S) measurements are excluded from complete evaluations.The excluded quick measurements are not part of complete nine-metric evaluations.Source: B Complete local evaluation vectors · p. 42
  • GlobalPIQA is unchanged between (M, S) and the complete strategy; their Overall difference reflects net changes in other metrics, not improvement in every metric.Table 23 presents the remaining metrics and complete Overall for the nine models.Source: B Complete local evaluation vectors · p. 42, Table 23 · p. 43

C Additional controls and mechanism results … C.3 Measurement populations and subsequent corrections

The appendix tests downstream-seed variability and target deletion, then clarifies which populations the measurements support and how several experimental confounds were addressed. Its corrections narrow claims about generalization preservation where overlap was found.

C.1 Downstream fine-tuning randomness

  • The complete strategy exceeds both the parent and (M, S) under each fine-tuning seed, with model weights held fixed.Table 25 reports equal-weight means of seven primary subtask metrics; seed 44 is supplementary and is not combined with other metrics to construct a new Overall.Source: §C.1 · p. 43, Table 25 · p. 43

C.2 Target deletion and source isolation

  • Table 26 reports local loss differences under source isolation in nats; positive values indicate that deleting source-absent supervision is worse on measured targets than deleting matched-copy supervision, with intervals based on record-specific resampling units.Source: Table 26 · p. 44
  • A loss difference is observed with pair isolation, while intervals under stricter document isolation include zero.Pair isolation excludes pairs used in the corresponding training data; document isolation also excludes other pairs from the same source document.Source: §C.2 · p. 44, Table 26 · p. 44
  • At 100M words, a separate evaluation measures seven tasks in single-seed target-deletion experiments with compact inputs held fixed.The two deletion conditions remove supervision from 48,105 and 48,387 BPE tokens, respectively; the supplied passages do not report task scores.Source: §C.2 · p. 44, Table 27 · p. 44

C.3 Measurement populations and subsequent corrections

  • An overlap check found that 6,992 rows used for ordinary-loss analysis also appeared in the historical stream, so they are not clean evidence of generalization preservation.An example outside the current continuation prefix may already occur in the parent’s training history, and distinct rows may share a source pair or document.Source: §C.3 · p. 44
  • Results identify whether they measure acquisition on training rows, source pairs outside the current prefix, pair-isolated targets, or document-isolated targets.A correction narrows the claim supported by the affected measurement without automatically invalidating experiments on other identified populations.Source: §C.3 · p. 44
  • KL self-comparisons require consistent forward-pass randomness, so additional preservation calls must save and restore the acquisition branch’s random-number state.Shared-initialization checks, query-dependence tests, text-overlap analysis, and forward-consistency checks address different confounds; their programs and records accompany the models for reuse.Source: §C.3 · p. 44

D Public leaderboard metrics … Systems and computation

The section examines how leaderboard aggregates, component metrics, research methods, and implementation checks characterize learning outcomes across experience, supervision, and context conditions. It presents diagnostic signals and controls as conditional evidence, distinguishing local capability changes from broader learning, generalization, and retention without treating observed gains as proof of general mechanisms or transfer.

D Public leaderboard metrics

  • Similar Overall scores can mask capability trade-offs, so aggregate and component metrics are complementary measures of improvement and its sources.Stronger grammatical judgments may coincide with weaker entity tracking, and the highest NLP average need not yield the highest Overall.Source: D Public leaderboard metrics · p. 45

E Research repository and use of materials

Models and representations

  • Masking granularity changes task trade-offs, no single mature-recipe component reproduced the full configuration, and recursive training did not match the leading configuration.The combined configuration’s late-training advantage was not explained by batch size alone, while position-related components were not independently isolated.Source: Models and representations · p. 48

Experience construction and compression

  • Relational organization and presentation layout affect learning differently across tasks and capabilities; compact views expand experience coverage, but compression and coverage effects require separation.Rule-based rewriting showed short-run capability trade-offs rather than joint gains, and visible token counts still need matching.Source: Experience construction and compression · p. 49

Learning objectives and supervision allocation

  • Local advantages from correct correspondence did not reliably yield broad gains, while target behavior could improve as broad language performance degraded, motivating joint learning-and-retention tests.If retrieval still solves a task, high scores cannot establish a new computation for state selection.Source: Learning objectives and supervision… · p. 49

Curricula and learning dynamics

  • Training-scheme rankings can reverse over time, curriculum ordering has no universal advantage, and context-cap results depended on the AoA reference without simultaneous gains across seven task families.Checkpoint averaging did not reliably improve on individual high-scoring models.Source: Curricula and learning dynamics · p. 50

Residual structures and incremental learning

Relational structure and identity

Conditional value of experience

  • Late-stage gains depended on experience type and architecture, and distinct sentences were not equivalent to distinct documents.Conditional comparisons included true, shuffled, repeated, and ordinary experience; register substitution also showed that opportunity costs depend on the material displaced, while relational-text candidates were limited and short-run results were not a mature training stream.Source: Conditional value of experience · p. 52

Supervision targets and sources

  • Local source-pair results held, but stricter document-disjoint intervals crossed zero, and source effects could occur without added supervision.The studies also varied source presence, prefix placement, gaps, and relative text positions; their context-geometry comparisons did not confirm specific mechanism effects.Source: Supervision targets and sources · p. 52

Functional mechanisms and learning signals

  • Decodable topic information did not guarantee state-dependent execution, and microworld interventions served as diagnostics rather than proof that natural-text selectors had formed.Comparisons included consistent prefixes, mismatches, entity substitutions, activation replacements, and controlled states, actions, contradictions, answer cues, and overwrites.Source: Functional mechanisms and learning… · p. 53
  • Auxiliary-objective fit was insufficient evidence of new capability, while local readout fit did not guarantee held-out transfer or show that the model used a correspondence.Local correlations were not stable prospective predictors of learning gains; limited editing examples also did not support broad mechanistic conclusions.Source: Functional mechanisms and learning… · p. 53

Measurement, transfer, and attribution

  • Correcting shared-parameter initialization weakened evidence for a large architecture interaction, and aggregate scores could obscure opposing patterns in item-level capability changes.Transfer correlations and radial growth did not establish capability conservation or a universal law; complete evaluation distinguished missing measurements, zero scores, displayed values, and model identity for comparability.Source: Measurement, transfer, and attribution · p. 53

Systems and computation

  • Word-budget checks required visible content to match accounted counts, while gradient checks validated limited implementation conditions rather than identical stochastic training trajectories.Reconstruction fixed weights, configuration, and code and separated training from search costs; same-seed reconstruction, replication across seeds, and compute savings were distinct conclusions.Source: Systems and computation · p. 54

Core research: extensions and empirical tests

The experiments test how prediction context, target relations, and functional access shape learning, then evaluate a preservation scheme that improved the nine-task Overall score for both continuation seeds.

R67 Selectivity in relation learning

  • Across three seeds, verbatim repetition and aligned reformulation had effects on compact reformulation targets that differed in direction, so repetition was not universally harmful.The comparison concerned source advantages from repetition and reformulation.Source: Core research: extensions and… · p. 54

R68 Prediction windows with matched materials

  • Splitting repetition and reformulation pairs into separate prediction windows substantially attenuated their large source effects despite matched content.The same-window and split-window versions were compared within each pairing condition.Source: Core research: extensions and… · p. 54

R69 Transfer conditions for natural reformulations

  • Aligned restatement improved source use for recurring target tokens, while nonrecurring targets showed no equally stable improvement over the reference.Recurrence was defined by whether the exact tokenizer ID appeared in the source; this lexical distinction does not define semantic novelty.Source: Core research: extensions and… · p. 54

R70 Familiar performance and functional access with unseen symbols

  • Recovering familiar behavior through continuation or weighted supervision did not guarantee restored functional reach for unseen symbols.The comparison included ordinary continuation, statically weighted supervision, and interleaved supervision.Source: Core research: extensions and… · p. 54

R71 Erasing and relocating functional signals

  • Results support a functional role for a specific attribute-position signal, but failed recovery with linear directions does not rule out distributed recoding.The experiments centered, zeroed, or rotated signals and tested refitted directions.Source: Core research: extensions and… · p. 54

R72 Dense inputs and sparse supervision

  • At equal exposure, dense inputs with separately controlled masking candidates and focused targets outperformed ordinary continuation for both seeds, though the comparison does not isolate masking.Complete evaluations were used for both seeds.Source: Core research: extensions and… · p. 54

R73 Preservation inputs and effective strength

  • With the same coefficient, preservation-gradient magnitudes differed by approximately 12× between ordinary and dense inputs, while the full comparison also changed target positions.Gradient magnitudes were measured at fixed shared target positions.Source: Core research: extensions and… · p. 54

R74 Principle-guided complete model

  • The complete preservation scheme improved the nine-task Overall score for both continuation seeds; additional exposure and compute were reported separately.The comparison rechecked ordinary continuation, relation learning, and the complete scheme from the same parent model.Source: Core research: extensions and… · p. 54

Related papers

Loading 2609.10702v1…