PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement

Shuxing Yang, Kaihao Zhu, Junjie Yang, Rui Zhao, Junyao Wu, Yize Wang, Wenhao Li, Fujia Chen, Taowen Deng, Shenzhan Hong, Yaqi Li, Zichen Li, Jincheng Mi, Yuang Pan, Hongsheng Chen, Yihao Yang

arXiv:2609.10702v1cs.CLcs.AI

TL;DR

The paper asks how limited text can teach models to use contextual relationships while retaining existing capabilities. Qiushi Engine conducts three connected research stages that build a frontier model, test learning principles, and guide a subsequent model. The principle-guided model exceeds ordinary continuation on complete evaluation, with public Overall rising from 42.02 to 42.25.

  • Problem

    The central question is what training experience teaches models to use contextual information relationships and keeps that ability usable during further learning.

  • Method

    Qiushi Engine connects frontier-model construction, controlled studies of familiar and unseen inputs, and principle-guided training that separates visible information, supervision, and preservation.

  • Results

    The second-generation principle-guided model exceeds ordinary continuation under both continuation seeds, with public Overall rising from 42.02 to 42.25.

  • Takeaways & Limitations

    The study supports organizing limited experience around contextual dependencies and evaluating acquisition, reuse, and retention together.

  • Takeaways & Limitations

    The complete evaluations use two continuations of one parent, and the full method includes extra preservation presentations; transfer across parent models remains a separate quantity.

Abstract

from arXiv · show

Learning from limited text requires models to use context, generalize to new inputs, and retain useful capabilities. Qiushi Engine conducted a long-horizon, end-to-end autonomous research program on BabyLM 2026 Strict-Small, within 10 million corpus words and 100 million cumulative word presentations. Three stages connected frontier advancement, principle discovery, and principle-guided model improvement. Stage I combined compact restatements, budget reinvestment, and residual incremental learning to build a frontier model. Stage II found that exact repetition and aligned restatement produce different patterns of context use, depending on target relations and prediction windows. In controlled tasks, recovering familiar performance did not ensure that unseen inputs could still use learned computations. These findings support a testable data-efficient learning principle: organize experience around the contextual dependencies needed for prediction; separately design visible information, supervision, and preservation; test learning, generalization, and retention. Stage III retained source text, masked more local clues, supervised selected targets, and preserved predictions on ordinarily masked inputs. Two continuation seeds from the same parent outperformed ordinary continuation on the complete nine-metric aggregate. Overall rose from 42.02 to 42.25 across two generations; the second achieved the highest Overall in the public Strict-Small snapshot of 8 September 2026. Further studies addressed compression, relational anchors, shared representations, and measurement. Models are available on Hugging Face; code and research records accompany the GitHub repository. Together, these stages illustrate Research RSI: recursive self-improvement of the research process. Scientific understanding and method innovations change subsequent questions and designs; new experiments test and refine them.

1 BabyLM 2026: Learning from Limited Data

BabyLM 2026 Strict-Small provides a shared setting for studying how limited text can teach models to use contextual information, generalize, and retain capabilities. Qiushi Engine’s program combined frontier-model construction, controlled learning studies, and principle-guided improvement, with both model scores and testable explanations as outcomes.

  • Limited-data language modeling must teach models to use relevant information, apply learning to new expressions, and retain useful capabilities during continued training.
  • The study distinguishes encountering relevant information from learning to use it, because repetition, restatement, and new sources consume budget while offering different learning opportunities.
  • The program presents models, methods, and scientific understanding as reusable outputs, while describing specialized-domain and low-resource transfer as plausible directions rather than established cross-domain results.
  • Stage I combined compact restatements, budget reinvestment, and residual incremental learning to produce a frontier model under restricted data and exposure.The broader exploration covered text construction, tokenization, representation, architecture, optimization, and training schedules.
  • The resulting principle organizes experience around contextual dependencies and treats visible information, supervision, and preservation as separate choices evaluated through acquisition, reuse, and retention.

2 Stage I: Frontier Advancement

Stage I built a frontier model by coordinating compact restatements, budget reinvestment, architecture, and continued training under strict data limits. Evaluation showed that coherent organization and late-training choices mattered, while local gains and early rankings did not reliably predict broad performance.

  • Compact restatements and budget reinvestment: 20.42% more source–restatement pairs fit the fixed 423,520-word replacement block after shortening each restatement.The collection grew from 10,094 to 12,155 pairs, adding 2,061 pairs while retaining source–restatement connections.
  • Training dynamics: The compact-restatement method shifted from 0.83 points below its reference at 20M words to 1.29 and 1.35 points above at 70M and 80M.This seven-metric screening average excluded (Super)GLUE and therefore was not the official NLP Average.
  • Residual structure and continued learning: A residual branch projected width 480 through a width-128 bottleneck, initialized its upward projection to zero, and scaled the correction by s = 1.75.The branch was trained jointly with the backbone before a later frozen-base incremental stage.
  • Coherent relations in late training: 44.1064 versus 43.1214 showed that coherent training beat segment shuffling by approximately 0.9850 on the seven-metric average.The control fixed the starting model, increment, exposure, and update count, isolating coherent organization as the changed condition.
  • Frontier model: 42.02 was the final first-generation local Overall after combining compact restatements, residual learning, and evaluation-based late-training selection.The released lineage used coordinated data and model choices under limited exposure, followed by a dedicated incremental stage.
  • Experimental basis: Stage I established a reusable source–restatement dataset, a trainable frontier model, and phenomena that motivated mechanism studies of learning, reuse, and retention.These observations moved the research from configuration selection toward principle discovery.

3 Stage II: Principle Discovery

Stage II tested how relation type, context windows, supervision targets, and representations affect whether models use learned information and reuse it on new inputs. The experiments supported separating visible information, supervision, and preservation when evaluating learning and generalization.

  • Relation type shapes context use: Aligned restatement changed true-source advantage by +0.7970 nats, whereas exact repetition changed it by −0.8138 nats; unrelated-source changes were much smaller.These values were averaged over three seeds, with standard deviations of 0.0758 and 0.1729 respectively.
  • Relation type shapes context use: Separating source and restatement passages attenuated both repetition’s negative effect and restatement’s positive effect, showing that shared prediction context mattered.The window controls retained each condition’s complete material while removing a shared context for a single prediction.
  • Familiar performance and reuse on unseen inputs: 83.3% versus 75.5% unseen-symbol accuracy showed that interleaved supervision preserved more reuse than static relation weighting, while familiar accuracy remained 100.0%.Full-sequence continuation reached 40.2% on unseen symbols despite 87.2% familiar accuracy.
  • Familiar performance and reuse on unseen inputs: Full-sequence continuation restored familiar-query use of a learned signal without comparable recovery for unseen queries, indicating selective access rather than general unfamiliar-context failure.Intervention tests found weaker unseen-query redirection after full-sequence continuation than after interleaved supervision.
  • Supervision and target selection: At 100M words, seven-metric averages were 43.896 with full supervision, 43.437 after deleting source-absent targets, and 42.792 after deleting matched copied targets.The comparison held the input fixed while changing supervised prediction targets, though deletion counts and target quantities were not identical.
  • Principle discovery: Stage II made visible information, supervised targets and weights, and preserved functions separately testable across natural-text and controlled reuse experiments.Together, the studies motivated coordinated input, supervision, and preservation design.

4 Stage III: Principle-Guided Model Improvement

Stage III converted findings about contextual information use into a training design that separately controls visible clues, supervised targets, and preservation. Across two continuation seeds, dense masking with sparse supervision and ordinary-input preservation improved complete nine-metric evaluation over ordinary continuation, while preservation effects involved measurable trade-offs.

  • 4.2 Separating Visible Inputs from Supervised Targets: The method retains source text, masks more local restatement clues, supervises selected targets, and constrains predictions on ordinarily masked inputs.These changes were applied to the first-generation model and its existing paired text.
  • 4.2 Separating Visible Inputs from Supervised Targets: Dense masking with sparse supervision separates missing local clues from the positions whose prediction errors update model parameters.The (M, S) and (M, M) comparison changes supervised targets while holding dense masking fixed.
  • 4.2 Separating Visible Inputs from Supervised Targets: Increasing target count did not improve Overall when the total focused-loss weight remained 0.15, because dense-target models scored below sparse-target models.This comparison changes both the target set and the relative weight received by each target.
  • 4.4 Complete Evaluation Across Two Continuation Seeds: GlobalPIQA contributed about 0.165 for both seeds, approximately 74% and 79% of the full method’s total gains, while other components partly offset improvements.The remaining eight components jointly contributed 0.0574 and 0.0428, so the net gain combined identifiable improvements and declines.
  • 4.3 Preserving Functions on Ordinary Inputs: Preservation limits change on ordinary inputs but must still allow improvement on correct targets, creating a trade-off between retaining prior behavior and learning the new task.Dense-input preservation sharply reduced model change while also suppressing improvement on correct targets; preservation configurations additionally varied in target positions and effective constraint strength.

5 Research Lineage and Independent Findings

The program produced independent research branches that turned model-building materials into controlled instruments for studying compression, relational identifiability, memory, addressing, and optimization. These studies supplied reusable methods and controls, while also exposing task-specific limitations and confounds.

  • Research Lineage and Independent Findings: Compact restatements and residual incremental learning became both model-improvement methods and experimental instruments for comparing expression relationships, objectives, and preservation.The shared model and material inheritance connected the research branches to the main investigation.
  • Compression and Coverage: Replacing child-directed/spoken material rather than adult prose lowered the four-metric score by 0.6125 points at 100M words despite identical added text.The respective changes from the reference were +0.0825 and +0.6950 on BLiMP, Supplement, EWoK, and COMPS.
  • Relational Anchors, Shared Representations, and Identifiability: Sparse anchors resolve binary-label orientation ambiguity, and shared representations provide a testable condition for propagating anchor information.Separate representations fit some local tasks but did not show the same propagation.
  • Entity Storage, Retrieval, and Semantic Addressing: Identity-interface supervision raised matching accuracy from 0 to 1 and restored relational responses for events without direct supervision.The experiments treated associating input names with the correct objects as a prerequisite for using learned relations.
  • Entity Storage, Retrieval, and Semantic Addressing: Adding retrieved content through h+gP(m) raised paired accuracy to 0.1375 and the slot-swap effect to 3.087, although connection and training changes were jointly modified.The reported gains therefore do not isolate the gate’s contribution.
  • Controls and Measurement: Task inspection withdrew an apparent compositional-binding result because a constant answer or first-slot shortcut could achieve high accuracy.The storage, addressing, and selection results came from separately completed experiments.
  • Optimization and Measurement: Optimization interventions sometimes improved short-run loss or internal measurements without establishing stable full-task gains, so aligned updates remain a hypothesis rather than a capability measure.The independent studies distinguish local measurements from task outcomes.

6 Discussion

The discussion links contextual dependencies, capability reuse, and training design, showing how findings from controlled studies informed a method that improves complete evaluation. It also frames the program as Research RSI, while limiting the evidence to a recursive cycle within this BabyLM program.

  • 6 Discussion: The proposed design organizes experience around contextual dependencies and separately controls visible information, supervision, and preservation.Ordinary-input preservation retained most of the new prediction gain while reducing loss changes under ordinary masking.
  • 6 Discussion: Familiar accuracy can diverge from unseen-symbol relation processing, so evaluation must test whether new inputs can access learned computations.Changing supervision and internal interventions altered unseen-symbol behavior.
  • 6 Discussion: The research program recursively reused methods, implementations, and explanations, allowing later experiments to revise earlier accounts and guide model construction.Compact restatements, residual increments, and preservation studies each served both practical and scientific purposes.
  • 6 Discussion: The evidence supports a recursive research cycle within this BabyLM program, but not sustained improvement of general research ability across tasks.Broader claims would require independent goals and comparisons of knowledge reuse, research cost, experiment selection, and outcome quality.
  • 6 Discussion: Two continuation seeds from one parent improved complete evaluation under the full method, while component analyses separated contributions from GlobalPIQA, Entity Tracking, and preservation.GlobalPIQA supplied about 74% and 79% of the full methods’ Overall gains relative to the parent; Entity Tracking supplied the largest positive contribution relative to ordinary continuation.

7 Conclusion

The conclusion presents three connected stages that investigate contextual learning, capability reuse, and principle-guided improvement under restricted data. The resulting method exceeded ordinary continuation on complete evaluation, while the program exemplified Research RSI within this BabyLM study.

  • 7 Conclusion: The study found that repetition and aligned restatement shape context use differently, depending on target relations and prediction windows.Recovering familiar performance did not guarantee that unseen inputs could use a learned computation.
  • 7 Conclusion: The resulting principle treats visible information, supervision, and preservation as separate design choices and evaluates new and existing capabilities together.It organizes experience around the contextual dependencies required by the target.
  • 7 Conclusion: 42.02 to 42.25 Overall marked the improvement across the two public representative model generations.The second-generation training method exceeded ordinary continuation under both continuation seeds from a shared parent.
  • 7 Conclusion: Research RSI describes how accumulated understanding, methods, and experimental experience changed later questions and designs, which were then tested and revised.The immediate object of improvement was the research process rather than the agent’s parameters or code.
  • 7 Conclusion: The repository preserves models, code, data construction, complete results, and scientific notes for further testing and questions.The report presents the model generations, learning principle, experimental methods, and independent findings as a reusable research foundation.

A.1 Computing environment and experimental support

The study used documented computational support for text processing, training, evaluation, and mechanism experiments. Release packages also record a CPU validation environment, while original runs retained their own program and dependency records.

  • A.1 Computing environment and experimental support: Two NVIDIA H100 GPUs supported model training, evaluation, teacher-generated text, and accelerated mechanism experiments.CPUs supported text processing, tokenization, data statistics, and result analysis.
  • A.1 Computing environment and experimental support: Table 19 documents the computing platform, with GPU use recorded in research records and CPU, memory, and operating-system specifications checked on 9 September 2026.The table provides the study’s computing-environment record.
  • A.1 Computing environment and experimental support: The released model packages record Python 3.12, PyTorch 2.11.0, CUDA 12.8, Transformers 4.57.6, Tokenizers 0.22.2, and Safetensors 0.8.0 for CPU validation.These dependency records describe release validation rather than every original training, generation, or evaluation run.

A.2 Model versions, teachers, and data sources

The model versions share a fixed architecture and data-processing framework while differing in continuation branches, target selection, masking, loss normalization, and preservation design. Stage III combines focused supervision on annotated source–restatement rows with ordinary training and preservation presentations.

  • Model versions: Stage III updates an existing incremental branch while complete-model evaluation loads that branch with the base network.Model identity includes weights, architecture configuration, tokenizer, and loading implementation.
  • Model versions: The shared architecture uses eight DeBERTa-v2-style layers, hidden size 480, eight attention heads, feed-forward size 1,920, and a 16,384-tokenizer vocabulary.Sequences contain up to 256 tokens, with AdamW and a residual branch of bottleneck size 128.
  • Teachers and data sources: The corpus includes 37,594 source–restatement pairs totaling 1,656,800 words and 12,155 compact pairs totaling 423,511 words within the ten-million-word corpus.Stage III input composition contains 20,475 packed rows and 3,162,742 words; rows can contain several paired segments.
  • Acquisition and preservation: Relation-focused rows use dense masking over Qm but supervise only Qf, whereas other rows receive ordinary whole-word masking.Qf is a subset of Qm; focused and ordinary losses are normalized by their respective target-token counts before combination.
  • Acquisition and preservation: Content groups are selected lexically from token-aligned spans, with at most 16 groups sampled sparsely per row and dense masking selecting up to 128 additional groups by default.If candidates exist but none is sampled sparsely, one group is retained; supervised groups are added to the dense set.
  • Optimization: Only 48 designated incremental parameter tensors are trainable during the complete update, which accumulates complete rows by prescribed word counts.Variable row lengths make row and token counts per update variable, and the learning-rate offset continues the existing schedule.

A.4 Evaluation aggregation, randomness, and trajectories

Evaluation uses a common nine-metric aggregation and records scorer, loading, and checkpoint configurations explicitly. The report distinguishes local scoring procedures, uncertainty sources, and shared-versus-endpoint trajectory checkpoints.

  • Aggregation: Complete-model interfaces generate outputs for masked prediction, Reading, downstream fine-tuning, and AoA before aggregation under a common nine-metric procedure.Four decimal places distinguish close local scores, while uncertainty is assessed separately for each measurement.
  • Aggregation: The evaluation configuration jointly depends on scorer version, model loading, and checkpoint lists, with the official evaluator and AoA revision recorded.The repository retains the scoring programs and loading adaptations used for the complete incremental model.
  • Scoring records: BLiMP local selected-index scoring gives 68.51 and 68.26 for the two generations, while released-answer-text matching gives 68.52 and 68.27.Seven items with identical answer text account for the differences between procedures.
  • Scoring records: Macro-averaged task scores and net item changes can move in opposite directions because subtasks have unequal sizes.Nonaccuracy measures such as Reading are excluded from item counts.
  • Uncertainty: Uncertainty is matched to the sampling unit: seeds for relation experiments, text-pair resampling for source-isolated losses, and two continuations from one parent for final-model results.Paired item-level intervals for GlobalPIQA are not provided.
  • Trajectories: AoA trajectories share 17 earlier checkpoints and add each model’s own endpoint to form an 18-point trajectory.The shared checkpoints span 1M–10M words individually and then every ten million from 20M to 80M.

B Complete local evaluation vectors

The complete local evaluation reports nine-metric vectors and Overall values for a shared parent and continuation models, while distinguishing sparse, dense, and preservation-inclusive strategies. GlobalPIQA remains unchanged between sparse supervision and the complete strategy, so Overall differences come from other metrics.

  • Evaluation vectors: Tables 22 and 23 report local task metrics and remaining metrics with complete Overall for the shared parent and continuation models.Table 23 displays values to four decimal places, while source tables retain full precision.
  • Evaluation vectors: The complete strategy adds ordinary-input preservation to sparse supervision, whereas dense supervision denotes the (M, M) configuration.Quick (S, S) measurements are excluded from complete nine-metric evaluations.
  • Evaluation vectors: GlobalPIQA is unchanged between (M, S) and the complete strategy, so their Overall difference reflects net changes in other metrics.The Overall difference does not represent simultaneous improvement in every metric.
  • Evaluation vectors: Table 24 isolates principal Overall differences and identifies additional preservation presentations as part of the complete strategy.The table compares the same nine-model evaluation set described for the complete vectors.

C Additional controls and mechanism results

Additional controls test downstream variability, source-isolation effects, generalization claims, and forward-pass consistency. These analyses narrow interpretation where overlap or sampling limitations affect the evidence.

  • Downstream controls: The complete strategy exceeds both the parent and (M, S) under each of two downstream fine-tuning seeds with continuation seed 62064 fixed.The supplementary seed 44 comparison measures downstream-training variability and is not combined into a new Overall.
  • Mechanism controls: At 20M exposure, removing source-absent targets produces higher loss on source-absent content words than removing whole-word-matched source-present targets under pair isolation.The stricter document-isolation intervals include zero, separating the observed pair-isolated effect from the stricter condition.
  • Mechanism controls: The seven-task deletion experiment at 100M words holds compact inputs fixed while removing supervision from 48,105 and 48,387 BPE tokens under two conditions.GlobalPIQA is abbreviated as GP in the corresponding table.
  • Measurement controls: A later overlap check found 6,992 rows shared with the historical stream, so affected measurements are no longer treated as clean evidence of generalization preservation.The report narrows claims by identifying the relevant population rather than invalidating experiments on other populations automatically.
  • Measurement controls: Preservation calls must save and restore the acquisition branch’s random-number state so KL self-comparisons use consistent forward-pass randomness.Shared-initialization, query-dependence, overlap, and forward-consistency checks address different confounds.

D Public leaderboard metrics

The report evaluates models through nine complementary metrics and documents the methods, results, and research materials needed to interpret them. Its catalog records both positive findings and conditional or exploratory outcomes across training choices.

  • D Public leaderboard metrics: The public leaderboard comparison covers language form, knowledge, entity tracking, composition, reading, and human-like measures.Tables 28 and 29 present the public submissions across these metric groups; Reading and AoA are not ordinary task accuracies.
  • D Public leaderboard metrics: Similar Overall scores can conceal different strengths, so aggregate and component metrics must be read together.Overall records combined improvement, while individual metrics reveal sources and trade-offs across capabilities.
  • D Public leaderboard metrics: The repository connects models, configurations, data construction, complete evaluations, research notes, and report-build files for further testing.Hugging Face provides model packages, while GitHub preserves implementations and research records; report-building commands do not run experiments or external models.
  • D Public leaderboard metrics: The catalog reports that structural, tokenization, masking, architecture, recursive-training, vocabulary, and representation changes do not yield a uniform broad advantage.Several entries are conditional or exploratory, with gains depending on controls, seeds, budget accounting, or task trade-offs.
  • D Public leaderboard metrics: Local structural readouts, auxiliary objectives, and representation fits do not by themselves guarantee broad capability gains or held-out transfer.The reported controls distinguish local recovery from stable generalization and caution against broad mechanistic conclusions.

Core research: extensions and empirical tests

The empirical program tests how repetition, restatement, prediction windows, supervision, and preservation shape learning and functional access. Its final complete preservation scheme improves the nine-task Overall score for both continuation seeds.

  • Core research: extensions and empirical tests: Repetition and aligned reformulation affect compact reformulation targets differently, and neither effect is universal across target conditions.The study compares source advantages across three seeds and reports direction differences by target class.
  • Core research: extensions and empirical tests: Separating repetition or reformulation pairs into different windows substantially attenuates their source effects despite matched content.The split-window controls retain each condition’s complete material while removing shared context for a single prediction.
  • Core research: extensions and empirical tests: Aligned restatement improves source use for recurring target tokens, whereas nonrecurring targets show no equally stable improvement over the reference.The distinction is lexical and is based on whether the exact tokenizer ID appears in the source.
  • Core research: extensions and empirical tests: Recovering familiar behavior does not guarantee restored functional reach for unseen symbols.The controlled comparison tests ordinary continuation, statically weighted supervision, and interleaved supervision.
  • Core research: extensions and empirical tests: Dense inputs with separately controlled masking candidates and focused targets outperform ordinary continuation at equal exposure, but masking is not isolated as a single factor.Complete evaluations support the comparison for both seeds.
  • Core research: extensions and empirical tests: The complete preservation scheme improved the nine-task Overall score for both continuation seeds.The comparison rechecked ordinary continuation, relation learning, and the complete preservation scheme from the same parent model.
Loading 2609.10702v1…