PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Transfer from Controlled Pretraining Mixtures to Code

Ohad Rubin

arXiv:2610.11548v1cs.LG

TL;DR

Loss reduction is often used to judge both whether synthetic tasks reveal model capability and whether more samples are worthwhile. The paper separates diagnosticity, teachability, and transfer in controlled pretraining mixtures; transfer peaks with a target-aligned source dominant, while loss-based scheduling underperforms fixed-mixture controls.

  • Problem

    Falling task loss is often treated as both evidence about model capability and a reason to sample more, although these diagnostic and economic questions can diverge.

    Source: §1 · p. 1

  • Method

    The study holds 70% general Python fixed, varies a 30% slice across three source families, and separately measures diagnosticity, teachability, and transfer.

    Source: §1 · p. 2, §4.4 · p. 5

  • Results

    HumanEval pass@20 peaks at 22.6 with OpenCodeInstruct at 75% of the slice; Ado drives its share below 5% and trails matched fixed controls by 2.4–11.0 pass@20 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For code generation, keep the target-aligned source dominant, use curated synthetic data as a bounded share, and choose proportions with a target-referenced signal.

    Source: §8 · p. 10

    The experiment estimates source-family value, but format-specific and content-specific effects remain entangled.

    Source: §9 · p. 11

Abstract

from arXiv · show

Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data. Both uses are often justified by loss reduction: falling loss is treated as informative, and faster loss reduction with more sampling as evidence that a task is worth sampling. We separate three signals. A task is diagnostic when its loss tracks global pretraining progress; it is teachable when its loss responds to its own token budget; and a data source transfers when including it improves a downstream target. We study controlled pretraining in which 70% of the corpus is fixed general Python and the remaining 30% is a simplex over three source families: OpenCodeInstruct, a curated suite of 12 code-adjacent synthetic tasks, and 15 literature-derived probe tasks. Across a task-budget sweep we detect teachability for 14 of 27 tasks, with a sharp asymmetry between the two synthetic families (10/12 curated versus 4/15 literature-derived). Teachability and downstream transfer give different rankings. On the mixture-simplex edge between the curated suite and OpenCodeInstruct, HumanEval pass@20 after a fixed fine-tuning stage rises from 15.9 at pure curated data to its highest observed value, 22.6, at a mixture that is 75% OpenCodeInstruct, then falls to 19.5 at pure OpenCodeInstruct. Curated synthetic data therefore has conditional value: it contributes as a limited share of a mixture that a target-aligned source still dominates. Finally, a loss-based adaptive scheduler exposes the mismatch between residual loss reducibility and downstream transfer. Across three 60k-step free-ratio runs, Ado drives the OpenCodeInstruct share below 5% within the first 5k steps and to 1.2--1.4% by the end of training, and underperforms its matched fixed-mixture controls by 2.4--11.0 percentage points. Optimizing near-term task-loss reduction moves the mixture away from the region that transfers.

1 Introduction

Loss reduction can indicate different things: a task may track general progress, respond to its own exposure, or transfer to a downstream target. In controlled pretraining, teachability and transfer diverge, and loss-based scheduling underperforms fixed-mixture controls.

1 Introduction

  • The study fixes 70% of pretraining to general Python and varies the remaining 30% across OpenCodeInstruct and two synthetic-task families.The controlled setup holds the base corpus, model family, training budget, and fine-tuning stage fixed.Source: §1 · p. 2
  • The paper operationalizes diagnosticity as loss tracking global progress, teachability as response to task-specific token budget, and transfer as a source share improving the downstream target.It also introduces an exposure-advantage diagnostic comparing task-specific exposure with global pretraining step.Source: §1 · p. 2
  • Teachability is detected for 14 of 27 tasks, with 10 of 12 curated tasks versus 4 of 15 literature-derived probes responding to their own token budgets.The asymmetry makes teachability family-dependent, not a guarantee of downstream transfer.Source: §1 · p. 2
  • HumanEval pass@20 peaks at 22.6 with OpenCodeInstruct holding 75% of the slice, while pure curated data scores 15.9, below the zero-synthetic baseline.The curated suite’s contribution is conditional on mixture context rather than a pure-source advantage.Source: §1 · p. 2
  • Ado drives OpenCodeInstruct below 5% within 5k steps and to 1.2–1.4% by training’s end, trailing matched fixed controls by 2.4–11.0 pass@20 points.Ado allocates budget by predicted remaining loss reduction.Source: §1 · p. 2
  • For code generation, the proposed rule is to keep the target-aligned source dominant, add curated synthetic data as a bounded share, and choose proportions with a target-referenced signal.This follows the distinction between a steadily falling loss and evidence that added task examples improve the downstream target.Source: §1 · pp. 1–2

2 Related Work

Related work covers data-mixture optimization, synthetic code data, controlled probes, and code evaluation. The paper distinguishes its controlled comparison of loss reducibility from downstream transfer.

2 Related Work

  • Prior mixture-selection work learns weights with proxy-model group-DRO, frames selection as prediction, or reweights online; Ado samples by predicted remaining loss reduction.The paper’s contribution is a controlled demonstration that this reducibility signal and downstream transfer can point in opposite directions.Source: §2 · p. 2
  • OpenCodeInstruct is treated as aligned instruction data, distinct from abstract synthetic task families, while probe training leaves the effects of added exposure and downstream transfer unestablished.The cited probe literature includes retrieval, state tracking, composition, and transduction tasks.Source: §2 · pp. 2–3
  • HumanEval is the downstream endpoint, evaluated with the unbiased pass@k estimator of Chen et al. [2].Source: §2 · p. 3

3 Experimental Setup

The experiments hold general Python fixed at 70% of pretraining and vary the remaining 30% across three source families. They compare fixed mixtures through task-budget sweeps and HumanEval evaluation after a common fine-tuning stage.

3.1 Base corpus and controlled slice

  • Each run fixes 70% of tokens to general Python and allocates the controlled 30% slice across OpenCodeInstruct, curated tasks, and literature probes.The slice weights sum to one, so increasing one source displaces the others.Source: §3.1 · p. 3

3.2 Source families

  • OpenCodeInstruct is target-aligned instruction data; the curated suite has 12 code-adjacent synthetic tasks, and the literature suite has 15 standard-probe tasks.Controlled pretraining and fine-tuning both use the same refined OpenCodeInstruct split.Source: §3.2 · p. 3

3.3 Model and training

  • The fixed-mixture sweep tests 15 mixtures along pairwise source edges in 25-point steps, with each synthetic task appearing in nine runs and a matched CodeParrot-only baseline.Runs use a sparse-expert decoder-only model and last 60k or 120k steps.Source: §3.3 · pp. 3–4

3.4 Downstream evaluation protocol

  • Every run receives the same 1,000-update OpenCodeInstruct fine-tuning stage before HumanEval evaluation on 164 problems with 20 sampled completions per problem.Best-over-steps selects the highest score among checkpoints 100 through 1,000; pass@5 and pass@10 accompany pass@20.Source: §3.4 · p. 4

4 Three Signals

The paper distinguishes task loss as a readout of global progress, as a response to task-specific exposure, and a source’s contribution to a downstream endpoint. It measures these signals separately, emphasizing that transfer depends on mixture context.

4.1 Notation

  • The analysis tracks held-out loss, cumulative task exposure, and downstream HumanEval pass@20 to distinguish task-level signals from source-level transfer.The task-analysis window runs from pretraining steps 5,000 to 55,000.Source: §4.1 · p. 4

4.2 Diagnosticity

  • Diagnosticity is how well a task’s held-out loss tracks global pretraining progress, summarized by pooled R2 from regressing logged validation loss on step.This measures how well a task reads out progress, not the effect of its own examples.Source: §4.2 · p. 4

4.3 Teachability

  • Teachability asks whether held-out loss responds to a task’s own token budget, tested across nine fixed-mixture runs with a median fourfold exposure range.The slope reports learning per doubling of exposure; detections use a one-sided permutation test with 2,000 exposure shuffles and are also reported after Benjamini–Hochberg adjustment.Source: §4.3 · p. 5
  • Diagnosticity and teachability are logically independent; seven tasks have positive exposure advantage, and five of the six largest values are curated tasks.Exposure advantage compares how well cumulative task exposure versus global step explains logged validation loss.Source: §4.3 · p. 5

4.4 Transfer, and why it is conditional

  • Transfer is defined locally as a source-share increase that improves the downstream endpoint relative to a feasible mixture, because the simplex forces other shares to fall.The paper estimates transfer from differences between completed mixtures on pairwise edges and across the mixture surface.Source: §4.4 · p. 5

4.5 The proxy-objective gap

  • Ado allocates more sampling to tasks with greater predicted near-term loss reduction, a task-level signal whose objective contains only pretraining loss.Its observed mixture moves away from the region preferred by the downstream surface for most of the 60k-step training horizon.Source: §4.5 · p. 5

5 Task-Level Results

Task-budget response varies across the synthetic tasks, with curated tasks more often teachable than literature-derived probes. Exposure advantage provides a distinct view of whether falling task loss reflects task-specific exposure rather than global progress.

Task-budget results

  • Figure 1 plots explained across-run variation against emitted tokens per example; filled markers clear a shuffled-budget test, while the shaded band shows shuffled-budget fits.Task exposure varies across the nine fixed-mixture runs, with a median fourfold range.Source: §5 · p. 5, Figure 1 · p. 6
  • 14 of 27 tasks clear the permutation test for teachability: 10 of 12 curated tasks and 4 of 15 literature-derived tasks; adjustment retains 6 and 2, respectively.The family difference survives Benjamini–Hochberg adjustment.Source: §5 · p. 6
  • Task length only weakly relates to explained variance (Pearson r = 0.34, p = 0.08; Spearman ρ = 0.27, p = 0.18), and the reported extremes contradict a simple length explanation.Copy and reverse sequence emit 69.3 tokens per example with R2 = 0.02, while one-token counting and year-span comparison tasks respond strongly.Source: §5 · p. 6

Exposure advantage

  • Seven tasks have positive exposure advantage, five of the six largest are curated, and seven of the ten lowest are literature-derived.The analysis notes that the run-set associations reflect a mixture sweep, so task budgets move with surrounding source composition.Source: §5 · p. 6

6 Mixture-Level Transfer

HumanEval transfer on the curated–OpenCodeInstruct edge peaks at an interior mixture with OpenCodeInstruct dominant, rather than at either endpoint. The result supports a bounded curated share in this setting, while the proposed complementarity mechanism remains a hypothesis.

6.1 The curated–OpenCodeInstruct edge

  • 15.9 → 18.9 → 20.7 → 22.6 → 19.5 pass@20 across the curated-to-OpenCodeInstruct edge, peaking at 75% OpenCodeInstruct and 25% curated.The best observed point exceeds pure OpenCodeInstruct by 3.1 percentage points.Source: Figure 2 · p. 7, §6.1 · pp. 7–8
  • At 25% curated, pass@20 is 3.1 points above pure OpenCodeInstruct; at 100% curated, it is 3.6 points below that endpoint and 0.6 below the zero-synthetic baseline.The ranking is consistent at pass@5, pass@10, and pass@20.Source: §6.1 · p. 8
  • Bootstrap intervals support an OpenCodeInstruct-dominant region with a limited curated share, but a denser grid is needed to identify a precise optimum.The intervals resample 164 problems 50,000 times and repeat Best-over-steps selection in each replicate.Source: §6.1 · p. 8

6.2 Interpretation: capability versus alignment

  • The proposed complementarity is that curated tasks exercise reusable program-like operations while OpenCodeInstruct supplies instruction form close to the adaptation bridge and downstream target.The paper explicitly treats this mechanism as a hypothesis consistent with the data.Source: §6.2 · p. 8

6.3 Off-edge behavior and interaction structure

  • The central edge claim is scoped to five curated–OpenCodeInstruct mixtures; Figure 2 also presents the full surface and lower-k endpoints.Its edge panels make pairwise tradeoffs visible at three values of k.Source: Figure 2 · p. 7, §6.3 · p. 8

6.4 Teachability and transfer rank sources differently

  • The curated suite has the highest teachability rate (10/12), yet its pure-source endpoint is 15.9%, below pure OpenCodeInstruct and the zero-synthetic baseline.In this setup, teachability does not order the observed downstream endpoints and is insufficient as the sole mixture objective.Source: §6.4 · p. 8

7 Adaptive Scheduling Optimizes Reducibility and Loses Transfer

The section tests whether loss-based adaptive scheduling preserves transfer-favorable source mixtures. Free-ratio Ado rapidly reallocates away from OpenCodeInstruct, and its downstream results trail matched fixed-mixture controls.

7 Adaptive Scheduling Optimizes Reducibility and Loses Transfer

  • Ado fits recent loss measurements, predicts near-term loss reduction, and gives greater sampling probability to domains with stronger predicted reductions.Free-ratio scheduling also changes source weights, moving the run across the mixture simplex.Source: §7 · p. 8

7.1 Setup

  • Each 60k-step profile compares a fixed shadow run, preserved-ratio adaptation, and free-ratio adaptation; a 120k branch tests allocation beyond the downstream-comparison horizon.Shadow mode records scheduler proposals while training on the fixed source mixture.Source: §7.1 · p. 9

7.2 The OpenCodeInstruct share collapses

  • Across all three 60k-step free-ratio runs, OpenCodeInstruct falls below 5% of the controlled slice within 5k steps and ends at 1.2%–1.4%.In the 120k branches, the scheduler begins moving mass back around step 90k.Source: §7.2 · p. 9, Figure 3 · p. 9

7.3 Results

  • Free-ratio Ado trails matched shadow controls by 2.4, 11.0, and 4.9 percentage points; its pass@20 endpoints are 9.1%–16.5%, below the 22.6% best-observed fixed mixture.Preserved-ratio Ado reaches 17.1%–18.9% pass@20 while keeping source budgets fixed.Source: §7.3 · p. 10, Table 2 · p. 10

7.4 Reading

  • Ado optimizes predicted near-term loss reduction, whereas fine-tuned HumanEval measures transfer; in this setup, OpenCodeInstruct scores low on the former early and high on the latter.The 1,000-step fitting horizon delays OpenCodeInstruct's appeal under Ado's own criterion until roughly 88k steps.Source: §7.4 · p. 10

8 Discussion

The discussion distinguishes diagnosticity, teachability, and transfer as separate questions requiring different measurements. It also characterizes curated synthetic data as useful at a bounded share, while literature-derived probes can remain controlled readouts despite weak evidence of marginal learning value.

8 Discussion

  • Diagnosticity tracks global progress, teachability tests response to task-specific exposure, and transfer measures whether changing a source share improves the post-fine-tuning endpoint.Each question requires its own intervention or comparison.Source: §8 · p. 10
  • At a 25% slice share, curated data is worth 3.1 pass@20 points over pure OpenCodeInstruct, while its 100% share reaches 15.9%, below the 16.5% zero-synthetic baseline.The observed bound reflects how much target-aligned data the curated source displaces.Source: §8 · p. 10
  • Four of fifteen literature-derived probes respond to their own budget, while seven of the ten lowest exposure advantages come from that family.Such probes can still provide controlled readouts of retrieval, state tracking, composition, and transduction when evidence of marginal learning value is weak.Source: §8 · pp. 10–11

9 Limitations •

The reported mixture findings are bounded by the tested model, data, evaluation target, grid resolution, and scheduler. The task-budget and checkpoint estimates also have specific design constraints.

9 Limitations •

  • The main comparisons use a sparse-expert decoder with CodeParrot as the fixed 70% source; other architectures, scales, or base corpora may yield a different mixture surface.Source: §9 · p. 11
  • OpenCodeInstruct's aligned instruction responses and the suites' abstract machine-generated problems change format and content jointly, leaving their effects entangled.Source: §9 · p. 11
  • Downstream transfer is evaluated on HumanEval only; transfer to other code benchmarks or general reasoning tasks remains unmeasured.Source: §9 · p. 11
  • The simplex is sampled at 15 points, so the best observed point, (0.75, 0.25, 0), is not an estimated argmax; that requires a denser grid.Source: §9 · p. 11
  • Only Ado is tested; other adaptive objectives and longer planning horizons may produce different allocation trajectories.Source: §9 · p. 11
  • Each task has nine run points and its budget varies with the surrounding mixture, so the permutation test estimates association across the completed grid rather than an isolated task effect.A task-isolated intervention remains future work.Source: §9 · p. 11
  • Selecting the best checkpoint over steps is optimistic by 0.2–1.6 percentage points across runs; bootstrap repeats selection to represent uncertainty, with the endpoint estimating attainable performance under checkpoint selection.Source: §9 · p. 11

10 Conclusion

The conclusion separates teachability from transfer: a highly teachable curated suite helps at a limited share of a mixture dominated by OpenCodeInstruct, while its pure-source endpoint falls below the zero-synthetic baseline. The appendices describe the task suites, task-level analysis, and training setup underlying these findings.

10 Conclusion

  • Under controlled pretraining where only a 30% slice varied, the best observed HumanEval pass@20 came from a mixture with 75% target-aligned source and 25% curated synthetic data; the suite alone fell below the zero-synthetic baseline.Source: §10 · p. 11

A Task Suite

  • The controlled suite comprises 15 literature-derived probe clusters and 12 curated synthetic task clusters, alongside aligned OpenCodeInstruct data.Each cluster combines difficulty variants with equal problem weights before source-level aggregation.Source: A Task Suite · p. 13

A.1 Literature-derived task clusters

  • Literature-derived clusters probe transduction, global readout, retrieval, stack state, composition, long-context retrieval, and mutable-state tracking.Examples include copy and reverse sequences, associative recall, Dyck completion, ListOps, passkey retrieval, variable tracing, SCAN, and bAbI tasks.Source: §A.1 · p. 13

A.2 Curated task clusters

  • Curated clusters cover program execution, reasoning, planning, rewriting, ranking, search, numerical comparison, and in-context binding.Examples include bounded Python execution, fictional-ontology reasoning, gridworld planning, Horn chaining, Markov rewriting, DPLL traces, TF-IDF ranking, and tree search.Source: §A.2 · pp. 13–14

B Task-Level Analysis Details

  • Task-level analysis measures loss change against task exposure, fits loss fall across exposure budgets, and calibrates detections by within-task exposure permutations.Exposure advantage compares the R2 for task exposure with the R2 for global pretraining step.Source: B Task-Level Analysis Details · p. 15
  • Curated tasks emit a median 10.2 supervised tokens per example versus 1.0 for literature probes, while emitted length and explained variance are weakly related.Across tasks, Pearson r = 0.34 (p = 0.08) and Spearman ρ = 0.27 (p = 0.18).Source: B Task-Level Analysis Details · p. 15

C Training Hyperparameters

  • The experiments use a 70/30 CodeParrot-controlled-data split, fixed 262,144-token updates, and 60k- or 120k-step runs.The four-, eight-, and sixteen-expert models share a 66.947M-parameter backbone with top-2 routing.Source: C Training Hyperparameters · pp. 15–16

D ADO Configuration

  • Ado reallocates the controlled 30% slice; free-ratio runs can also shift mass across the three source families.The base fraction stays fixed, and Ado updates its allocation every 1,000 steps using answer-token loss.Source: D ADO Configuration · p. 16

E Evaluation Details

Evaluation fine-tunes every pretrained checkpoint on the same OpenCodeInstruct split, then measures HumanEval pass@k across checkpoints. The appendices specify checkpoint selection and uncertainty estimates, task construction, and where the reported quantities appear.

E Evaluation Details

  • Each pretrained checkpoint receives the same 1,000-update fine-tuning stage on the refined OpenCodeInstruct training split before downstream evaluation.Examples are shuffled and first-fit packed, with batch size 256, cosine decay, and peak learning rate 10^-5.Source: E Evaluation Details · p. 16
  • HumanEval evaluation samples 20 completions per problem, executes them in isolation, and counts a completion as correct when it passes the corresponding tests.The test split contains 164 problems, and evaluation uses a 2,048-token context with at most 512 generated tokens.Source: E Evaluation Details · p. 16
  • Best over steps selects the highest pass@k across checkpoints 100–1,000, inflating the endpoint by 0.2–1.6 percentage points depending on the run.Bootstrap replicates repeat checkpoint selection; paired mixture-edge intervals resample the 164 problems 50,000 times.Source: E Evaluation Details · p. 16

F Curated Task Construction Prompt

  • Curated task clusters were elicited one at a time with a prompt requesting a self-contained description of an automatically sampleable task.The prompt asks for enough detail that someone could implement the task from its description.Source: F Curated Task Construction Prompt · p. 17

G Provenance of Reported Numbers

  • The provenance map assigns Table 1 to five curated–OpenCodeInstruct pass@20 values, Figure 2 to the 15-mixture surface, Figure 1 to budget detections, and Table 2 to scheduler endpoints and shares.Figure 3 shows scheduler trajectories; evaluation selection and interval conventions are specified in Appendix E.Source: G Provenance of Reported Numbers · p. 18
Loading 2610.11548v1…