PaperlayerView exact revision on arXiv

Source-linked AI summary

Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training

Yunpeng Xu, Kun Zheng

arXiv:2609.09081v1cs.AI

TL;DR

The paper asks whether per-domain mid-training coverage choices create differences that later alignment can undo. Across controlled multi-domain allocations and fixed-budget post-training procedures, domain gaps persist, coverage has interior optima, and zero coverage can cause large checkpoint-specific collapses, though some effects are confounded by generic drift.

  • Problem

    Per-domain mid-training data composition is typically determined by availability rather than principled design, raising whether later SFT or RL can reduce coverage-associated differences.

  • Method

    The study evaluates 30 allocations across a five-domain simplex, including six withheld configurations, while holding downstream procedures fixed and measuring token-share coverage.

  • Results

    Coverage gaps survive tested alignment: compensatory SFT bridges 0/240 pairs at the 5 pp threshold, every domain has a moderate-band optimum, and zero coverage collapses mid-training-only accuracy.

  • Takeaways & Limitations

    Within this setting, budget placement—not only budget size—preserves coverage-induced gaps, while moderate coverage appears better than either extreme for each domain.

  • Takeaways & Limitations

    Per-domain coverage is a zero-sum mixture-level marginal rather than an independently manipulable dose, and zero-coverage effects are co-mingled with generic distributional drift.

Abstract

from arXiv · show

Mid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it. In a controlled logical-reasoning setting (Qwen3-8B-Base, with a 4B replication; five semantically rule-disjoint KOR-Bench domains) we train 30 allocations spanning the five-domain simplex, 24 sweep configurations plus six withheld from the fit, at five seeds each. Three findings emerge. First, every domain has an interior coverage optimum: the moderate band ($10\%$-$40\%$) is best for all five domains, and a calibrated permutation test for quadratic interiority gives $P\approx0.010$; the fitted mid-training-only curves, with 8B peaks between $9.9\%$ and $35.1\%$, reproduce for curve shape but not peak location. Second, the gaps survive a fixed-budget alignment pass: compensatory SFT raises 116/120 cells (mean $+4.32\%$) yet bridges $0/240$ pairs at a $5\%$ threshold and $30/240$ at a $10\%$ ratio, an equal-budget uniform control behaves almost identically, and a permutation null would bridge $13.8\pm3.3$ and $77.9\pm8.5$ pairs ($P<0.001$). Third, zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is commingled with generic drift. An exploratory $θ^*$ allocation attains the largest full-pipeline gain ($+4.36\%$ vs. $+0.80\%$/$+0.64\%$\,pp) but is marginal under Welch test.

1 Introduction

This study tests whether principled mid-training coverage choices shape domain performance and whether fixed downstream alignment can reduce the resulting gaps. Across controlled five-domain experiments, coverage produces interior optima, while compensatory SFT largely preserves the gaps and zero coverage can cause severe mid-training-only degradation.

  • Alignment and gap closure: 116/120 configuration–domain cells improve after compensatory SFT, yet it bridges 0/240 pairwise gaps at 5 pp and only 30/240 at the 10% ratio.An equal-budget uniform-SFT control behaves almost identically, while the permutation null expects 13.8 ± 3.3 and 77.9 ± 8.5 bridged pairs, respectively.
  • Zero-coverage effects: Zero mid-training coverage collapses mid-training-only accuracy, but the FineWeb-Edu-only control shows that the collapse is co-mingled with generic distributional drift.The tested downstream recipe partly repairs the effect, so the paper does not attribute the collapse specifically to coverage starvation.
  • Interior coverage optima: Moderate coverage between 10% and 40% is best for all five domains, with calibrated quadratic-interiority P ≈0.010.Fitted 8B mid-training-only peaks range from 9.9% to 35.1%, while held-out allocations reproduce curve shapes but not peak locations.
  • Study question and design: 24 sweep configurations plus six held-out allocations test five-domain coverage mixtures while holding the downstream mid-training→SFT→RL recipe fixed.The study also carries three allocations through the full pipeline and applies compensatory versus uniform SFT to all 24 checkpoints.

2 Methodology

The paper studies a fixed multi-stage pipeline in which mid-training produces a checkpoint that initializes SFT and then GSPO-based RL. Coverage is measured as token allocation within a fixed-budget mixture, with domain-specific deltas quantifying downstream effects.

  • Pipeline: The fixed pipeline proceeds from mid-training to SFT and then reinforcement learning, with later stages refining the preceding model state.The study focuses on how the mid-training distribution shapes later behavior rather than evaluating the recipe itself.
  • Training stages: Mid-training applies continued causal language modeling to Dmid, producing checkpoint θmid for subsequent stages.SFT starts from θmid in the mid-training condition and from θ0 in the no-mid-training baseline.
  • Training stages: SFT uses conditional next-token likelihood, followed by GSPO with sequence-level group optimization and binary verifier rewards.The reward is 1 for a correct answer and 0 otherwise.
  • Coverage quantification: Coverage is the fraction of total mid-training tokens assigned to a domain, rather than a measure of unique examples, data quality, task difficulty, or an independently manipulable dose.The fixed total-token mixture includes an internal five-domain component and a fixed external component.
  • Coverage quantification: The study tracks Δmid(d) as the net mid-training effect and Δrl(d) as the RL contribution.Both compare the full mid-training+SFT+RL result with an appropriate preceding stage.

3 Experimental Setup

The experiments use synthetic, rule-disjoint logical-reasoning data and controlled token allocations across five KOR-Bench domains. They evaluate mid-training checkpoints, downstream stages, external benchmarks, and fixed-budget SFT reweighting while accounting for mixture and design constraints.

  • Benchmarks and data: KOR-Bench spans five semantically rule-disjoint domains, each with 25 rule types, verifiable answers, and three difficulty levels.Rule-disjointness reduces semantic-transfer confounding but does not make domains statistically or neurally independent.
  • Benchmarks and data: The evaluation includes KOR-Bench plus ProofWriter, ZebraLogic, and CounterBench as limited external consistency checks.The external benchmarks differ in exposure and structural similarity and are not treated as definitive independent validation.
  • Experimental design: All trainable configurations start from Qwen3-8B-Base, and the coverage sweep trains 24 controlled configurations spanning the five-domain simplex.Mid-training-only accuracy is measured before SFT/RL, and compensatory SFT is applied to every configuration.
  • Experimental design: Coverage percentages represent internal allocation while a fixed external component contributes shared formal-deduction exposure.Because domain proportions vary jointly, the curves describe mixture-level associations rather than isolated causal dose-response functions.
  • Experimental design: Each domain samples uniformly across its 25 rule types, but configurations are not balanced for difficulty, answer length, or trace length.The compensatory-SFT gap counts are descriptive because the configurations are purposively selected design points.
  • Experimental design: Table 1 compares shared no-mid-training baselines with three mid-training stages across three experimental allocations.Identical balanced SFT data lets Mid-training+SFT versus SFT only isolate insertion of a mid-training checkpoint before SFT.

4 Results

Across controlled coverage sweeps and selected full-pipeline comparisons, domain accuracy peaks at moderate own-domain coverage, while later SFT generally improves scores without reliably closing coverage-induced gaps.

  • 4.1 The Impact of Mid-Training Data Coverage: Held-out allocations broadly track the fitted curves, with residuals of 1.0–1.8 pp for four domains and 3.0 pp for Operation.The held-out checkpoints were withheld from fitting and carried through the full pipeline.
  • 4.2 Vertical Comparison: Mid-Training Contribution Across Stages: +4.36 pp is the largest exploratory θ∗ full-pipeline gain versus SFT+RL, compared with +0.80 pp for balanced and +0.64 pp for imbalanced coverage.The ordering is descriptive because neither comparison survives Bonferroni correction; the full-pipeline comparison uses only three configurations.
  • 4.2 Vertical Comparison: Mid-Training Contribution Across Stages: Zero-coverage checkpoints can collapse mid-training-only accuracy, while later stages partly repair the deficit in the tested recipe.Operation reverses from −27.2 pp versus Base to +5.2 pp over SFT-only at mid+SFT; Counterfactual turns positive only at the RL row, at +1.2 pp within its SD.
  • 4.3 Horizontal Comparison: Can Alignment Close Mid-Training Gaps?: 116/120 configuration–domain cells improve after compensatory SFT, but 0/240 pairs close at 5 pp and only 30/240 close at a 10% ratio.Uniform SFT behaves similarly, with 0/240 closures at 5 pp and 32/240 at 10%.
  • 4.3 Horizontal Comparison: Can Alignment Close Mid-Training Gaps?: Compensatory SFT raises accuracy but does not reliably close pairwise gaps created by mid-training, and its tested budget-proportional family has limited dynamic range.The same closure pattern holds on six held-out allocations.

5 Discussion

The study finds that coverage gaps persist through the tested fixed-budget alignment procedures, while several design constraints limit how broadly the findings should be interpreted. Zero-coverage effects are partly confounded with generic distributional drift, and the simplex design does not identify independent domain dose-response curves.

  • Alignment robustness: Fixed-budget compensatory SFT leaves coverage-induced gaps essentially intact, while an equal-budget uniform control behaves almost identically.The tested policy family and short RL leg do not justify the broader claim that alignment generally cannot repair coverage choices.
  • Alignment robustness: 0/240 pairs were bridged at the 5 pp threshold, while closure was attainable only by trading roughly half the mean gain in a sharpening sweep.Six held-out allocations reproduced the gap-preserving pattern out-of-sample, including after RL.
  • Scope: The reported patterns are confined to the tested KOR-Bench/Qwen3-8B-Base setting and should be treated as hypotheses rather than rules for allocation.No held-out allocation validates the fitted peaks, intervals, or θ* allocation.
  • Simplex confound: The five-domain simplex makes each coverage curve a mixture-level marginal, so its apparent inverted-U peak is not an independently identified domain dose-response.A joint response surface gives saddle-shaped stationary points, but the Hessian is too weakly determined to distinguish a saddle from a shallow flat region.
  • Control confound: Zero coverage collapses mid-training-only accuracy, but the FineWeb-Edu-only control shows that the effect is co-mingled with broader distributional drift.The study therefore frames interference as a within-setting observation rather than isolated evidence that selectively starving a KOR-Bench domain is the sole cause.

6 Conclusion

The paper studies how mid-training coverage across knowledge-orthogonal reasoning domains shapes later performance in a multi-stage training pipeline. It finds persistent coverage-sensitive gaps, interior per-domain optima, and important confounds that make the conclusions exploratory within the tested setting.

  • Main findings: Three findings hold in this setting: alignment leaves gaps essentially intact, zero coverage causes checkpoint-level collapses, and every domain has an interior coverage optimum.The moderate band yields higher mean accuracy for all five domains, while fitted peaks range from 9.9% to 35.1%.
  • Relation to prior work: Continued pretraining can specialize reasoning capabilities, but this study extends that line to multi-domain allocation rather than single-domain specialization.Prior work establishes domain-specific continued pretraining as a path to specialization; here, allocation is the primary planned coordinate.
  • Research question: The study asks how per-domain data coverage during mid-training determines whether structured reasoning capabilities are reliably acquired.KOR-Bench's knowledge-orthogonal design reduces semantic-transfer confounding, while domains still share language, parameters, and general reasoning procedures.
  • Training stages: SFT and RL are treated as refinement stages whose effectiveness can remain bounded by capabilities established earlier in training.The paper's experiments test whether later alignment procedures can undo mid-training coverage choices.
  • Evaluation perspective: Aggregate reasoning scores can obscure divergent per-domain gains and losses, motivating evaluation that preserves domain-level structure.The paper therefore emphasizes per-domain coverage and structured reasoning benchmarks rather than relying only on a single aggregate score.

A.6 Knowledge-Orthogonal and Structured Reasoning Benchmarks

The paper situates KOR-Bench among structured reasoning benchmarks that test explicit rule application and distinguishes its contribution as a study of training-data allocation. Its knowledge-orthogonal domains provide a controlled instrument for examining acquisition across domains.

  • Benchmark landscape: Structured reasoning benchmarks test applying explicit rules rather than retrieving memorized world knowledge.Examples include formal entailment, relational rule-following, and complex real-world rule systems.
  • KOR-Bench's role: KOR-Bench formalizes knowledge-orthogonal reasoning through domains whose training gains do not automatically transfer across domains.This property supports the paper's focus on whether domain-specific mid-training coverage determines capability acquisition.
  • Paper's distinction: Unlike benchmark-construction work, this paper studies how the proportion of domain-specific mid-training data affects structured reasoning capabilities.The contribution is on the training side rather than introducing new evaluation criteria.

B Training and Evaluation Details

The study allocates substantial budget to broad mid-training coverage sweeps and validates moderation across model scales, while treating RL as a short secondary stage.

  • Training budget: 160 mid-training runs use a fixed ≈1.5B-token budget, including 24 sweep configurations, six held-out allocations, θ∗, and FineWeb-Edu controls.Each configuration uses five seeds.
  • Training budget: The design prioritizes mid-training breadth over RL depth, making the RL leg a fixed 200-step secondary observation.A coverage sweep plus compensatory and uniform SFT passes is identified as the cheapest informative replication subset.
  • Model scale: The Qwen3-8B-Base choice reflects measured post-training headroom, while 4B provides a comparable-headroom replication and larger models compress observable effects.The 4B and 8B checkpoints have the largest measured headroom among the evaluated scales.
  • Cross-scale replication: At 4B, the moderate 10–40% coverage band is best for all five domains and each fitted quadratic has an interior vertex.The replication uses eight mid-training-only allocations spanning low, moderate, and high coverage bands.
  • Cross-scale replication: The 4B fitted peak locations shift relative to 8B, so allocation guidance is not scale-invariant; alignment and θ∗ ordering were not retested at 4B.The reported peak shifts are −1.1, +1.3, +3.8, −3.5, and −17.5 pp across the five domains.
  • Pipeline details: RL uses GSPO with group-normalized binary verifier rewards and a fixed 200-step schedule, while compensatory and uniform SFT share a matched three-epoch token budget.The sequence-level probability ratio is formed from autoregressive token ratios, with ε = 0.2.

C.6 RL Budget: Interpretive Limitations and Training Instability

The short RL stage yields modest gains and does not establish general post-training repair: its interpretation is limited by budget, verifier sparsity, and instability in some exploratory runs.

  • Interpretive limitations: +1.1–+2.3 pp are the observed GSPO gains, with the largest gain of +2.30 pp under θ∗.The paper treats this RL leg as a secondary observation rather than a headline claim.
  • Interpretive limitations: The experiments cannot distinguish competence-limited RL from an insufficient 200-step budget for reshaping domain gaps.A longer schedule could in principle provide more opportunities for exploration and verifier feedback, but that possibility was not tested.
  • Pipeline limitation: The reported RL conclusions are scoped to the tested 200-step schedule and should not be read as evidence about post-training repair in general.The binary correctness verifier provides no dense per-token reward signal from chain-of-thought.
  • Pipeline limitation: On six held-out allocations, the complete pipeline gains +1.19 pp over compensatory SFT, closes 0/60 pairs at 5 pp, and 7/60 at a 10% ratio.Whether compensatory SFT plus RL closes gaps across the full 24-configuration sweep remains open because exploratory sweep runs were excluded for instability.
  • RL behavior: The reward curve wanders around ≈62–67% without a consistent upward trend, matching only a +1.20 pp evaluation gain from SFT-only to SFT+RL.Figure 5 reports the shared no-mid-training baseline over 200 steps.

D Data Accounting

Coverage is defined as each domain’s token share within a fixed five-domain internal component, with total budgets and internal/external ratios held constant across the sweep.

  • Coverage definition: Coverage percentages fully determine each domain’s token allocation within the internal five-domain component because the downstream recipe and total token budget are fixed.The corresponding allocations for the full-pipeline comparison are recorded separately.
  • Fixed accounting: The 24-configuration sweep holds the internal/external token ratio and per-component token budgets fixed across configurations.The table records measured internal-component token shares rather than sample counts.

E Decontamination and Split Isolation

The study isolates training and evaluation instances through rule-family-specific splits and deduplication, while acknowledging pretrained-model contamination as an uncontrolled boundary.

  • ProofWriter split: ProofWriter-family training uses regenerated items of depth at most 5, while evaluation uses depths 5–9, with depth 5 explicitly in distribution.This creates a depth-based separation rather than complete family-level isolation.
  • KOR-Bench split: KOR-Bench SFT and evaluation use different random seeds, and no identical instance appears in both sets.Evaluation instances are held out by instance ID.
  • External benchmarks: CounterBench and ProofWriter evaluation families are excluded from training, but ZebraLogic has 0.3% same-family mid-training exposure.The ZebraLogic training instances are distinct from evaluation instances, so it is not a zero-exposure benchmark.
  • Contamination: The study does not control for Qwen3-8B-Base pretraining exposure to public benchmarks.The authors identify this as a limitation applying to work with pretrained models.
  • Teacher isolation: SFT chain-of-thought traces come from symbolic solvers rather than an LLM, and no teacher model accesses evaluation instances.This separates teacher-trace generation from evaluation data.

F Mid-Training Coverage Sweep

Across the coverage sweep, per-domain accuracy is best at interior, moderate coverage rather than at an extreme. The fitted curves are stable descriptively and generalize to held-out allocations, but joint optimization over the simplex remains unsupported.

  • F.1 Model-Free Check: Coverage Bands: The moderate 10–40% coverage band is best for all five domains, while a calibrated quadratic interiority test gives P ≈0.010.The model-free comparison pools 30 allocations; the test uses permutation of whole accuracy rows.
  • F.2 Compositional (Aitchison) Reanalysis: The compositional ilr reanalysis fits well but finds a saddle in every domain, so the paper does not claim a joint interior optimum.A reduced surface is predominantly concave, but establishing a joint optimum would require more allocations than the 42 trained here.
  • F Mid-Training Coverage Sweep: The fitted peaks range from 9.9% to 35.1% and change by at most 0.5 pp across down-weighting thresholds.Removing a post-hoc configuration shifts peaks by at most 0.20 percentage points, leaving reported intervals materially unchanged.
  • F.3 Fitting Procedure: The exploratory θ* allocation is a descriptive probe based on fitted marginal curves rather than a globally optimal mixture.It is obtained with SLSQP using 2,000 random initialisations and FineWeb-Edu-only baseline accuracies.
  • F Mid-Training Coverage Sweep: The split-Gaussian fits substantially outperform flat baselines, with weighted RMSEs of 0.63–1.44 pp versus 3.28–12.51 pp across domains.The quadratic form is used for the calibrated interiority test because it cannot manufacture an interior optimum as easily as the flexible split-Gaussian.
  • F Mid-Training Coverage Sweep: Held-out residuals are 1.0–1.8 pp for four domains and 3.0 pp for Operation, showing curve-shape generalization across all five domains.These diagnostics validate interpolation over the finite sweep, not peak locations or joint optima on new mixtures.

G.4 Results

Fixed-budget compensatory SFT improves most cells but does not reliably close the capability gaps created by mid-training coverage. Its behavior is similar to an equal-budget uniform pass, and the conclusions are scoped to this fixed policy and budget.

  • G.4 Results: Compensatory SFT raises 116 of 120 configuration–domain cells by a mean of +4.32 pp, but bridges only 0/240 pairs at 5 pp and 30/240 at a 10% ratio.The five cells receiving zero compensatory allocation show no meaningful gain.
  • G.4 Results: Redirecting SFT budget toward coverage-deficient domains raises accuracy but does not reliably close pairwise gaps created by mid-training.This summarizes the fixed-policy result without extending it to other reweighting formulas or budgets.
  • G.4 Results: Uniform SFT raises all 120 cells by a mean of +4.20 pp and bridges 0/240 pairs at 5 pp and 32/240 at a 10% ratio, closely matching compensation.Compensation materially exceeds uniform SFT only for Counterfactual and is slightly weaker for Cipher and Puzzle.
  • G.4 Results: Within-domain compensatory gains rise with remedial SFT proportion and saturate at domain-specific ceilings, with per-domain fits explaining R2 = 0.71–0.97.The pooled fit explains only R2 = 0.31 because domains differ more in overall gain level than in response to remedial proportion.
  • G.4 Results: The conclusions are limited to the fixed-budget, fixed-formula setting, where the within-configuration gain differential reaches at most ≈7 pp.More aggressive allocations and scaled SFT volumes were not tested.

H.1 Coverage Sweep: External Benchmark Accuracy

External evaluations provide limited, descriptive checks of whether coverage effects extend beyond KOR-Bench. ProofWriter shows depth-related transfer, ZebraLogic shows same-family gains, while CounterBench results do not establish a definitive negative control.

  • Scope of external checks: External benchmark results are limited consistency checks rather than independent confirmation of a general transfer law.The external evaluation uses three benchmarks and nine coverage configurations plus the FineWeb-Edu baseline.
  • ProofWriter depth extrapolation: +5.60 pp overall after SFT was observed for the D5-heavy ProofWriter mixture, including +5.70 pp on D5 and +4.64 pp on D6.D7–D9 strata were too small for strong claims.
  • ZebraLogic: 0.3% ZebraLogic coverage, corresponding to 201 direct samples, yielded higher Mid-training+SFT accuracy than SFT+RL at every reported house count.This is a descriptive, exposure-uncontrolled comparison within a related formal-deduction family.
  • CounterBench: Non-positive CounterBench deltas are consistent with no transfer but do not constitute a definitive negative control.Differences in procedures, prompt format, difficulty, and base-model prior may contribute to the pattern.

I Limitations and Ethical Considerations

The study’s conclusions are bounded by its benchmark suite, Qwen3-Base models, reported mixture, and finite-budget recipe, with several model-selection results lacking held-out validation. Ethical claims are limited to research use and should not be equated with deployment benefits such as safety or fairness.

  • Scale: The primary experiments use Qwen3-8B-Base with a mid-training-only Qwen3-4B-Base replication, leaving other scales and architectures untested.Whether the qualitative patterns persist at other model scales is unknown, and fitted intervals may shift with model size.
  • Scope and inference: Claims are scoped to KOR-Bench, three external benchmarks, the Qwen3-Base family, the reported mixture, and the tested finite-budget recipe.The paper identifies these as explicit scope boundaries for its inferences.
  • Scope and inference: The fitted peaks, 95%-of-peak intervals, θ* allocation, compensation thresholds, and full-pipeline gain reuse the same sweep for model selection.This creates an inferential limit because those results lack held-out validation in the cited limitation passage.
  • Ethical considerations: The study uses public or solver-generated data and research checkpoints, but benchmark balance should not be equated with deployment safety, calibration, fairness, or user benefit.The ethical discussion states that the models are not intended for deployment.
Loading 2609.09081v1…