Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

1 Introduction

• Because longer serial mid-training can plateau or degrade downstream performance, the paper asks whether distributing a fixed budget across branches can outperform a longer run.The introduction identifies combining intra- and inter-trajectory merging as underexplored.Source: §1 · p. 2
• At matched token budgets, Trajectory Soup surpasses the strongest single-trajectory average and inter-trajectory baselines, with a widening margin as budgets expand.The advantage persists after supervised fine-tuning.Source: §1 · pp. 2–3
• Controlled recipe variations produce geometrically distinct trajectories, and interpolation between them improves on both endpoints.The paper frames trajectory count as an allocation axis that remains productive after a single trajectory saturates.Source: §1 · p. 3
• Trajectory Soup averages validation-selected checkpoints within each branch, then averages branch anchors into one model.Its local bias–variance analysis separates the contributions of the two averaging levels and predicts a finite preferred checkpoint count that decreases as trajectories are added.Source: §1 · p. 3

2 Empirical Motivation: Saturation and Complementary Trajectories

• Serial training on a fixed architecture and corpus plateaus and can eventually decline, while within-trajectory averaging has diminishing returns once local fluctuations are attenuated.The section discusses limited-corpus exposure and optimization-path fluctuations as plausible contributors.Source: §2 · p. 3
• Five screened branches show positive but limited alignment in their leading directions after within-trajectory averaging.The branches differ through controlled changes such as data order, learning rate, batch size, schedule, or optimizer.Source: §2 · pp. 3–4, Figure 2 · p. 4
• Inter-trajectory interpolation achieves lower validation loss than intra-trajectory averaging alone.The comparison interpolates between two branches’ averaged anchors at matched token horizons.Source: Figure 3 · p. 4, §2 · p. 4

3.1 Problem Formulation and Budget Accounting

• Each branch starts from a shared checkpoint, uses the common training distribution and a controlled recipe, and is admitted only if its training-loss gap stays within the compatibility tolerance.The implementation uses a relative training-loss gap tolerance of 0.01; data-mixture changes are left for future work.Source: §3.1 · pp. 4–5

3.2 Trajectory Soup

• Trajectory Soup first averages each branch’s validation-selected checkpoints, then averages the resulting anchors across branches into one model.The two-stage construction is equivalent to uniformly averaging the selected checkpoints; N = 1 recovers intra-trajectory averaging, while K = 1 averages each branch’s best checkpoint.Source: §3.2 · p. 5, §3.1 · p. 5, Figure 4 · p. 5

4.1 A Local Validation-Loss Model

• Under a local quadratic loss near a stationary minimizer, expected excess loss separates mean displacement from model variation without requiring Gaussian or optimizer-specific assumptions.The analysis assumes checkpoints and averages lie in the modeled region around that minimizer.Source: §4.1 · p. 6

4.2 Branchwise Decomposition of Validation Loss

• Branchwise loss separates selection-dependent bias, residual within-trajectory variation, and persistent branch variation; only the within-trajectory term is reachable by merging one branch.The shared bias and recipe- or selection-specific offsets are not removed by either averaging stage.Source: §4.2 · p. 6

4.3 Hierarchical Variance Reduction

• Effective count Keff(K) measures the reduction in within-trajectory variance; temporal correlation and ranking dependence can keep it below K.It reaches K only when selected fluctuations are uncorrelated with equal marginal contributions.Source: §4.3 · p. 7
• With independent branch anchors, temporal averaging reduces within-branch variance and averaging across N branches reduces both stochastic contributions; uniform weights attain the variance bound under the stated covariance condition.The soup's bias depends on the selected branch collection, and Table 1 compares averaging operations under the local model and homogeneous independent-anchor assumptions.Source: §4.3 · p. 7, Table 1 · p. 7

5.1 Experimental Setup

• Experiments fork branches from a shared checkpoint with controlled recipe variations, compare four merging scopes, and evaluate both compute-matched and expanded budgets.The default setup uses Ling-3.0-Tiny and saves checkpoints every 25B tokens; Limited divides a fixed total budget, while Extended adds fully trained trajectories.Source: §5.1 · p. 8

5.2 Main Results of Trajectory Soup

• Trajectory Soup (Extended) reaches an overall average of 68.96, exceeding the strongest single-trajectory merge and Model Soup (Extended); the compute-matched Limited setting also beats both baselines.The comparison reports the best evaluated configuration for each merging strategy.Source: Table 2 · p. 9, §5.2 · p. 9

5.3.1 Robustness across models and learning-rate schedules.

• On a 2B-parameter MoE with WSD, Trajectory Soup reaches overall averages of 49.64 in Limited and 50.07 in Extended, exceeding the strongest intra-trajectory result in both settings.This reproduces the aggregate trend on a smaller model and a different learning-rate schedule.Source: §5.3.1 · p. 10, Table 4 · p. 10

5.3.2 Ablation Studies on the Trajectory Soup Configuration

• Uniform coefficients outperform tested rank-based weights, default Top-K selection beats three alternatives, and symmetric 8:8 allocation achieves the highest average among tested splits.The ablations favor uniform averaging, within-trajectory Top-K selection, and balanced cross-trajectory representation.Source: §5.3.2 · pp. 10–11, Figure 6 · p. 11

5.3.3 Trajectory Diversity over Sampling Density

• With candidate-pool size and merge size matched at 12, Trajectory Soup beats dense single-trajectory merging at comparable budgets, isolating trajectory diversity rather than denser sampling as the gain source.The controlled comparison samples every 25B tokens from each of two branches, versus every 12.5B tokens in a single branch.Source: §5.3.3 · pp. 11–12, Figure 7 · p. 11

5.3.4 Scaling the Number of Trajectories and Merged Checkpoints

• The best score rises from 68.79 with two trajectories to 69.08 with five, while gains shrink; all groups peak around 10–16 merged checkpoints, then oscillate or decline.The four-trajectory soup eventually falls below the two-trajectory optimum when the checkpoint pool grows.Source: §5.3.4 · p. 12, Figure 8 · p. 12

5.3.5 Directional Diversity Predicts Interpolation Gain

• Weaker alignment between branches predicts larger interpolation gains, with Pearson r=−0.80 (p=0.005) and Spearman ρ=−0.94.The authors identify directional diversity as a pre-merging criterion for selecting branches.Source: §5.3.5 · p. 12, Figure 9 · p. 12

6.1 Mid-Training and Horizon-Extensible Scaling

• Scaling-law and data-constrained work studies compute, model size, and token use, while WSD and mid-training studies address extending schedules and improving post-training starting points.This paper adds trajectory count and selective weight-space aggregation under matched and expanded budgets.Source: §6.1 · p. 13

6.2 Checkpoint Averaging Along a Single Trajectory

• Prior methods aggregate checkpoints along a run, including SWA (Izmailov et al., 2018), TWA, and language-model averaging studies (Sanyal et al., 2023; Li et al., 2025; Tian et al., 2026; Zhou et al., 2026).The paper builds on temporal aggregation by combining selected averages across independently evolved branches.Source: §6.2 · p. 13

6.3 Combining Multiple Training Trajectories

• Prior work combines models from shared initializations or specialized branches, while mode-connectivity studies examine low-loss paths; this paper analyzes straight-segment interpolation between mid-training anchors.The cited examples include Model Soups (Wortsman et al., 2022), Branch-Train-Merge (Li et al., 2022), and Branch-Train-MiX (Sukhbaatar et al., 2024).Source: §6.3 · p. 13

• The comparison fixes processed tokens but omits checkpoint storage, validation, search, and accelerator-utilization costs, so it is not a full development-cost account.The authors identify token-priced trajectory costs as a next step for choosing allocations by optimization rather than candidate sweeps.Source: §7 · p. 14
• The theory is exact only under a local quadratic validation-loss model and inherits its error across the region spanned by selected checkpoints and their averages.Direction angles and interpolation profiles are complementary but do not establish that the region containing all branches is jointly well behaved.Source: §7 · p. 14
• The geometric diagnostics are measured only after branches exist, leaving recipe perturbations that generate removable error unpredicted before training.The authors identify predicting such perturbations as a way to design diversity rather than screen for it afterward.Source: §7 · p. 14
• Averaging cannot remove shared branch error; highly correlated trajectories leave a floor, while overly aggressive perturbations can make the merged model worse than its members.At a fixed budget, more branches also mean shorter runs, and additional checkpoints can admit weak ones; the method currently uses validation search to choose compatible branches and checkpoint counts.Source: §7 · p. 14

8 Conclusion

• Trajectory Soup selects strong checkpoints within shared-initialization branches and consolidates their anchors into one model, framing trajectory count as a compute-scaling dimension beyond longer single runs.The conclusion presents weight-space consolidation as applicable beyond mid-training, while identifying perturbation design, trajectory pricing, and compatibility beyond the shared origin as open questions.Source: §8 · p. 14

A Additional Theory for Hierarchical Merging

• The appendix makes the random-effects assumptions explicit, allows heterogeneous or correlated branches, restores fixed-budget branch-length coupling, and states covariance conditions for uniform averages.Its identities are exact under the local quadratic validation-loss model.Source: A Additional Theory for Hierarchical… · p. 19

A.1 Conditional Random Effects and General Covariances

• With correlated anchors, off-diagonal covariance terms remain in the merged variance; a positive average contribution leaves an error floor as branch count grows.Independent selected anchors make those off-diagonal terms vanish, while separate within-branch selection preserves anchor independence when the shared setup is fixed.Source: §A.1 · p. 20
• Inverse-variance weights minimize variance for independent anchors with unequal contributions; equal weights minimize it under exchangeable covariance, while checkpoint correlations determine the effective count.For nonnegative rank covariances, the effective count lies between 1 and K; independent residuals attain K, whereas negative covariances can push it above K.Source: §A.1 · p. 20

A.2 The Tradeoff in Selecting More Checkpoints

• An added checkpoint improves expected loss only when its residual-variance reduction exceeds its bias increment; under an illustrative quadratic bias model, the preferred count decreases as branches are added.The predicted count depends on the assumed bias law, available checkpoints, and residual correlations; other geometries can yield monotone improvement or decline.Source: §A.2 · p. 21

A.3 The Fixed-Budget Allocation Tradeoff

• At fixed total token budget T, N branches each receive T/N tokens; the loss identity accounts for the tradeoff but does not itself determine an optimal N or K.More branches can increase averaging gains but shorten runs, and the preferred balance depends on horizon-specific bias, variances, and feasible checkpoint counts.Source: §A.3 · pp. 21–22

A.4 Variance-Optimal Weights at Both Levels

• Uniform branch and temporal weights minimize the variance term under the stated covariance conditions, but they minimize total loss only when temporal reweighting leaves bias constant.That bias condition holds when selected checkpoint means coincide; heterogeneous or correlated branches may favor inverse-variance branch weights.Source: §A.4 · pp. 22–23

A.5 A General Criterion for Uniform Weighting

• For a positive semidefinite covariance matrix, uniform weights minimize its quadratic form exactly when row sums are equal; under exchangeable residual covariance, full correlation eliminates temporal variance reduction.With exchangeable covariance, the effective checkpoint count depends on the correlation level.Source: §A.5 · p. 23

A.6 Scope of the Theoretical Claims

• The theory requires a local quadratic model, finite second moments, and shared parameterization; it addresses population validation loss, not benchmark accuracy or post-training outcomes.Its assumptions are sufficient, not necessary, for empirical merging gains, and the theory does not imply every additional branch improves the model.Source: §A.6 · pp. 23–24

B Proofs of the Main-Text Results

• The appendix proves the main theorems and corollary under the local quadratic model, with finite second moments wherever the relevant quantities occur.The positive semidefinite curvature matrix may be singular; the proofs do not require it to be invertible.Source: B Proofs of the Main-Text Results · p. 24

B.1 Proof of Theorem 4.1

• Under the local quadratic model, expected excess loss separates into squared bias and a nonnegative variance term.The proof obtains the decomposition by expanding around the mean and using the centered fluctuation's zero expectation.Source: §B.1 · p. 24

B.2 Proof of Theorem 4.2

• The covariance proof separates the conditionally mean-zero residual from the shared shift, yielding distinct residual and persistent-variation contributions.Under the shared-shift condition, the inter-trajectory variance is independent of K.Source: §B.2 · p. 24
• Top-K merging improves on a branch’s best single checkpoint exactly when the within-trajectory term falls more than the bias rises.The proof obtains this comparison by subtracting the K = 1 case.Source: §B.2 · pp. 24–25

B.3 Proof of Theorem 4.3

• The soup’s bias term is the squared curvature-weighted displacement of its mean from the single-checkpoint mean.The proof applies the total-covariance identity to represent the residual covariance and then derives the soup expression.Source: §B.3 · p. 25

B.4 Proof of Corollary A.1

• Uniform weights attain the variance minimum at both averaging levels; the joint minimizer is unique under the stated positive-definiteness conditions.The bias term can still depend on the weights.Source: §B.4 · p. 26

C Compute-Scaling and Geometry Protocols

• Appendix C describes the compute accounting and scaling fits for Figure 1b, alongside geometric measurements used in the trajectory analyses.It also covers the pairwise analysis in Section 5.3.2.Source: C Compute-Scaling and Geometry Protocols · p. 26

C.1 Compute Axis and Scaling-Law Fits

• Compute uses estimated cumulative mid-training FLOPs on a logarithmic axis, counting branch tokens after the shared fork and excluding the common pretraining prefix.The empirical inverse-power-law fits map cumulative compute to Overall Average Accuracy; the Raw fit uses 17 checkpoints and the Soup fit uses 10 frontier points.Source: §C.1 · pp. 26–27
• At 100B-equivalent compute, fitted Overall Average Accuracy is 68.36% for Trajectory Soup and 66.52% for Raw training.The Raw and Trajectory Soup frontier fits have R2 = 0.594 and R2 = 0.965, respectively.Source: §C.1 · p. 27

C.2 Trajectory Directions and Pairwise Alignment

• Each branch direction is its centered merged-checkpoint sequence’s leading principal component, oriented along its net displacement; pairwise cosine measures alignment.Cosines near 1 indicate aligned directions, while values near 0 indicate nearly orthogonal directions.Source: §C.2 · p. 27

C.3 Interpolation Between Branch Anchors

• At matched horizons, validation scores across interpolated branch anchors are compared with their endpoints and equal-weight midpoint to assess merging benefit.The best interpolation is retained at each horizon.Source: §C.3 · p. 27

C.4 Pairwise Cosine and Relative Interpolation Gain

• Pairwise relative gain is the merged score minus the midpoint of endpoint scores; the scatter relates that gain to branch-direction cosine.The plotted gains are averaged over 17 token-budget offsets, with an OLS fit and 95% confidence band for the mean response.Source: §C.4 · p. 28

D.1 Default Model Configuration

• The default sparse MoE model has 7.9B total parameters and 1.3B activated per token; its listed architectural values come from the supplied description.The table’s caption says those values were not independently verified through a public model release.Source: §D.1 · p. 28, Table 6 · p. 29

D.2 Training Settings Shared by All Branches; D.3 Branch Identity and Recipe Mapping

• Branches share the pretrained starting checkpoint and common settings, while EXP1–EXP5 vary admitted recipe dimensions; all five pass the compatibility screen at 600B tokens.The smaller-model reproduction uses separate, locally assigned branch labels.Source: §D.2 · p. 28, §D.3 · p. 28

D.4.0.1 Pre-training evaluation.

• Pre-training evaluation uses 41 benchmark configurations across five capability categories, including reasoning, language understanding, professional knowledge, mathematics, and coding.Source: §D.4.0.1 · pp. 29–30

D.4.0.2 Post-training evaluation.

• Post-training evaluation tests 16 configurations across six scenarios, including coding, mathematics, reasoning, knowledge, instruction following, and function calling.Source: §D.4.0.2 · p. 30

D.5 Merging Baselines

• Single-Trajectory Merge averages each branch’s top-ranked checkpoints, while Model Soup uniformly averages branch endpoints without validation-based member selection.The comparisons separate within-trajectory averaging from cross-trajectory combination and checkpoint selection.Source: §D.5 · pp. 30–31
• Reported tables select checkpoints and choose N and K using validation accuracy, so the selection signal and reported score are not fully independent.The accuracy-selected tables do not establish whether validation-loss ranking would agree; that variant requires a rerun.Source: §D.5 · p. 31

D.6 Ablation Schemes

• The coefficient ablation varies within-branch weights while keeping selected sets fixed; EQUAL is the variance-optimal default under the stated covariance condition.The other schemes tilt weights toward lower-ranked checkpoints or concentrate them on higher-ranked ones.Source: §D.6 · pp. 31–32
• The Top-K-each rule keeps the highest-ranked K checkpoints per branch; alternatives remove either the quality criterion or the per-branch quota.Global top-NK drops the quota, Tail-K uses recent checkpoints instead of quality ranking, and All checkpoints tests averaging the pooled set without selection.Source: §D.6 · p. 32

E Per-Trajectory Results of Single-Trajectory Merging

• The appendix expands the strongest Single-Trajectory Merge into per-branch results, selecting the reported branch separately for each table by Overall Average.Comparing Trajectory Soup with the strongest branch is conservative for the Overall Average; other branch comparisons would widen the margins.Source: E Per-Trajectory Results of… · p. 32

E.1 Default Model at Mid-Training

• 68.72 (Limited) and 68.96 (Extended) are the Trajectory Soup Overall Averages, exceeding all three individual branches’ 68.34–68.55 range.EXP1 is the strongest individual branch, but no branch leads in every category.Source: §E.1 · pp. 32–33, Table 9 · p. 33

E.2 Default Model after Post-Training

• 61.39 (Limited) and 61.52 (Extended) are Trajectory Soup’s post-training Overall Averages, above every branch after the same SFT procedure.EXP2 is the strongest individual branch after SFT at 61.11; EXP1 and EXP3 tie at 60.67.Source: Table 10 · p. 33, §E.2 · p. 33

E.3 Smaller MoE Model with a WSD Schedule

• 49.64 (Limited) and 50.07 (Extended) are Trajectory Soup’s Overall Averages on the smaller model, exceeding both individual branches.The two branches differ only by data-shuffling seed and score 49.49 and 49.55.Source: Table 11 · p. 33, §E.3 · p. 33