PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola

arXiv:2603.12228v1cs.LGcs.AI

TL;DR

The paper asks why random parameter perturbations can improve pretrained language models despite random search being ineffective from scratch. It measures density and diversity around pretrained weights and proposes RandOpt, a parallel sample-and-ensemble method; for contemporary large models, RandOpt is competitive with PPO, GRPO, and ES.

  • Problem

    Random guessing is ineffective for finding useful solutions from scratch, leaving open why it can work for post-training pretrained language models.

  • Method

    The paper measures task-expert density and diversity near pretrained weights and introduces RandOpt, which samples N perturbations, selects the top K, and ensembles their predictions.

  • Results

    RandOpt achieves accuracy competitive with PPO, GRPO, and ES under the same post-training FLOPs for current models.

  • Takeaways & Limitations

    Large pretrained neighborhoods contain dense, diverse task specialists, so multiple post-training methods can work once a strong pretrained representation is available.

  • Takeaways & Limitations

    RandOpt is unsuitable for training from scratch, struggles on small pretrained models, and may saturate when further improvements require searching beyond local thickets.

Abstract

from arXiv · show

Pretraining produces a learned parameter vector that is typically treated as a starting point for further iterative adaptation. In this work, we instead view the outcome of pretraining as a distribution over parameter vectors, whose support already contains task-specific experts. We show that in small models such expert solutions occupy a negligible fraction of the volume of this distribution, making their discovery reliant on structured optimization methods such as gradient descent. In contrast, in large, well-pretrained models the density of task-experts increases dramatically, so that diverse, task-improving specialists populate a substantial fraction of the neighborhood around the pretrained weights. Motivated by this perspective, we explore a simple, fully parallel post-training method that samples $N$ parameter perturbations at random, selects the top $K$, and ensembles predictions via majority vote. Despite its simplicity, this approach is competitive with standard post-training methods such as PPO, GRPO, and ES for contemporary large-scale models.

1 Introduction

The paper argues that pretraining creates neighborhoods containing increasingly dense and diverse task-improving experts, making random post-training effective for large models. It introduces RandOpt, which samples, selects, and ensembles perturbations in parallel, achieving competitiveness with standard post-training methods.

  • After pretraining, task-improving parameter perturbations become increasingly dense as model scale and performance grow.Small models remain in a needle-in-a-haystack regime, while large pretrained models enter a high-density thicket regime.
  • Sampled perturbations are specialists: improvements on one task can coincide with degradation on others, and diversity increases with model size.The paper characterizes these perturbations as complementary task experts rather than uniform generalists.
  • RandOpt samples N random perturbations, evaluates them, selects the top K, and ensembles their predictions by majority vote.The method exploits neighborhood density through random sampling and neighborhood diversity through ensembling.
  • RandOpt achieves accuracy competitive with PPO, GRPO, and ES under the same post-training FLOPs.It trains in O(1) wall-clock time versus O(T) for baselines requiring T sequential update steps, but inference costs K times more because of ensembling.
  • For current models, the density of task-improving solutions is high enough that random guessing can support effective post-training.Ensembling multiple guessed solutions often improves performance substantially.

2 Structure of the Multi-task Loss Landscape Around Pretrained Weights

Around pretrained weights, larger models contain denser and more diverse task-improving solutions. Sampled perturbations behave as specialists, with complementary strengths across tasks.

  • Solution Density: Larger models occupy accuracy valleys with many nearby higher-performing peaks, unlike small models near local accuracy maxima.The comparison spans models from 0.5B to 32B parameters and three reasoning tasks.
  • Solution Density: Solution density increases monotonically with model size across multiple improvement margins, including perturbations yielding +5% accuracy.Large-scale pretrained weights therefore lie in dense basins populated by abundant high-quality nearby solutions.
  • Solution Diversity: Spectral discordance increases with model size, indicating increasingly diverse solution rankings across tasks.The analysis evaluates 500 perturbations across seven tasks in mathematics, code, creative writing, and chemistry.
  • Solution Diversity: Performance spectra are spiky rather than flat, showing that sampled perturbations specialize across tasks instead of improving uniformly.PCA clustering further reveals distinct expert groups with shared strengths and complementary capabilities.
  • Solution Diversity: Distinct PCA clusters contain perturbations with task-specific strengths, such as excelling at math while failing at chemistry.Perturbations from different clusters offer complementary capabilities.
  • Solution Diversity: Mottled RGB task-accuracy landscapes indicate that task landscapes are largely uncorrelated rather than behaving identically.Gray shades would instead indicate similar behavior across tasks.

3 A Minimal Setting In Which Thickets Emerge: Autoregressive Modeling of 1D Signals

A minimal autoregressive model of synthetic 1D signals reproduces the thicket phenomenon. Diverse pretraining creates nearby functional alternatives, whereas insufficient or overly narrow pretraining does not.

  • Experimental Setting: Pretraining mixes sinusoidal, linear, harmonic, sigmoidal, sawtooth, and square-wave functions for next-value prediction.The multilayer perceptron maps a preceding context window to the next signal value.
  • Experimental Setting: With N = 1000 Gaussian perturbations, the experiment compares base predictions, random perturbations, and the top 5 perturbations against a test function.Perturbations use σ = 0.002, and the top five best-fitting models are highlighted.
  • Three Regimes: No pretraining produces a needle-in-the-haystack regime, while diverse pretraining produces a thicket of nearby function-specific continuations.Pretraining only on linear signals creates a plateau because the pretrained model already nearly solves the test task.
  • Interpretation: The phenomenon also appears in simpler models, with varied signal-shape pretraining critical for nearby solutions and functional diversity.Too little pretraining yields no nearby solutions, whereas single-type pretraining yields little functional diversity.

4 A Practical Algorithm: Random Guessing & Ensembling (RandOpt)

RandOpt samples parameter perturbations in parallel, selects the best performers, and ensembles their predictions. It uses dense, diverse pretrained neighborhoods rather than sequential gradient updates or a single solution.

  • Algorithm: RandOpt samples N perturbations, evaluates them on post-training data, selects the top K, and aggregates their predictions by majority vote.The method exploits density to find useful candidates and diversity to combine complementary experts.
  • Algorithm: Each perturbed model is formed as θ_i = θ + σ_iϵ(s_i), evaluated on D_train, and ranked by its performance score.Noise scales are sampled from a set of available scales, with each perturbation tied to a random seed.
  • Inference: Inference generates predictions only from the selected set and combines them through majority voting.The ensemble output is computed from the selected perturbations rather than all sampled models.
  • Design Difference: RandOpt is fully parallel, avoiding gradient steps and sequential updates while retaining a set of ensembled solutions.This distinguishes it from standard single-vector adaptation and connects it to population-based evolutionary methods.

5 How Does RandOpt Compare to Standard Methods for Post-Training?

RandOpt mostly matches or outperforms established post-training methods across model scales and task categories, while avoiding sequential optimization through parallel computation. Its performance depends substantially on ensembling, and benchmark gains can include both reasoning and formatting improvements.

  • Interpretation of gains: RandOpt’s benchmark gains are partly attributable to reasoning improvements and partly to formatting improvements, and performance can be sensitive to minor stylistic changes.The paper explicitly cautions that benchmark performance may reflect both sources of improvement.
  • Comparison with standard methods: RandOpt mostly matches or outperforms established methods across model scales from 0.5B to 8B and diverse task categories at equal training FLOPs.The evaluation covers language and vision-language settings, with comparisons against TT-MV, PPO, GRPO, and ES.
  • Comparison with standard methods: RandOpt uses no sequential optimization steps, offering a potentially large wall-clock advantage when sufficient parallel compute is available.A 200 GH200 deployment trained Olmo-3-7B-Instruct on Countdown in 3.2 minutes with N = 2000 and K = 50, reaching 70% accuracy.
  • Ensembling: K = 50 is substantially more effective than K = 1, showing that ensembling is crucial to RandOpt’s performance.The method’s inference-time ensemble combines predictions from selected perturbations.
  • Ensembling: 50-pass test-time ensembling also improves PPO, GRPO, and ES, with ES + TT-MV raising Qwen2.5-0.5B GSM8K accuracy from 42.6% to 61.2%.The comparison is asymmetric because standard 1-pass PPO/GRPO use a single test-time sample, while RandOpt and ES + TT-MV ensemble 50 samples.

6 Scaling Properties of RandOpt

RandOpt improves with larger search populations under low selection ratios, while the optimal selection ratio decreases as population size grows. Its effectiveness also depends strongly on pretrained model scale and sufficient pretraining.

  • Population size and selection ratio: For sufficiently low selection ratios, performance improves monotonically with population size N on the Countdown task.The reported interaction is measured with Qwen2.5-3B-Inst, and its precise shape may vary across tasks and models.
  • Population size and selection ratio: The optimal selection ratio K/N decreases as population size increases, creating a trade-off between search budget and ensemble selection.A practical strategy when avoiding hyperparameter tuning is to keep K small and make N as large as the budget permits.
  • Model scale: RandOpt fails to improve very small models, begins increasing rapidly at around 1.5B parameters, and remains near zero when applied from scratch.The results indicate that sufficient pretraining and model scale are essential for RandOpt to work.

7 Distillation

Distillation addresses RandOpt’s higher inference cost by compressing the top-K ensemble into one model. In the reported GSM8K experiments, the distilled model achieves performance comparable to the ensemble at relatively small additional cost.

  • Motivation: RandOpt requires K forward passes at test time, so the paper explores distilling the top-K models into a single model.This targets the method’s principal inference-time cost.
  • Procedure: The distillation procedure uses hard samples generated by the top-50 models, then performs supervised fine-tuning on the base model.It generates candidate answers, retains inputs where most candidates are incorrect, and trains on reasoning traces and final answers.
  • Results: The distilled model achieves performance comparable to the top-K ensemble on GSM8K.The comparison is reported for Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct.
  • Cost: Distillation costs about 2% of the training cost under the reported setup.Training uses a population of 5,000, whereas distillation uses the top 50 models and 10 SGD iterations.

8 Types of Thickets

RandOpt gains arise from multiple kinds of thickets rather than reasoning alone. On GSM8K, both formatting corrections and newly correct numerical answers contribute substantially to performance.

  • Reasoning and format thickets: The paper tests whether RandOpt’s gains reflect shallow changes such as answer formatting rather than fundamentally different reasoning.The analysis compares retained correctness, reasoning corrections, and format fixes relative to the base model.
  • Diversity of thickets: Thickets may involve answer formats, reasoning approaches, personalities, or domain knowledge because benchmark expertise combines multiple task-relevant behaviors.The paper defines task expertise operationally through benchmark performance, which can reflect several factors.
  • Reasoning and format thickets: GSM8K accuracy decomposition shows that both format correction and solving previously missed problems contribute to RandOpt and GRPO gains.The figure distinguishes strictly correct answers, corrected answers, and gains from fixing format.

9 Related Work

The paper frames neural thickets as a multi-task property of pretrained weight neighborhoods, connecting task-specialist density and diversity to prior work on loss landscapes, lottery tickets, reweighting, and random search.

  • Multi-Task Loss Landscapes: Neural thickets describe structure in multi-task loss landscapes, complementing prior analyses focused primarily on single-objective neural-network landscapes.The paper characterizes nearby weight-space optima across downstream tasks rather than output-space probability reweighting.
  • On Lottery Tickets and Neural Thickets: After pretraining, good task-specific weights become abundant near pretrained weights, unlike the sparse solutions encountered when training from scratch.This contrasts the Lottery Ticket Hypothesis’s rare favorable initializations with the post-pretraining neighborhood regime.
  • Connections: The paper relates thickets to reweighting pretrained behavior, self-improvement by trace selection, adaptable initializations, Bayesian parameter sampling, and direct weight-space manipulation.Its distinct focus is the geometry of nearby weight-space optima and specialists already latent around pretrained weights.
  • Random Search: RandOpt differs from sequential random search by demonstrating a fully parallel search-and-ensemble procedure for post-training language models.Prior work established sequential random search for control and some language-model settings, but the paper reports limited precedent for parallel search beyond simple scenarios.

10 Implications

The paper proposes treating pretrained weights as a distribution of diverse task specialists rather than a single model. This perspective motivates parallel adaptation and reframes pretraining as creating neighborhoods where downstream improvement can be easy.

  • Rethinking Pretraining: Pretrained weights specify a distribution over models containing diverse specialists whose behavior differs qualitatively from the singular pretrained model.The paper argues that this distribution cannot be characterized adequately by its mean alone.
  • Multi-Task Loss Landscapes: Downstream adaptation depends on multiple task-specific loss landscapes, which a scalar pretraining landscape can obscure.The pretrained weights need not minimize any individual downstream objective.
  • Pretraining Is All You Need?: Once representations are sufficiently pretrained, simple adaptation methods can work remarkably well, while finding those representations may still require structured learning.The paper places RandOpt’s success within this broader distinction between adaptation and representation discovery.
  • Decentralized, Parallel Adaptation: RandOpt’s workers communicate only through inference-time ensembling, making the method attractive when compute nodes are inexpensive but communication is constrained.This decentralized design is contrasted with methods requiring communication during sequential updates.

11 Limitations

The paper’s limitations concern dependence on pretraining, uncertain scaling beyond local thickets, inference-time ensembling costs, restricted structured-prediction support, and incomplete understanding of how thickets arise.

  • Pretraining Dependence: RandOpt performs negligibly from scratch and struggles on small pretrained models, so its success depends on a sufficiently good pretrained representation.Structured search may still be needed to discover that representation.
  • Capacity for New Skills: Further improvement may saturate with model size and search resources, potentially requiring searches beyond local thickets where structured methods become necessary.The paper presents this as an open possibility rather than a demonstrated failure mode.
  • Inference-Time Cost: RandOpt typically requires K > 1 forward passes at inference, and distillation adds training compute, reduces full parallelism, and is tailored to categorical LLM reasoning.These tradeoffs limit how broadly the proposed distillation approach transfers.
  • Structured Prediction: Majority-vote ensembling is straightforward mainly for single-class or integer answers, while structured outputs require modified aggregation methods.The image-generation extension is presented only as a proof of concept, not as the best choice.
  • Open Mechanism Question: The experiments do not fully explain when or why pretraining creates thickets, including whether multi-task pretraining is the critical factor in large language models.The paper identifies this mechanism question as requiring further investigation.

D Additional Experimental Analysis

Additional experiments examine full-model results, ensembling, solution-density distributions, scaling with population and selection, baseline parallelism, and compute-matched comparisons.

  • Full Results: RandOpt’s full LLM results cover multiple base models, adaptation methods, and tasks, with comparisons reported in Table 4.The broader evaluation includes reasoning, code, creative writing, and chemistry tasks across several model families and sizes.
  • Effect of Ensembling: Ensembling over many perturbations is critical for competitive performance on most tasks, while selecting only the top perturbation still substantially improves over the base model.Figure 11 compares RandOpt with K=1 and K=50 alongside baselines.
  • Solution Density: 64% of GSM8K and 60% of Countdown perturbations match or exceed base performance at 32B, versus 0% and 8% respectively at 0.5B.The distributions show increasing task-improving density with model size; longer tails in smaller models do not imply higher absolute accuracy.
  • Scaling Population and Selection: Population size improves Countdown performance log-linearly, while the benefit of increasing the selection ratio saturates as N grows.For sufficiently large N, selecting only the top 1% can yield high performance and reduce inference-time costs.
  • Scaling Baseline Parallelism: 87.1% accuracy was achieved by RandOpt with N = 3000 on GSM8K, exceeding all tested GRPO and PPO settings under one-step training.GRPO peaked at 83.5%, while PPO peaked at 78.0% in the reported grid search.
  • Compute Matching: Compute-matched comparisons normalize methods by total training FLOPs, with ES and RandOpt using forward-only population evaluations.The accounting distinguishes policy, reference, critic, and backward passes for GRPO and PPO from gradient-free evaluation.

F Additional Results on 1D Signals

Additional 1D-signal experiments reproduce three regimes: untrained perturbations fail, mixed-type pretraining enables improvement through RandOpt, and single-type pretraining can already reach ceiling performance.

  • Experimental design: The 1D experiments test both approximation on the fitted function and generalization to newly sampled functions of the same type.Generalization results also report mean performance over 256 randomly sampled test functions.
  • Needle in haystack regime: Without pretraining, Gaussian perturbations fail to produce effective models under Xavier or Kaiming initialization.Even larger σ values create prediction variation but not good continuations of the test functions.
  • Thicket regime: With mixed-signal pretraining, RandOpt improves performance on a selected function type when the base model cannot reliably infer that type from limited context.The experiment selects perturbations after post-training on the target type.
  • Plateau regime: With single-function-type pretraining and same-type testing, the base model reaches ceiling performance, leaving no need for further adaptation.This regime is evaluated in both approximation and generalization experiments.
Loading 2603.12228v1…