PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning, Fast and Slow: Towards LLMs That Adapt Continually

Rishabh Tiwari, Kusha Sareen, Lakshya A Agrawal, Joseph E. Gonzalez, Matei Zaharia, Kurt Keutzer, Inderjit S Dhillon, Rishabh Agarwal, Devvrit Khatri

arXiv:2605.12484v2cs.LGcs.AI

TL;DR

Parameter-only adaptation can make LLMs absorb task-specific information into persistent weights, while fixed-parameter context adaptation typically cannot match parameter-updating gains. Fast-Slow Training jointly optimizes slow model weights and fast textual context, reaching matched performance with 1.4–3× fewer optimizer steps and a higher asymptote than RL alone.

  • Problem

    Using parameter updates as the sole adaptation mechanism can entangle persistent model behavior with task-specific information, while context adaptation alone may not match parameter-updating gains.

  • Method

    Fast-Slow Training jointly co-optimizes persistent model parameters through reinforcement learning and rapidly changeable textual context through prompt evolution.

  • Results

    Across CodeIO, Math, and HoVer-hard, FST reaches matched performance with 1.4–3× fewer optimizer steps, a higher asymptote, and lower KL displacement than RL alone.

  • Takeaways & Limitations

    The results support training fast textual and slow parametric adaptation together to preserve base-model generality and plasticity while improving continual learning on changing tasks.

  • Takeaways & Limitations

    The study evaluates one primary fast-slow instantiation using CISPO and GEPA, leaving the effects of alternative prompt and weight optimizers for future work.

Abstract

from arXiv · show

Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls.

1 Introduction

The paper frames LLM adaptation as joint optimization of slow model parameters and fast textual context, rather than parameter learning followed by prompt tuning. Fast-Slow Training (FST) combines these channels to improve data efficiency while reducing forgetting and preserving future adaptability.

  • Motivation: Parameter-only adaptation forces reusable skills, task heuristics, and transient rollout lessons into the same model parameters.LLMs are commonly adapted through SFT or RL, both of which modify parameters.
  • Method: Fast-Slow Training combines persistent slow model weights with cheaply changeable fast prompts, instructions, and task context.FST interleaves slow reinforcement-learning updates with fast context optimization using GEPA.
  • Results: Up to 3× fewer rollouts are needed for fast-slow training to match RL reward, while it consistently reaches a higher performance ceiling.Fast weights incorporate task-level signal rapidly, reducing dependence on slow parameter updates.
  • Results: Up to 70% lower KL to the base policy is observed at matched reward than with RL-only baselines.The textual channel carries part of the adaptation, so parameters need not move as far from the base policy.
  • Results: FST-trained models adapt effectively to a second task, whereas RL-trained models collapse to near 0%, indicating greater retained capacity for future learning.The comparison starts from checkpoints trained on one task with either RL-only or FST.
  • Results: In continual-learning settings with changing tasks, FST adapts more quickly to changing objectives.The tasks change on the fly in the reported evaluation.

2 Preliminaries

The framework separates model parameters θ as slow weights from textual scaffolds ϕ as fast weights, optimizing them through a joint task objective. It is optimizer-agnostic, with the paper instantiating slow learning via RLVR and fast learning via reflective prompt evolution.

  • Fast and slow weights: a general framework: Slow weights are model parameters θ, while fast weights are textual scaffolds ϕ drawn from the discrete text space Σ∗.Given query x, responses are generated by a policy conditioned on θ, ϕ, and x.
  • Fast and slow weights: a general framework: The framework supports multiple optimizers for both factors and is instantiated with RLVR for θ and reflective evolutionary prompt optimization for ϕ.Slow-side options include SFT, preference optimization, PPO, and GRPO; fast-side options include APE, OPRO, DSPy/MIPROv2, and GEPA.
  • Slow weights: RL with verifiable rewards: In conventional RLVR, an automatic verifier supplies rewards in [0, 1], group-relative advantages guide updates, and ϕ remains fixed to a generic system prompt while only θ changes.The slow-weight update follows ScaleRL with grouped rollouts and a truncated importance-sampling REIN-FORCE objective, cispo.
  • Fast weights: reflective prompt evolution: Fast weights ϕ are optimized with GEPA, which evolves textual prompts using rollout-based critiques from a frozen reflection LM and retains the top-m candidates from a Pareto frontier.GEPA mutates prompts to improve expected reward on an anchor set from D, returning complementary prompts as fast weights.

3 Fast-Slow Training (FST)

Fast-Slow Training jointly updates slow model weights through RL and fast textual prompt weights through GEPA. It maintains a population of complementary prompts and alternates prompt optimization with RL updates using mixed prompt and sampling variation.

  • Fast-Slow Training: FST jointly optimizes slow weights θ through RL and fast weights Φ through GEPA.The fast weights are represented by a population of K textual prompts.
  • Fast-Slow Training: A prompt population preserves complementary Pareto-frontier behaviors, enabling RL to sample multiple conditioning strategies on each problem.Group-relative advantages compare prompt-induced and sampling-induced variation within the same problem.
  • Fast-Slow Training: Each training cycle runs GEPA on a lookahead batch to produce top-K prompts, then holds them fixed while updating θ for the next T RL steps.GEPA uses the current policy as rollout model, a frozen reflection LM as proposer, and the previous prompt population as its seed.
  • Fast-Slow Training: 3.0× fewer training steps on CodeIO and HoVer-hard, and 1.4× fewer on Math (Polaris), let FST reach RL’s running peak.These are matched-step validation-accuracy results using running maximum and mean@4.

4 Advantages of Fast-Slow Training

Fast-Slow Training improves data efficiency and asymptotic performance while keeping slow weights closer to the base model. This reduced displacement preserves plasticity and supports stronger continual learning across changing tasks.

  • Advantage 1: Data Efficiency: FST reaches RL’s running peak in 3.0× fewer optimizer steps on CodeIO, 1.4× on Math, and 3.0× on HoVer-hard, then exceeds RL’s peak.The OOD average remains essentially flat between FST and RL despite FST using 1.4× to 3.0× fewer training samples.
  • Advantage 3: Base-Model Proximity: FST achieves higher performance at lower KL divergence than RL across all four tasks, shifting the accuracy/KL frontier closer to the base model.Textual fast weights absorb task adaptation, reducing displacement of the slow weights from the base configuration.
  • Advantage 4: Plasticity: FST initialization outperforms RL initialization through 400 steps on subsequent tasks, while prior RL can collapse HoVer-hard learnability to near-zero.On Physics → HoVer-hard, FST-init reaches 24.2% at step 400 versus 19.9% for RL-init.
  • Advantage 5: Continual Learning: During sequential HoVer, CodeIO, and Physics training, FST reaches near-peak performance in every stage, whereas RL barely improves during the CodeIO stage.Tasks are swapped every 200 steps during a single uninterrupted 600-step training pass.

5 Why Does Fast-Slow Training Work?

Fast-Slow Training works because fast weights acquire task signal earlier than slow weights, while jointly optimizing fast and slow channels raises the performance ceiling. Distillation alone does not match FST, indicating that both channels must optimize against reward together.

  • Fast weights acquire task signal faster than slow weights: At around step ∼50, FST reaches measurable reward, whereas parameter-only RL remains near zero for roughly the first ∼300 steps.FST’s early progress is driven almost entirely by the first few GEPA cycles, before θ moves appreciably, and leverages textual failure feedback.
  • Fast and slow weights both raise the performance ceiling: On HoVer-hard, slow weights raise pass@1 from 2.0% to 11.6%, fast weights to 10.6%, and combining both to 21.2%.The decomposition evaluates slow and fast contributions using FST-trained weights and FST-evolved prompts.
  • Fast and slow weights both raise the performance ceiling: On CodeIO, combining slow and fast channels follows the same pattern of jointly increasing performance beyond either channel alone.The passage states that the HoVer-hard pattern also holds on CodeIO, though the supplied excerpt truncates its values.
  • Fast and slow weights both raise the performance ceiling: Distillation alone plateaus well below FST, confirming that both fast and slow channels must optimize against reward jointly to lift the ceiling.The initial naive-distillation results do not substitute for direct RL on the slow weights.

6 Discussion

Fast-slow training offers continual-learning benefits by keeping models near their base state while separating task-specific information across weights and prompts. The framework remains broad, with efficiency, trajectory reuse, and optimizer choices identified as future directions.

  • Benefits: FST maintains proximity to the base model, supporting plasticity and reduced forgetting in continual learning.The discussion argues that finite-capacity models need not distill all task-specific information into their weights.
  • Benefits: On HoVer-hard and CodeIO, both weight and prompt channels contribute, whereas Math (Polaris) gains come almost entirely from slow weights.The joint FST-weight-plus-FST-prompt configuration dominates on HoVer-hard and CodeIO.
  • Benefits: Fast-slow updates separate declarative and procedural knowledge, an important distinction for general-purpose reasoning.The discussion frames this separation as broader than the specific fast-slow implementation studied.
  • Future work: The fast-slow framework is highly general, with prompt and weight optimizers, compute efficiency, and trajectory reuse proposed as future work.The study uses CISPO and GEPA as one capable instantiation of weight and prompt optimization.

7 Related Work

Related work spans slow parameter-based learning, fast textual-context optimization, and complementary fast–slow learning systems. Recent LLM methods combine textual feedback with reward-driven weight updates, while this work applies fast–slow decomposition to verifiable-reward RL.

  • Slow learning: RL for LLM reasoning: Slow-learning RL writes improvements into LLM parameters, while prolonged adaptation can shrink output entropy, raise KL to the base policy, and erode plasticity.These effects are identified as the plasticity loss phenomenon.
  • Fast learning: prompt and context optimization: Fast-learning research improves behavior by editing textual context through prompt search, prompt proposers, evolutionary methods, compound LM programs, evolving agent context, and reflective self-feedback.This work uses GEPA, which maintains a per-instance Pareto frontier of candidate prompts.
  • Fast and slow weights: complementary learning systems: Complementary learning systems and fast-weight architectures motivate separating learning into fast and slow channels.Here, the fast channel is an evolving population of textual prompts, and the slow channel is the model parameters.
  • Modern fast–slow methods for LLM RL: Recent LLM methods combine textual feedback with reward-driven weight updates, including alternating supervised fine-tuning and prompt optimization and co-evolving prompt populations with policies.BetterTogether alternates SFT with prompt optimization, while this work extends the paradigm to verifiable-reward RL with textual-feedback-created Pareto-frontier prompts.

8 Conclusion · A GEPA · B Algorithm pseudocode

The paper concludes that Fast-Slow Training jointly optimizes slow model parameters with RL and fast textual contexts through reflective prompt evolution, improving efficiency, asymptotic performance, KL preservation, plasticity, and continual learning. GEPA implements the fast-weight search with a diverse prompt population, while Algorithm 1 interleaves prompt evolution and RL updates.

  • 8 Conclusion: Fast-Slow Training jointly optimizes slow parameters θ via RL and fast textual contexts Φ via reflective prompt evolution.The two channels are interleaved during post-training.
  • 8 Conclusion: 1.4–3× fewer optimizer steps, a higher asymptote, and lower KL displacement distinguish co-optimization from RL alone across CodeIO, Math, and HoVer-hard.These gains translate into preserved plasticity and stronger continual-learning behavior.
  • A GEPA: GEPA optimizes fast weights ϕ by searching textual prompts with a frozen reflection LM πref that proposes mutations from rollout critiques.The reflection model is separate from the capable model used for task execution.
  • A GEPA: GEPA maintains candidate prompts in a population P and tracks their per-instance fitness vectors on an anchor set.This supports population-based comparison across instances.
  • A GEPA: Each GEPA generation selects a Pareto-frontier parent, samples rollouts, elicits a textual child edit, evaluates it, and prunes dominated candidates.After a fixed metric-call budget, GEPA returns the top-m candidates from the resulting frontier.
  • A GEPA: The GEPA frontier preserves diverse candidates that perform best on different data slices, enabling RL to exploit complementary fast weights simultaneously.The passage identifies this diversity as the reason the RL phase can use several fast weights together.
  • B Algorithm pseudocode: Algorithm 1 interleaves GEPA prompt evolution with T RL minibatch updates over a data stream, using initial slow weights θ0, a seed prompt ϕseed, population size K, and reflection LM πref.The pseudocode also specifies cycle length T and GRPO group size G with K | G.
  • B Algorithm pseudocode: For each minibatch and fast-weight candidate, the algorithm assembles G/K rollouts, computes group-relative advantages, and updates θ with Lcispo.All G rollouts are placed in one group before the advantage computation.

C Star-graph dataset construction

The star-graph dataset varies source degree, path length, and node-pool size while constructing a unique gold path alongside decoy branches. Its first-hop branching makes exploration difficult, especially at d = 25, where random success is only 4%.

  • Graph construction: Each graph instance is parameterized by (d, p, n), with a unique length-(p −1) gold path and d−1 decoy chains rooted at the source.Source and intermediate nodes are sampled without replacement, while decoy nodes are drawn fresh.
  • Exploration difficulty: At the source, choosing any decoy first hop commits the solver to a chain that never reaches the goal, with no built-in backtracking.The source is the only node with degree d; all other nodes have degree 2 along their arms.
  • Exploration difficulty: 4% is the success rate of a uniformly-random first hop when d = 25, making the first branching decision the task’s central exploration challenge.Only the source branches; selecting the correct neighbor is necessary to reach the destination.
  • Experimental splits: The headline experiments use (d, p, n) = (25, 20, 500), with 10,000 training examples and 200 held-out test examples.Difficulty is swept by varying source degree, path length, and node-pool size.

D Hyperparameters and compute

The appendix specifies shared RL and GEPA settings for reproducibility, alongside cluster resources and wall-clock costs for headline runs. Most headline experiments use a single 8-GPU node, while a Polaris ablation uses 32 GPUs.

  • Shared RL configuration: RL uses GRPO with cispo loss, group-based advantage normalization, a 10−3 KL penalty, AdamW at 10−6 learning rate, and 10-step warm-up.Each RL step samples G = 8 rollouts per problem with train_batch_size = 32 problems.
  • Shared GEPA configuration: GEPA runs optimize every T = 6 RL steps, score all K population prompts on each question, and use question-level advantage grouping.The reflection LM is OpenAI gpt-5.2.
  • Per-domain overrides: Polaris uses a continued-SFT base, disables GRPO advantage normalization by group standard deviation, and doubles the training batch to 64 problems/step.The continued-SFT base further SFTs Qwen3-8B on Nemotron to recover math performance.
  • Compute and wall-clock: ∼60 s is the mean per-RL-step wall-clock for RL-only under the headline configuration, compared with ∼100 s for FST without rollout reuse.These headline timings are reported for HoVer-hard with K = 8.
  • Compute and wall-clock: ∼47 s is the RL-step cost when rollout reuse is enabled, making FST slightly faster than RL-only.All runs use 8× H100 (80GB) per node; headline runs use 1 × 8 GPU, while the Polaris K = 8 ablation uses 4 × 8 = 32 GPUs.

E Design ablations · F Rollout reuse: same accuracy at lower wall-time and rollout cost

The ablations identify population size, advantage normalization, cycle length, and GEPA recipe as important design choices, with K=8 and T=6 forming the headline configuration. Rollout reuse recycles recent GEPA trajectories to reduce wall-time while preserving FST accuracy.

  • E Design ablations: At matched step 500 on CodeIO, RL-only scores 39.65% mean@4 and serves as the reference for all design ablations.All sweeps use Qwen3-8B thinking with the light-recipe defaults.
  • E Design ablations: K=8 reaches 42.84% mean@4, while K=1 already improves over RL-only by +1.5 pp to 41.10%.Performance is non-monotonic across K ∈ {1, 2, 4, 8}, with gains saturating at K=8.
  • E Design ablations: With K=2, the Problem baseline reaches 40.65%, outperforming the Prompt baseline by +1.4 pp and RL-only by +1.0 pp.The Prompt baseline scores 39.30%, slightly below RL-only, whereas the Problem baseline shares one statistic across rollouts under both prompts.
  • E Design ablations: At K=8 with the Problem baseline, T=6 reaches 42.84%, whereas T=12 lowers mean@4 by 1.7 pp to 41.13%.The longer cycle increases prompt staleness as the policy changes between GEPA optimizations.
  • E Design ablations: The light GEPA recipe uses K=4, a 192-example evaluation budget, 960 maximum metric calls, and incremental prompt edits rather than full rewrites.The full recipe uses K=1, max_metric_calls=1922, doubled metric budget, and an open-ended proposer.
  • F Rollout reuse: same accuracy at lower wall-time and rollout cost: Reuse is possible because GEPA evaluation tuples contain each problem, candidate prompt, sampled response, and reward, matching rollouts needed by the subsequent RL phase.Without reuse, these tuples are discarded when GEPA selects the next prompt population.
  • F Rollout reuse: same accuracy at lower wall-time and rollout cost: The cache claims matching problem–prompt slots before live vLLM generation, concatenates cached and live trajectories, and clears when the next GEPA population is produced.Reused trajectories are at most T RL steps old; the headline configuration uses T=6.
  • F Rollout reuse: same accuracy at lower wall-time and rollout cost: ∼29% speedup reduces HoVer-hard wall-time per RL step from ∼66 s to ∼47 s, primarily by eliminating generation work while leaving GEPA cycle cost unchanged.The comparison covers the first 300 training steps and uses otherwise identical FST runs with and without reuse.

G KL-vs-reward, full four-task results · H Explicit fast-to-slow distillation

Across four tasks, the KL–reward comparison includes a Polaris-specific exception tied to its different base model. Explicit fast-to-slow distillation transfers fast-channel signal into slow weights, but remains below jointly optimized FST.

  • G KL-vs-reward, full four-task results: Figure 12 compares validation reward against KL(πtrain ∥πbase) for CodeIO, Math (Polaris), HoVer, and Physics.It uses the same axes, smoothing, and conventions as Figure 5.
  • G KL-vs-reward, full four-task results: On Polaris, RL and FST overlap in KL–reward space rather than showing FST shifting the frontier left.The passage attributes this qualitative difference to Polaris using a distinct base model setup.
  • H Explicit fast-to-slow distillation: FST-distill tests whether fast textual-channel gains can be folded into parameters without applying RL directly to slow weights.It replaces the slow-weight policy-gradient update with an on-policy reverse-KL distillation loss.
  • H Explicit fast-to-slow distillation: FST-distill uses a FST-evolved fast-weight prompt as teacher while the student sees only the problem, updating slow weights through per-token reverse KL.The student samples outputs on-policy, and the teacher has frozen parameters conditioned on the evolved prompt.
  • H Explicit fast-to-slow distillation: FST-distill rises above the prompt-only ceiling by transferring fast-weight signal into parameters across updates, but plateaus below FST.FST jointly optimizes both fast and slow channels, while FST-distill transfers signal without direct reward-based policy-gradient updates on θ.
  • H Explicit fast-to-slow distillation: On HoVer, Figure 13 compares FST with FST-distill on validation reward and actor entropy.The figure reports that both methods preserve healthy entropy during training.

I Evolved GEPA prompts during FST training

During FST, GEPA generally preserves each seed prompt’s role and format while adding failure-mode-specific guidance derived from low-reward rollouts. The evolved prompts therefore become task-specific checklists, including precise execution and retrieval instructions for CodeIO and HoVer-hard.

  • I Evolved GEPA prompts during FST training: GEPA almost never rewrites the seed, instead preserving its basic role and output format while adding layered guidance through small targeted edits.The K={4, 8} Problem-baseline recipe constrains the proposer to small targeted edits.
  • I Evolved GEPA prompts during FST training: The additions target observed failure modes in low-reward rollouts, producing task-specific checklists rather than generic system prompts.Examples include avoiding placeholder numbers in CodeIO, skipping parenthetical disambiguators in HoVer-hard, and off-by-one errors in Polaris.
  • I.1 CodeIO: The evolved CodeIO prompt requires exact step-by-step Python execution, including state changes, control flow, recursion, mutation, aliasing, and numeric behavior.It explicitly forbids estimation, intuition, placeholders, fabricated numbers, and abandoning long computations.
  • I.1 CodeIO: CodeIO guidance requires deriving the actual return value and JSON serialization from the code, preserving the returned type and avoiding wrappers unless the function returns them.The prompt distinguishes bare scalars, strings, booleans, null, lists, and dictionaries, and prohibits following conflicting natural-language output requirements.
  • I.1 CodeIO: CodeIO arithmetic instructions preserve full-precision intermediate values, Python float64 semantics, exact rounding behavior, and algorithm-specific integer or Decimal operations.The prompt also rejects rough guesses, unjustified algebraic simplifications, and approximate trigonometric, exponential, or logarithmic evaluations.
  • I.4 HoVer-hard: HoVer-hard retrieval guidance strengthens ambiguous-title queries with distinctive anchors and requires explicitly including both event titles when claims involve separate events.For songs or episodes, it also adds the relevant artist, band, or series title to the query.
Loading 2605.12484v2…