PaperlayerView exact revision on arXiv

Source-linked AI summary

The Reasoning-Creativity Trade-off: Toward Creativity-Driven Problem Solving

Max Ruiz Luyten, Mihaela van der Schaar

arXiv:2601.00747v1cs.LG

TL;DR

Correctness-focused training can collapse reasoning distributions into a few semantic templates, raising questions about why diversity decays and how it can be preserved. DCR models training over distributions of solution traces, derives algorithm-specific collapse modes, and identifies diversity designs with convergence guarantees.

  • Problem

    Correctness-focused training can shrink reasoning distributions into a few semantic templates, while existing approaches leave open how to explain and predict collapse and guarantee diverse reasoning portfolios.

    Source: §1 · p. 1

  • Method

    DCR models training as gradient flow over distributions of solution traces and combines utility, regularization, and a diversity energy in a unified objective.

    Source: §1 · p. 2

  • Results

    The Diversity Decay Theorem predicts winner-takes-all fixation for STaR, neutral drift for GRPO, and homogenization of correct strategies for DPO; incorporating DCR diversity energy guarantees convergence to a unique, stable, diverse interior equilibrium.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The creativity kernel and coefficients α and β are design levers for seeking policies that are both correct and creative.

    Source: §1 · p. 2

    Entropy alone is blind to trace content and cannot distinguish syntactic rephrasings from genuinely distinct strategies.

    Source: §6.1 · p. 7

Abstract

from arXiv · show

State-of-the-art large language model (LLM) pipelines rely on bootstrapped reasoning loops: sampling diverse chains of thought and reinforcing the highest-scoring ones, mainly optimizing correctness. We analyze how this design choice is sensitive to the collapse of the model's distribution over reasoning paths, slashing semantic entropy and undermining creative problem-solving. To analyze this failure, we introduce Distributional Creative Reasoning (DCR), a unified variational objective that casts training as gradient flow through probability measures on solution traces. STaR, GRPO, and DPO, as well as entropy bonuses, and other methods, all constitute special cases of the same loss. The framework delivers three core results: (i) the diversity decay theorem, describing how correctness-based objectives lead to distinct modes of diversity decay for STaR, GRPO, and DPO; (ii) designs that ensure convergence to a stable and diverse policy, effectively preventing collapse; and (iii) simple, actionable recipes to achieve this in practice. DCR thus offers the first principled recipe for LLMs that remain both correct and creative.

1 Introduction

Correctness-focused reinforcement learning can contract reasoning distributions into a few semantic templates, motivating a framework that explains collapse and designs for preserving diverse, useful strategies. DCR models training over distributions of solution traces and unifies collapse analysis, diversity-preserving guarantees, and practical design guidance.

1 Introduction

  • Correctness-focused reinforcement learning can sharply reduce output entropy, concentrating reasoning in a handful of semantic templates.The paper asks how to explain this collapse, predict its algorithm-specific modes, and design provably diverse portfolios of reasoning paths.Source: §1 · p. 1
  • DCR models training as gradient flow over the full conditional distribution of solution traces and unifies utility, regularization, and diversity in one framework.It encompasses STaR, GRPO, and DPO as specific instantiations.Source: §1 · p. 2
  • The Diversity Decay Theorem predicts winner-takes-all fixation for STaR, neutral drift for GRPO, and homogenization of correct strategies for DPO.These are distinct collapse modes under scalar-only objectives.Source: §1 · p. 2
  • Incorporating DCR diversity energy guarantees convergence to a unique, stable, diverse interior equilibrium that neutralizes the predicted collapse modes.The stated guarantee concerns the diversity energy's effect on learning dynamics.Source: §1 · p. 2
  • DCR offers practical design levers in the creativity kernel and coefficients α and β for training policies that are both correct and creative.The paper presents these choices as a principled recipe rather than ad hoc diversity heuristics.Source: §1 · p. 2

2 Related Work

Prior work documents diversity loss during reward optimization and explores diversity-aware remedies, but the cited approaches leave gaps in principled prediction and control. DCR contributes a unified dynamical analysis of collapse and a designed diversity objective.

2 Related Work

  • Reported collapse spans RLHF, reasoning optimization, and other settings, with reduced entropy, semantic clustering, or narrower strategy portfolios.The cited work also reports entropy, type–token ratio, and embedding spread falling after RLHF, and similar narrow attractors across PPO, Expert Iteration, and GRPO.Source: §2 · pp. 2–3
  • Existing diversity-aware methods include entropy bonuses, novelty search, and Preference-Matching, but lack a unified account of collapse and sufficient counterforce.Entropy bonuses retain stochasticity without targeting qualitatively distinct ideas; novelty-search outputs are managed separately and redistillation can regress gains.Source: §2 · p. 3
  • Replicator dynamics and information-theoretic RL offer partial lenses, but neither provides a comprehensive analysis of creativity in LLMs.The former suggests utility can concentrate mass on the highest-fitness type, while the latter does not guarantee entropy captures structural novelty.Source: §2 · p. 3
  • DCR combines a variational diversity regularizer with a dynamical-systems analysis of how scalar-reward objectives erode diversity under STaR, GRPO, and DPO.Its design guidance selects the kernel and weights α and β to shape the policy distribution.Source: §2 · p. 3

3 Distributional Creative Reasoning

DCR treats an LLM policy as a distribution over finite solution traces, then optimizes utility alongside regularization and a composite diversity energy. Its Shahshahani gradient flow has a convergence guarantee under stated conditions, while its parameterized implementation uses stochastic gradients and kernel estimates.

3 Distributional Creative Reasoning

  • DCR frames training as dynamics over probability distributions on solution traces, making diversity analyzable alongside correctness.For a fixed vocabulary and maximum trace length, the trace set is finite; the policy is a probability mass function represented in a simplex.Source: §3 · p. 3, §3.1 · p. 3

3.2 The DCR Objective

  • The DCR objective combines expected trace utility, diversity energy, and a KL penalty relative to a reference policy, with coefficients tuning the balance.The reference can be the SFT checkpoint, and the KL term promotes stability.Source: §3.2 · p. 4

3.3 The Diversity Energy Functional D[p]

  • D[p] combines Shannon entropy for probabilistic breadth with a kernel coverage term that penalizes concentration on semantically similar traces.With a PSD kernel, D[p] is concave; strict concavity follows when α > 0 or when β > 0 and the kernel is strictly positive definite on the tangent subspace.Source: §3.3 · p. 4
  • Entropy alone is blind to trace content, so the kernel term is needed to promote qualitatively different reasoning strategies.A small entropy barrier can ensure strict concavity and positive trace probabilities during optimization.Source: §3.3 · p. 4

3.4 Learning Dynamics: Gradient Flow

  • Under the stated regularity assumptions, the Shahshahani gradient flow converges from an interior policy to the unique maximizer of a strictly concave objective.The objective increases unless the policy is already at that maximizer, and the resulting policy balances utility, diversity, and regularization.Source: §3.4 · p. 4

3.5 Parametric Realization and Scalability

  • The parametric realization uses softmax logits and stochastic gradient descent; a mini-batch U-statistic estimates the kernel-term gradient at O(B^2) cost.Projection or clipping constrains policies to a trimmed simplex, and embedding-based kernels and gating are among the practical design strategies discussed.Source: §3.5 · pp. 4–5

4 Collapse Under Scalar Objectives

This section analyzes common scalar-driven cases—negligible diversity energy and small entropy weight—and derives predictable, algorithm-specific diversity decay that motivates the full DCR objective.

4.1 Scalar-Driven Dynamics: The SRCT Framework

  • Log-ratio dynamics reflect competing forces: score differences exert selective pressure, while entropic damping pushes trace probabilities toward equalization.In the pure-selection limit, incorrect traces are suppressed, while diversity among correct traces follows algorithm-specific dynamics.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 Stochastic Dynamics: Fixation Under Noise

  • Under batch noise, STaR’s collapse trajectory is robust, GRPO can drift to fixation, and DPO can converge with a subset of solutions dominating.For DPO, the dominating subset may contain semantically redundant solutions.Source: §4.3 · p. 6

4.2 Deterministic Diversity Decay (Small ε)

  • Under scalar-objective training, STaR deterministically collapses onto a single dominant correct trace.An initial probability advantage creates positive feedback that rapidly amplifies the dominant trace.Source: §4.2 · p. 5, §4.4 · p. 6
  • GRPO preserves correct traces’ initial relative probabilities, but stochastic drift can eventually fixate on a low-diversity subset.Its neutral stability provides no active protection against mini-batch fluctuations.Source: §4.2 · p. 5, §4.4 · p. 6
  • DPO equalizes probabilities across high-utility traces rather than promoting structured semantic diversity.Noise can break equalization’s symmetry, allowing a semantically redundant subset to dominate.Source: §4.2 · p. 5, §4.4 · p. 6, §4.3 · p. 6

4.4 Synthesis: The Diversity Decay Theorem

  • The theorem covers STaR, GRPO, and DPO—not every scalar-only objective—and minimal entropy does not prevent their decay modes.Finite-batch noise can accelerate collapse.Source: §4.4 · p. 6

5 The Diversity Energy Effect on the Equilibrium Structure

DCR’s diversity energy reshapes the optimization landscape toward equilibria containing simultaneously correct and diverse traces. Its entropy and kernel components create a trade-off between probabilistic breadth, semantic diversity, and suppression of incorrect traces.

5 The Diversity Energy Effect on the Equilibrium Structure

  • DCR’s diversity energy shifts learning toward equilibria containing multiple simultaneously correct and diverse traces.The full objective adds diversity energy to utility and a KL term.Source: §5 · p. 6, §5.1 · p. 6

5.1 From Collapse to Structured Diversity

  • Adding a positive diversity weight yields a structured-diversity equilibrium shaped by the entropy and kernel components.The effective kernel can gate semantic similarity by correctness, applying diversity pressure among correct traces.Source: §5.1 · p. 6, §5.2 · p. 6

5.2 The Dual Levers of Diversity Energy: Shaping p⋆

  • Entropy promotes probabilistic breadth and equalization among correct traces but also increases mass on incorrect traces, whereas the kernel penalizes semantic similarity among correct traces.Entropy alone cannot produce this structured diversity.Source: §5.2 · p. 7

5.3 Balancing Correctness and Structured Diversity at Equilibrium

  • The equilibrium trade-off requires the kernel penalty among correct traces not to overwhelm their utility gain: 2λβ(K_eff p⋆)_c < 1.This condition helps suppress incorrect traces.Source: §5.3 · p. 7
  • Increasing ε_total aids equalization among correct traces but weakens suppression of incorrect traces, so λα and λβ must be balanced.The target is a regime where incorrect traces are suppressed while diverse correct solutions remain.Source: §5.3 · p. 7

6 The Creativity Kernel

DCR pairs probabilistic breadth with a kernel that penalizes semantic redundancy, because entropy alone cannot distinguish genuinely different strategies or ensure correctness. The section describes how to design and implement that kernel, while noting that ideal target-shaping kernels are typically infeasible.

6 The Creativity Kernel

  • The creativity kernel complements entropy’s probabilistic breadth with content-sensitive pressure for semantic diversity.Entropy responds to trace probabilities, not conceptual content, and cannot distinguish paraphrases from distinct strategies.Source: §6 · p. 7, §6.1 · p. 7

6.1 Limitations of Entropic Diversity

  • Entropy alone can preserve probability on incorrect traces and cannot ensure structured variation among correct solutions.A mechanism that promotes semantic dissimilarity is needed alongside probabilistic dispersion.Source: §6.1 · p. 7

6.2 Sculpting Semantic Diversity

  • The kernel penalizes probability concentrated on semantically similar traces, but ideal kernels for sculpting a target equilibrium require stringent global, equilibrium-dependent conditions that are typically infeasible.This motivates practical, learnable semantic kernels.Source: §6.2 · p. 7

6.3 Practical Design of the Semantic Kernel

  • The effective kernel gates semantic similarity with a correctness verifier, restricting kernel interactions to correct traces.Possible semantic kernels include embeddings of full reasoning traces or task-specific structural similarity such as proof-step or lemma-dependency graphs.Source: §6.3 · pp. 7–8

6.4 Implementation and Desiderata

  • A minibatch U-statistic estimates the kernel term at O(B^2) per step, while effectiveness depends on grouping equivalent solutions and distinguishing different correct strategies.The kernel should ignore syntactic differences within a solution category while assigning low similarity across qualitatively different approaches.Source: §6.4 · p. 8

7 Concluding Insights

The paper concludes that scalar reward maximization collapses strategic diversity and presents DCR’s diversity energy as a remedy with a unique, stable interior policy. It also states testable predictions about algorithm-specific decay and the distinction between entropy-only and kernel-inclusive approaches.

7 Concluding Insights

  • DCR’s diversity energy counters scalar-objective decay and ensures convergence to a unique, stable, interior policy, with kernel and entropy weights balancing valid diversity against incorrect traces.The effective kernel gates semantic diversity to correct traces.Source: §7 · p. 8

7.1 Testable Predictions

  • Under scalar-only objectives, STaR predicts winner-takes-all fixation, GRPO stochastic erosion of diversity, and DPO homogenization across preferred traces.These are presented as testable algorithm-specific decay modes.Source: §7.1 · p. 8
  • An entropy-only approach preserves indiscriminate breadth at a cost to correctness, whereas a kernel-inclusive approach can prevent collapse and increase semantic diversity among correct solutions.Source: §7.1 · p. 8

A Mathematical Foundations and Problem Formalism

This section develops the mathematical foundations for policies over finite trace spaces, covering simplex geometry, objective curvature, and replicator-gradient dynamics. It also characterizes the soft-max parameterization and clipping-based probability floors, and establishes conditions and bounds related to diversity preservation and stable convergence.

A.2.1–A.2.3 Trace space, floors, and inequalities

  • On the trimmed simplex, entropy has bounded mean-log values and Lipschitz behavior, while clip–renormalize induces an effective policy floor.The trace space is finite for a fixed horizon, and policies lie on its probability simplex.Source: §A.2.1 · p. 10, §A.2.3 · p. 11, §A.2.2 · p. 11

A.3.1–A.4.1 Objective curvature and boundary barriers

  • The diversity functional is concave, with strictness supplied by positive entropy weight or kernel curvature; entropy and full-support KL penalties exclude boundary maximizers.The quadratic kernel slice is strictly concave along feasible directions exactly when its kernel has no nonzero tangent vector.Source: §A.3.1 · p. 11, §A.3.3 · p. 12, §A.3.2 · p. 12

A.4.2 No finite-time boundary hitting

  • Under bounded fitness, an interior initial policy stays strictly positive at every finite time.The log probability derivative is bounded, so integrating it precludes finite-time boundary hitting.Source: §A.4.2 · p. 12

A.5.1–A.5.2 Metric and integrability

  • Replicator dynamics are Shahshahani gradients, and a field is integrable exactly when its anchored cross-partials or projected Jacobian satisfy the stated symmetry conditions.When integrable, the objective is unique up to an additive constant and gauge.Source: §A.5.1 · p. 13, §A.5.2 · p. 13

A.6.2–A.6.4 Lyapunov identity and global convergence

  • With A > 0 and the stated assumptions, the flow has a unique global solution in a compact trimmed simplex and converges exponentially to its unique interior maximizer.The appendix also gives a strict Lyapunov identity and log-ratio contraction with time-uniform floors and caps.Source: §A.6.2 · p. 13, §A.6.4 · p. 14, §A.6.3 · p. 14

A.7 Special Case: Replicator Flow with Single-Site Scores

  • For single-site scores, the interior equilibrium is uniform when all score functions are identical and strictly monotone, but need not be uniform when they differ.Source: §A.7 · p. 14

A.8–A.8.2 Barrier–Dominance

  • Barrier–dominance conditions ensure the trimmed simplex is forward invariant, using an entropy face gap whose value decreases with the floor and trace-space size.At δ = 1/S, the gap is zero and the trimmed set is the uniform point.Source: §A.8.2 · p. 15, §A.8.1 · p. 15, §A.8 · p. 15

B.1–B.2 Logit-space geometry

  • The logit appendix specifies deterministic soft-max geometry and shows that centered logits map real-analytically and bijectively to the simplex interior; stochastic effects are deferred.Soft-max is invariant to adding a constant to every logit.Source: §B.1 · p. 16, §B.2 · p. 16

B.2 Soft-max Map: Gauge, Inverse, and Log-ratio

  • Soft-max logits in Θ are uniquely recovered from log-ratios: θi − θj = log(pi/pj), and G(p) centers log p.If two logits in Θ have the same soft-max, they are equal.Source: §B.2 · p. 16

B.3 Geometry and Conditioning of the Soft-max Jacobian

  • The soft-max Jacobian has kernel span{1} and rank S−1; on T, its eigenvalues are bounded below by pmin and above by 1/2.The upper bound is attained for S = 2 at p = (1/2, 1/2), while the lower bound is attained at p = 1/S.Source: §B.3 · p. 17
  • Each soft-max Jacobian coordinate has a bound of 1/3, and this bound is optimal already for S = 2.Source: §B.3 · p. 17
  • The map θ ↦ Jθ is globally Lipschitz, and its dimension-free lower-bound example establishes optimality in the ℓ1 domain norm.The bound’s extremal example uses θ(t) = (t, −t, 0, . . . , 0).Source: §B.3 · p. 17

B.4 Clip–Renormalize and the Logit Lift

  • Clip–renormalize gives every output coordinate a sharp floor δmin = δ⋆/(1 + (S −1)δ⋆), and the lifted map P and softmax ◦P are globally Lipschitz.Their respective ℓ2 Lipschitz constants are 1/(2δ⋆) and 1/(4δ⋆).Source: §B.4 · p. 18
  • If every coordinate of pθ0 is at least δ⋆ + ε and ∥θ − θ0∥2 ≤ ε, no coordinate is clipped and P(θ) = θ.The probability change is bounded by ε/2 in ℓ∞.Source: §B.4 · p. 18
  • Under Assumption (A), the gradient of the clipped parameterized objective Φ is Lipschitz in θ.Assumption (A) requires a Lipschitz probability-space gradient and a finite uniform bound on its norm over [δmin, 1]S.Source: §B.4 · p. 19

B.6 Quadratic Approximation and Hessian Suprema

Section B.6 derives second-order bounds and sharper Hessian suprema on the trimmed simplex. Section B.7 collects parametric constants and bounds for reference.

B.6 Quadratic Approximation and Hessian Suprema

  • For S = 2, both Hessian suprema are attained, while for S > 2 they are strict suprema approached by concentrating residual mass.The proof reduces the extrema to 2×2 or 3×3 blocks after maximizing a convex Rayleigh quotient at simplex vertices.Source: §B.6 · p. 20
  • The second-order expansion supplies remainder bounds, sharpened on the δ-interior by replacing global constants with c2(δ, S) and c1(δ, S).The refinement assumes the parameter path stays in the trimmed simplex.Source: §B.6 · p. 20

B.7 Reference table: Parametric Constants

  • The reference table gathers canonical parametric constants, operator bounds, clipping regimes, and ℓ1/ℓ2 Hessian suprema.Composite bounds in §B.5 are evaluated on [δmin, 1]^S, with δmin set by clip–renormalize.Source: §B.7 · p. 21

C The Self-Reinforcing Correctness Training (SRCT) Framework

The SRCT framework formulates a Shahshahani gradient flow on the simplex and establishes its dynamical properties. Under stated conditions, it gives confinement and convergence results, including for separable score fields.

C.2 SRCT objective, correct variational derivative, and canonical drift

C.3 Operator facts for S and the entropic map E

  • The selection covariance annihilates the all-ones vector, and its quadratic form is the variance of the sampled component.Under the stated regularity conditions, the drift admits a unique local absolutely continuous solution; tangency and barrier–dominance give global confinement.Source: §C.3 · p. 22, §C.4 · p. 22

C.5 Mass balance and log-ratio calculus

  • Mass remains one, while pairwise log-ratios obey a linear relaxation identity with a comparison envelope when A > 0 and the forcing is bounded.The envelope decays exponentially from the initial log-ratio, up to a term determined by the forcing bound.Source: §C.5 · p. 22

C.6 Positivity and face invariance on the closed simplex

  • Coordinates initialized positive cannot hit zero in finite time, whereas coordinates initialized at zero remain zero.The results distinguish interior trajectories from invariant boundary faces.Source: §C.6 · p. 23

C.7 Barrier–Dominance and confinement on ∆K−1

  • Barrier–dominance compares outward selective pressure with the entropy face gap to ensure forward invariance of the trimmed simplex.Under the strict inequality, trajectories starting in the simplex interior remain interior.Source: §C.7 · p. 23

C.8 Existence/uniqueness on the mass hyperplane

  • A Lipschitz extension to the mass hyperplane and Picard–Lindelöf yield a unique global absolutely continuous solution, with trajectories remaining in the simplex under (C.1).Source: §C.8 · p. 23

C.9 Single-site score fields: Lyapunov structure and convergence

  • For separable single-site scores, the Lyapunov function is strictly concave on each face and trajectories converge; with A > 0, strict barrier–dominance gives global convergence to p†.For A = 0, the framework instead characterizes water-filling and support selection under (CR+SM).Source: §C.9 · pp. 23–24

C.10 Safe denominators (linear-functional floor)

D.2 The STaR score field: bounds, Jacobian, and Lipschitzness

This section analyzes STaR score-field bounds and Jacobian behavior alongside STaR confinement and SRCT results for GRPO and DPO. It also examines DPO–SRCT boundary invariance, contraction, cross-class bounds, and Lyapunov structure, identifying conditions for stable interior behavior and eventual trimming.

D.2 The STaR score field: bounds, Jacobian, and Lipschitzness

  • For M ≥1, the STaR score is smooth on the interior simplex, centered, and satisfies ∥ϕSTaR(p)∥∞≤1.The stated quadratic upper bound is tight as ρ approaches 1 with all correct mass on one index.Source: §D.2 · p. 25
  • The STaR Jacobian has zero columns for incorrect indices and rank at most M; its 1/ρ² factors preclude global Lipschitzness near ρ=0.On compact domains with ρ bounded below by a positive constant, the passage states that uniform Lipschitz control is available.Source: §D.2 · pp. 25–26

D.3 STaR as an SRCT flow: well-posedness, Lipschitz drift, and confinement

  • STaR’s SRCT flow has a unique global interior solution, avoids finite-time boundary hitting, and, for ε > 0, maintains a uniform coordinate floor.Under either stated Barrier–Dominance condition, the trimmed simplex is forward invariant.Source: §D.3 · p. 26, ϕSTaR · p. 27

E GRPO through the SRCT Lens

  • The GRPO analysis establishes barrier–dominance, rank-one score Lipschitz constants, cross-class envelopes, and convergence to a unique two-level equilibrium under a slope condition.Its characteristic hG is C1 on [0, 1] and constant when G ∈{2, 3}.Source: E GRPO through the SRCT Lens · p. 28, §E.1 · p. 28, §E.2 · p. 29

E.3 SRCT drift: global Lipschitzness and mass conservation

  • The GRPO SRCT drift conserves mass, and under the exact or sufficient Barrier–Dominance conditions, the trimmed simplex is forward invariant with global existence and uniqueness.Interior trajectories cannot reach the boundary in finite time.Source: §E.3 · p. 29, §E.4 · p. 31

E.5 Log-ratio dynamics, envelopes, and scalar reduction

  • GRPO equalizes within-class proportions exponentially at rate ε; under Barrier–Dominance, cross-class dynamics reduce to a scalar with a unique zero and an interior class-uniform limit when feasible.Otherwise, the limit lies on the corresponding face; when either class is degenerate, the score vanishes and the active-coordinate equilibrium is uniform.Source: §E.5 · p. 31, §E.6 · p. 32

F DPO through the SRCT Lens

  • The DPO SRCT analysis develops score and full-drift Lipschitz bounds, Barrier–Dominance confinement, intra-class contraction, cross-class envelopes, eventual trimming, and global convergence to a unique two-level equilibrium.The appendix summary specifies that eventual trimming and convergence hold under a slope condition.Source: F DPO through the SRCT Lens · p. 32

F.2 Uniform size and Lipschitz bounds for the DPO score

  • DPO’s score has operator-norm Lipschitz bounds, its full drift is Lipschitz, and its SRCT ODE conserves mass.The supplied DPO passages state these properties but do not provide the remaining positivity or well-posedness results in detail.Source: §F.2 · p. 33, §F.3 · p. 33, §F.4 · p. 33

F.4 DPO–SRCT ODE, mass conservation, and positivity

  • Interior initial conditions yield global solutions that remain in the simplex interior at every finite time.The boundary argument bounds each negative log-coordinate over finite time.Source: §F.4 · p. 33

F.5 Barrier–Dominance (BD)

  • Barrier–Dominance provides a necessary-and-sufficient test for nonnegative flow on a boundary face, with strict inequality ensuring strict interior invariance.The passage also gives uniform sufficient templates, including a sharp ℓ∞ test.Source: §F.5 · p. 34

F.6 Intra-class contraction

  • Within class C, log-probability differences contract at rate ε; the slope condition ε > copen ensures contraction in both classes.For class I, the theorem gives a separate contraction criterion involving copen on the open simplex and cmax on a trimmed domain.Source: §F.6 · p. 34

F.7 Cross-class envelopes, trimming sharpenings, and a static cap

  • The cross-class analysis establishes that the static cap dominates a half-gap and that, under BD compatibility, the asymptotic lower envelope lies strictly below that cap.A stronger trimmed constant is also available by replacing gβ(0) with g⋆.Source: §F.7 · p. 35

F.8 Lyapunov structure and eventual trimming (under SC)

  • Under the slope condition, the dynamics have a Lyapunov identity and eventually keep every coordinate bounded below by a positive constant.The eventual-trimming time and bound may depend on the initial policy and system parameters.Source: §F.8 · pp. 35–36

F.9 Two-level equilibrium: existence, uniqueness, and global convergence

This section analyzes the two-level dynamics through equilibrium and stochastic perspectives, including global convergence, boundary behavior, and invariant laws. It also characterizes stationary concentration toward the fittest face and derives algorithm-specific log-ratio dynamics, with concentration rather than fixation under a positive probability floor.

F.9 Two-level equilibrium: existence, uniqueness, and global convergence

  • Under (SC), a unique gap z⋆ determines the two-level equilibrium, and every interior trajectory converges to it.The proof uses compact-trim entry, Lipschitz drift, and a Lyapunov identity with strict concavity of L.Source: §F.9 · p. 36
  • When N = 0, the uniform equilibrium is globally attractive; when M = 0, it is unique and globally attractive provided ε > β/4.A compatible floor δ⋆ can be chosen below LI(z⋆) without obstructing BD.Source: §F.9 · p. 36

G Dynamics on Coarse-Grained “Lumps”

G.3 Small-ε perturbation: trace and lump bounds

  • For any partition, the lump-level perturbation is bounded by the trace-level perturbation; the bound holds for all t ≥ 0 when the trim is forward invariant.The all-time condition is supplied by the forward-invariance result.Source: §G.3 · p. 38

G.4.1 STaR; G.4.2 GRPO; G.4.3 DPO (sign-pure lumps)

  • The appendix derives lump ODEs for STaR, GRPO, and sign-pure DPO, with a log-ratio relation for DPO coordinates sharing a label.The supplied passages identify these algorithm-specific dynamics but do not provide their full formulas.Source: §G.4.1 · p. 38, §G.4.2 · p. 38, §G.4.3 · p. 39

G.6 Open problems

  • For ε > β/4, the DPO flow has a unique two-level interior equilibrium, and its gap decreases with ε.The sensitivity and gap claims are stated in the open-problems discussion.Source: §G.6 · p. 40

H.4 Continuous–time limits (correct scaling)

  • As the step size vanishes, mini-batch updates converge to the ODE when η/B → 0 and to a Wright–Fisher-type SDE when η/B → γ > 0.Both limits are stated on each fixed time interval [0, T].Source: §H.4 · p. 42

H.5 Boundary behavior: entropy gap and BD conditions

  • Deterministic BD conditions make the trimmed simplex forward invariant, but inward drift alone does not ensure almost-sure non-attainment of its floor in the unreflected diffusion.The unreflected model has finite-horizon high-probability non-exit bounds; with reflection, the diffusion has a unique invariant law and exponential mixing.Source: §H.5 · pp. 42–43

H.8 Small centred bias: concentration toward the fittest face

  • The invariant law of the reflected biased diffusion concentrates near the fittest face, with a symmetric band estimate also reported.The theorem passage identifies the invariant law and concentration result; the accompanying passages give a bound parameter and the band estimate, but their formulas are incomplete here.Source: §H.8 · p. 44
  • With a positive floor δ⋆, probability mass remains at least |I|δ⋆, so the result is concentration toward—not fixation on—the fittest face.A fixation statement is reserved for the vanishing-floor limit δ⋆↓0.Source: §H.8 · p. 44

H.9 Log–ratio SDEs (algorithm–specific)

  • Itô’s formula gives an exact log-ratio identity; GRPO simplifies under equal within-class scores, STaR when score differences track probability differences, and DPO for same-sign pairs near equal probabilities.The DPO passage specifies β(x)∈[−β/4,0); the supplied excerpts do not show the full SDEs.Source: §H.9 · p. 44

H.10 Regime dictionary (concise)

The appendix characterizes how noise and boundary conditions shape regime behavior, then develops idealized and practical kernel designs for diverse equilibria. It also describes the metrics and shared trace universe used in insight experiments.

H.10 Regime dictionary (concise)

  • For r ≪1, the law concentrates tightly inside; r ≍1 gives a unique invariant law, while r ≫1 broadens it without making faces attracting.If BD♯ fails, boundary approach and absorption may occur, so interior-concentration claims no longer apply.Source: §H.10 · pp. 44–45
  • On the trimmed simplex, mini-batch noise has variance O(1/B); the limits are an ODE when η/B →0 and a Wright–Fisher-type diffusion when η/B →γ.BD♯ yields ODE invariance and finite-horizon high-probability confinement for the unreflected SDE; the reflected diffusion is strictly invariant and exponentially ergodic.Source: §H.10 · p. 45

I Kernel Design Strategies for SRCT

  • The SRCT appendix develops kernel-design results spanning stationarity, curvature, convergence, suppression, and practical learned-kernel analysis.It includes a block-constant PSD construction and an explicit global Lipschitz modulus for the SRCT drift.Source: I Kernel Design Strategies for SRCT · p. 45

I.1 Idealized Kernel for a Two–Level Equilibrium

  • In the two-level ansatz, stationarity is equivalent to classwise kernel-average constancy plus a correct–incorrect gap; positive entropy curvature gives a unique interior maximizer.The gap condition is stated in Proposition I.1; strong concavity establishes uniqueness and interiority.Source: §I.1 · pp. 45–46
  • The idealized objective converges exponentially, with a δ-free rate available in the Shahshahani metric.The ODE is globally well-posed and the trimmed simplex is forward-invariant.Source: §I.1 · p. 46

I.2 Practical Design with a Learnable Semantic Kernel

  • For learnable semantic kernels, kernel-average disparities across correct traces create diversity pressure when the kernel separates semantic groups; tuning balances suppression, equalization, and diversity.Smaller εtot strengthens incorrect suppression but weakens equalization; λβ controls semantic diversity pressure and should avoid over-penalizing coherent high-utility traces.Source: §I.2 · pp. 47–48

J Insight Experiments

  • The insight experiments use a shared 12-trace universe with eight correct traces in three clusters and four incorrect traces, and track entropy, cluster inequality, and incorrect mass.Finite multinomial mini-batches supply the stochasticity; the appendix describes these experiments as validating parts of the theory.Source: J Insight Experiments · pp. 48–49

J.1 Experimental Implementation and Reproducibility

Experiments on a shared synthetic trace universe compare theoretical and algorithm-faithful updates, documenting distinct collapse patterns for scalar objectives. DCR has a parameter band with near-zero error, full coverage, and negligible between-seed variation, while safety depends on design choices.

J.1 Experimental Implementation and Reproducibility

  • Experiments use a 12-trace universe with eight correct traces in three clusters and four incorrect traces; finite multinomial batches provide stochasticity, and runs generally span 5,000 steps across five seeds.Study B adds a gated diversity kernel to a GRPO-like base and compares Entropy-only and Ungated ablations.Source: §J.1 · pp. 49–50
  • Study A+ compares replicator-theory tracks with algorithm-faithful STaR, GRPO, and DPO procedures, calibrating procedural steps against theoretical one-step changes.Calibration seeks higher mean cosine while keeping the norm ratio near 1.Source: §J.1 · pp. 49–50

J.2 Strategy–simplex overview (Fig. 1)

  • On the three-strategy simplex, STaR flows to a corner, GRPO meanders before noise-driven fixation, DPO equalizes correct-strategy mass, and DCR retains multi-strategy support.The figure marks early and late states at steps 200 and 5000.Source: §J.2 · p. 50, Figure 1 · p. 51

J.3 Study A: scalar–objective dynamics (Fig. 2)

  • STaR collapses essentially immediately, GRPO’s median fixation is ≈4.7k steps at B=16 with none by 5k at B=64, and DPO homogenizes correct strategies while keeping incorrect mass at zero.Figure 2 tracks entropy, fixation, cluster Gini, and incorrect mass across the three methods.Source: §J.3 · p. 50, Figure 2 · p. 52

J.4 Study B: overlays and alignment diagnostics (Figs. 3, 4, 5)

  • Procedural updates track theory in event timing, though DPO has higher entropy and modest directional cosine despite strong sign agreement; GRPO steps are near-neutral, while STaR aligns closely.Batch-size summaries show shrinking one-step JS divergence alongside decreasing DPO cosine, with event timing synchronizing.Source: §J.4 · p. 50, Figure 3 · p. 53, Figure 4 · p. 54

J.6 Objective and safety trajectories (Fig 8)

  • DCR reaches a stable interior solution with safety ≳0.93, compared with Ungated’s ≈0.48; Entropy-only has safety fixed at 1.Figure 8 overlays the objective proxy and safety trajectories.Source: §J.6 · p. 51, Figure 8 · p. 55

J.7 Safety–margin distribution (Fig. 9)

  • Within the DCR band, every run’s minimum safety margin stays positive, with the worst case ≈0.267.The distribution supports the tuning rule that kernel pressure must not overwhelm the unit utility signal.Source: §J.7 · p. 51, Figure 9 · p. 56

J.5 Study C: DCR phase diagrams (Fig. 6) and ablations (Fig. 7)

  • Across a broad (α, β) band, DCR combines near-zero incorrect mass, full cluster coverage, negligible between-seed JSD, and a unique interior equilibrium; ablations expose trade-offs in distinctiveness and safety.Entropy-only preserves breadth without targeted distinctiveness, while Ungated penalizes incorrect–incorrect similarity and degrades safety.Source: §J.5 · pp. 50–51, Figure 6 · p. 55, Figure 7 · p. 55
Loading 2601.00747v1…