Center-of-Mass Interpretation of Evolution Strategies

1 Introduction and Related Work

• The standard smoothed-gradient interpretation is correct but says little about the effect of a single update in parameter space.ES samples and scores perturbed models, then combines their perturbations using rewards.Source: §1 · p. 1
• For any finite population, the mean-subtracted ES update is the displacement from the sampled centroid to the reward-weighted center of mass, scaled by a factor depending on mean reward.Rewards can be shifted by a constant to make them nonnegative with positive total without changing the update.Source: §1 · pp. 1–2
• In large populations or with antithetic sampling, the centroid coincides with the model, so the update moves the model toward its population’s reward-weighted center of mass.Figure 1 illustrates antithetic candidates and the reward-weighted shift; the displacements are exaggerated for visibility.Source: §1 · p. 1, Figure 1 · p. 2
• Unlike earlier ES, CMA-ES, and Price’s-equation accounts of weighted means, this interpretation explains the mean-subtracted update as a step between two centers of mass.Earlier accounts describe the new mean as a weighted average of candidates; mean subtraction makes the sampled centroid the step’s starting point.Source: §1 · pp. 1–2

2 Background: Anatomy of an ES Update

• ES perturbs the current model with independently sampled Gaussian directions, with σ controlling perturbation scale, to produce candidate models.The perturbations are sampled around θ from a normal distribution.Source: §2 · p. 2
• Each candidate receives a reward, with higher reward indicating better performance; the population’s mean and reward standard deviation can also be computed.The supplied passage begins the definition of the mean reward but does not give its full expression.Source: §2 · p. 2
• The update combines sampled directions using reward-derived coefficients, and the resulting Δθ is added to the current parameters.The learning rate α also absorbs the perturbation scale; the reward transformation determines each candidate’s contribution.Source: §2 · p. 2
• Common coefficient rules subtract or standardize mean reward or use candidate ranks; the main derivation assumes coefficients sum to zero, while raw rewards are treated separately.The zero-sum condition is stated for the covered derivation variants.Source: §2 · p. 2

3 The Sampled Population and Its Centroid

• The centroid of the sampled models is their equal-weight center of mass.Assigning each sampled model the same particle mass yields the population centroid.Source: §3 · p. 3
• Across repeated populations, the centroid lies at the current model, while an individual finite population can be displaced from it.The finite-sample displacement is σ times the mean sampled perturbation.Source: §3 · p. 3
• As the population grows, its centroid converges to the model; mirrored perturbation pairs make the centroid equal the model exactly at any population size.Under mirrored pairs, the mean perturbation is zero.Source: §3 · p. 3

4 ES Moves the Model Along a Center-of-Mass Shift

• The physical center-of-mass interpretation requires nonnegative rewards with a positive total, which can be arranged by adding a constant to every reward without changing the update.Source: §4 · p. 3
• The mean-subtracted ES update is exactly the displacement from the sampled centroid to the reward-weighted center of mass, scaled by a factor depending on mean reward.This identity holds for any finite population.Source: §4 · p. 4
• With an infinite population or antithetic sampling, the sampled centroid coincides with the current model, so the update moves the model toward its population’s reward-weighted center of mass.Source: §4 · p. 4

4.1 Centered and Raw Rewards Use Different Reference Points

• Raw-reward ES measures displacement from the current model to the reward-weighted center of mass, whereas mean subtraction uses the realized sample’s equal-weight centroid as its reference.Source: §4.1 · p. 4
• Mean subtraction removes the raw update’s additional term tied to the finite-sample centroid’s distance from the model; that term vanishes as the population grows and also disappears under antithetic sampling.The additional term depends on total reward and the centroid-to-model displacement, rather than reward differences.Source: §4.1 · p. 4

• Mean-subtracted ES is exactly the reward-weighted center-of-mass displacement from the equal-mass centroid, scaled by a factor depending on mean reward, for any finite population.Source: §5 · p. 4
• In the infinite-population regime or under antithetic sampling, the centroid coincides with the model, so the model moves toward its population’s reward-weighted center of mass.Source: §5 · p. 4
• The term removed by mean subtraction has expectation 1/N of the expected raw update and is typically noise-dominated when d ≫ N.Source: §5 · p. 4

Limitations

• The physical reading requires nonnegative rewards with a positive total, achievable by adding a constant to every reward without changing the identity.The identity itself does not require this condition; the physical picture is a convenience.Source: Limitations · p. 5
• Analysis of the term removed by mean subtraction assumes independent Gaussian perturbations with fixed σ and excludes adaptive step sizes and covariance updates such as CMA-ES.Source: Limitations · p. 5
• The claim that the removed term is dominated by noise applies when d ≫ N, the LLM fine-tuning regime, and may not hold for small problems.Source: Limitations · p. 5

A Appendix

• Under the stated conventions, the listed ES weighting rules produce coefficients that sum to zero.Source: Table 1 · p. 6
• The centered-rank formula matches released OpenAI-ES code, while the paper specifies only a rank transformation.Source: Table 1 · p. 6
• CMA-ES uses positive, truncated rank utilities that sum to one rather than zero.The utility is proportional to log(µ + 1/2) − log i.Source: Table 1 · p. 6