PaperlayerView exact revision on arXiv

Source-linked AI summary

What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor

Abhisek Keshari

arXiv:2609.32532v2cs.LGcs.AI

TL;DR

Generalization gaps are usually reported without a reference point, leaving unclear how their returns compare with chance. This paper measures a matched uniform-random floor and examines action rules, entropy diagnostics, and ProcGen evaluation code. Its results show that these choices can change the interpretation of held-out returns and motivate reporting the action rule and floor alongside each gap.

  • Problem

    Generalization gaps are usually reported without a reference point, although the metric is a headline transfer measure in reinforcement learning.

    Source: §1 · p. 1

  • Method

    The paper measures a uniform-random policy floor under the same evaluation harness and level draws, then compares action rules, entropy diagnostics, and ProcGen evaluation practices.

    Source: §1 · pp. 1–2, §3.2 · p. 3

  • Results

    In miner, sampling scores 5.1× the held-out floor while argmax scores below it in all 6 runs, and raw entropy flags six of eight environments.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The authors recommend stating the action rule, seeding evaluation, specifying tests before analysis, and reporting the random floor on both level sets.

    Source: §8 · p. 10

    A uniform-random floor calibrates only the lower end and is a weak reference point.

    Source: §7 · p. 10

Abstract

from arXiv · show

A generalization gap in reinforcement learning, return on training levels minus return on held-out levels, is usually reported without a reference point. We argue that it should be read against a measured random floor: the return of a uniform-random policy on the same levels under the same evaluation harness. On eight ProcGen environments with PPO at a compute-limited budget (8M steps, 16 parallel environments; three games extended to 25M), the floor changes what standard numbers mean. The test-time action rule decides which policy is measured: in miner, the sampled policy scores 5.1x the floor on held-out levels while its argmax scores below it in every run, and greedy evaluation places two environments significantly below the floor. Used as a convergence diagnostic, raw policy entropy flags six of eight environments, but 32-66% of that entropy lies on actions with identical effects; against the floor, five of eight sampled policies are clearly above it on held-out levels and heist's is not distinguishable from it. An audit of twelve ProcGen codebases finds that nine sample test-time actions with no explicit choice at the evaluation call site. We recommend that every reported gap state its action rule, seed its evaluation and specify its tests before analysis, and report the floor on both level sets.

1 Introduction

The paper argues that ProcGen generalization gaps should be interpreted against a measured uniform-random floor, because test-time action rules and raw-entropy diagnostics can change what the reported returns mean. It measures this floor across ProcGen environments and audits evaluation practice to motivate reporting the action rule, evaluation protocol, and floor.

1 Introduction

  • In miner, sampled evaluation scores 5.1× the held-out random floor, while argmax scores below it in all 6 runs.The same checkpoints and levels are evaluated under both rules.Source: §1 · pp. 1–2
  • Raw entropy flags six of eight environments, but 32–66% of measured entropy lies on actions with identical effects.Heist’s sampled held-out return is not distinguishable from the floor, despite higher training-level return at 25M steps.Source: §1 · p. 2
  • Eleven of twelve audited codebases evaluate held-out levels, and all eleven sample policy-gradient actions; nine have no explicit choice at the evaluation call site.The action rule is often inherited rather than deliberately set.Source: §1 · p. 2
  • The study measures a uniform-random floor on the same seeded level draws and under the same harness on both training and held-out levels.The floor is measured for eight ProcGen environments at 8M steps and three at 25M.Source: §1 · p. 2
  • The paper analyzes the effect of test-time action rules, raw entropy as a convergence diagnostic, and two protocol choices in its own case study.The self-audit examines how the level-draw seed and choice of test move a conclusion.Source: §1 · p. 2

2 Related Work

The paper situates its measured random floor within established return-normalization and evaluation-methodology work, while distinguishing its focus on the meaning of the two returns in a generalization gap. It also relates action selection, entropy, and encoder design to prior ProcGen research.

Reference points for returns

  • Unlike published fixed reference constants, this paper measures the random floor under the policy’s harness, on the same seeded draws, and on both level sets.This allows the floor to be subtracted from both terms of a train-minus-test difference.Source: §2 · p. 2

Evaluation methodology in RL

  • Prior RL evaluation work addresses reporting rigor and variability, while this paper asks which policy produced the differenced returns and where chance lies.The cited work includes reproducibility, confidence intervals, interquartile means, baselines, hypothesis testing, and multiple comparisons.Source: §2 · p. 2

Why the default exists

  • ProcGen protocols designed for ranking use stochastic action selection consistently, but a paper-level train-minus-test gap also depends on the rule and checkpoint measured.The original benchmark and competition average episodes for stable leaderboards.Source: §2 · p. 3

Policy entropy and stagnation

  • Prior work documents PPO stagnation and loss of plasticity under distribution shift; this paper focuses on how residual entropy interacts with action rules and the floor.Test-time exploratory actions have also been reported to improve ProcGen generalization (Zisselman et al., 2023).Source: §2 · p. 3

Encoders

  • The paper places IMPALA CNNs, feature pyramid networks, and VSOP-3D among encoder designs and ProcGen architectural work.The cited work associates VSOP-3D gains with architectural scale-up combined with an algorithmic change (Jesson et al., 2024).Source: §2 · p. 3

3 Method

The method compares agents and evaluation rules on fixed checkpoints, using matched level draws and a measured random floor. It separately defines an entropy screen and statistical tests for interpreting policy performance relative to that floor.

3.1 Agents

  • Two agents share a three-block IMPALA backbone, while the multiscale FPN fuses feature maps before the shared 256-dimensional projection.PPO is trained on 200 easy-mode levels; held-out evaluation uses 100 fixed levels, with 8M-step runs across eight environments and selected extensions to 25M.Source: §3.1 · p. 3

3.2 Evaluation protocol

  • Each final checkpoint is evaluated by sampling and argmax on both level sets, with the floor measured on the same three seeded draws.Each draw has 128 episodes; a run’s value averages the three draws, and sampling is the primary rule.Source: §3.2 · p. 3

3.3 Entropy screen and above-floor check

  • The entropy screen tiers raw-action entropy at below 2.0, 2.0–2.3, and above 2.3 nats, while the above-floor check tests sampled held-out return.The screen uses final-checkpoint states visited during sampled-policy evaluation and averages across both level sets; entropy alone cannot establish failure to learn.Source: §3.3 · p. 4

3.4 Statistical analysis

  • A pre-specified confirmatory re-analysis uses Welch tests, bootstrap intervals, Holm correction, and a TOST equivalence margin of max(0.1 |floor|, 0.25).The plan followed an initial single-draw analysis, so the authors call it confirmatory re-analysis rather than pre-registration.Source: §3.4 · p. 4

4 Field-Wide Audit

The field-wide audit examines released ProcGen evaluation code for stochastic actions and in-loop held-out returns. It finds that sampling is common and often implicit, while bounding its conclusions to code practice rather than the correctness of published results.

4 Field-Wide Audit

  • All eleven audited codebases that evaluate held-out levels sample policy-gradient actions, and six report in-loop test returns.In-loop returns average episodes across policy updates rather than evaluating a fixed checkpoint.Source: §4 · p. 4, Table 1 · p. 5
  • Nine of the eleven codebases have no explicit action-rule choice at the evaluation call site.Three inherit sampling through a default argument, while six use the training rollout’s sampling call.Source: §4 · pp. 4–5
  • The audit covers released evaluation code, not reported paper results, and does not measure how in-loop averages compare with fixed-checkpoint evaluation.The authors do not claim that any published number is wrong.Source: §4 · p. 5

5 Results

Across ProcGen results, the measured gap depends on the test-time action rule and on whether returns are read against a same-level random floor. Entropy screens, longer-run comparisons, and encoder differences also require protocol and configuration context.

5.1 The action rule decides which policy is measured

  • Sampled held-out return exceeds greedy in miner, coinrun, and heist, while greedy exceeds sampled in fruitbot and bossfight, so the rule’s effect has no fixed sign.Starpilot, bigfish, and dodgeball are within noise; no single correction factor converts one rule’s numbers into the other’s.Source: §5.1 · p. 5
  • Miner’s sampled policy scores 5.93 on held-out levels, 5.1× the 1.17 floor, while argmax scores 0.60 on the same checkpoints.The sampled result is above the floor in all six runs; argmax is below it in all six.Source: §5.1 · p. 5, Table 2 · p. 6
  • At 8M, heist’s sampled held-out return is not distinguishable from the floor, while greedy returns on both level sets are below it.The sampled gap is 1.54 versus 0.71 under greedy; that greedy gap compares two below-floor returns.Source: §5.1 · p. 6

5.2 The entropy screen against the above-floor check

  • Five of eight sampled policies are above the floor on held-out levels; heist is not distinguishable, while bigfish and bossfight remain inconclusive.The entropy screen and above-floor check agree at the extremes but diverge for some middle-tier environments.Source: Table 3 · p. 7, §5.2 · p. 7
  • The middle-tier screen over-calls: Miner is flagged by raw entropy (2.087 nats, 77.1% of maximum), yet its sampled policy is above the floor in all six runs.Across environments, 32–66% of raw entropy comes from probability spread across raw actions with identical effects.Source: §5.2 · p. 7

5.3 Longer training under a different configuration

  • In the 25M configuration, heist’s training return rises to 6.94 against a 3.54 floor, while held-out return is 3.00 against 3.15 and not distinguishable.The runs also use 256 rather than 16 parallel environments, changing batch size and policy updates; the comparison is between configurations, not training lengths alone.Source: §5.3 · pp. 7–8, Table 4 · p. 8

5.4 Protocol choices in our own harness

  • On heist at 25M, the sampled encoder difference in held-out return varies from p = 0.028 to 0.27 across three level draws, while the gap difference stays significant on every draw.The authors seed every draw, average each run over three draws, and do not treat re-evaluations of identical weights as independent tests.Source: §5.4 · p. 8

5.5 Encoder comparison, read against the floor

  • On heist at 25M, IMPALA’s gap exceeds FPN’s by 1.06 with sampling and 1.37 with greedy selection, but sampled held-out returns sit near the floor.The sampled gap difference partly reflects training-level return and partly FPN’s higher held-out return; a significant gap difference between near-chance policies is weak evidence about generalization.Source: §5.5 · p. 9

6 Discussion

The discussion argues that a generalization gap needs a measured random-floor reference and that action rules and entropy affect what its numbers support. It recommends matched evaluation and reporting the floor on both level sets.

6 Discussion

  • Miner’s sampled held-out return is 5.1× the floor, yet its argmax falls below the floor; the floor crossing shows the action rule changes which policy is measured.Sampling measures the trained policy, while argmax measures a deterministic policy derived from it.Source: §6 · p. 9
  • Raw entropy flags broad distributions over raw actions, but the above-floor check is needed to assess whether held-out performance exceeds chance.The discussion notes that a high-entropy policy is not thereby illegitimate; the issue is what the number supports.Source: §6 · p. 9
  • Report the action rule, apply it identically to both level sets, seed evaluation draws, measure both floors, and specify statistical tests before analysis.The floor costs minutes to measure.Source: §6 · p. 9

7 Limitations

The findings are bounded by the compute-limited checkpoints, evaluation choices, and study coverage. In particular, the random floor is a weak lower reference, and the study does not test pretrained encoders.

7 Limitations

  • At 8M steps, agents average 0.19 normalized held-out score (0.25 training); at 25M, the three-game mean is 0.33, with 8M settings and early checkpoints bounding interpretation.The 8M configuration uses no reward normalization or clipping and 16 parallel environments; 25M uses 256, so step-budget comparisons also change batch size and update count.Source: §7 · pp. 9–10
  • With six pooled runs and Holm correction, moderate above-floor effects that differ by encoder remain unresolved in bigfish and bossfight.Most conditions use three seeds per encoder; the encoder comparison is presented as a case study.Source: §7 · p. 10
  • The uniform-random floor calibrates only the weak lower end: above-chance scores do not establish competent transfer, and no upper reference is reported.The discussion names stricter lower references and a scripted or strongly trained upper reference, neither measured here.Source: §7 · p. 10
  • The screen uses one set of cuts on eight of ProcGen’s sixteen easy-mode games with PPO and raw actions; five-step equivalence tests could miss later effects.Source: §7 · p. 10
  • Both encoders are trained from scratch; pretrained encoders and newer designs such as Impoola’s globally pooled IMPALA encoder are not tested.Source: §7 · p. 10
  • The audit covers released evaluation code at pinned commits, not reported results, because papers may have used unreleased scripts.Source: §7 · p. 10

8 Conclusion

The conclusion treats the gap as a measurement that needs a reference point. In ProcGen, the random floor changes how action rules and entropy flags should be interpreted, motivating a reporting protocol that makes those choices explicit.

8 Conclusion

  • Miner’s sampled policy scores 5.1× chance on held-out levels, while its argmax scores below chance; the raw-entropy screen also over-calls, with four of six flagged environments clearly above the floor.Bossfight is inconclusive at six runs, and heist’s held-out return is not distinguishable from chance.Source: §8 · p. 10
  • Nine of eleven audited ProcGen codebases sample test-time actions without an explicit choice at the evaluation call site.Source: §8 · p. 10
  • The conclusion recommends stating the action rule, seeding evaluation, specifying tests before analysis, and reporting the random floor on both level sets.Source: §8 · p. 10

A Pooled Significance of the Gap

The section describes how pooled generalization gaps are summarized and tested, and identifies the floor and pooled returns reported on both level sets and budgets.

A Pooled Significance of the Gap

  • The primary gap summary is the mean across runs with a 95% percentile bootstrap confidence interval; with six runs, intervals ending near zero warrant caution.The bootstrap distribution is coarse and can understate uncertainty.Source: A Pooled Significance of the Gap · p. 12
  • Secondary tests use per-run Welch tests and combine two-sided p-values with Fisher’s method, with one-sided Stouffer’s Z as a directional check and Holm correction within each rule-by-budget family.Fisher can detect a difference in any run, while Stouffer additionally requires agreement in direction; these tests can disagree with the bootstrap interval.Source: A Pooled Significance of the Gap · p. 13
  • Tables 5 and 6 report train-minus-test gaps for the three extended environments at both budgets and for all eight environments at 8M, respectively.Table 5 distinguishes sampled (primary) and greedy evaluation.Source: Table 5 · p. 13, Table 6 · p. 13

B Floor on Both Level Sets

  • Table 7 reports the uniform-random floor and pooled returns on both level sets at both budgets, using the same level draws and the primary calls of Table 2.Floor uncertainty is reported as per-episode SEM over 384 episodes; returns use SEM over runs.Source: B Floor on Both Level Sets · p. 13, Table 7 · p. 13

C Threshold Sensitivity of the Screen

The screen’s sensitivity is examined by sweeping its entropy threshold, while calibration compares raw policy entropy with entropy after functionally equivalent actions are merged.

C Threshold Sensitivity of the Screen

  • Table 8 sweeps the low/intermediate cut from 1.8 to 2.2 while holding the high cut at 2.3 and reports each environment’s tier under raw and merged entropy.The table covers sampled-evaluation states at 8M steps; L/I/H denote low, intermediate, and high residual entropy.Source: Table 8 · p. 14, C Threshold Sensitivity of the Screen · p. 14

D Calibration: Effective Actions and Merged Entropy

  • The calibration tests action equivalence by replaying each action from cloned states and comparing observations, rewards, and episode boundaries over the action and four subsequent no-ops.Across sampled states, distinct outcome classes range from 4 in fruitbot to 11 in starpilot; merged entropy sums probabilities within each class.Source: D Calibration: Effective Actions and… · p. 14
  • Normalizing raw entropy by ln k is not meaningful because raw entropy can exceed ln k when probability mass is spread across equivalent actions.Merging probabilities within outcome classes first provides the valid comparison.Source: D Calibration: Effective Actions and… · p. 14
  • Table 9 reports raw and merged entropy, the share of raw entropy within classes, and raw entropy on greedy-evaluation and restricted sampled-evaluation states.It separates 8M and 25M sampled-evaluation results into upper and lower blocks.Source: Table 9 · p. 14

E Exploratory: Merged-Argmax Evaluation

The appendix describes an exploratory merged-argmax evaluation, normalized-score reporting, audit evidence, and additional implementation and evaluation details. Its tables specify comparisons and protocols, but the supplied passages do not provide their cell values.

E Exploratory: Merged-Argmax Evaluation

  • Table 10 compares held-out returns for merged argmax, plain argmax, sampling, and the floor, alongside paired merged-minus-greedy differences and class-disagreement rates.Returns are reported at 8M over six runs as mean ± SEM, with Holm correction within this exploratory analysis.Source: Table 10 · p. 15, E Exploratory: Merged-Argmax Evaluation · p. 15

F Normalized Scores on the Scale of Cobbe et al.

  • Table 11 reports normalized scores as (R − Rmin)/(Rmax − Rmin) using Cobbe et al.’s easy-mode constants, with sampled evaluation primary and the floor measured on the same draws.The measured uniform-random floor is usually negative on this scale because Rmin comes from a policy trained with observations masked out.Source: F Normalized Scores on the Scale of… · p. 15, Table 11 · p. 15

G Audit: Commit Hashes and Evidence Locations

  • Table 12 records each audited codebase’s commit and evidence location, distinguishing stochastic test evaluation from in-loop test return.The passage identifies the twelve repositories examined and notes where default-False deterministic-action settings are defined when applicable.Source: Table 12 · p. 16, G Audit: Commit Hashes and Evidence… · p. 16

H Additional Details and Tables

  • The encoder details specify a shared three-block IMPALA backbone and an FPN variant with fewer parameters: 521k versus 622k.The baseline projects the flattened final feature map to 256 dimensions; the FPN fuses multi-scale maps before a 256-dimensional projection.Source: H Additional Details and Tables · p. 16
  • Of 1,008 evaluation cells, 93 ran on CPU because of memory limits, and sampled actions there were reproducible on CPU but not bit-identical to GPU runs with the same seed.Greedy cells draw no random numbers.Source: H Additional Details and Tables · p. 16
  • The appendix reports PPO hyperparameters in Table 13 and final-checkpoint results in Table 14, but the supplied captions do not state their values.Table 15 covers heist at both budgets under both action rules, pooled over encoders, with the floor on identical level draws.Source: Table 13 · p. 17, Table 14 · p. 17, Table 15 · p. 17

I Representation Diagnostics

The representation diagnostic’s train-versus-test CKA comparison lacks valid row pairing, and controls show that its large values track timestep alignment rather than level set.

I Representation Diagnostics

  • 5.6–7.0× lower fruitbot values after shuffling one side’s rows show that the train-versus-test CKA statistic tracks episode-timestep alignment rather than level set.The timestep-aligned train-versus-train control gives the same value as train versus test.Source: I Representation Diagnostics · p. 18, Table 16 · p. 18
  • Under a uniform row permutation, the CKA expectation is √(PR_X PR_Y)/(n − 1), so between-encoder differences track effective rank.On fruitbot at 8M, IMPALA’s raw value is 3.44× FPN’s, but 1.14× in units of each side’s null.Source: I Representation Diagnostics · p. 18
  • Changing only collection geometry moves within-encoder CKA by 4.6× for IMPALA and 6.3× for FPN on heist at 25M.Cross-encoder CKA on identical observations is correctly paired but does not contribute to the paper’s conclusions.Source: I Representation Diagnostics · p. 18

Related papers

Loading 2609.32532v2…